Logging standards

March 19, 2026 · View on GitHub

Libraries

The following libraries implement the logging standards defined below:

What we should be logging

Apps should log all important events, especially failures, as well as enough events to be able to trace the execution logic through each major step without being overly noisy. Any information which adds value is worth including in the event. If you're unsure, log more rather than less.

A few ideas of useful things to log are:

  • application startup (with sanitised application config output under data)
  • log all failures
    • a failure caused by a bad request (client error) that the system deals with (i.e. issues a 4XX response) is not an error and should be logged as INFO only if deemed appropriate.
  • http requests
  • requests to data stores and other third party services (e.g. mongo, postgres, kafka, neptune, etc.)
  • log events at significant steps throughout the execution flow
    • sometimes it is useful to have started x and finished x for very long running events, but usually a single event can suffice
    • avoid back to back log statements as a general rule, especially if the second event implies the first

Always consider the performance impact of logging:

  • loops
  • log-specific conditionals ("should I log this")
  • JSON serialisation
  • deeply nested data structures

Sensitive information

NEVER log passwords, credentials, auth headers, or other sensitive information in any form.

In some situations it may be necessary to log a user's IP address, such as during authentication, for the purposes of auditing. This should only be done in specific situations and only when necessary.

Logging specification

We have defined the formatting and data structures, including field names and types, we use for logging.

This is important to allow the centralised logging service to properly index the data and make it searchable. Any logs not adhering to the specification may be lost.

Changes to the spec

Any changes, including the addition of new fields, must be agreed by the team as a whole before being added to this spec. Any changes to this spec will require changes to all of the logging libraries as well as the centralised logging service.

Output formatting

Although "human readable" formatting is useful for local development, all apps deployed to an environment must comply to the following formatting rules.

  • All logs must be JSON Lines
  • All logs must be UTF8 encoded
  • All logs must not be colour formatted
  • All logs must not contain newlines
  • All logs must not contain tabs
  • All logs must not be string escaped

Log structure

Common fields

The following are the only top level fields. If you have app specific details you wish to add, it is likely that they should be added under data rather than creating a new field.

Field nameRequiredTypeExampleDescription
authNoauthAuthentication event data
classificationNostring"PROTECTIVE_MONITORING"Log classification for platform routing and special handling
created_atYesdatetime1"2019-01-21T16:19:12.356Z"Date and time of the log event (ISO8601 extended date and time string)
dataNoobject2Arbitrary key-value pairs
errorsNo[]errorError event data
eventYesstring"connecting to mongodb"The event being logged
httpNohttpHTTP event data
namespaceYesstring"dp-frontend-router"Service name or other identifier
rawNostringLog message captured from a third party library
severityYesint82The event severity level code
spec_versionNostring"v1"Major version of this logging specification used by the emitting library
span_idNostringSpan ID from OpenCensus
trace_idNo3stringTrace ID from OpenCensus

Spec version

The spec_version field is an optional identifier for the major version of this logging specification that the emitting library claims conformance with.

If spec_version is not provided, it will be treated as v1.

The logging libraries should set spec_version automatically on every emitted log line. Services should not override it in normal operation.

Classification

The classification field is an optional, single value used to mark logs that require special platform handling.

This is primarily intended for platform routing (for example sending protective monitoring logs to Sentinel). It is not a severity indicator and must not be used as a substitute for severity.

If classification is not provided, the event is treated as unclassified and normal routing applies.

Allowed values (controlled vocabulary):

ValueMeaning
PROTECTIVE_MONITORINGSecurity relevant events intended for protective monitoring / SIEM routing

Rules:

  • values must be uppercase letters separated by underscores (e.g. EXAMPLE_VALUE)
  • services must not invent new values; new values must be added to this specification
  • values must describe the category/intent, not the action to be taken by the platform

HTTP event data

HTTP events are so common that we've defined a specific top-level field to capture it. The http field defines the common http event data.

This can be used to log inbound or outbound HTTP event data.

Field nameRequiredTypeExampleDescription
durationNoint64236578Difference between started_at and ended_at in nanoseconds
ended_atNodatetime4"2019-01-21T16:19:12.356Z"ISO8601 date string of the request end time
hostYesstring"10.100.20.1"The hostname or ip from the URL
methodYesstring"GET"The HTTP method used
pathYesstring"/healthcheck"The request path from the URL
portYesint32443The port number from the URL
queryNostring"x=1&y=1"The unparsed query string from the URL
response_content_lengthNoint6434The length of the response
schemeYesstring"https"The scheme or protocol from the URL
started_atYesdatetime4"2019-01-21T16:19:12.356Z"ISO8601 date string of the request start time
status_codeNoint16200The HTTP status code

Auth event data

The auth field defines the common auth event data.

Field nameRequiredTypeExampleDescription
identityYesstring"dp-import-tracker"The user id or service id
identity_typeYesstring"service"The identity type (i.e. user or service)

Error event data

Errors are logged as an array. Each element of the array represents a single error and has the following fields.

Field nameRequiredTypeExampleDescription
dataNoobject5Arbitrary key-value pairs
messageYesstring"connection refused"The error cause
stack_traceNo[]stack_traceStack trace as an array
Stack trace element

The stack trace is logged as an array. Each element of the array has the following fields.

Field nameRequiredTypeExampleDescription
lineNoint1618The line in the source file that is throwing the error
fileNostring"/some/path/main.go"The source file containing the code throwing an error
functionNostring"main.main"The function in which the error is occurring

Effective event types

The event field is a brief description of the type of event.

A good event description is:

  • lowercase
  • not a sentence (i.e. is brief and without a full stop)
  • generic (e.g. http request)
  • kept consistent where possible
  • not a formatted string (additional information should be provided via other fields)

For example:

'event': 'file could not be read from s3'

These are guidelines only - there are some exceptions where it's ok to ignore them, for example when using uppercase to refer to a constant or environment variable:

'event': 'DEBUG is not a boolean'

Severity levels

The event severity levels are used to identify how critical an event is, especially failure events.

There are four severity levels:

CodeSeverityDescription
0FATALAn catastrophic failure event resulting in the death of the application
1ERRORAn unrecoverable failure event that halts the current flow of execution
2WARNA failure event that can be managed within the current flow of execution (e.g. retried)
3INFOAll non-failure events

For example, if in the process of serving an HTTP request an action is retried three times, the first two failures would have a severity of WARN, whereas the third time would have an ERROR.

Third party logs

The log libraries attempt to intercept all logs from third party libraries in order to prevent collision with our logging specification. These intercepted log messages should be output as a string under the raw field with the event as "third party log" and severity of 3 (INFO). All other relevant fields (e.g. created_at, namespace, trace_id, span_id, etc) should be set as they would normally.

For example:

{
  "created_at" : "2019-02-01T13:45:24.157Z",
  "namespace" : "dis-example-service",
  "spec_version" : "v1",
  "trace_id": "1105cb0c04f86a4b6a1abaf74246b87f",
  "severity" : 3,
  "event": "third party log",
  "raw" : "Started ServerConnector@7f4fedd{HTTP/1.1,[http/1.1]}{0.0.0.0:4567}"
}

Routing examples

Normal log (no special routing):

{
  "created_at": "2026-01-01T09:15:30.123Z",
  "namespace": "dis-example-service",
  "spec_version": "v1",
  "severity": 3,
  "event": "starting service"
}

Protective monitoring log (eligible for routing to Sentinel):

{
  "created_at": "2026-01-01T09:15:30.123Z",
  "namespace": "dis-example-service",
  "spec_version": "v1",
  "severity": 2,
  "event": "permission denied",
  "classification": "PROTECTIVE_MONITORING",
  "data": {
    "action": "read_dataset",
    "reason": "missing_permission"
  }
}

Arbitrary data fields

There are two arbitrary data fields in the logging spec. One at the top level for general event specific data and one under error for error specific data. These fields are a map/object in code, but will be stored as a string field by the centralised logging service to allow additional detail to be added to an event without needing to worry about key name collision. These details can still be searched using a general text search on the data field.

The data fields are a means of adding event specific details that help in understanding the event, but are not common enough to justify a field in the spec. These fields are an important part of the event data and it is important that the data be useful and well structured.

Kafka

TODO: Add kafka log data to the logging spec

  • Events should refer to an ID to tie the events across services together - whether this is the same ID as used in context, or another identifier, which has previously been logged alongside the context
  • log topic and (if consuming) group and offset

Footnotes

  1. All dates must be UTC and in ISO8601 extended date time format with at least millisecond precision: yyyy-MM-dd'T'HH:mm:ss.SSSZZ (e.g. 2019-01-21T16:19:12.356Z). ↩

  2. The event-specific details in data are input as an object/map in code, but are stored as a text string by the centralised logging service to allow additional detail to be added to an event without needing to worry about key name collision. These details can still be searched using a general text search on the data field. ↩

  3. Not mandatory, but must be included for all events created during the handling of a request. ↩

  4. All dates must be UTC and in ISO8601 extended date time format with at least millisecond precision: yyyy-MM-dd'T'HH:mm:ss.SSSZZ (e.g. 2019-01-21T16:19:12.356Z). ↩ ↩2

  5. The error-specific details in data are input as an object/map in code, but are stored as a text string by the centralised logging service to allow additional detail to be added to an event without needing to worry about key name collision. These details can still be searched using a general text search on the data field. ↩