Structured Logging Example with Logback
This is a Java project that shows how to use Logback effectively for structured logging. It should show how you configure Logback, and how you can reduce the amount of complexity in your end projects by packaging your logging appenders and configurators in a distinct project.
The project is configured into several modules:
guice. The most relevant ones to start with are
classic module contains all the logback code and the appenders, and is intended to be deployed as a small helper library for your other projects, managed through Maven and an artifact manager, or just by packaging the JAR. The
example project depends on
classic, and contains the "end user" experience where log levels are adjusted and JSON can be pretty printed or not.
example project cannot touch the appenders directly, and has no control over the format of the JSON appender -- console and text patterns can be overridden for developer convenience. By enforcing a separation of concerns between logger configuration and logging levels, it is easy and simple to manage appenders in one place, e.g. going from file appenders to TCP appenders, adding filters for sensitive information, or collapsing repeated log information.
This is not intended to be a drop in replacement or a straight library dependency. You will want to modify this to your own tastes.
What is Structured Logging?
It's logging in JSON. Technically, you could be logging in another structure like XML or JSON, but almost everyone uses JSON. It's been around for a while. Technically, since there are several JSON objects all in one file / stream, this is called "newline delimited JSON" or NDJSON or jsonlines. In this project, both text and JSON formats are rendered independently, but if you only output JSON it's not a huge deal, because you can read JSON logs as text with a special log viewer such as jl.
Semantically, a log entry typically has multiple pieces of information associated with it, described as "high cardinality" by observability geeks. Structured logging means that the cardinality goes from "closed" -- you can only log things that you have defined fields for -- to "open", where you can add arbitrary fields and objects to your log entry as long as it's JSON.
Structured logging is really all about giving yourself — and your team — a logging API to help you provide consistent context in events. An unstructured logger accepts strings. A structured logger accepts a map, hash, or dictionary that describes all the attributes you can think of for an event.
Logs are different from events. All events can be represented as logs, but not all logs are events. Many logs are only portions of events. An event is a conceptual abstraction and a log is one possible representation of that abstraction.
Logs are also different from metrics. A metric represents a single number. You can extract metrics from logs, but it's a very expensive way of going about it.
There is a question of what you want to add when you log. This is a matter of taste, but in general you should log so that you create a consistent narrative. As previously mentioned, a log may indicate a portion of an event, so you want to log where doing so would help tell a story of what happened afterwards.
There are some things you should always add to an event, such as who is talking to your service, what they're asking, business relevant fields, additional context around your service / environment, response time and particulars. You should add units to your field names when you measure a quantity, i.e.
response_time_ms, and add a "human readable" version of internal information if available.
You should add context to your logs that helps differentiate it from its peers, so you never have to guess where the source of a log is coming from.
Adding a correlation id helps you design for results and tie your logs into a coherent event. You don't need to use a UUID: a flake id will probably be better for you. I'm using idem here, but most things will work.
So, we know what structured logging is now. What does it look like in SLF4J?
Adding Structure to Logging
SLF4J doesn't have specific support for structured logging, but logstash-logback-encoder does. It's complete and comprehensive, but buri