The Problem

JBoss and Wildfly like most application servers, doesn’t come with built-in support for adding a correlation ID to requests out of the box. However, this capability is often crucial in complex distributed systems for tracking and debugging purposes.

Why Add a Correlation ID?

In a distributed system, requests may traverse multiple services and components.

Adding a unique correlation ID to each request allows you to:

1. Trace Requests: Easily track the flow of a specific request across various services, making troubleshooting and debugging more efficient.
2. Identify Dependencies: Quickly identify which requests are related to one another, helping to pinpoint bottlenecks or issues in your system.

TL;DR

As always, if you want to skip right to the complete example, you can find the source code covered in this blog at GitHub.

A container level solution (I’ve used in production for many years)

There are many ways an application developer could solve this problem for an individual application (e.g. servlet request filters). Still, what if we want to add this for the entire application platform, so any application deployed on our server inherits this functionality? In the absence of this centralized solution for your many services, there are many paths to have inconsistencies across your services. As this is a critical feature operationally for triaging production issues across service boundaries, I believe the best solution is to implement a custom solution to add a correlation ID to each incoming request that is also logged in the shared logging provided by the application server so all logs associated with a request across all services can be quickly filtered.

High-level code overview

To achieve a container-level solution in JBoss, we will install a “global module” that adds a custom “HttpHandler” to the Undertow subsystem. Additionally, we will modify the container logging configuration to tag all application logs with a CorrelationId.

• CorrelationIdGenerator

  • The CorrelationIdGenerator class intercepts incoming requests as an Undertow HttpHandler. It first checks if a “X-Correlation-Id” header is present in the request. If not, it generates a new UUID as the correlation ID. Additionally, it looks for a “uuid” header and uses its value if present, falling back to the generated correlation ID.

• MDC (Mapped Diagnostic Context) Configuration: Once generated (or read from incoming requests) the correlation ID is stored in the threads MDC , which is a context map that utilizes a thread-local storage mechanism. Undertow threads do not service the entire request but instead delegate to another “worker” thread, so a little bit of code is needed to wrap this handoff to ensure the MDC map is copied to the new thread (see WrappingMdcExecutor.java which is responsible for propogation of the MDC context).

• Timing Metrics: One additional value add in this provided module is that each incoming request starts a timer to measure the response time of the request and logs this timing information when the request is completed. This can be useful for performance monitoring. This consistent information coming from your collection of services makes it very easy to develop very nice monitoring dashboards from your log output (ideally funneled to a tool like Splunk/Loki for eash search).

Building the code example

A simple mvn clean install will do the trick.

Installing mdc-undertow-ext-3.0.1.jar as a global module in JBoss EAP 7

Follow these steps to install the target/mdc-undertow-ext-3.0.1.jar as a global module in JBoss EAP 7 (WildFly 10) using the JBoss CLI:

1. Start the JBoss CLI

Navigate to the bin directory in your JBoss EAP 7 installation and run the jboss-cli script (use jboss-cli.bat on Windows or jboss-cli.sh on Linux/macOS).

2. Connect to the running JBoss instance

In the JBoss CLI, enter the following command to connect to the running JBoss instance:

connect

3. Add the module using the module add command

In the JBoss CLI, use the module add command to create the module and add the JAR file as a resource. Make sure to specify the required dependencies as well. Replace /path/to/mdc-undertow-ext-3.0.1.jar with the actual path to the JAR file:

module add --name=com.rhc.mdc --resources=/path/to/mdc-undertow-ext-3.0.1.jar --dependencies=org.slf4j,io.undertow.core,javax.api,org.jboss.logging,org.jboss.modules,com.google.guava

4. Register the module as a global module

In the JBoss CLI, enter the following command to register the module as a global module:

/subsystem=ee:write-attribute(name=global-modules,value=[{name=com.rhc.mdc}])

5. Modify the Undertow subsystem configuration

To make all applications benefit from the CorrelationIdGenerator and WrappingMdcExecutor classes, you need to modify the Undertow subsystem configuration. In the JBoss CLI, enter the following commands:

/subsystem=undertow/configuration=filter/custom-filter=correlation-id-generator:add(class-name=com.rhc.mdc.CorrelationIdGenerator,module=com.rhc.mdc)
/subsystem=undertow/server=default-server/host=default-host/filter-ref=correlation-id-generator:add

6. Modify the pattern-formatters to include the correlationId:

In the JBoss CLI, enter the following commands to modify the pattern-formatters of the logging subsystem to include the correlationId from the Mapped Diagnostic Context (MDC) in each log message:

/subsystem=logging/pattern-formatter=PATTERN_FORMATTER_NAME:write-attribute(name=pattern, value="%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%c] (%t) [correlationId=%X{correlationId}] %s%e%n")
/subsystem=logging/pattern-formatter=COLOR-PATTERN:write-attribute(name=pattern, value="%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%c] (%t) [correlationId=%X{correlationId}] %s%e%n")

7. Reload the server

To apply the configuration changes, enter the following command in the JBoss CLI:

reload

After completing these steps, the mdc-undertow-ext-3.0.1.jar will be installed as a global module in JBoss EAP 7, and all applications deployed on the server will benefit from the CorrelationIdGenerator and WrappingMdcExecutor classes.

Example output

2023-04-18 16:49:43,642 INFO  [com.rhc.mdc.CorrelationIdGenerator] (default I/O-1) [correlationId=121221fc-0c59-4eaa-81b7-37305a02e83c] [METRICS] Starting timer for requestMethod=GET requestPath=/helloWar-1.0.0/resources/ping request
2023-04-18 16:49:43,655 INFO  [com.rhc.mdc.CorrelationIdGenerator] (default task-1) [correlationId=121221fc-0c59-4eaa-81b7-37305a02e83c] [METRICS] Response Time for requestMethod=GET requestPath=/helloWar-1.0.0/resources/ping: 12 ms, httpResponseCode=200

As you can see, this output includes a correlationId tag for the application logs as well as information about the incoming request and how long the request takes to complete (12 ms). This information would be available for all applications deployed on the modified jboss installation.