Opentelemetry Datadog integration (TraceId/SpanId issues)

[jimbogithub]

I'm trying to convert from using the Datadog java agent to using Quarkus' integrated Opentelemetry.

Following the Quarkus docs I have the Traces publishing successfully and Logs are being scraped same as before. I am now trying to enable the Log/Trace correlation but am hitting the issue of incompatible TraceId/SpanId formats as discussed in https://docs.datadoghq.com/tracing/other_telemetry/connect_logs_and_traces/opentelemetry/?tab=java.

Datadog wants unsigned 32bit decimal (not hex values). Interestingly it seems to handle the raw Opentelemetry values OK for the Trace ingestion - it just truncates and converts the Ids, e.g. 000000000000000013b0237284e4397e (I'm deliberately setting the hi-long to 0 using an IdGenerator but it still works if I don't) becomes 1418672857384499582.

So inspired by the IdGenerator producer example and this exporter example: #29143, I hoped something similar would be possible for the MDCProvider. I've tried installing variations of the code below via a CDI producer and via META-INF/services/org.jboss.logmanager.MDCProvider.

import org.jboss.logmanager.MDCProvider;
import io.quarkus.vertx.core.runtime.VertxMDC;

/**
 * Default OTEL TraceId is 128bit but Datadog can only handle 64bit and dislikes hex.
 */
public class DatadogCompatibleMDCProvider implements MDCProvider {

	private static final MDCProvider DELEGATE = VertxMDC.INSTANCE;
	private static final Set<String> KEYS_TO_CONVERT = Set.of("traceId", "parentId", "spanId");

	@Override
	public String get(String key) {
		return DELEGATE.get(key);
	}

	...

	@Override
	public String put(String key, String value) {
		return DELEGATE.put(key, convertForDatadog(key, value));
	}

	...

	private String convertForDatadog(String key, String value) {
		if (KEYS_TO_CONVERT.contains(key)) {
			String hexString = value.substring(value.length() - 16);
			long datadogId = Long.parseUnsignedLong(hexString, 16);
			return Long.toUnsignedString(datadogId);
		}
		return value;
	}

}

Unfortunately this appears to not get installed. Is there a proper way to do it?

[quarkus-bot[bot]]

/cc @Ladicek (tracing), @brunobat (opentelemetry,tracing), @radcortez (opentelemetry,tracing)

[jimbogithub]

Solved using a log Filter (https://quarkus.io/guides/logging#console-log-handler) instead:

import java.util.logging.Filter;
import java.util.logging.LogRecord;

import org.eclipse.microprofile.config.inject.ConfigProperty;
import org.jboss.logmanager.ExtLogRecord;

import io.quarkus.logging.LoggingFilter;

/**
 * Can be enabled with {@code quarkus.log.console.filter=dd-otel-id-converter}.
 */
@LoggingFilter(name = "dd-otel-id-converter")
public final class DatadogOpenTelemetryIdConversionFilter implements Filter {

	private final String traceIdKey;
	private final String spanIdKey;
	private final String parentIdKey;

	public DatadogOpenTelemetryIdConversionFilter(
			@ConfigProperty(name = "dd-otel-id-converter.trace-id-key", defaultValue = "traceId") String traceIdKey,
			@ConfigProperty(name = "dd-otel-id-converter.span-id-key", defaultValue = "spanId") String spanIdKey,
			@ConfigProperty(name = "dd-otel-id-converter.parent-id-key", defaultValue = "parentId") String parentIdKey) {
		this.traceIdKey = traceIdKey;
		this.spanIdKey = spanIdKey;
		this.parentIdKey = parentIdKey;
	}

	@Override
	public boolean isLoggable(final LogRecord record) {
		if (record instanceof ExtLogRecord extLogRecord) {
			convert(extLogRecord, traceIdKey);
			convert(extLogRecord, spanIdKey);
			convert(extLogRecord, parentIdKey);
		}
		return true;
	}

	private void convert(ExtLogRecord extLogRecord, String key) {
		String hexString = extLogRecord.getMdc(key);
		if (hexString != null) {
			if (traceIdKey.equals(key)) {
				hexString = hexString.substring(hexString.length() - 16);
			}
			long datadogId = Long.parseUnsignedLong(hexString, 16);
			extLogRecord.putMdc(key, Long.toUnsignedString(datadogId));
		}
	}

}

Custom IdGenerator also no longer required.
Hope that helps anyone with the same issue. Let me know if you find a better way.

[radcortez]

Great! Thanks :)

[brice-laurencin]

don't you need to also relocate trace id and span id to dd.trace_id and dd.span_id?

[jimbogithub]

don't you need to also relocate trace id and span id to dd.trace_id and dd.span_id?

No you don't have to because DD can be mapped to something other than those defaults. But you can if you want to via the @ConfigProperty settings on the above conversion filter. Or just hard code them to what you want.

[brice-laurencin]

I'm confused with this one doc page: https://docs.datadoghq.com/tracing/other_telemetry/connect_logs_and_traces/opentelemetry
DD sounds like it wants those relocated, plus I feel it would help correlate the calls with different stacks.

BTW, thanks a lot for this gist 👏

[brunobat]

Ok, to overcome this Datadog limitation, we could introduce an optional property that could add to the logs the relevant IDs here:

quarkus/extensions/opentelemetry/runtime/src/main/java/io/quarkus/opentelemetry/runtime/OpenTelemetryUtil.java

Line 90 in 9d5e5b2

spanData.put(SPAN_ID, spanContext.getSpanId());

[maxandersen]

@brunobat would this make sense to be captured in a quarkiverse.datadog extension?

[brunobat]

Yes @maxandersen @jimbogithub. This would be the right place for a Datadog extension: https://github.com/quarkiverse/quarkus-opentelemetry-exporter

[brunobat]

see comment above. We don't need an extension if we just need to add the right data to the logs.

[jimbogithub]

FYI, this is no longer required. DataDog is now natively supporting the OTEL formats (Trace 128 bit hex / Span 64 bit hex) when parsing injected logs. So the only requirement is to remap the names which can be done in DataDog Log Profiles or in your Quarkus log formats (depending on whether you are using JSON or console logging).

DataDog docs still refer to requiring integer format and although they recently added the 128 bit Trace ID support, their OTEL specific docs still say it needs to be 64 bit. However I've just tested using the full length native hex formats and it all works. I'm assuming that despite the docs, this is official support due to their OTEL partnership and it just makes sense as they use the hex formats elsewhere.

[brunobat]

Thanks!