September 17th, 2024

OpenTelemetry Tracing in < 200 lines of code

OpenTelemetry tracing simplifies to structured logging and context propagation, using spans for metadata. It supports distributed tracing via HTTP headers and requires instrumentation to capture trace data for monitoring.

Read original articleLink Icon
OpenTelemetry Tracing in < 200 lines of code

OpenTelemetry tracing can often seem complex to developers, who may view it as a "black box." However, it can be simplified into a model that resembles structured logging combined with context propagation. Tracing involves creating spans, which are essentially enhanced log entries that include essential fields like name, timestamp, duration, and IDs (trace ID, span ID, and parent span ID). Each trace is a collection of these spans, allowing developers to track the flow of requests through their applications.

To implement tracing, developers can create a simple library that generates spans and manages context. This library can be expanded to support distributed tracing, which allows trace context to be passed between different systems. This is achieved by including trace information in HTTP headers, following a standardized format. Instrumentation, or wrapping existing code to track operations, is also crucial for effective tracing.

The article concludes with a demonstration of how to send trace data to a service like Honeycomb, emphasizing the importance of structured data for effective monitoring and debugging in distributed systems.

- OpenTelemetry tracing simplifies to structured logging and context propagation.

- Spans are key components of tracing, containing essential metadata.

- Distributed tracing allows tracking across multiple systems using standardized HTTP headers.

- Instrumentation is necessary for wrapping code to capture trace data.

- Sending trace data to services like Honeycomb enhances monitoring capabilities.

Link Icon 8 comments
By @vrosas - 5 months
While the libraries and the documentation for otel are bloated messes, I maintain that any platform that isn’t using some sort of tracing system is practically negligent in their engineering duty. If you’re still out there querying logs with some giant sql statements you’re missing out. The pure wonder of being able to click on an http request and seeing every service it touched, every application log outputted, and every database query it ran, and all the timings of each of those is magical.
By @Ciantic - 5 months
The utility of tracing is great, I've been using Azure Application Insights with NodeJS (and of course in .NET). This is relatively simple because it monkey patches itself everywhere if you go through the "classic" SDK route. Then adding your own data to logs is just a few simple functions trackTrace, trackException, trackEvent, etc.

However, if you want to figure out how it works you might be scared, it is not lightweight. I just spent a few days digging through the Azure Application Insights NodeJS code base which integrates with OpenTelemetry packages. It's an utter mess, a huge amount of abstractions. Adding it to the project brought 100 MB and around 40 extra packages.

By @thewisenerd - 5 months
the "true spec is the data" is very powerful.

for example, we translate our loosely OTEL-based telemetry into a format which was consumable by any otel collector.

shim a few fields, et voila! can be read by Jaeger-UI. free trace tree visualization.

By @krashidov - 5 months
It's so easy in node. I miss node.

Setting this up in the mess that is Python/Gunicorn/asgi/wsgi/celery/Django has not been as easy

By @sandelz - 5 months
While otel is really nice and easy to integrate (at least on .net and node) into software, the collector/UI side seems to be overly complex.

I have used application insights on Azure on my day job but I was wondering is there a simple self hosted collector/UI to use?

By @WuxiFingerHold - 5 months
I've done something similar for a small service running in Docker compose. Maybe I didn't think through it but the complexity and huge dependencies introduced by the official libs immediately turned me off. DIY was very easy and simple.
By @h1fra - 5 months
I have one biff against otel, it's not possible to stream a trace. You have to build a big object in memory that is not suitable for a long-running process. And because of that it's not possible to start it somewhere and finish it somewhere else
By @caseyw - 5 months
OpenTelemetry (OTel) and the OpenTelemetry Protocol (OTLP) are immensely powerful tools. The ability to emit telemetry data from any source, coupled with a receiver that can sample, filter, pipe, and potentially reshape the data to suit any need, is a game changer. This flexibility revolutionizes how we approach observability and monitoring across diverse systems.