Observability 2.0 and the Database for It

If I understand correctly the "big idea" here is:

1. Juice up your Traces with every attribute possible 2. Use a telemetry backend that relies on cheap object storage so that your costs don't explode. 3. ...profit?

Ok, but now we are exporting and storing everything about every request just so we can derive some previously cheap metrics like server CPU consumption? I guess for most applications the overhead of buffering, formatting and sending all of this telemetry data doesn't matter for folks?

Almost 8 years ago, when I was working as a Monitoring SRE at Google, I wrote a proposal to use compressed sensing to reduce storage and transmission costs from linear to logarithmic. (The proposal is also available publicly, as a defensive publication, after lawyers complicated it beyond recognition https://www.tdcommons.org/dpubs_series/954/)

I believe it should be possible now, with AI, to train online tiny models of how systems behave in production and then ship those those models to the edge to use to compress wide-event and metrics data. Capturing higher-level behavior can also be very powerful for anomaly and outlier detection.

For systems that can afford the compute cost (I/O or network bound), this approach may be useful.

This approach should work particularly well for mobile observability.

Opensearch and Elasticsearch do most/all of what this proposes. And then some.

The mistake many teams make is to worry about storage but not querying. Storing data is the easy part. Querying is the hard part. Some columnar data format stored in S3 doesn't solve querying. You need to have some system that loads all those files, creates indices or performs some map reduce logic to get answers out of those files. If you get this wrong, stuff gets really expensive and costly quickly.

What you indeed want is a database (probably a columnar one) that provides fast access and that can query across your data efficiently at scale. That's not observability 2.0 but observability 101. Without that, you have no observability. You just have a lot of data that is hard to query and that provides no observability unless you somehow manage solve that. Yahoo figured that out 20 years or so ago when they created hadoop, hdfs, and all the rest.

The article is right to call out the fragmented landscape here. Many products only provide partial/simplistic solutions and they don't integrate well with each other.

I started out doing some of this stuff more than 10 years ago using Elasticsearch and Kibana. Grafana was a fork that hadn't happened yet. This combination is still a good solution for logging, metrics, and traces. These days, Opensearch (the Elasticsearch fork) is a good alternative. Basically the blob of json used in the article with a nice mapping would work fine in either. That's more or less what I did around 2014.

Create a data stream, define some life cycle policies (data retention, rollups, archive/delete, etc.), and start sending data. Both Opensearch and Elasticsearch have stateless versions now that store in S3 (or similar bucket based storage). Exactly like the article proposes. I'd recommend going with Elasticsearch. It's a bit richer in features. But Opensearch will do the job.

This is not the only solution in this space but it works well enough.

I'm biased because I recently introduced ClickHouse at my company, but everything I've seen so far makes me think analytical and observability use cases like this "just work" in ClickHouse.

Just like Postgres became the default choice for operational/relational workloads, I think ClickHouse is (or should) quickly become the standard for analytical workloads. In both cases, they both "just work". Postgres even has columnar storage extensions, but I still think ClickHouse is a better choice if you don't need transactions.

A rule of thumb I think devs should follow would be: use Postgres for operational cases, and ClickHouse for analytical ones. That should cover most scenarios well, at least until you encounter something unique enough to justify deeper research.

Author here. Thanks @todsacerdoti for posting this.

I am big fan of the idea to have original data and context as much as possible. With previous metrics system, we lost too much information by pre-aggregation and eventually run into the high-cardinality metrics issue by overwhelming the labels. For those teams own hundreds of millions to billions time series, this o11y 2.0/wide event approach is really worth it. And we are determined to build an open-source database that can deal with challenges of wide events for users from small team or large organization.

Of course, database is not the only issue. We need full tooling from instrument to data transport. We already have opentelemetry-arrow project for larger scale transmission that may work for wide events. We will continue to work in this ecosystem.

This sounds a lot like structured logging with a fresh coat of paint. Wide events are nice conceptual model, but if you’ve been doing structured logs seriously, especially with something like Loki or ELK stack, you’re already capturing rich context per event — including things like user info, request paths, even DB queries if needed.

I’ve been using Loki recently and really like the approach: it stores log data in object storage and supports on-the-fly processing and extraction. You can build alerts and dashboards off it without needing to pre-aggregate or force everything into a metrics pipeline.

The real friction in all of these systems is instrumentation. You still need to get that structured event data out of your app code in a consistent way, and that part is rarely seamless unless your runtime or framework does most of it for free. So while wide events are a clean unification model, the dev overhead to emit them with enough fidelity is still very real.

It looks like what the grafana stack does but it's linking specialized tools instead of building one big tool (eg linking traces [0]).

The only thing then is that there is no link between logs and metrics, but I guess since they created alloy [1] they could make it so logs and metrics labels match, so we could select/see both at once ?

Oh ok here's a blog post from 2020 saying exactly this: https://grafana.com/blog/2020/03/31/how-to-successfully-corr...

[0]: https://grafana.com/docs/grafana/latest/datasources/tempo/tr... [1]: https://grafana.com/docs/alloy/latest/

At my company we seem to have moved a little in the opposite direction of observability 2.0. We moved away from the paid observability tools to something built on OSS with the usual split between metrics, logs and traces. It seems to be mostly for cost reasons. The sheer amount of observability data you can collect in wide events grows incredibly fast and most of it ends up never being read. It sucks but I imagine most companies do the same over time?

This model seems super expensive! I interpret it as traces on steroids which will make query complex and slow!

A lot of businesses haven't even nailed simple histograms with prometheus. I wouldn't like observability to become a full set of problems on its own!

Also timeseries is powerfull in observability because a lot of issues can be represented as cheap counters, gauges and distributions. I want to see a paradigm complimentary to this simple principle instead of producing nested documents with nested objects.

There are a whole bunch of attempts to unify metrics, logs and traces into a single DB now.

* InfluxDB (the newest Rust rewrite)

* http://openobserve.ai/

* https://uptrace.dev/

* Clickhouse powered solutions (eg https://signoz.io)

* ... ?

I'm quite skeptical about the "store raw data" approach. It makes querying much more complex and slower, storage much more expensive, etc.

Columnar databases that can store the data very efficiently are the way to go, IMO. They can still benefit from cheap long-term storage like S3.

The point that the trinity of logs, metrics and traces wastes a lot of engineering effort to pre-select the right metrics (and labels) and storage (by having too many information triplicate), is a good one.

> We believe raw data based approach will transform how we use observability data and extract value from it. Yep. We have built quuxLogging on the same premise, but with more emphasis on "raw": Instead of parsing events (wide or not), we treat it fundamentally as a very large set of (usually text) lines and optimized hard on the querying-lots-of-text part. Basically a horizontally scaled (extremely fast) regex engine with data aggregation support.

Having a decent way to get metrics from logs ad-hoc completely solves the metric cardinality explosion.

This article leaves me confused. The “wide event” example presented is a mishmash of all the different concerns involved with a business operation: HTTP request, SQL query, business objects, caches, …. How is this any better than collecting most of this information as separate events on a technical level (with minimal, if any, code changes: interceptors, middleware etc) and then aggregating afterwards?

From my perspective, this is just structured logging. It doesn’t cover tracing and metrics, at all.

> This process requires no code changes—metric are derived directly from the raw event data through queries, eliminating the need for pre-aggregation or prior instrumentation.

“requires no code changes”? Well certainly, because by the time you send events like that your code has already bent over backwards to enable them.

Surely I must be missing something.

> We believe raw data based approach will transform how we use observability data and extract value from it.

Perhaps we need to have generic database framework that properly and seamlessly cater for both raw and cooked (processed) for observability something similar to D4M [1].

[1] D4M: Dynamic Distributed Dimensional Data Model:

https://www.mit.edu/~kepner/D4M/

After reading this post I'm left wondering: you want to capture events. You want to have different views of them. Why don't you use Kafka and create a consumer per "view"?

just one word : uptrace, https://uptrace.dev/

a very satisfied user : trace, metrics, log in a perfect way