State of the Observability Market

Both technologically and commercially, there are changes coming in the wider observability market that seem to be following a few trends.

S3-native storage

Both Observe and OpenObserve are natively using object-storage for telemetry storage and Parquet files to efficiently store and query telemetry data. ClickStack, Honeycomb and others are increasingly allowing either native rehydration from S3 or archiving of cold data from the platform to object-storage based on various thresholds. As volumes of telemetry data increases coming into these platforms, offloading to lower-cost tiers of storage are becoming key means of reducing cost of the platforms, and improving the ability to query and analyse ever-increasing datasets.

BYOB (Bring your own bucket)

Where these providers are increasingly changing their business models is the use of customers bringing their own storage and cloud environments for storage. I imagine a combination of both commercial constraints and a sense of data ownership are jointly driving this change. Most recently, Datadog introduced this capability, ultimately breaking the walled garden of data they previously had to allow customers local control of their data. Reduced costs of transport and ingestion from the source through the platform again driving the changes.

OTel, and the use of wide-structured events are converging the use of previously disparate systems into single sources for indexing and querying. A flow-on effect of this seems to be the use of columnar storage and more specifically Apache Iceberg/Parquet as a standard for efficient storage at scale. With newer providers settling on this format I imagine we will see a longer-term convergence from older providers on these technologies if they prove as efficient and performative as claimed. Given the scale of several clients of Observe, I wouldn’t be surprised to see this prove to be true.