T
Takuya Kajiwara
Guest
This is Part 2 of 3 in the Lenses series.
In Part 1, I introduced the Presentation Lens: how Datadog shows data back to users.
Datadog has a lot of products. Agents, tracing libraries, RUM SDKs, AWS integration, log pipelines, retention filters⦠the list goes on.
Itβs not easy to remember how all of these pieces fit together.
Thatβs where the Lenses idea comes in.
In my previous post, I introduced this mental model and zoomed in on the Presentation Lens.
Now, letβs move on to the second one: the Collection Lens.
A Lens is not about a whole product, but about the part of its functionality that plays a certain role.
The Collection Lens is about how data makes its way into Datadog.
It covers the features that gather, refine, and shape signals before theyβre stored and shown.
I like to think of data collection in three phases:
Responsibility: get signals from your systems into Datadog.
Examples: the Datadog Agent on hosts and containers, tracing libraries inside applications, the Lambda Forwarder, RUM SDKs, AWS integration, synthetic bots, private locations, Workflow Automation (when itβs used to gather data), or even custom API calls.
Why it matters: without ingestion, thereβs nothing to observe. These components sit close to your systems and act as the first touchpoint for your telemetry.
Responsibility: adjust telemetry so it carries the right context β by adding whatβs missing and removing what doesnβt belong.
Examples: log pipelines, event pipelines, and Sensitive Data Scanner, which strips out credentials or personal information so only meaningful context remains.
Why it matters: telemetry without context is just noise. Enrichment makes the data actionable by adding what was missing, and tools like Sensitive Data Scanner make it safe and clean by removing details that arenβt needed. Together, they ensure the data is complete, relevant, and trustworthy.
Responsibility: manage volume and cost by filtering, sampling, or transforming data.
Examples: log indexes, generate metrics, APM retention filters, RUM retention filters, tag configurations for metrics.
Why it matters: collecting everything forever is ideal in theory, but rarely possible in practice. Reduction features give you levers to control scale and budget while still keeping the insights you need.
Trade-off: reduction always means some loss of fidelity. The art is to reduce smartly, without cutting away the signals youβll need later.
Looked at by product name, features like pipelines, indexes, or SDKs can feel scattered.
But through the Collection Lens, they form a simple flow: ingest β enrich β reduce.
This also highlights priorities:
So instead of asking, βWhich product does what?β, you can ask:
Once thatβs clear, you can decide which ingestion, enrichment, and reduction features to use.
This is the second post in the Lenses series.
Weβve now covered Presentation and Collection β how data is shown, and how data gets in.
Next up is the Execution Lens, which focuses on how Datadog takes action on your systems.
That one will be shorter, but it completes the picture.
Iβd love to hear your thoughts: does the Collection Lens help make Datadog easier to navigate?
Continue reading...
In Part 1, I introduced the Presentation Lens: how Datadog shows data back to users.
Datadog has a lot of products. Agents, tracing libraries, RUM SDKs, AWS integration, log pipelines, retention filters⦠the list goes on.
Itβs not easy to remember how all of these pieces fit together.
Thatβs where the Lenses idea comes in.
In my previous post, I introduced this mental model and zoomed in on the Presentation Lens.
Now, letβs move on to the second one: the Collection Lens.
Whatβs the Collection Lens?
A Lens is not about a whole product, but about the part of its functionality that plays a certain role.
The Collection Lens is about how data makes its way into Datadog.
It covers the features that gather, refine, and shape signals before theyβre stored and shown.
I like to think of data collection in three phases:
- Ingestion β bringing signals in.
- Enrichment β shaping context so the data makes sense (adding whatβs missing, removing what doesnβt belong).
- Reduction β trimming or transforming data to keep things practical.
Phase 1: Ingestion
Responsibility: get signals from your systems into Datadog.
Examples: the Datadog Agent on hosts and containers, tracing libraries inside applications, the Lambda Forwarder, RUM SDKs, AWS integration, synthetic bots, private locations, Workflow Automation (when itβs used to gather data), or even custom API calls.
Why it matters: without ingestion, thereβs nothing to observe. These components sit close to your systems and act as the first touchpoint for your telemetry.
Phase 2: Enrichment
Responsibility: adjust telemetry so it carries the right context β by adding whatβs missing and removing what doesnβt belong.
Examples: log pipelines, event pipelines, and Sensitive Data Scanner, which strips out credentials or personal information so only meaningful context remains.
Why it matters: telemetry without context is just noise. Enrichment makes the data actionable by adding what was missing, and tools like Sensitive Data Scanner make it safe and clean by removing details that arenβt needed. Together, they ensure the data is complete, relevant, and trustworthy.
Phase 3: Reduction
Responsibility: manage volume and cost by filtering, sampling, or transforming data.
Examples: log indexes, generate metrics, APM retention filters, RUM retention filters, tag configurations for metrics.
Why it matters: collecting everything forever is ideal in theory, but rarely possible in practice. Reduction features give you levers to control scale and budget while still keeping the insights you need.
Trade-off: reduction always means some loss of fidelity. The art is to reduce smartly, without cutting away the signals youβll need later.
Why This Model Helps
Looked at by product name, features like pipelines, indexes, or SDKs can feel scattered.
But through the Collection Lens, they form a simple flow: ingest β enrich β reduce.
This also highlights priorities:
- Ingestion and enrichment are essential. They make observability possible.
- Reduction is optional. Itβs the set of knobs you turn to keep data useful and affordable.
So instead of asking, βWhich product does what?β, you can ask:
- What data do I need to know my system is healthy, and to find the root cause when it isnβt?
Once thatβs clear, you can decide which ingestion, enrichment, and reduction features to use.
Closing Thoughts
This is the second post in the Lenses series.
Weβve now covered Presentation and Collection β how data is shown, and how data gets in.
Next up is the Execution Lens, which focuses on how Datadog takes action on your systems.
That one will be shorter, but it completes the picture.
Iβd love to hear your thoughts: does the Collection Lens help make Datadog easier to navigate?
Continue reading...
