Observability & Policy Tooling

This is the toolchain that turns “the agent did something” into evidence a reviewer can query. It is also the least glamorous page in this set and the one that most reliably distinguishes an engineer from an enthusiast.

The claim you eventually want to be able to make is narrow and testable: for any agent action in the last ninety days, you can produce the trace that led to it, the policy decision that allowed it, the version of the code and prompt that ran, and the dependency inventory of the artifact that served it. Every tool below exists to support one clause of that sentence.

OpenTelemetry

• Link: https://opentelemetry.io
• What it is: A vendor-neutral standard and set of SDKs for generating traces, metrics, and logs. The Python SDK instruments your application; the Collector receives, processes, and exports telemetry; the semantic conventions define agreed attribute names, including a working set for generative AI and agent systems.
• Install: pip install opentelemetry-sdk opentelemetry-exporter-otlp for Python; the Collector ships as a binary and container image from the project.
• Why it matters for security: A span with a stable, conventional name is the difference between a log you can search and a log you can only read. When every agent emits a tool-call span with the same attributes, “show me every filesystem write an agent made last week” becomes a query rather than a project.

OpenTelemetry Collector

• Link: https://github.com/open-telemetry/opentelemetry-collector
• What it is: A standalone service that receives telemetry, applies processing pipelines, and exports to one or more backends.
• Install: Run the official container image or install a distribution binary; configuration is a YAML pipeline of receivers, processors, and exporters.
• Why it matters for security: The Collector is where you redact. Prompts and tool arguments frequently carry secrets and personal data, and a processor that scrubs them before export is a control you can point at during a privacy review. It is also the natural chokepoint for sampling policy, so security-relevant spans are never the ones dropped under load.

Jaeger

• Link: https://www.jaegertracing.io
• What it is: An open-source distributed tracing backend and UI for storing and exploring traces.
• Install: Run the official all-in-one container image for local work; production deployments use the separate components documented on the site.
• Why it matters for security: A multi-agent run is a distributed system whether you meant it to be or not. The Jaeger waterfall view is how you see that a delegated sub-agent made a tool call the parent never intended, which is the visual that makes a confused-deputy finding land in a room of engineers.

Grafana Tempo

• Link: https://github.com/grafana/tempo
• What it is: A high-scale distributed tracing backend that stores traces in object storage and integrates tightly with Grafana.
• Install: Deploy via the official container image or Helm chart per the project documentation.
• Why it matters for security: Retention economics decide whether you have evidence. Cheap object-storage-backed traces are what make a ninety-day or one-year lookback affordable, and an investigation is only as good as the window it can reach back into.

Prometheus

• Link: https://prometheus.io
• What it is: A metrics collection and alerting system built on a pull model and a time-series query language.
• Install: Run the official binary or container image; the Python client installs with pip install prometheus-client
• Why it matters for security: Detection at the aggregate level catches what per-request review misses. A sudden change in tool-call rate, refusal rate, or token consumption per session is a signal, and alerting on it turns your telemetry from forensic into preventive.

Grafana

• Link: https://grafana.com/oss/grafana/
• What it is: A dashboarding and visualization platform that queries Prometheus, Tempo, and many other sources in one place.
• Install: Run the official container image or install from the project’s packages.
• Why it matters for security: A dashboard is a communication artifact. When you can show a product owner the rate of blocked tool calls next to the rate of successful sessions, the guardrail conversation stops being abstract and starts being a tradeoff you can negotiate.

structlog

• Link: https://github.com/hynek/structlog
• What it is: A structured logging library for Python that emits key-value events, with processor pipelines and JSON rendering.
• Install: pip install structlog
• Why it matters for security: Free-text log lines are unqueryable at exactly the moment you need them. Emitting event="tool_call" with the trace ID, agent identity, tool name, and policy decision as fields is what lets an auditor answer a question without reading your code, and processors give you one place to redact before anything reaches disk.

Open Policy Agent and Rego

• Link: https://www.openpolicyagent.org
• What it is: A general-purpose policy engine. Policies are written in Rego, a declarative language, and evaluated against JSON input to return a decision.
• Install: Install the opa binary from the official releases or run the container image; the CLI includes opa test for unit-testing policies.
• Why it matters for security: Policy-as-code is what stops authorization logic from being scattered through prompt text. “Can this agent, acting for this user, call this tool with these arguments” becomes a versioned, unit-tested file with a decision log, and the decision log is itself audit evidence. Keeping the policy outside the model also means a prompt injection cannot argue with it.

Sigstore and cosign

• Link: https://www.sigstore.dev and https://github.com/sigstore/cosign
• What it is: Sigstore is a set of services for signing and verifying software artifacts using short-lived, identity-bound certificates. cosign is its CLI for signing and verifying container images and other artifacts.
• Install: Install the cosign binary from its official releases or via Homebrew.
• Why it matters for security: Agent deployments pull images, models, and increasingly MCP servers from places nobody vetted. Signature verification at admission is the control that turns “we think this is the build we reviewed” into a claim the pipeline enforces.

Syft and Grype

• Link: https://github.com/anchore/syft and https://github.com/anchore/grype
• What it is: Syft generates a software bill of materials from a container image, filesystem, or archive. Grype scans those same targets, or a Syft SBOM, for known vulnerabilities.
• Install: Install both binaries from their official releases, Homebrew, or the official container images.
• Why it matters for security: AI dependency trees are deep and change weekly. An SBOM produced at build time and stored with the artifact is what makes the next ecosystem-wide CVE a query against inventory rather than a week of archaeology.

Terraform

• Link: https://www.terraform.io
• What it is: An infrastructure-as-code tool that provisions and manages infrastructure from declarative configuration files with a planned, reviewable diff.
• Install: Install the terraform binary from the official downloads or via a package manager such as Homebrew.
• Why it matters for security: A lab that cannot be rebuilt identically produces results nobody can check. Declaring your lab environment means a reviewer can reproduce your finding, and it means the isolation controls around a code-executing agent are themselves reviewable code rather than a sequence of console clicks you half remember.

Putting it together

This stack is the point of Lab 4, where the deliverable is a queryable audit trail rather than a working agent. The compliance framing lives in evidence automation; the systems you are instrumenting come from the agent frameworks page, and the attacks you are trying to detect come from the red team page. Back to the tools index.