Lab 4: Building an Agent Audit Trail

After an incident involving an agent, someone will ask a question you cannot answer from application logs: why did it call that tool, with what, under whose authority, and what data did it touch on the way. Traditional logging captures the call; it does not capture the decision. This lab instruments an agent with OpenTelemetry so that every run reconstructs as a tree of spans — one per run, one per model call, one per tool call — carrying the attributes an investigator actually needs. The design choice that defines the lab is that arguments are hashed rather than logged: an audit trail that copies sensitive tool inputs into your observability platform has moved your data problem, not solved it.

Objective

Instrument the agents from Labs 1 and 2 with a consistent span model and attribute schema, view a real trace locally, write investigation queries you would run after an incident, and export one sanitized trace as a portfolio artifact that demonstrates you can make agent behavior auditable.

Prerequisites

Python 3.10+ and a working agent from Lab 1 or Lab 2. Background on why this matters for compliance evidence is in Evidence Automation; the tooling landscape is surveyed in Observability Tools.

cd ~/labs/soc-mcp && source .venv/bin/activate
pip install opentelemetry-sdk opentelemetry-exporter-otlp
export OTEL_SERVICE_NAME="soc-agent"
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"

Step 1: Define the span model

Decide the tree shape before you write instrumentation, because inconsistent span names make queries impossible later. Three levels are enough:

agent.run                          (root — one per invocation, one trace id)
├── agent.model_call               (one per request to the model)
├── agent.tool_call                (one per tool invocation)
│   └── agent.authorization        (the allow/deny decision for that call)
├── agent.model_call
└── agent.human_approval           (one per approval prompt shown to a person)

Two rules keep this queryable. Span names are a small fixed vocabulary — never interpolate a tool name or agent name into the span name, put it in an attribute, because agent.tool_call with tool.name=create_ticket is filterable while create_ticket_span is not. And every span in a run shares one trace id, including the approval span, so “show me everything that happened in the run that created ticket 41” is a single lookup.

Step 2: Set the required attributes

This is the schema. Each attribute exists to answer a question an investigator will actually ask.

Hash arguments rather than logging them, and be explicit with yourself about why. Tool arguments frequently contain the exact data you spent Lab 1 redacting — log queries, ticket summaries, file paths, customer identifiers. Observability backends are typically broader-access and longer-retention than the systems the data came from, so copying raw arguments there quietly widens who can read that data and for how long, and it drags your traces into the scope of every data-protection obligation that covered the original. A digest preserves what investigations actually need — correlation and integrity, “the ticket was created with the same arguments the human approved,” “these forty calls were identical” — while carrying no payload. Where you genuinely need a human-readable hint, record a coarse derived field (tool.args.query_length=180, tool.args.asset_id=web-edge-01 for a non-sensitive enum) rather than the full argument object. Salt the digest per environment if the argument space is small enough to brute-force; an unsalted hash of a short enum is not a secret.

Step 3: Instrument the agent

# tracing.py
import hashlib
import json
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

AGENT_VERSION = "2026.03.14-3"


def init_tracing(service_name: str, console: bool = False) -> trace.Tracer:
    provider = TracerProvider(resource=Resource.create({"service.name": service_name}))
    exporter = ConsoleSpanExporter() if console else OTLPSpanExporter()
    provider.add_span_processor(BatchSpanProcessor(exporter))
    trace.set_tracer_provider(provider)
    return trace.get_tracer(service_name)


def digest(value) -> str:
    """Stable 16-hex digest. Sorted keys so equal arguments hash equal."""
    blob = json.dumps(value, sort_keys=True, default=str).encode()
    return "sha256:" + hashlib.sha256(blob).hexdigest()[:16]

Wrap the loop. Note that the tool span is created before the authorization decision, so denied calls still produce a span — a trail that only records successful actions cannot answer “what did it try to do?”

# instrumented_agent.py
from opentelemetry.trace import Status, StatusCode
from tracing import init_tracing, digest, AGENT_VERSION

tracer = init_tracing("soc-agent")

SYSTEM_PROMPT_ID = "investigator-system-v4"


def run_agent(agent_name: str, alert_id: str, system_prompt: str, messages: list) -> dict:
    with tracer.start_as_current_span("agent.run") as run:
        run.set_attribute("agent.name", agent_name)
        run.set_attribute("agent.version", AGENT_VERSION)
        run.set_attribute("agent.alert_id", alert_id)
        run.set_attribute("agent.prompt.template_id", SYSTEM_PROMPT_ID)
        run.set_attribute("agent.prompt.hash", digest(system_prompt))
        run.set_attribute("data.classification", "internal")

        for _ in range(MAX_ITERATIONS):
            with tracer.start_as_current_span("agent.model_call") as call:
                call.set_attribute("gen_ai.request.model", MODEL)
                response = client.messages.create(
                    model=MODEL, max_tokens=4000,
                    system=system_prompt, tools=TOOL_SCHEMAS, messages=messages,
                )
                call.set_attribute("gen_ai.usage.input_tokens", response.usage.input_tokens)
                call.set_attribute("gen_ai.usage.output_tokens", response.usage.output_tokens)
                call.set_attribute("gen_ai.response.stop_reason", response.stop_reason)

            messages.append({"role": "assistant", "content": response.content})
            if response.stop_reason != "tool_use":
                run.set_attribute("outcome.status", "ok")
                return extract_result(response)

            results = []
            for block in (b for b in response.content if b.type == "tool_use"):
                results.append(execute_tool_span(agent_name, block))
            messages.append({"role": "user", "content": results})

        run.set_status(Status(StatusCode.ERROR, "iteration budget exhausted"))
        run.set_attribute("outcome.status", "error")
        return {"error": "iteration budget exhausted"}


def execute_tool_span(agent_name: str, block) -> dict:
    with tracer.start_as_current_span("agent.tool_call") as span:
        span.set_attribute("tool.name", block.name)
        span.set_attribute("tool.args_digest", digest(block.input))

        with tracer.start_as_current_span("agent.authorization") as authz:
            decision, rule = authorize(agent_name, block.name)
            authz.set_attribute("authz.decision", decision)
            authz.set_attribute("authz.rule", rule)

        span.set_attribute("authz.decision", decision)
        if decision == "deny":
            span.set_attribute("outcome.status", "denied")
            span.set_status(Status(StatusCode.ERROR, f"denied by {rule}"))
            return {"type": "tool_result", "tool_use_id": block.id,
                    "content": f"ERROR: not permitted ({rule})", "is_error": True}

        if block.name in IRREVERSIBLE_TOOLS:
            with tracer.start_as_current_span("agent.human_approval") as gate:
                approval_ref = request_approval(block.input)
                gate.set_attribute("approval.granted", approval_ref is not None)
                gate.set_attribute("approval.ref", approval_ref or "")
                gate.set_attribute("tool.args_digest", digest(block.input))
            if approval_ref is None:
                span.set_attribute("outcome.status", "denied")
                return {"type": "tool_result", "tool_use_id": block.id,
                        "content": "ERROR: human denied approval", "is_error": True}
            span.set_attribute("approval.ref", approval_ref)
            block.input["approval_ref"] = approval_ref

        output = ALLOWED_TOOLS[block.name](**block.input)
        span.set_attribute("outcome.status", "ok")
        span.set_attribute("tool.result_digest", digest(output))
        return {"type": "tool_result", "tool_use_id": block.id, "content": output}

Recording tool.args_digest on both the approval span and the tool span is what makes approval bypass detectable. If the digests differ, the arguments changed between what the human saw and what executed — which is exactly the Lab 3 T5 finding, now visible in telemetry instead of only in a code review.

Step 4: Run a collector and view the trace

Start with the console exporter — it needs nothing running and shows you the exact JSON your backend will receive.

python -c "
from tracing import init_tracing
init_tracing('soc-agent', console=True)
import instrumented_agent as a
a.run_agent('ticketer', 'ALERT-2026-03-14-A', a.SYSTEM_PROMPT, a.seed_messages())
"

A single tool span looks like this. Read it as an auditor would: everything needed to reconstruct the decision, nothing that leaks the data.

{
  "name": "agent.tool_call",
  "context": {
    "trace_id": "0x7d3f9a12c4be8817a01f5b6c9d2e3f40",
    "span_id": "0x2b91c0de77a4f318"
  },
  "parent_id": "0x91aa4c0e2f118b73",
  "start_time": "2026-03-14T09:41:22.118Z",
  "end_time": "2026-03-14T09:41:29.902Z",
  "status": { "status_code": "OK" },
  "attributes": {
    "agent.name": "ticketer",
    "agent.version": "2026.03.14-3",
    "tool.name": "create_ticket",
    "tool.args_digest": "sha256:4ab0f71c9d2e5583",
    "tool.result_digest": "sha256:c19d4477ab30e2f1",
    "authz.decision": "allow",
    "authz.rule": "role:ticketer",
    "approval.ref": "APPROVAL-8f2c1ad04b19",
    "data.classification": "internal",
    "outcome.status": "ok"
  }
}

Once the shape looks right, point OTEL_EXPORTER_OTLP_ENDPOINT at a locally running collector or trace UI and browse the tree visually. Any OTLP-compatible backend works; see Observability Tools for options.

Step 5: Write your investigation queries

Instrumentation you have never queried is a guess about what you will need. Write the queries now, run them against real traces, and fix the schema where a query turns out to be impossible. Pseudo-query form below — translate to your backend’s syntax.

Q1 — Which runs called the ticket tool without an approval reference? This is the direct detection for the human-in-the-loop bypass from Lab 3.

span.name = "agent.tool_call"
  AND tool.name = "create_ticket"
  AND (approval.ref = "" OR approval.ref IS NULL)
  AND outcome.status = "ok"
-> return trace_id, agent.name, agent.version, start_time

Q2 — Where did approved arguments differ from executed arguments? Catches modification between the gate and the call.

JOIN span("agent.human_approval") AS h
  ON span("agent.tool_call") AS t USING (trace_id)
WHERE h.tool.args_digest != t.tool.args_digest
-> return trace_id, t.tool.name, h.tool.args_digest, t.tool.args_digest

Q3 — Which agent and prompt version were in force when denials spiked? Correlates a behavior change with a deploy, which is usually the first question after “did something change?”

span.name = "agent.authorization" AND authz.decision = "deny"
GROUP BY agent.name, agent.version, agent.prompt.hash
ORDER BY count DESC, window = last 7d

Two more worth adding once these work: runs where a single trace contains more than N agent.tool_call spans (the denial-of-wallet signal from Lab 3 T7), and runs touching data.classification = restricted that had no approval span at all.

Step 6: Export a sanitized trace as an artifact

Run one clean end-to-end pipeline, capture the console exporter output, and save the full trace as JSON. Before publishing, strip anything environment-specific: real hostnames, internal service names, API endpoints, employer identifiers. Keep the digests — they are meant to be publishable, which is a large part of why the design hashes rather than logs.

Pair the JSON with a short annotated walkthrough: here is the run span, here is the model call that decided to create a ticket, here is the authorization decision, here is the human approval and its reference, here is the tool call carrying the matching digest. Then add the one paragraph that makes it a security artifact rather than a demo — what an investigator can now determine that they could not before, and what this trace still would not tell you. Store it alongside your other Portfolio Artifacts.

Interpreting the results

A healthy trace is boring and complete. Every run has a root span with an agent name, version, and prompt hash. Every tool call has an authorization decision. Every irreversible call has an approval span with a matching digest. Every run ends with an explicit outcome.status, including the failures.

The gaps are what to look for. Tool calls with no authorization child mean a code path bypasses your policy check. Runs that end with no terminal status mean the process died and you cannot tell whether the work completed. Model calls with no parent run span mean something is invoking the model outside your instrumented entry point — often a retry wrapper or a helper written after the instrumentation. Approval spans that always show granted=true within a second or two of being created mean the human is not reading the prompt, which is a control failure that no amount of telemetry fixes but only telemetry reveals.

Finally, audit your own trail for leakage before you trust it. Grep an exported trace for the patterns you redacted in Lab 1 — key prefixes, email addresses, IP addresses. Finding one means an attribute is carrying raw content somewhere, and the fix belongs in the instrumentation, not in the backend’s masking rules. Then carry the resulting controls and gaps into the risk register in Lab 5, where “agent actions are reconstructable” becomes a control with an evidence artifact attached.

Checklist

• Span model documented before instrumentation, with a fixed span-name vocabulary
• Root run span per invocation; all spans share one trace id
• Agent name, agent version, model, prompt template id, and prompt hash recorded
• Tool arguments hashed or digested, never logged raw
• Authorization decision and rule recorded, including denials
• Human approval reference recorded and bound to an argument digest
• Data classification recorded on every run
• Trace viewed end to end via console exporter or a local collector
• Three investigation queries written and actually run against real traces
• Exported trace sanitized, grepped for leaked secrets, and annotated