Lab 5: Threat Model & AI Risk Register

This lab produces no code. It produces two documents that decide whether the code you wrote in Labs 1 through 4 is allowed to run anywhere real: a threat model for one agent, and a risk register that translates its gaps into terms a business owner can act on. This is the work that separates an engineer who builds agents from one who can be trusted to approve them, and it is disproportionately what interview panels probe — anyone can bolt on a filter, but drawing the trust boundaries correctly and stating residual risk honestly is a distinct skill. Pick one agent, work it end to end, and produce artifacts short enough that someone will actually read them.

Objective

Draw the data flow for one agent you built, mark its trust boundaries, run STRIDE per element to produce a threat table, classify each gap by blast radius and reversibility, build a risk register with full worked rows, and write a one-page executive summary a non-specialist can act on.

Prerequisites

One agent from Lab 1 or Lab 2, your findings table from Lab 3, and the trace schema from Lab 4 — the confirmed findings become risks and the traces become evidence artifacts. No installation is required beyond a place to write, but a repo keeps the documents versioned alongside the system they describe.

mkdir -p ~/labs/threat-model && cd ~/labs/threat-model
git init
touch data-flow.md stride-analysis.md risk-register.md exec-summary.md

Ground the vocabulary in NIST AI RMF and check obligations against EU AI Act and ISO 42001 before you publish a register you intend to show an employer.

Threat model only systems you own or are authorized to assess. Publishing a threat model of an employer’s or a vendor’s system — even one you built — discloses their architecture and control gaps, and that is a disclosure decision belonging to that organization, not to you. For a portfolio, model your own lab agent and say so explicitly.

Step 1: Draw the data flow and mark trust boundaries

Diagram what actually exists, not the architecture you intended. Then apply the rule that makes agent threat modeling different from ordinary application threat modeling: a trust boundary is crossed wherever data of a different trust level enters the context window. Retrieved documents, log lines, tool results, error strings, and memory entries all become tokens sitting next to your system prompt, and once they are in context the model has no reliable way to tell instruction from data. Network segments and process boundaries still matter, but they are not where the interesting agent risk lives.

    flowchart TD
    subgraph TRUSTED [Trusted zone - operator and host code]
        OP[Operator] --> HOST[Host process and agent loop]
        POL[Authorization policy and tool allow list]
        RED[Redaction pass]
        GATE[Human approval gate]
    end

    subgraph MODEL [Model provider - external]
        LLM[LLM inference API]
    end

    subgraph UNTRUSTED [Untrusted data sources]
        LOGS[Log files - attacker influenced]
        MEM[Persistent memory store]
        ERR[Tool error strings]
    end

    subgraph ACTIONS [State changing systems]
        TIX[Ticket store]
        OTEL[Trace backend]
    end

    HOST -->|prompt plus context| LLM
    LLM -->|tool call request| HOST
    HOST --> POL
    POL -->|allow| TOOLS[MCP tool server]
    TOOLS --> LOGS
    TOOLS --> MEM
    LOGS -->|TB1 untrusted content into context| RED
    MEM -->|TB2 stale claims into context| RED
    ERR -->|TB3 error text into context| RED
    RED --> HOST
    HOST --> GATE
    GATE -->|approval ref| TIX
    HOST --> OTEL

Name every boundary so the STRIDE table can reference it. In this diagram: TB1 is untrusted log content entering context, TB2 is persisted memory re-entering context in a later session, TB3 is tool error text entering context, TB4 is the host-to-model-provider boundary where data leaves your infrastructure, and TB5 is the approval gate where an agent proposal becomes an authorized action.

TB5 deserves a note. It is the only boundary in the diagram where a human is the control, which makes it both the strongest control and the one most likely to erode — an approval prompt that is always approved has the shape of a control and the function of a rubber stamp.

Step 2: Run STRIDE per element

Walk each element and each boundary through Spoofing, Tampering, Repudiation, Information disclosure, Denial of service, and Elevation of privilege. Not every category applies to every element; skip the empty cells rather than inventing content. Copy this shape:

Element	STRIDE	Agent-specific threat	Existing control	Gap
TB1 log content → context	Tampering	Indirect prompt injection: attacker writes a log line instructing the agent to change its verdict or call a different tool	System prompt untrusted-data rule; `<log_line>` framing; tool allow-list	Probabilistic only. Lab 3 A-02 showed delimiter escape partially succeeded. Host does not strip framing tags from ingested content
TB1 log content → context	Information disclosure	Injected text asks the agent to echo raw log contents into its summary	Host-side redaction before the model sees text	Coverage limited to known regex patterns; novel secret formats pass through
TB2 memory → context	Tampering	Memory poisoning: a false “approved scanner” note persists and suppresses real detections in later sessions	None	Confirmed in Lab 3 A-04. No provenance, no TTL, no write review on the memory store
TB3 tool error → context	Tampering	Error string carries instructions the agent follows on a failure path	System prompt rule only	Error path not covered by the tool-result framing applied to normal results
TB4 host → model provider	Information disclosure	Sensitive log content leaves the trust boundary in the request body	Redaction pass; no raw-argument logging	Residual and accepted. Requires contractual and retention controls, not code
Tool server: `create_ticket`	Elevation of privilege	Report writer or investigator induces ticket creation outside its role	Server-side role check; distinct credentials per agent	Holds. Lab 3 A-05 denied
TB5 approval gate	Spoofing	Agent fabricates an approval reference the server accepts	Prefix check on `approval_ref`	Confirmed weak. Lab 3 A-06: any `APPROVAL-*` string passes
Agent loop	Denial of service	Induced loop drives repeated tool calls, consuming budget	Iteration cap of 6	Cap is per run; nothing limits runs per hour or total spend
Whole system	Repudiation	Cannot reconstruct why an action was taken after an incident	OpenTelemetry trace with authorization, approval, and argument digests	Holds for instrumented paths; retry helper is uninstrumented

Two habits make this table useful. Cite the evidence — “Lab 3 A-06” beats “possibly weak,” and it means a reviewer can check you. And write “holds” where a control works; a threat model that lists only gaps gives no credit for the controls that are load-bearing, and the next engineer will not know which ones they must not remove.

Step 3: Classify each gap by blast radius and reversibility

Severity for agent risks is driven less by exploit difficulty than by what the agent can reach and whether the damage can be undone. Score both axes coarsely.

Blast radius — Contained (one run, one output a human reads before acting), Local (one system or dataset), Broad (multiple systems, or all future runs). Reversibility — Reversible (undo with no external effect), Costly (undo requires human work or notifying someone), Irreversible (external state changed, data left the boundary, a person acted on it).

Gap	Blast radius	Reversibility	Priority
Approval reference is prefix-checked only	Local — the ticket store	Costly — tickets page humans and must be retracted	High
Memory poisoning, no provenance or TTL	Broad — every future run reading the store	Costly — requires auditing and purging the store	High
Delimiter escape in log framing	Contained — one run’s verdict	Reversible — a human reads the summary	Medium
Error-path instructions not framed	Contained — one run	Reversible	Medium
No rate limit across runs	Local — spend	Costly — money already spent	Medium
Redaction misses novel secret formats	Local — data leaves the boundary	Irreversible — cannot unsend	High
Retry helper uninstrumented	Broad — audit blind spot on an unknown fraction of runs	Irreversible — cannot reconstruct the past	Medium

Memory poisoning outranks the delimiter escape despite being harder to notice, and that is the point of scoring reach rather than cleverness: the injection changes one answer, the poisoning changes every answer until someone finds it. Anything Broad plus Irreversible goes to the top regardless of how unlikely it looks.

Step 4: Build the risk register

The register is the artifact a risk owner reads, so risk statements go in business terms — consequence first, mechanism second. “Prompt injection in logs” means nothing to a service owner; “an attacker who can write a log line can suppress a security alert” gets a decision.

ID	Risk statement	OWASP	ATLAS tactic	Likelihood	Impact	Control	Owner	Evidence artifact	Status
R-01	An attacker who can write to a monitored log can cause the triage agent to suppress a real finding, delaying incident response	LLM01 Prompt Injection	Defense Evasion	Medium	High	Host-side content framing with tag stripping on ingest; human review of all `clean` verdicts on high-value assets	SOC tooling lead	`evidence/P4-log-triage.txt`; Lab 3 findings table	Open
R-02	A false claim written into agent memory silently suppresses a class of detections in all future runs until manually discovered	LLM04 Data and Model Poisoning	Persistence	Medium	High	Provenance and TTL on every memory entry; agent must cite memory provenance; quarterly memory review	Agent platform owner	`evidence/poison-followup.txt`; Lab 4 trace showing memory read	Open
R-03	A compromised or injected agent creates incident tickets without human authorization, eroding trust in the alerting pipeline	LLM06 Excessive Agency	Privilege Escalation	Low	Medium	Server-minted, signed approval references bound to the investigation digest; Lab 4 query Q1 as a detection	Agent platform owner	Lab 3 finding A-06; Lab 4 Q1 query output	In progress
R-04	Sensitive customer data in logs leaves the organizational boundary in a request to an external model provider	LLM02 Sensitive Information Disclosure	Exfiltration	Medium	High	Host-side redaction before every request; periodic redaction coverage review against sampled logs; provider retention terms reviewed	Data protection lead	Redaction module + coverage test results; provider terms record	Mitigated — residual accepted
R-05	An induced tool-call loop consumes the API budget, degrading availability for legitimate triage	LLM10 Unbounded Consumption	Impact	Medium	Low	Per-run iteration cap; per-hour run cap; spend alert at 80 percent of monthly budget	Agent platform owner	Lab 4 query on tool-call counts per trace	Open
R-06	Agent actions cannot be reconstructed after an incident because some code paths are uninstrumented	LLM09 Misinformation (accountability)	Defense Evasion	Low	Medium	Instrument every model entry point; alert on model-call spans lacking a parent run span	Agent platform owner	Lab 4 exported trace; span-coverage query	In progress

Every row needs a named owner and a real evidence artifact. A register with an owner column reading “team” and an evidence column reading “documented” is a compliance prop. Pointing at a specific trace, transcript line, or test result is what makes it auditable — and it is what Evidence Automation is ultimately about.

Note R-04’s status. Some risks are accepted, not fixed. Writing “mitigated — residual accepted” with a named accepting owner is more honest and more professional than leaving it open forever or pretending redaction is total.

Likelihood and impact bands need definitions written down somewhere in the document. “Medium likelihood” means nothing until you say whether it means once a quarter or once a year. Two people scoring the same risk differently is fine; two people using the same word to mean different things is not.

Step 5: Write the one-page executive summary

One page, no jargon, decisions at the top. The audience is someone who will not read the register and must still make a correct call.

AGENT RISK SUMMARY — SOC triage pipeline (lab environment)
Prepared 2026-03-16 by <name>. Scope: three-agent pipeline reading log files
and creating incident tickets, with human approval before ticket creation.

WHAT IT DOES
Reads security logs, summarizes suspicious activity, and proposes an incident
ticket that a person approves before it is created.

WHAT WE FOUND
Six risks. Two are high priority. The system's role separation and human
approval step both work as designed and should not be removed.

  1. Anyone who can write a line into a monitored log can influence what the
     agent reports. Our tests changed the agent's framing of a finding but did
     not cause an unauthorized action.
  2. False information written into the agent's memory persists and affects
     future runs. This is our highest-priority gap because it degrades quietly
     and affects every later analysis, not just one.
  3. The approval step can be satisfied by a value the agent itself can
     produce. No approval was actually bypassed in testing, but the control is
     weaker than intended and is being rebuilt.

WHAT WE ARE DOING
  - Adding provenance and expiry to agent memory, and requiring the agent to
    cite it (owner: platform, target: 30 days)
  - Replacing approval references with server-issued signed values bound to the
    specific proposal (owner: platform, in progress)
  - Reviewing redaction coverage against sampled production logs each quarter
    (owner: data protection)

WHAT WE ACCEPT
Log content is sent to an external model provider after redaction. Redaction
covers known secret formats and cannot be proven complete. Accepted by
<name/role> on <date>, subject to quarterly coverage review.

RECOMMENDATION
Suitable for advisory use with human review of every ticket. Not suitable for
autonomous ticket creation until items 2 and 3 are closed.

The recommendation line is the whole document. Everything above it exists to justify a single sentence about what this system may and may not be trusted to do unattended.

Interpreting the results

Read the two documents together and check three things. Does every high-priority gap have a control that is architectural rather than a prompt change — if the remediation column is mostly “improve the system prompt,” you have written a wish list, since probabilistic controls belong behind deterministic ones and never in front of them. Does every risk row trace to concrete evidence from Labs 3 and 4, or are some rows theoretical — theoretical rows are legitimate, but mark them so a reviewer knows which claims are tested. And does the executive summary say what the system may not do, since a summary with no stated limit is marketing.

Then reread your data flow with the register in hand. Most late-discovered risk comes from a boundary that was drawn wrong or omitted — a cache nobody diagrammed, a retry path that skips the policy check, a log sink that receives raw arguments. If the register surfaced something the diagram does not show, fix the diagram; it is the artifact everything else derives from.

Keep both documents versioned next to the code. A threat model dated the same week as the architecture is evidence of practice; one dated eighteen months ago is evidence of an audit. Present the pair together in interviews — see Portfolio to Offer and What Employers Screen For for how this reads to a panel.

Checklist

Data flow diagram reflects what exists, not the intended design
Every trust boundary named and numbered, including context-window crossings
STRIDE run per element, with empty categories skipped rather than padded
Controls that hold are recorded, not only the gaps
Every gap cites evidence from Lab 3 or Lab 4 where it exists
Each gap scored for blast radius and reversibility
Risk statements written in business consequence terms
Every register row has a named owner and a real evidence artifact
Likelihood and impact bands defined in writing
Accepted residual risk stated explicitly with an accepting owner
Executive summary fits one page and ends with a clear may/may-not recommendation

Lab 4: Building an Agent Audit Trail