Layer 2 — Engineering Craft – Agentic AI Security Career

Layer 2 — Engineering Craft

Security people who cannot build lose arguments to people who can. In agentic security specifically, the artifacts that convince anyone — a working red team harness, a sandboxed agent runtime, an audit trail that survives an auditor’s questions — are all engineering deliverables. This layer is the ordinary modern toolchain, with one skill promoted well above the rest: observability. Agent behavior is only auditable if it was instrumented, and instrumentation is a design decision made before the incident, not after.

Skill → minimum bar → how you demonstrate it

Skill	Minimum bar for this field	How you demonstrate it
Python	Write async code, type-annotate, package a CLI, test it. Read a framework’s source when docs are wrong	A repo where the agent, the tools, and the tests are all yours
Git	Branching, rebase, resolving conflicts, meaningful history, signed commits	Clean commit history on a portfolio project; PRs with review comments
REST / GraphQL	Design and consume APIs; understand auth headers, pagination, error semantics; know why GraphQL’s flexible query surface is an authorization problem	A tool server your agent calls, with authz enforced at the endpoint
Docker	Multi-stage builds, non-root users, minimal base images, no secrets in layers	A Dockerfile that runs your agent as an unprivileged user with a read-only root filesystem
Kubernetes	Pods, services, namespaces, RBAC, network policies, secrets handling	A network policy that denies egress by default for the agent workload
CI/CD	Pipelines that lint, test, scan, and build; understand what a compromised runner can reach	A pipeline that fails the build on a red team regression test
Terraform	Write and read modules, understand state as a sensitive artifact, plan/apply discipline	The infrastructure for your lab environment, in code, reviewable
Observability	Instrument with OpenTelemetry, emit structured logs, propagate trace context across services and tool calls	A trace where a single agent task is one connected tree, tool calls and all

Sequence advice: Python and Git are prerequisites for everything else, containers before Kubernetes, and observability while you build rather than after. Retrofitting tracing onto an agent you already wrote is roughly twice the work and teaches you half as much.

Observability is the differentiating skill

Every other skill in this layer is table stakes shared with any backend engineer. Instrumenting an agent so that its behavior is reconstructable is not — and it is exactly what separates a security engineer who can support an agentic platform from one who can only review its design docs.

The test is a single question, asked after the fact: which tool did the agent call, with what arguments, on whose authority, and why? If your telemetry cannot answer all four, you do not have an audit trail. You have logs.

The question	What must be in telemetry	Failure if missing
Which tool?	Tool name and version, target server, MCP server identity	Cannot scope an incident to affected integrations
With what arguments?	Argument digest, and redacted or field-level structure	Cannot tell a benign call from a malicious one
On whose authority?	Actor identity, delegated principal, credential scope, approval reference	Cannot answer “was this user allowed to do that?”
Why?	Prompt hash, retrieved document IDs, plan step, parent span	Cannot find the injection that caused it

Do not log raw prompts, raw tool arguments, or retrieved document contents into general-purpose telemetry. You will replicate the sensitive data you are supposed to be protecting into a system with looser access controls than the original. Log hashes and identifiers, keep the payloads in a restricted store with its own retention policy, and make the trace point at them.

What an agent trace should contain

Model each agent step as a span, with tool calls as children. A minimal, useful span for a tool invocation:

{
  "name": "agent.tool.invoke",
  "trace_id": "…", "span_id": "…", "parent_span_id": "…",
  "attributes": {
    "agent.id": "triage-agent",
    "agent.step": 4,
    "gen_ai.model": "…", "gen_ai.temperature": 0.2,
    "prompt.hash": "sha256:…",
    "tool.name": "ticket.update", "tool.version": "1.3.0",
    "tool.args.digest": "sha256:…",
    "authz.principal": "user:1042", "authz.decision": "allow",
    "authz.policy": "agent-tools/ticket-write",
    "approval.ref": "hitl-2291",
    "retrieval.doc_ids": ["kb:114", "kb:207"]
  }
}

Notes that matter more than the exact schema:

Trace context must propagate across process boundaries — orchestrator to tool server to downstream API. A trace that stops at the tool server boundary hides the part you care about.
authz.decision belongs in the trace, and it should come from a real policy evaluation, not from the model deciding it was fine.
approval.ref links to the human-in-the-loop record. For any high-impact action, the trace should point at who approved it and what they were shown.
Hashes make traces comparable. Identical prompt.hash across a spike of calls is a signal; the raw prompts are a liability.
Structured logs carry the same correlation IDs, so log and trace queries meet in the middle.

Lab 4 is this section as a build exercise; the tooling landscape is in observability tools; the reason auditors care is in evidence automation.

Sandboxing and containment is an engineering skill

Containment is where security design becomes code you can be wrong about. Treat the agent runtime as hostile-by-assumption: not because the model is malicious, but because its input is attacker-reachable and its output drives execution.

Layer	What you configure	What it buys you
Process	Non-root user, dropped capabilities, seccomp profile, no privileged mode	A code-execution tool cannot escalate to the host
Filesystem	Read-only root, a single writable scratch mount, no host mounts, no credential files in the image	An agent cannot read secrets it was never meant to see, or persist across runs
Network egress	Default-deny, explicit allowlist by destination, no wildcard DNS, egress through a logging proxy	The dominant exfiltration path becomes a short, auditable list
Resource	CPU/memory limits, wall-clock timeout, token and step budgets	Runaway loops and cost-based denial of service become bounded
Credential	Short-lived, per-task, per-user tokens injected at call time — never baked into the image or environment	A leaked context window leaks something that has already expired
Lifecycle	Ephemeral containers, one task per instance, no reuse of a compromised runtime	Cross-task contamination through leftover state stops being possible

Two rules worth internalizing:

Egress allowlisting is the highest-leverage single control for an agent that touches untrusted content. Most exfiltration in this space is the agent making an ordinary outbound request it was persuaded to make.
Code execution tools need their own blast radius, separate from the orchestrator. If your “run Python” tool shares a filesystem and network namespace with the planner, you have one process, not two components.

# Sketch: the shape of the guarantee, not a copy-paste manifest
securityContext: { runAsNonRoot: true, readOnlyRootFilesystem: true,
                   allowPrivilegeEscalation: false, capabilities: { drop: ["ALL"] } }
resources: { limits: { cpu: "1", memory: "1Gi" } }
# plus: NetworkPolicy with default-deny egress and an explicit allowlist

CI/CD and IaC, with the agentic twist

Both are standard practice with one addition each that people miss.

For CI/CD: your pipeline is now a place where agents run. AI coding assistants and automated review agents execute inside runners that hold repository write access, package registry credentials, and sometimes deployment keys. Treat the runner as a workload with an identity and scope it accordingly — short-lived tokens, no long-lived deploy keys in the environment, and separation between the job that builds and the job that deploys. The second addition: put a red team regression suite in the pipeline. A prompt injection you fixed last month should fail the build if it comes back.

For Terraform: the state file is a sensitive artifact — it can contain secrets in plaintext and it describes your entire attack surface. Remote state with encryption, access control, and locking is the baseline. The agentic addition is that your agent’s IAM role, its network policy, and its egress allowlist should all be in code, so that a change to the agent’s blast radius shows up as a reviewable diff rather than a console click nobody saw.

# The point is the review artifact, not the syntax:
# a widened agent permission becomes a visible line in a PR.
resource "aws_iam_role_policy" "agent_tools" {
  # scope changes here require a human to approve a diff
}

Common gaps by background

Coming from	Usual gap	What to build to close it
Security analyst / SOC	Writing production Python, Git workflow, CI	Lab 1: a log triage agent with tests and a pipeline
Backend engineer	Network policy, IAM scoping, containment mindset	Take an agent you wrote and lock its egress to an allowlist without breaking it
Data / ML	Containers, IaC, distributed tracing	Wrap a model-calling service in OTel and read your own traces
Infrastructure / SRE	Agent-specific span attributes, human approval flows	Add `authz.decision` and `approval.ref` to an existing traced service

A portfolio repo that is containerized, traced, egress-restricted, and CI-tested does more hiring work than any certification in this layer. It is also the substrate every later layer plugs into — see portfolio artifacts.

Self-check

My agent emits a single connected trace per task, across process boundaries.
I can answer which tool / what arguments / whose authority / why, from telemetry alone.
No raw prompt, argument, or retrieved document lands in general telemetry.
My agent container runs non-root with a read-only root filesystem.
Egress is default-deny with an explicit allowlist I can recite.
Credentials are short-lived, per-task, and injected at call time.
My CI fails on a red team regression test, not just on unit tests.
My infrastructure is in Terraform and reviewable by someone else.

Next: Layer 3 — AI Tooling Mastery, where the thing being instrumented gets interesting.

Layer 1 — Security Foundations Layer 3 — AI Tooling Mastery