Agent engineering: the seven skills that survive production
Agent engineering is system engineering, not sentence writing: an agent that survives production needs system design, precise tool contracts, retrieval engineering, reliability patterns, security and safety, evaluation and observability, and product thinking. Prompting is the recipe; the agent engineer is the cook who understands ingredients, timing and failure modes.
'Prompt engineer' is an outdated job title
A job posting asked for a prompt engineer with experience in distributed systems, API design, MLOps, security and product management. That is five jobs, not one. But the posting was not wrong — it was just badly named. Building agents is system engineering, not sentence writing.
The chef analogy makes it precise. Prompt engineering is the recipe. Agent engineering is the cook: someone who understands ingredients, techniques, timing, safety and improvisation. A recipe gets you a meal only in the hands of someone who can handle a kitchen.
An agent is not a chatbot that answers questions. It acts: it books flights, processes refunds, queries databases, makes decisions. Building something that acts in the world is a different discipline from getting a model to talk.
The seven skills, in the order a production agent needs them
Each of these is a discipline on its own. The point of the list is that all seven live in the same system — and the failure of any one of them becomes a production incident.
- System design — the agent is an orchestra: the model decides, tools act, databases hold state, and specialists coordinate. How data flows and what happens when a component fails is architecture, not prompting.
- Tool and contract design — every tool has a contract: give me these inputs, I return this output. A vague contract makes the model improvise, and you do not want model imagination near financial transactions.
- Retrieval engineering — most production agents use RAG, and retrieval quality sets the ceiling on agent performance. Chunking, embedding models and re-ranking are the levers.
- Reliability engineering — APIs fail. Retries with backoff, timeouts, fallback paths and circuit breakers are the patterns that keep an agent from hanging or hammering a dead endpoint.
- Security and safety — prompt injection is real, and an agent is an attack surface. Input validation, output filters and permission boundaries are the defences.
- Evaluation and observability — trace every decision, every tool call, every retrieval. Run evaluation pipelines with known answers. 'It seems better' is not a deployment criterion; metrics are.
- Product thinking — humans must trust the agent. Clarity about when it is sure and when it is not, graceful failure handling, and clear escalation to a person when needed.
Where the highest leverage actually is
Two practical moves beat endless prompt tuning. The first is to read your tool schemas aloud. If a new engineer could not tell exactly what each tool does from the schema, the model cannot either — and the model will fill the gap with imagination. Tighten the contracts.
The second is to trace one real failure backwards instead of rewriting the prompt. Was the right document retrieved? Was the right tool chosen? Was the schema clear? In nine of ten cases the cause is the system, not the words. That is the difference between optimizing a sentence and engineering a system.
The practical start for a prompt engineer becoming an agent engineer
- Tighten the tool schemas first. The highest-leverage change in most agents. A schema that says only 'userID: string' lets the agent pass anything. Add patterns, examples and required fields.
- Trace one failure backwards. Instead of optimizing the prompt, follow the evidence: was the document retrieved, the tool chosen, the schema clear? In most cases the system is the cause, not the prompt.
- Add reliability patterns before they are needed. Retries with backoff, timeouts, fallbacks and circuit breakers are backend skills that most agent builders learn the hard way in production. Put them in before the incident.
- Instrument from day one. Tracing and evaluation pipelines are not add-ons. If you cannot measure success rate, latency and cost per task, you cannot improve them.
You cannot improve what you cannot measure — and you cannot secure what you have not threat-modelled.
The Pexon view: agents that survive production
Our position is that the seven skills are exactly the boundary between demo agents and production agents — and that boundary is where we work. We design and build agent systems where tool contracts are precise, retrieval is engineered, reliability is designed in, security is treated as a threat model, and observability is a given.
That is the same standard we hold for the platform underneath: agents monitored, evaluated and kept inside guardrails, with operational responsibility for the whole stack. We are not here to make demo agents shine; we are here to make production agents survive.
The honest risks. Prompting is only the beginning — without system design, agents fail on complexity. Vague contracts are genuinely dangerous: model imagination near financial transactions is a real risk. Reliability is underestimated; most agent builders learn failure handling in production, expensively. Security is not optional, because prompt injection is real and agents are attack surfaces. And without metrics there is no progress — 'it seems better' is not a deployment criterion.
Keep reading
- AI workflows vs AI agents: who decides — the level decision that precedes the engineering
- Why AI agents are hard to scale — the architecture discipline once agents work
- Enterprise RAG platform: retrieval for your own data — the money page for this cluster
- Company Brain hub: retrieval and agents for your data — the cluster this post belongs to
Questions we get asked about building production agents
Is prompt engineering dead?
No — it is the foundation, the recipe. But the agent engineer, the cook, needs more: system design, tool contracts, retrieval, reliability, security and metrics. Prompting alone does not keep an agent working when an API fails or a user input is hostile.
Do I really need all seven skills?
For agents that survive production, you need awareness of all of them. Some you build as you go; security and reliability are not optional. The highest-leverage starting point is the tool schemas — vague contracts make the model improvise where it must not.
Where do I start?
Two moves. First, tighten your tool schemas — read them aloud; if a new engineer would not know exactly what each tool does, fix that. Second, trace one real failure backwards instead of optimizing the prompt: in nine of ten cases the cause is the system, not the words.
Why is a vague tool contract dangerous?
Because the model fills the gap with imagination. A schema that says only 'userID: string' lets the agent pass 'John', 'user 123' or anything else. You do not want model imagination near financial transactions. The contract is what keeps the agent precise.
Next step
Get the agent skills gap assessed before the build
Two weeks, fixed price. We audit your agent stack against the seven skills — tool contracts, retrieval, reliability, security, metrics — and hand over the fix list with the highest-leverage changes first. The audit is yours whether or not we build the fixes.