BAM! Three Keystones of an AI Agent
When I started building AI agents, I approached it from a software engineering perspective: how do I write agents? How do I run them? There must be a lot of clever code to write here. But here’s what surprised me: creating a successful AI agent is not all that much about writing software. There’s already a host of great libraries and frameworks to bootstrap the technical foundation—it’s much more about iterating the details that make my agent unique.
After building and iterating on several agents, I’ve identified three keystones that capture much of what makes my agents unique: Brain, Actions, and Memory. Get these three right, and I’m well on my way to building agents that actually work.
Brain: The Model and Its Instructions
The Brain is the foundation—the LLM model and the system prompt that guides it. This is what gets used over and over again in the agent loop. This is where I initially vastly underestimated the power of iteration.
Choosing a suitable model for the job matters, of course. But the real magic happens when you iterate on the system prompt. Small changes in how you instruct your agent can lead to dramatic improvements in behavior.
Good system prompts are not 1-2 sentences—they’re easily hundreds of lines long. The challenge is that longer is not better, but more comprehensive is. This isn’t your regular report writing where you pad it to look “comprehensive.” It’s about packing as much information into as short and unambiguous a form as possible. And that inevitably takes iteration, learning, and tweaking. Not surprisingly, prompt engineering has become a discipline of its own.
Actions: The Tools Your Agent Can Use
An agent is only as capable as the tools given to it. This is where you decide what the agent can actually do in the world—read files, update databases, send emails, schedule meetings, whatever your use case demands.
But here’s the catch: it’s not just about creating a big bunch of tools. You need to iterate on the right set of tools AND provide the right instructions on how to use them—both in tool descriptions and in your system prompt.
The right tools, clearly described, turn a clumsy agent into a precise one. And yet again, getting it right most likely requires iterations, learning, and tweaking.
Memory: The Most Complex Keystone
Memory is where things get really interesting. This is about how the agent retrieves and stores information across interactions.
There are multiple layers to consider:
- Data retrieval strategy: From simple context dumping to full RAG pipelines
- Agent memories: Semantic, episodic, and procedural memories (more on these later)
- Feedback loops: Setting up a system to incorporate learnings from past interactions to future runs
From what I’ve found so far, everything I’ve tossed under memory in my simple framework requires by far the most actual software engineering. It’s a combination of building the technology for storage and retrieval pipelines, and tuning the instructions for what to store and how to fetch relevant information.
With feedback loops potentially affecting performance exponentially over time, this is a huge part of building a successful agent—and something I need to devote much more time to understanding.
My Conclusion: Less Software Engineering, More Agent Design
As an engineer, I definitely have a bias toward “building software.” The more I work with agents, the more I realize I need to fight that urge very deliberately. Sure, there’s some software required to enable working on the three keystones, and sure, when things need to be delivered at production grade and scale, there’s a healthy amount of good software engineering to be done.
But to get the agent to that point, I’ve come to the conclusion that success is much more about getting the software side out of the way as quickly as possible and devoting maximum time to iterating, learning, and tweaking agent design.
I think of it as “prompt engineering at scale”—tuning the three keystones until they work together effectively. Not debugging syntax errors, but observing behavior, forming hypotheses, and running experiments.
Brain, Actions, and Memory—only when I get those right enough through iteration do I really need to worry about how to get them into production.