Things worth knowing about LLMs
Working with Large Language Models (LLMs) is easy: all we need to do is send a message. But if we look closer, we'll find there are things we can do to get better results, and that can be pushed far beyond direct interaction.
Mechanics of Large Language Models
To take the way we work with agents to the next level, we need a clear understanding of their core mechanics so we can use their strengths and avoid their weaknesses. Request: Each request to an LLM includes context and settings.
Platforms and Harnesses
Most of us will work with AI tools such as Cursor, Claude Code, Pi, or T3 Code. While each offers some unique features, they all share the same foundation: a language model.
Generalized Steering
You may have heard of prompting techniques such as chain of thought, few-shot prompting, or tree of thoughts, which were popular before the rise of LRMs (Large Reasoning Models). Then came general instruction files such as AGENTS.md, containing broad rules…
Takeaways
As I mentioned, this lesson should give you a broad, general overview of the core mechanics, and at this point, not everything needs to be perfectly clear. You can think of it as climbing a hill and looking around: you can see everything around you, just not…
Harness Personalization
Since LLMs know nothing about us and our project when a session starts, we want them to automatically get at least a general understanding of our context. Most harnesses currently support an AGENTS.md file that is loaded at the beginning of each conversation.
How does the context work?
One of the most important things to understand about the context is that even if the information exists in a context, it does not mean it will be used. And that goes even further, because initially a given instruction may be respected by the agent, but later…
Designing and generating skills
We already know that a skill is just a Markdown file loaded into context by a user or an agent, and that it may include additional scripts. We can find many examples on skills.sh and some great tips at agentskills.io, but when choosing them or creating our…
Tool use under the hood
At this point we know that LLMs can write structured responses, but also that they're really good at writing code. We use both to let them interact with the environment.
Subagents and Workers
Despite growing context windows, for many reasons, we may prefer keeping the interactions short and, above all, focused on a task. Both may be achieved in many ways, and the most important ones are: Subagents: many harnesses let an agent to spawn another…
Takeaways
At this point, it should be clear that managing context is one of the most important aspects of working with agents and we need to learn as much as possible about it. At the same time we need to keep in mind that AI is increasingly better at managing context.
Text Files and Cloud Platforms
Since LLMs' native "language" is text, Markdown and HTML got incredibly popular, despite all their limitations. Agents can easily access and manage them, and Markdown can be easily converted to other formats and even websites using Astro or Next.
Beyond Markdown
Markdown and HTML are not the only formats we can use for knowledge bases. There are also mdx and svx, which let us use components within markdown files.
Purpose of the knowledge bases
Knowledge systems such as internal wikis were built for many reasons: document organization, employee onboarding, or improving communication and processes. Their main users were humans, so we could assume they had at least a general understanding of the…
Self-management
Before ChatGPT, knowledge bases could be partially maintained by automations that pulled in data from external services or took care of archiving old notes. These automations could get pretty advanced, although building the logic was difficult.
Broader context
Agents' understanding is limited to what's already in their context. And even if we have a knowledge base, they can see only what's currently in it, which means they lose track of what came before and what's coming next.
An example
Let's say we want a personal knowledge base, strictly for us and our "personal AGI agent" that has broad context about us, our work, and our personal life. It aims to become our "second brain", a concept that's been widely known for years, but now managed…
Takeaway
Even with AI, knowledge bases are "easy to start and hard to master", but that shouldn't stop us from starting. And a personal knowledge base is one of the best things we can try, because it will help us explore many possibilities, but also run into issues,…
Harnesses
Looking at the available options, we can find many CLI harnesses, such as Codex, Claude Code, OpenCode, Pi, DeepSeek Harness, Grok Build; UI harnesses, such as Cursor, Zed, and Devin; as well as apps like t3code, humanlayer, and superset or agents such as…
Settings
Once we have good control over the harness and agents, so that we don't have to manage each of them separately, things like switching between providers or model routing become much easier, because all we need to do is set a few recommendations for the agent…
Teamwork
Working with agents in a team of people isn't obvious, and right now there are no clear answers about how to do it right. That's why we need to use our understanding of generative AI and the available tools to build workflows that serve us best and evolve…
Takeaways
The most important thing about working with an AI harness is that we can use AI to build and manage the setup, but we still need to keep it as simple as possible so we can iterate on it fast. Also, even though we discussed a setup with Pi and Herdr, that…
Speed of complexity growth
The simple answer to the question is that we move faster now, and that applies both to how quickly we ship features and to how quickly technical debt grows. And even if AI workflows take care of incremental improvements and maintenance, the app will have…
Executable Markdown vs Code
Skills may be seen as executable markdown files. When we consider what agents can do with even a simple set of instructions, it's fair to say that many products may be a single .md file.
Quality
When we think about systems created and maintained by agents at a scale that's impossible for humans to review, what can we do to ensure the quality of the product, while also staying on top of what's going on so we can make good decisions about its overall…
Getting remote
NOTE: the next section covers setting up a VPS with Tailscale, Pi and Herdr. It's entirely optional, and you can skip it because Cursor and Devin already offer remote agents out of the box.
System security
Remote agents open up many possibilities, but they also bring many risks. We can push the system toward greater autonomy, but we need to be aware of the risk of mistakes and attacks from outside.
Business value
The idea of running agents in the background may be interesting, but it may also raise concerns about costs, quality, and reliability, as well as its actual value. With the help of AI, it is easy to set up routines and triggers that wake agents to produce…
Agentic Loop
We already know that agents, at their core, are loops that keep working until the LLM decides not to use any tools and instead returns text and finishes the task. Between iterations we can manipulate context, settings and trigger logic we need.
Personality and Environmental Awareness
Models are getting more intelligent, and this may lead us to conclude that agents don't need our guidance. To some extent, that's true.
Context and Memory
Given all the context management issues we discussed last week, it should be obvious that managing context well is a critical part of any harness. Last week, we learned how language models handle context and how we can build a knowledge base.
Tools
Letting an agent interact with the outside world can be implemented in many ways. In general, we know the agent needs to be aware of its capabilities and able to use them together to accomplish the task in as few steps as possible, while maintaining its…
Takeaways
We work with agents every day, sometimes even without realizing how much happens under the hood. After this lesson, whether we build our own agents or keep using existing ones, we have a better understanding of what we may expect from them and what makes…
Architecture
Agents can be built from scratch, and with today's coding agents, that's very feasible. We can also use agentic frameworks such as Mastra or Agents SDK to make this process easier, since they handle common mechanics we don't need to build ourselves.
Core features
Designing agents is different from designing typical apps. As models get better, cheaper and faster, we can give them more freedom and space.
Providers and configuration
When building agents, we shouldn't really commit to a specific provider, not only because of vendor lock-in but also because we may want to use multiple models at once, or models that are specific to a given provider. For example, we might use Gemini for…
Designing tools
Agents can easily create tools for themselves, and all they need is documentation and some test requests. But if we look closer, there are some things we may want to consider to improve the efficiency and reliability of tools generated this way.
Context management
Here we're reaching the point where our harness can make a big difference compared to general agents, not because we teach the model what to do, but because we show it what's available, dynamically. We know that cache hit rate is one of the most important…
Workflows and long-horizon tasks
Building a good ambient agent is a great exercise of building reliable systems that involve non-deterministic logic. An additional challenge is to keep these systems lean by relying on primitives that AI can combine to go beyond the scenarios the agent was…
Takeaways
The example from this lesson should clearly present why we may want to build our custom harness or learn how to build extensions for existing ones to make them specialized for specific tasks. And even if this leads us to stop working on the agent, or new…
Personalization
Personalization often happens through custom instructions (such as AGENTS.md), skills and tools, but we know it can also happen through harness modifications, extensions and sandboxes. First, let's take a closer look at agents.md and some of its examples.
Model Context Protocol
We've mentioned MCP (Model Context Protocol) at least a couple of times so far and even had a chance to use it. Now it's time to take a closer look at the protocol and how it works under the hood.
Designing a custom MCP
The agents we have today can build an entire MCP server and deploy it if we simply point them to the documentation of the service we want to build it for. Our job, then, is to make this process easier for them (suggesting the template I mentioned, etc.),…
Skills & Tools
So far, we've mostly discussed skills as a way to avoid repeating the same instructions and to align the agent's behavior with our preferences. By now it should be clear that we want to avoid skills that tell the agent exactly what to do, so we don't limit…
Takeaways
At this point it should be clear that generalization is one of the most important skills to develop if we want to build good agents. But generalization also has to be "embedded" in the agents themselves.
Multi-agent architectures
The idea of multi-agent systems isn't new. It goes back well before the ChatGPT release, to the 1970s and 1980s and distributed artificial intelligence, and most of those concepts can be combined with the language models we have today.
Agent2Agent
Agents have to work together freely, but in most cases we don't want them to share their internals. The reason may be context management, specialization, our internal policies or the fact that the second agent does not belong to us.
Multi-agentic collaboration
The latest models are getting really good at long-horizon tasks, but all it takes is trying to work with agents within a team to see how difficult it is to manage work that can happen so fast. Even the simplest activities easily overlap and conflict with…
Orchestration
Working directly with agents makes us responsible for managing their work, helping them figure out issues, and steering their actions toward a bigger goal. For agents to operate on their own, most of these activities need to be translated into the system…
Takeaways
Now, putting this all together with what we've learned so far: - Agents: even though a single agent could potentially handle the entire process, splitting responsibilities between agents gives us control over their context, capabilities and permissions, as…
Guardrails: preventing the unpreventable
In S01E05, we discussed mechanisms for protecting against prompt injection. They're examples of guardrails: logic that aims to reduce behavior we don't want or increase the chance that we'll get what we want.
Observability in agentic systems
S02E04 mentioned that "activity tracking" helps us understand the logic and behavior of multi-agent systems. In fact, observability is critical because it opens the door to evaluations, moderation and self-improvement.
Evaluation
Agentic systems can be tested, but not in exactly the same way as classic apps, because their behavior varies between runs and their outcomes often cannot be verified with code alone. This leads us to the concept of evals, which are repeatable tests that run…
Takeaways
Observability, guardrails and evals are crucial parts of agentic systems, especially those deployed to production and running at scale. But even if we keep working with existing harnesses, we may still want to evaluate our tooling (CLI, MCP and skills), even…
Agent card boundary
Another agent asks Angie's agent about the calendar and asks it to delete an event, with an open agent, a rule in the prompt and an agent card.
Agent loop
Runs a minimal agent, a loop in which the model calls a tool to keep going or answers with text to stop.
AGENTS.md onboarding
Give a new agent the same task with no AGENTS.md, with pointers to the docs, or with all rules, and check the files it edits.
Approval gate
Runs six night jobs with no approval policy, a policy in the instructions, or a gate in the harness, and counts irreversible actions that run without a yes.
Background Workers
A Coordinator runs three long jobs while the user writes, once with blocking calls and once with background Workers, and code counts how long each message waits.
Chat template
Shows chat messages as the one text a model reads, with the special tokens that mark each role.
Cleanup routine drift
A nightly wiki cleanup with no review can lose facts when the model rewrites whole notes, but not when code checks each small edit.
Code mode
Runs one task twice, once with 30 function tools and once with only find and execute, and compares the input tokens and cost.
Confabulation probe
Asks one question with no context and with two different contexts, asks about an npm package that does not exist, and asks for the latest version of a package.
Context compaction
Compacts a long coding thread into a summary three times, as a harness does, and checks in code which facts each summary keeps.
Context self-report
Asks a sample coding agent how it sees the conversation, and checks which parts of its context the answer names.
Context window budget
Counts each block of one agent request and shows where the request sits in a model's context window.
Contextual recall
Find a memory from the whole conversation and the user profile, not only from the last message.
Effort routing
A Coordinator selects low or high reasoning effort for ten arithmetic tasks, and code compares the cost per solved task with fixed effort.
Did the track play?
A music service returns 200 even when nothing plays; a tool that checks the player stops the agent's false claims.
File picker
A coding agent answers four questions about a made-up codebase of 469 files, alone and after a small, fast model picks files for it, and code checks each number.
Fork or fresh subagent
Sends one task in four ways and checks which earlier user decisions each release plan keeps.
Function calling
Shows one function-calling round trip, where the model writes a tool call as JSON, our code runs the tool, and the result goes back to the model.
Guided components
A component library and a template in the instructions make agent-written .svx notes use the same components, and code checks that every fact stays in.
Heavy guidance
A fixed 50-line summary procedure and a short goal answer the same questions from the same documents, and code checks both answers.
Hybrid search
Searches the notes of a small team wiki three ways (keyword, semantic and hybrid) and checks each ranking against an answer key.
Image detail probe
Code draws seven kinds of simple pictures that vision models often read wrong, and checks how many answers a model gets right at low and at high image detail.
Image sidecar notes
A grep agent cannot find facts that exist only in images, and a text note written once per image at intake fixes this.
Inbox filing
One agent files eight inbox items into a small knowledge base with one set of rules, and code checks each result.
Injection defense layers
An inbox agent reads 12 items, five of them attacks, and code shows which attacks make it send Dana's address out and which defense layer stops each one.
Instruction drift
Makes one conversation longer and longer, and checks in code if the model still follows a rule from its instructions at each size.
Job end notifications
A Coordinator watches four background jobs, once by polling and once with a message from the harness, and code counts how late it sees each job end.
Knowledge health check
A script finds the mechanical problems in a wiki, a model finds the problems of meaning, and the script checks every fix.
Loop error recovery
One fault stops a loop with no error handling; a retry and a saved conversation let it finish.
MCP and CLI tool bridge
Shows that a tool from an MCP server and a local command both reach the model as function tools.
Memory extraction
Three memory modes read the same six conversations, and a score shows which mode keeps outdated facts, mixes up two people, or saves duplicates.
Next-token probabilities
Shows the most likely next tokens after a text and the probability of each token.
pass@1 and pass^k
Run ten checked tasks k times each and compare the share of passing trials with the tasks that pass every trial.
Per-user workspaces
Three ways to keep an agent inside one user's workspace, and the two that let it reach another user's files.
Pointer notes
A pointer note does not copy data from a tracker; it says where the data lives and how to read it, so the agent reads the current value instead of an old copy.
Fresh pull-request reviews
Reviews eight pull requests twice, with the diff only and with the team rules and touched files, and counts the seeded problems each review catches.
Premise order in a famous riddle
Gives a model the Monty Hall riddle with the same four lines in two orders, and checks if it answers the riddle it read or the one it knows.
Primitives vs specific tools
Runs six tasks for a tea shop's agent with eight specific tools and with four primitives, and checks the files it saves on foreseen and unforeseen tasks.
Privacy boundary
Routes 24 requests with a model router and with a rule in code, and counts the private requests that reach a cloud model.
Prompt cache prefix
The prompt cache reuses a request only while its start stays the same.
Prompt eval loop
A model rewrites a routing prompt from its training failures, and held-out cases show if each rewrite works on new messages.
Reasoning effort
Sends one checkable task at each reasoning effort level and compares reasoning tokens, time and cost with whether the answer is correct.
Responses API request
Sends one Responses API request with plain fetch and prints the JSON request and the JSON response.
Resumable run
A run waits for approval, crashes after the action, and resumes in three recovery modes; only a saved intent prevents a second action.
Retail agent: collaboration
The checkpoint where the lesson's Mastra retail agent delegates over A2A to delivery, supplier and personal shopping agents to solve a missing item.
Retail agent: exchanges
The checkpoint where the lesson's Mastra retail agent handles exchanges, with inventory lookup, policy checks in code, quotes and confirmation.
Retail agent: MCP and skills
The checkpoint where the lesson's Mastra retail agent gets MCP tools and resources, workspace files, code mode, elicitation, MCP Apps and Agent Skills.
Retail agent: order lookup
The first checkpoint of the lesson's Mastra retail agent, one agent that looks up orders, to compare instructions, memory and tools.
Retail agent: reliability
The checkpoint where the lesson's Mastra retail agent gets guardrails, OpenEnv evaluations, Jev response scoring, security cases and a reliability dashboard.
Rule count in a code review
Give a model 3 to 50 review rules, or 5 principles, and let code check every rule and test the fix.
A rule with its reason
Give two support agents the same privacy rule, one with its reason, and check which one keeps contact details out of a public reply.
Same prompt, many runs
Sends one prompt many times with the same settings and counts the distinct answers and the wrong answers.
Secret references
Runs one night job twice, with an API key in the agent's context and with a reference that the harness fills in, and counts the requests that carry the key to the model provider.
Self-configuring agent
An agent writes the settings of a made-up harness from memory, with its docs, and with a validator, and code checks six wishes and every key.
Self-querying
Asks a model for the sum of four atomic numbers, directly and after it recalls each number itself; code scores each answer against the periodic table.
Self verification
Two agents fix the same bug report with the same green tests, and only the agent that can open the app as the user sees her data and fixes all four books.
Semantic anchors
Sends one task with no guidance, with a format's rules, and with the format's name only, then checks the shape of each answer.
Session mining
A weekly routine reads past sessions of a coding agent and proposes skill changes, once from every transcript and once from what a script extracts first.
Silent context loss
The harness cuts or clears a tool result, with or without a note, and code checks the model's answers against the files.
Skill selection
Sends test requests to a model that sees six skills as names and descriptions, and scores which skill it loads for each one.
Skill vs saved script
Runs one daily brief for five mornings as a skill and as a saved script, and compares correct IDs, model calls, cost and time.
Subscribe to another schedule
A morning brief on a 07:00 clock sends yesterday's research when research runs late; a brief that subscribes to research.done sends today's facts, later.
Step replay
Ask an agent why it took a recorded step, then replay that step with one edit to check if the answer is true.
Structure test loop
A designer agent files 24 raw items into folders, a separate tester checks the structure against a clear definition of good enough, and a loop sends the misses back.
Structured output
Asks a model for JSON that follows a schema, then parses and checks the answer for a real city and for an invented one.
Subagent asks back
A booking subagent asks its parent for a missing date, then resumes its original task.
Subagents as tools
One agent and one subagent per contract read the same sixteen look-alike contracts; code checks every field and shows how small the main thread stays.
Summary index
The model writes index.md, one line per note, and an agent that reads it finds notes that grep misses.
Synthetic skill eval
Tests one edit to a refund skill on 3 hand-picked tickets and on 36 generated tickets, and scores both versions against the policy in code.
System prompt blocks
Run one background task with five system prompts, each one block longer, and count which runs write a correct report.
Task board context
Asks an agent for the same design with its task note only and with the whole task board, and checks each design for facts from the board.
Tokenization round trip
Encodes text into tokens, shows the ID of each token, and decodes the IDs back to the same text.
When an eval is too easy
Title-shaped questions hide a search tool's failure on everyday phrasing.
Tool feedback
Runs one task in four modes to show how errors, hints and large results in tool results change the model's next step.
Tool search
An agent finds one tool among 41 and loads its schema only when it needs it.
Tool and skill links
An image tool and its style skills name each other, so the agent loads the skill before the call, and still uses either one alone.
Unreached notes
A map of the wiki in the instructions sends an agent to the notes that its search does not reach; links in metadata help less.
Wake on signal
Two agents watch the same 28 days of a reading app, one every day and one only when a code check trips, and compare the real problems they catch, their false alarms and their cost.
Worker result handoff
A Coordinator reads six finished Workers as full screens, as the last lines, or as reports the Workers file, and code counts the facts it gets right.
Worker steering
A Coordinator sends a busy Worker one correction for all its texts in three ways, and code checks which texts use the new price at the end.
Worktree isolation
Two coding Workers edit the same file, first in one shared checkout, then in one Git worktree each, and tests count the edits that survive.
somebody slow
A message waits on the Challenge page. Your harness sends the course a shorter version that keeps its paragraph about the 26th exactly as written, then answers one question on a form.
who said what
Two recordings from one night on one product, one audio and one video. Your harness reports what each one says, and where, and files one record of the night.
what it shows
The support desk merged 412 reports about dates into one ticket, #4471, and marked it known. Its knowledge base (kb) comes as an export, not a site.
before i looked
A team is handing its repo over to you: a small service, its plan, its notes, and a skill that has to run in your harness. With your harness, start a new public repo from the course template, turn the plan into issues there, let your agents build it, and…
answer where i asked
With your harness, fix what is still open in the course's rig project and put an agent on a host of your choice. When the course reviewer asks where the rig stands, your agent answers on its own.
found out late
Two teams rebuilt the nightly reorder loop of stock-5 and left their sessions behind. The repo holds the plan as it was handed over.
nobody asked
With your harness, build a small agent people can reach by email: it gets mail at its own address, does what it is asked with its own tools, and replies in the same thread. Hand its address to the course.
schema first
Give your tools a home: with your harness, build and deploy a remote MCP server of your own — memo, upload and transcribe — and connect it to your own harness. Then hand it to the course.
two places
The course has two jobs and a skill for each. With your harness, hand each job to a worker of its own — one on this machine, one elsewhere.
while you're away
While you're away, your agent keeps working. With your harness, move it into a sandbox on a machine that stays up, give it a way to ping your phone, and register a channel the course can write to.