How We Built a Self-Improving AI Ops Repo

An AI agent that's 90% accurate at each step only has a 59% chance of completing a five-step task. At ten steps, it drops to 35%. By twenty, you're looking at 12%. The math is simple — multiply the probabilities and watch them shrink. The more you ask the LLM to do end-to-end, the more errors compound until your "intelligent automation" spends more time failing than working.

We ran into this firsthand. We'd ask Claude to research a topic, draft an email, format it in HTML, and send it via Gmail SMTP. Four steps. It would nail three and hallucinate the fourth — a wrong API call, broken HTML, or a fabricated email address. Every time we fixed one step, another would drift. The agent was capable at each individual step, but chaining them together turned small inaccuracies into cascading failures.

So we stopped treating the LLM as a do-everything machine and started treating it as a decision-maker that calls reliable tools. The result is an operations repo — a single folder of Markdown instructions and deterministic scripts — that gets stronger every time it runs.

The Core Idea: Separate Decisions From Execution

Most AI agent setups fail because they blur two fundamentally different types of work. Decision-making — figuring out what to do, in what order, and how to handle edge cases — is inherently fuzzy. It requires judgment, context, and adaptability. This is what LLMs are good at. Execution — calling an API, formatting data, sending an email — is inherently deterministic. It should work the same way every single time, regardless of who or what triggers it.

When you force an LLM to handle both, you're using a probabilistic tool for deterministic work. That's the mismatch. Our solution is a 3-layer architecture that gives each type of work to the right system:

• Directives — Markdown files that define what to do. Goals, inputs, steps, edge cases. Like SOPs you'd hand to a new employee.
• Orchestration — The LLM itself. It reads directives, decides the right sequence, calls the right tools, handles errors, and communicates with the operator.
• Execution — Python and bash scripts that do the actual work. API calls, file operations, data processing. Testable, reliable, deterministic.

The LLM makes the decisions. Code does the work. Markdown ties them together. Let's look at each layer.

Layer 1: Directives

Directives live in a directives/ folder as plain Markdown files. Each one describes a single repeatable task — the goal, the inputs it needs, the steps to follow, the execution scripts to call, and the edge cases that have come up in past runs.

Here's a simplified example:

# Send Onboarding Email

## Inputs
- Recipient name (required)
- Recipient email (required)
- Role / title (optional)

## Steps
1. Compose HTML email using the template below
2. Write it to `.tmp/onboarding_email.html`
3. Run: `python execution/send_email.py --to "email"
     --subject "Welcome!" --html-file .tmp/onboarding_email.html`
4. Confirm delivery

## Edge Cases
- Gmail App Passwords require 2FA to be enabled
- Google Workspace uses the same SMTP settings as consumer Gmail

No code. No abstractions. Just clear instructions that any LLM — or human — can follow. The directive tells the agent what to do without requiring it to figure out the underlying mechanics. It also accumulates knowledge over time: every edge case that's been encountered is documented right there in the file, so no future run hits the same problem twice.

Layer 2: Orchestration

This is the LLM — in our case, Claude running through Claude Code, Anthropic's coding CLI. Its job is intelligent routing: read the directive, figure out the right sequence of tool calls, handle errors, and ask for clarification when something is ambiguous. The LLM excels here because these are judgment calls. Should I use this template or that one? The script threw an error — is it a transient issue or a bug I need to fix? The user's request is vague — should I ask for more detail or make a reasonable assumption?

What the LLM doesn't do is touch APIs directly, format data, or manage file operations. Every time the work crosses from "decide" into "do," it hands off to a script in Layer 3.

Layer 3: Execution

Deterministic Python and bash scripts live in an execution/ folder. They handle the mechanical work — SMTP calls, file I/O, git operations, data transformations. Each script takes explicit inputs, does one thing, and returns a clear success or failure.

# execution/send_email.py
# Takes --to, --subject, --html-file
# Reads credentials from .env
# Sends via Gmail SMTP
# Returns success or explicit error with stack trace

These scripts don't hallucinate. They don't vary between runs. They either work or they throw an error you can debug. And because they're just regular scripts, you can test them independently, version them in git, and reuse them across completely different directives.

Why the File System Makes This Work

This architecture isn't tied to any specific framework or vendor. It runs on the file system — a folder with Markdown files and scripts. That's it. And this is exactly why it works well with AI coding CLIs like Claude Code, Cursor, Windsurf, or anything else that can read files, run commands, and edit code.

The orchestration layer needs to read instructions from somewhere, execute tools somehow, and remember what it learned. In most agent frameworks, you'd wire up tool definitions, build a state machine, or configure a DAG. Here, the file system handles all of it natively. Directives are just files the agent reads. Scripts are just commands it runs. Environment variables are in .env. Intermediate outputs go to .tmp/. There's no agent runtime to maintain because the repo is the runtime.

This also makes the system portable. The entire directives/ and execution/ folder can be copied into a different project, a different repo, or a different machine. A directive for sending emails works the same whether it's called from your operations repo, your marketing repo, or a completely separate project. The scripts carry their own logic; the directives carry their own instructions. Nothing is coupled to a specific environment beyond what's in .env. We've already done this — reusing execution scripts across different projects without modification.

Repeatable and Portable by Default

The combination of these layers makes tasks both repeatable and portable. Repeatable because every run follows the same directive with the same deterministic scripts — there's no improvisation in the execution. The LLM's creativity is channeled into decisions (which directive to use, how to interpret the request, how to handle an error), not into the mechanics of the work itself.

When a new team member needs to be onboarded, the agent doesn't reinvent the process. It reads directives/send_onboarding_email.md and follows it. The email template is consistent. The SMTP configuration is in .env. The send logic is in send_email.py. Nothing is left to chance or to the model's mood that day.

Portable because adding a new capability to any project follows the same pattern:

1. Write a directive in directives/ describing the task
2. Create execution scripts in execution/ for the deterministic parts
3. Test it once with the agent
4. The task now works reliably, everywhere

You can move a directive from one project to another and it carries its full context — the steps, the edge cases, the tool references. You're not rebuilding from scratch every time. You're building a library of operations that any LLM-based agent can pick up and run.

Self-Annealing: Getting Stronger From Errors

The system improves itself. When something breaks — an API rate limit, a changed endpoint, an edge case nobody anticipated — the agent doesn't just retry. It follows a loop: read the error and stack trace, fix the script, test the fix, and update the directive with what it learned.

We call this self-annealing, borrowing from metallurgy where heating and slowly cooling metal removes internal stresses and makes it stronger. Same idea here — every failure becomes permanent knowledge baked into the system.

A real example: we hit Gmail's SMTP rate limits during a batch onboarding. The script failed. Claude read the error, researched the Gmail API docs, found the batch sending limits, rewrote the script with rate limiting, tested it, and added a note to the directive: "Gmail limits to 500 recipients per day. For batches over 50, add a 2-second delay between sends." That knowledge now lives in the directive. The next time this runs — by any agent, on any model — it accounts for rate limits automatically. No one has to remember. No one has to rediscover it.

The key is that improvements target scripts and documentation, not model weights. Everything is auditable in git, reversible with a revert, and readable by humans. If the agent writes a bad fix, you can see it in the diff and roll it back. This is a much safer form of self-improvement than anything that modifies the model itself.

The Repo Structure

The whole system lives in a single repository:

directives/          # Markdown SOPs — the instruction set
execution/           # Python/bash scripts — the tools
.env                 # API keys and configuration
.tmp/                # Intermediate files (gitignored)
CLAUDE.md            # Agent operating principles

CLAUDE.md contains the meta-instructions — how the agent should use the repo. Check for existing tools before writing new ones. Self-anneal when things break. Update directives as you learn. This file loads automatically when you open Claude Code in the directory, so the agent always knows how to operate without being told each session.

Deliverables don't stay local. They go to cloud services — Google Sheets, email inboxes, deployed websites — where the operator can access them. Everything in .tmp/ is ephemeral and regenerable.

Start Building

The architecture is simple: Markdown files for instructions, scripts for execution, an LLM for decisions. No frameworks. No vendor lock-in. A repo that gets smarter every time it runs and can be carried anywhere.

If you're running operations where an AI agent keeps getting things 80% right, you don't need a better model. You need a better structure. Push the deterministic work into code, keep the LLM focused on judgment, and let the system teach itself from its mistakes. That's how you go from a demo to something you'd actually trust with your business.

We build systems like this for companies that want AI automation they can actually rely on. Let's talk about what repeatable operations look like for your team.