From the very first commit of my Python mission, one rule mattered greater than some other: the coding agent mustn’t ever see the acceptance standards. So I constructed two brokers that by no means examine notes: one writes the checks from the complete specification, and a separate coding agent writes the code from the identical specification with the factors lower out. In most coding environments, the identical agent writes each, which compromises the validity of the check outcomes. I compelled the lower to occur in code through the use of context meeting, the step that builds the coding agent’s immediate, the place the factors part of the duty file is lower out earlier than the immediate is constructed.
For some time I believed that lower was hermetic. Then I discovered {that a} single remark line broke it: the lower stopped there, and each criterion after the remark reached the coding agent. Every week later, a criterion wrapped onto a second line did the identical factor. Not good. Each bugs had been mine and I used to be aggravated that the identical core downside had resurfaced. So this time, after fixing each bugs, I added checks to make sure that any future construct would fail if the lower was not utilized as supposed.
Expertise and MCP servers are sometimes weighed in opposition to one another. One yr in the past, on the day Anthropic launched Expertise, Simon Willison known as them “possibly an even bigger deal than MCP”. They don’t seem to be actually rivals, and for the rule my mission is dependent upon, the reply turned out to be neither, as I opted for context meeting. I nonetheless ended up transport all three, as every carried a separate benefit.
I needed customers to have the ability to attain qikly, my command-line device, from a chat setting, by means of the AI assistant of their editor or terminal, equivalent to Claude Code or Cursor. That places two LLMs in play: the assistant, which drives qikly for the person, and qikly’s personal coding agent, which writes the code. Neither ought to see the acceptance standards, as a result of both might then write code to go the checks reasonably than to unravel the issue. The assistant might already run qikly from the command line. What it lacked was figuring out when to make use of it. I figured a Ability, a folder of Markdown directions the assistant hundreds when a request matches its description, would do the trick. A Ability can be a great way to bundle the perfect practices for utilizing it, so the assistant picks them up with out anybody having to recollect to stick them in.
Writing the Ability made me separate two jobs, and solely certainly one of them belongs in a Ability.
The Ability’s job is recommendation: what goes within the necessities, which the coding agent reads, and what goes within the acceptance standards, which it by no means sees. That’s judgement, so it may possibly solely be taught: the assistant will often observe it, and you may measure whether or not it helps, however you can’t promise it.
Context meeting’s job is reducing the factors out, each time it builds the coding agent’s immediate, whether or not or not a Ability is put in.
The recommendation nonetheless issues though the lower is computerized, as a result of the lower removes solely the factors part of the spec. If a criterion can be written into the spec’s necessities, it reaches the coding agent anyway. So I added a examine to qikly --validate that warns when a requirement restates a criterion.
The identical cut up applies to any rule you give an LLM: some should maintain no matter it decides, and others solely work if it understands them. Two frequent fixes fall brief. A greater mannequin follows recommendation extra usually however not at all times. Setting the temperature to zero makes its solutions extra repeatable, no more obedient.
MCP Server vs Ability
Each are described as “giving the LLM a brand new functionality”, and each do, so the distinction is simple to overlook. Here’s what each really is.
An MCP (Mannequin Context Protocol) server is a course of: consider it as an API for AI assistants. It exposes typed features as instruments (mine are Python) over a stay client-server protocol. The host, the app the assistant runs in, equivalent to Claude Code or Cursor, connects to the server by means of its built-in MCP consumer and lets the LLM name these instruments the way in which your code calls a perform. A server both runs by yourself machine, began by the host as a neighborhood course of, or is hosted by its supplier as an online service at a URL; qikly’s is the native variety.
For those who write your individual agent loop, the code that calls the mannequin and runs the instruments it asks for, you don’t want a server for this: you go the device schemas in your individual API name. An MCP server is for when the loop belongs to an assistant equivalent to Claude Code or Cursor, whose code you can’t edit. My mission, qikly, runs as a command-line device and wishes no server. Its MCP server is an non-compulsory further, value including when an assistant drives qikly for you. It installs with qikly --install-mcp and has 4 instruments: qikly_run to begin a run, qikly_status to ballot it, qikly_validate to examine a job file and qikly_scaffold to draft one from a Python file.
A Ability is a static, transportable instruction set, often a SKILL.md saying when to make use of one thing and the way, with non-compulsory reference information beside it. It’s utilized in two steps: the LLM at all times sees the Ability’s identify and outline, and opens the complete file solely when that description appears to suit what it’s doing. So, not like the system immediate, which the LLM at all times sees in full, a Ability can fail by going unread: if its description doesn’t look related to a request, the remainder is rarely opened. A Ability folder may also ship scripts, however the assistant decides whether or not to run them, so even a Ability’s script is a alternative the assistant makes, not code its output should go by means of.
The query that issues
Most comparisons weigh the 2 on information versus entry. That framing misses a extra helpful query. When including something to an LLM utility, ask:
What stops the LLM from disobeying the rule?
My MCP server is code, so it passes that check for what goes by means of it. It by no means arms the acceptance standards to the assistant, nevertheless it asks, as a result of the refusal is written into the server reasonably than requested in a immediate.
The information on disk are one other matter. The factors sit in a job file, and the assistant can open that file with its personal instruments, with out going by means of qikly’s MCP server in any respect, so the server solely guards its personal entrance door. qikly’s coding agent by no means trusted that door: it by no means reads the disk, and the factors lower occurs in context meeting, which it can not get round. For the assistant there’s a mechanism too: the docs present a permissions rule that retains it out of the generated checks, that are written from the factors.
The actual query, then, just isn’t “ought to this be a Ability or a server”. It’s the one above, requested of the factor you might be including, after which selecting whichever format solutions it the way in which you want. I went on the lookout for what else was on supply and located a very good many codecs. Each certainly one of them falls into the identical two teams: those that inform the LLM of a rule, and those that implement it in code.
The 2 tables differ in a single factor: what occurs when the LLM would reasonably do one thing else. Each row of the primary ends on the similar place, the LLM’s personal judgement: nevertheless firmly a rule is worded, the LLM can nonetheless determine to not observe it. Within the second desk its desire doesn’t come up, as a result of one thing in its path refuses by itself authority. Every desk runs from the strongest row on the prime to the weakest on the backside. So what stops the LLM from disobeying? With a Ability, nothing does. Code in its path does, as a result of the LLM by no means will get the selection.
qikly makes use of eight of the twelve rows above, together with the one which holds its central declare. One of many eight has a reputation already: fastened management move, the place the sequence of calls is code reasonably than a plan, is what Anthropic’s workflow-versus-agent distinction calls a workflow, as in opposition to an agent that plans its personal steps, and by that definition qikly is a workflow. Fastened management move settles which immediate runs subsequent with out settling what any of them comprises, which is why the factors lower nonetheless needed to be a code mechanism reasonably than an ordering.
The mission this got here from
-
The code is open supply, and the touchdown web page hyperlinks the repository.
-
The Ability is described within the docs and installs with
qikly --install-skill. -
The MCP server is described within the docs and installs with
qikly --install-mcp.
Splitting the specification this fashion mirrors safety-critical apply, the place verification must be impartial of whoever constructed the factor, and goes one step additional. On most groups a developer reads the entire ticket, standards included, and the one independence is that any person else writes the checks. A developer who has learn the factors codes to them, so the checks can not present whether or not the necessities alone had been sufficient. In qikly each roles are performed by LLMs, so independence can not come from separate individuals. The independence comes as an alternative from what each is proven: the check generator will get the factors, and the coding agent by no means does. When a check fails, the coding agent sees what pytest prints. By default that features the failing check’s supply, which frequently restates its rule. A staged setting in qikly, off by default, begins with a one-line error as an alternative and widens solely as makes an attempt hold failing. Half 1 of this sequence defined that trade-off.
The coding agent by no means edits the checks. It emits a diff, and a diff touching something outdoors the code is refused. The combination and system suites are written earlier than any code exists and keep as written. The explanation that the unit suite is generated final, with the code in view, is that unit checks additionally cowl the helpers the coding agent wrote, which the duty file’s interface part doesn’t identify and which don’t exist till the code does.
In Half 3 I known as this Impartial Spec-Pushed Improvement (ISDD): the coding agent is given the choices, the check generator can be given the penalties, and the independence lies in what every of them is aware of. That solely works if the factors lower at all times occurs.
The mission, which now ships as a command-line device, an MCP server and an agent Ability, has grown to incorporate over 2,000 checks, and about 70 of them guard that one rule, masking each immediate the coding agent receives, each reply the MCP server sends, and every approach of writing a job file that has damaged the lower or might.
The excellence this mission taught me
What code executes can implement. What an LLM reads can solely inform. A Ability is learn by the LLM, so it might by no means have held the rule this entire mission rests on. A server is executed, so it might have. Mine refuses to return the factors too, however the rule qikly is dependent upon wanted no format in any respect: context meeting retains the factors out of the coding agent’s immediate nevertheless qikly is began, from a shell or from a device name.
Something that should be assured no matter what the LLM decides has to stay in code, behind an interface the LLM can not attain round, equivalent to entry management, the factors lower, fee limits, audit trails. In different phrases, something you’ll wish to say “it’s unimaginable for this to occur” about.
Something that’s information, judgement or process belongs in a Ability, as a result of a device name has nowhere to place it. Three concrete examples from my SKILL.md:
-
which line of a specification is a choice any person made versus which is a consequence that follows from it.
-
what it means when the restore loop, qikly’s fix-and-retest cycle, stalls and retains resubmitting the identical patch.
-
why a set that passes can nonetheless be value little or no.
Write the sentence you wish to be true, then ask what stops the LLM from disobeying it.
Why I shipped each anyway
As a result of they do completely different jobs.
The server begins and tracks lengthy runs. A run takes minutes to hours, relying on its scope and complexity, and a device name can not block for that lengthy. So qikly_run begins the run within the background, in its personal course of, and returns an id directly, whereas qikly_status studies what turned of it. A number of duties can run in parallel this fashion, one course of every. That start-and-poll cut up is the server’s primary comfort: the assistant begins and checks runs as unusual device calls as an alternative of driving a shell. The command line can do the identical with qikly --run and qikly --status, and the server is constructed on them. What the server provides is arguments checked in opposition to a schema, annotations telling the assistant which calls are secure to make unattended, and entry for an assistant with no terminal in any respect. Its replies attain the assistant’s personal context, so they’re yet one more place a criterion might leak, and so they have checks of their very own.
The Ability teaches what the instruments can not. An assistant that calls qikly_scaffold will get again a job file with two sections to fill in, and nothing within the device says which of the person’s sentences belong through which. That’s the decision-versus-consequence line above, the toughest judgement on this mission. There isn’t any perform signature for it, so it must be written for the LLM to learn, and that’s what a Ability is. It issues as a result of a choice left hidden within the standards makes runs stall extra usually, because the coding agent can not guess a choice it by no means sees. On one driver-assistance job, with the choice left within the standards, 3 of 10 runs converged, reaching code that passes each check; moved into the necessities, 10 of 10 did, and 9 of 10 on a rerun the subsequent day. Serving to draw that line nicely is most of what the Ability’s 4 and a half thousand phrases are for. It deserves an article of its personal, and can get one.
A Ability may also attain the assistant earlier than any device is in play. Mine opens with recommendation for a request that by no means mentions qikly: if requested to put in writing checks and there’s no specification, don’t write them but, and supply the 2 routes that get one written first. A device’s description is about utilizing the device, so it has nothing to say to an assistant about to put in writing checks by hand. That is nonetheless recommendation, not a assure: the assistant will often observe it, which is all a Ability can promise.
Why not RAG (Retrieval-Augmented Era)?
An information scientist’s first query is usually whether or not RAG might do that job as an alternative. At most it might stand in for the Ability. Retrieval just isn’t an alternative choice to the server in any respect. RAG places textual content within the context window and enforces nothing, so it fails the identical check a Ability does, and the assure might no extra stay there than it might stay in a Ability.
Towards a Ability it’s a honest query, and it comes all the way down to choice and dimension. Retrieval guesses relevance from the question, whereas a Ability declares it in an outline written by its writer, which is why a retriever looking out on “write me some checks” wouldn’t pull up the Ability’s opening instruction, a passage saying to not write checks but. And a Ability arrives entire, in its personal order, every rule nonetheless connected to the instance that explains it, the place retrieval arms over fragments ranked by rating and step three with out the 1st step is worse than nothing. That solely works whereas the entire thing is sufficiently small at hand over. At two thousand pages you haven’t any alternative however to retrieve fragments and stay with the rating, which is strictly when retrieval earns its infrastructure. Retrieval is for information you might have an excessive amount of of. A Ability is for information all of which issues.
Widespread mix-ups
When you begin trying, this will get combined up in public continuously, and it goes fallacious each methods.
Servers which might be actually Expertise. Quite a lot of what ships as an MCP server is information sporting a protocol. Even a single device can earn a server: it may possibly examine its inputs in opposition to a schema, maintain a credential the LLM by no means sees, or attain an assistant that has no terminal to run instructions in. But when its one device solely wraps a command-line program the assistant might already run, the one factor it provides is that the assistant now is aware of this system exists. A Ability says that simply as nicely, in a SKILL.md naming this system and when to make use of it, with no course of to configure, begin and hold alive.
Expertise which might be actually guarantees. A SKILL.md saying “by no means commit secrets and techniques”, “at all times ask earlier than deleting” or “don’t learn information outdoors the mission” is a hope with good formatting. Each a type of has to carry regardless of the LLM decides, so it wants a mechanism that may refuse: a permission system, a hook, a sandbox. Left as an instruction, it’s ornament.
If a rule must be true, it belongs someplace the LLM’s output should go by means of. If it solely must be recognized, an instruction the LLM reads is the best device, and it is best to cease calling it a safeguard.
Different widespread codecs for guarding a rule
Just a few issues get not more than a point out right here. Constrained decoding and structured outputs implement with none server in any respect. You go a JSON schema or a grammar with the API request, and because the mannequin writes, any subsequent token that might break it’s faraway from the alternatives, so an invalid reply can’t be produced. It ensures the answer is format, although, not that the reply is correct. Steady integration (CI) gates implement after the very fact reasonably than within the path: the LLM does no matter it does, and the construct fails afterwards. On this mission that’s the backstop, catching a regression within the standards lower reasonably than doing the lower. Output validators and guardrail libraries sit in between, imposing solely what they had been advised to examine.
And at last, an MCP server can inform in addition to implement. In addition to the instruments defined on this article, the Mannequin Context Protocol lets a server supply prompts (ready-made directions) and assets (paperwork the LLM can learn). These attain the LLM as textual content, so they’re recommendation, precisely like a Ability; solely the server’s instruments run code that may refuse. Transport a server doesn’t by itself imply a rule is enforced: that is dependent upon which a part of the server holds it.
Additional studying
This text describes my very own expertise with a spread of present AI matters, and builds on Anthropic’s free course, Constructing with the Claude API.
Expertise have a separate free course of their very own, Introduction to agent abilities.
What this text provides, from actual points I needed to analysis and assume by means of, is one query for sorting any rule you give an LLM, a map of twelve frequent codecs by whether or not they inform or implement, and a labored case from transport each a Ability and an MCP server.
Earlier articles on this sequence
Half 1 makes the case for the cut up: when one AI writes each the code and its checks, “the inexperienced check suite can imply nothing”. It explains how qikly writes the checks from standards the coding agent by no means sees.
Half 2 follows one full run of qikly, through which checks the coding agent by no means noticed caught a bug in its vary examine on radar information.
Half 3 introduces a brand new idea, ISDD, Impartial Spec-Pushed Improvement, and evaluations a Google analysis crew’s latest measurement of SDD.
Concerning the Writer
Gal Arav is the writer of Utilized Statistics for Information Science (qikly.com) and maintains qikly, a brand new open-source Python mission for Spec-Pushed Take a look at Automation.
Readers are inspired to provide it a spin, and any (human) suggestions is far appreciated because the codebase continues to evolve.
A word on the quilt
The duvet makes use of a nonetheless of Charlie Chaplin and Jackie Coogan in The Child (1921), public area in america, through Wikimedia Commons: https://commons.wikimedia.org/wiki/File:Chaplin_The_Kid_4.jpg . The duvet picture was impressed by a reproduction of this picture in my room in my childhood.
