What agent harnesses record, and what they send home

We are starting a study of AI coding agent harnesses. We want to know two things about each one: what it records on your machine, and what it sends off your machine. Then we compare the answers with what the vendor says.

Why the harness matters

An AI coding agent is two things. There is the model, which runs on a server somewhere. And there is the harness: the program on your machine that reads your files, runs commands, talks to the model and keeps a log. Claude Code, Codex, Kilo, Qwen Code and Pi are all harnesses.

Most public discussion is about models. But the harness makes two decisions that matter a great deal when something goes wrong.

• It decides what gets recorded. If an agent deletes a file, leaks a key or changes its own settings, the harness log is often the only record of what it was asked, what it did and who approved it. If the log is thin, or the agent can edit it, an investigation starts with a gap.
• It decides what leaves the machine. Every harness sends prompts and file contents to a model. Some also send usage data, crash reports, device identifiers or other fields. Users and their employers need to know what goes where, and whether the switches that are meant to turn it off actually do.

We record what AI agents do and investigate when it goes wrong. The harness is where that record begins, so this is where we are starting.

The questions we test

We ask the same questions of every harness.

1. What does it record on the machine, and where?
2. Does the record include the prompt, every tool call, the tool output, the model used and the time?
3. Does the record say whether each action was approved, and by whom?
4. Is the record protected? Who can read it, and could the agent itself change or delete it without trace?
5. How long is the record kept, and what deletes it?
6. Besides the model call, what does the harness send off the machine, and to whom?
7. Are extra fields or identifiers added to model calls?
8. If a fake secret is placed in the working folder, does it leave the machine, and where does it go?
9. Do the documented opt-out settings stop what they say they stop?
10. Does resuming an old session, or using a feedback command, upload anything the user might not expect?
11. Does what we observe match the vendor's privacy statement and documentation?

How we test

The method is simple and we will publish it in full.

• Isolated installs. Each harness is installed fresh, from the vendor's normal channel, with its own clean home folder for every run. We note the exact version.
• Fake secrets. We put canary values in the working folder: fake API keys and passwords that look real but unlock nothing. We ask the agent to do an ordinary task near them. If a canary turns up somewhere, we know how it got there.
• Wire capture. We route the harness through a local recording proxy and also log every network connection it opens, so traffic that skips the proxy is still seen.
• Reading the package. We review the code the vendor actually ships, not only the public repository, for hard-coded addresses, telemetry settings and opt-out switches.
• Opt-out checks. We run again with each documented opt-out set, one at a time and then together, and compare.
• Comparison with the vendor's words. We save a dated copy of the privacy policy and relevant documentation, and set our observations beside it.

We test on our own machines, with our own accounts and fake secrets only. We do not probe vendor servers beyond normal use. We can see what leaves the machine. We cannot see what a vendor does with it after that, such as how long it is kept or whether it is used for training. Where that matters, we say it is unknown unless the vendor has stated it.

What the results are, and are not

Results describe what one version of a harness did, in our setup, on the date we tested it. Harnesses change often. We will re-test when new versions ship and show the version and date beside every result. We do not certify products, award badges or rank vendors, and a result is not a judgement of a vendor's intent. Where a vendor gets something right, we will say so in the same note.

Results will be collected on a harness scorecard. The first results will follow in October.

How we handle findings

• Vendors hear first. If we find something a vendor should know about, we tell them privately before we publish.
• Time to reply. We allow 14 to 21 days, with extra time for holidays or where a fix is under way.
• Their reply is published. If a vendor responds, we publish the response alongside our findings.
• No payment. We do not accept payment from vendors, for inclusion, for review or for anything else connected with this study.
• We publish whatever the results show. Good, bad or unremarkable, results go on the scorecard.

Ask us to include a harness

We will start with the harnesses listed on the scorecard. If you build a harness, or use one we have not listed, you can ask us to include it. Vendors are welcome to tell us about settings or documentation we should check. Nobody pays, and a request does not change how a harness is tested or what we publish.

Write to research@agenticthinking.uk with the harness name and where to get it.

Disclosure: Agentic Thinking is an independent research lab. We steward the open-source AgentHook standard and maintain the open-source HookBus project. Agentic Thinking also owns AgentProtect, commercial software available for licensing. These interests may overlap with the recording and governance issues discussed above.

Related: Harness scorecard · Two labs, one failure · Our research

Collaborate with us →