My tool-calling agent looked amazing in demos and then double-charged people

AI agent quality assurance should look a lot more like integration testing than prompt tweaking. With OpenAI Structured Outputs, gpt-4o-2024-08-06 reportedly hit 100% schema adherence on complex JSON evals, which means malformed tool arguments are now a bug you can regression-test, not just a weird model habit.

I knew I was in trouble when the demo agent did the exact thing I asked for on Friday, then tried to do it twice on Monday.

Same prompt. Same API. Same cheerful little trace that made it look like everything was under control.

Except this time the second call wasn’t harmless. It hit a real downstream action. That was the moment I stopped thinking about agents as “smart prompts” and started treating them like messy software integrations with a language model bolted onto the front.

That shift changed everything I now believe about ai agent quality assurance.

Most teams are still testing agents like it’s 2023. They run a few happy-path prompts, watch Claude or GPT-4o call a function correctly, maybe screen-record the result for Slack, and call it done. Then production introduces malformed JSON, retries, stale memory, a 500 from Stripe, a timeout from HubSpot, and a duplicate action that nobody designed around.

And suddenly the model didn’t “go rogue.” Your QA did.

The demo was never the hard part

Getting a tool-calling agent to look smart in a demo is easy now. n8n makes it easy. OpenAI makes it easy. Anthropic makes it easy.

You wire up an HTTP Request Tool, maybe a Custom Code Tool, maybe let the agent call an entire n8n workflow through the Call n8n Workflow Tool, and within an hour you’ve got something that can search, summarize, enrich, and post to Discord like it has a tiny soul.

But demos are built on one giant lie: the outside world behaves.

Production is where auth expires. Production is where APIs return partial payloads. Production is where memory drags yesterday’s assumptions into today’s run. n8n’s Chat Memory Manager is a good reminder that a lot of “reasoning failures” are really state bugs wearing a trench coat.

That was the first surprise for me. The ugliest failures were not philosophical. They were boring. Bad parameters. Duplicate requests. Missing fields. Half-finished tool round trips.

Which is good news, because boring failures are testable.

What changed when OpenAI made strict schemas real?

This is the part I think a lot of people have not fully updated on.

OpenAI’s Structured Outputs materially changed the baseline for llm tool use reliability. If you set strict: true inside the function definition, OpenAI says gpt-4o-2024-08-06 hit 100% schema adherence on its complex JSON-schema evals, versus less than 40% for gpt-4-0613.

That is not a tiny model improvement. That is a QA philosophy change.

If your agent is still emitting malformed arguments under a strict schema, I would stop calling that “just how LLMs are” and start calling it a regression.

Here’s the shape of what that looks like:

{
  "type": "function",
  "function": {
    "name": "query",
    "description": "Execute a query.",
    "strict": true,
    "parameters": {
      "type": "object",
      "properties": {
        "table_name": {"type": "string"},
        "columns": {"type": "array", "items": {"type": "string"}}
      },
      "required": ["table_name", "columns"]
    }
  }
}

That snippet should immediately suggest a test fixture: same prompt, same intended action, once under loose JSON handling and once under strict schema enforcement. The old version fails. The strict version passes. If it stops passing later, you caught a regression before a customer did.

Strict schemas do not solve wrong tool choice, stale memory, bad business logic, or unsafe retries. A tool call can be perfectly valid JSON and still be the worst possible decision.

But once argument shape becomes more deterministic, the rest of your failures get easier to isolate. And that leads to the next trap.

Why did my agent do the same thing twice?

Because I let it.

Anthropic’s tool-use docs are refreshingly blunt about something demo builders ignore: Claude can emit one or more tool_use blocks in a turn. If your app treats that like a cute implementation detail instead of an orchestration contract, you are begging for duplicate side effects.

And if your downstream APIs are not idempotent, parallel or repeated calls are not a “model alignment” problem. They are an integration design failure.

Anthropic even gives you a lever for this:

"tool_choice": {
  "type": "auto",
  "disable_parallel_tool_use": true
}

If I’m touching anything that charges money, creates records, sends messages, or mutates state, I start from parallel disabled and add idempotency keys before I get fancy.

Also: enforce the dumb little validation rules. Anthropic requires user-defined tool names to match ^[a-zA-Z0-9_-]{1,64}$. That sounds trivial until some generated config slips through with a bad name and your agent fails before the interesting part even starts.

The real production pattern is the round trip: Claude returns a tool_use block, your app executes the operation, then you send back a tool_result block in a second request. That handoff is where the bodies are buried.

That’s where you inject fixtures for:

• malformed tool inputs
• partial API failures
• duplicate execution attempts
• missing tool_result payloads
• slow responses and timeouts
• successful call with semantically wrong result

Most teams test the first request. The breakage lives in the second.

The checklist I wish I had first

I don’t think most teams need a giant eval empire on day one. But I do think every serious agent needs a small, ruthless replay suite.

LangSmith’s evaluation guidance gets this exactly right. Break the system into critical components like tool invocation and output formatting. Start with 5-10 curated examples of what good looks like. Then use offline evaluation for regression testing and backtesting against historical traces.

That is just software testing with better nouns.

My minimum viable agent QA checklist

1. Schema fixture
   Test every high-value tool call against strict JSON Schema rules. If you use OpenAI Structured Outputs, malformed args should be rare enough to treat as failures, not folklore.

2. Wrong-tool fixture
   Give the agent a prompt where two tools look plausible. Make sure GPT-4o or Claude picks the right one for the business rule, not just the nearest keyword.

3. Duplicate-action fixture
   Replay the same tool call twice and verify your app blocks, deduplicates, or safely replays it.

4. Partial-failure fixture
   Simulate a 500, timeout, or partial payload from the HTTP Request Tool in n8n. Confirm the agent does not happily continue with corrupted assumptions.

5. Round-trip fixture
   Test the full tool_use → execution → tool_result flow. Most “agent bugs” are really handoff bugs.

6. State fixture
   Seed stale memory and verify the agent does not drag old context into a new job. If you use n8n memory features, test memory boundaries explicitly.

7. Regression dataset
   Save every interesting failure as a replayable case. Not a screenshot. Not a Slack thread. A dataset.

What I actually compare

That table looks boring. Good. Boring is what keeps agents from embarrassing you.

Do you need a whole eval stack right away?

Probably not.

If you’re building a narrow internal automation in n8n that reads a Google Sheet, calls an HTTP endpoint, and posts to Discord, you may not need a heavyweight setup. A curated replay suite plus tracing might be enough.

But “small” does not mean “casual.” It means fewer fixtures, chosen more carefully.

OpenAI Evals makes the larger point well: evals are one of the highest-leverage activities for LLM systems, especially for prompt chains and tool-using agents. If you’re swapping models, editing prompts, or changing tool schemas without versioned evals, you are outsourcing QA to production.

Even their setup instructions tell you what kind of work this is. It’s engineering work.

pip install evals
# or for local development
pip install -e .
# fetch registry data when cloning the repo
git lfs fetch --all
git lfs pull

OpenAI Evals lists Python 3.9 as the minimum required version. That tiny detail matters because it signals the mindset: this is not prompt art. This is test infrastructure.

And honestly, broad replay testing gets much easier when compute cost is predictable. When every regression run feels like it’s ticking a token meter, teams mysteriously become “comfortable” with less testing than they should be.

The opinion I wish someone had forced on me earlier

If your agent can call real APIs, it is not a chatbot with extra steps. It is an integration surface.

Treating agent QA like prompt craftsmanship is how you end up with gorgeous demos and haunted production logs. Treating it like software integration testing is how you sleep.

The counterintuitive part is that better models make discipline more necessary, not less. Once GPT-4o gives you strict schema adherence and Claude gives you cleaner tool orchestration, the remaining failures stop being mysterious. They become your responsibility.

Which is actually great news.

Because “the model is weird” is hard to fix. A replayable fixture for malformed JSON, duplicate actions, partial API failures, and stale memory is fixable tomorrow morning.

That’s the practical takeaway I wish I had on day one: don’t ask whether your agent is smart. Ask whether it passes the same ugly cases twice in a row.

That’s when you know it’s ready.