How to Evaluate n8n AI Agents Before Production

An n8n agent evaluation is four pieces: a dataset, a trigger, a metric, and a threshold.The dataset holds inputs and expected outputs. The Evaluation Trigger node replays every row through the same nodes production uses. The Evaluation node scores what came out. The threshold is the number you refuse to ship below. Miss any one of those and you don't have an eval, you have a demo.

I built my first one after a lead-qualification agent quietly started tagging warm leads as cold. Nothing errored. The workflow was green in every execution. It took nine days and an annoyed client for anyone to notice, and the cause was a two-word prompt tweak I'd made to fix something else entirely. A thirty-case suite would have caught it in ninety seconds.

Deterministic automations fail loudly. An API returns 401, a node turns red, you get an alert. Agents fail quietly. The model still returns a well-formed sentence, the JSON still parses, the workflow still completes — the content is just subtly worse than it was last week. There is no exception to catch.

That's why manual testing stops working around the time an agent gets useful. You paste in three inputs, eyeball the answers, and ship. Three inputs can't detect a five-percent drop in routing accuracy, and the three you pick are always the three you already know the agent handles. I've written before about how n8n control flow hides agent failures — evaluations are the other half of that problem. Control flow catches the crash; evals catch the drift.

The trigger for building one is simple: the moment you change a prompt and feel nervous, you need an eval. That nervousness is you correctly noticing you have no way to verify the change.

• The dataset: An n8n Data Table or a Google Sheet, one row per test case. Minimum two columns: the input your agent receives and the output you'd accept as correct. Add a category column early — you'll want to slice results by case type once you have more than thirty rows.
• The Evaluation Trigger: A trigger node you add to the workflow that already runs your agent. It reads the dataset and pushes each row through the real nodes. This is the part people get wrong: don't duplicate the workflow. A copy drifts within a week and then your suite is grading a workflow nobody runs.
• The Evaluation node: Three operations. Set Metrics writes scores to n8n's Evaluations tab. Set Outputs writes the agent's actual answer back into the dataset so you can diff runs. Check If Evaluating branches the execution so a test run doesn't email a real customer.
• The threshold: One number, written down, agreed on before you start tuning. Without it you'll rationalize every regression as 'close enough' — I have, more than once.

Check If Evaluating is the operation that makes the whole thing practical. Put it right before the first node with a real-world side effect — the Gmail send, the HubSpot update, the Stripe call — and route evaluation runs down a branch that skips it. Everything upstream, including the agent, the tools, and the retrieval, runs exactly as production does.

n8n ships five built-in metrics. Two are LLM-judged and cost a model call; three are deterministic and effectively free. Most people reach for the judged ones first and then complain about cost — usually the deterministic metric was the right answer.

Track two metrics per agent, not six.One that measures whether the answer is right, one that measures whether the agent behaved right. For a lead-router that's Categorization plus Tools Used. For a support-reply drafter it's Helpfulness plus a custom schema check. Six metrics gives you six numbers that all move slightly and no clear signal about whether to ship.

Full details on scales and configuration are in n8n's metrics documentation.

1. Mine your execution history for real inputs. Twenty to fifty rows. Real ones — the messy phrasing, the missing fields, the customer who wrote in all caps. Don't invent test cases; you'll invent easy ones.
2. Write the expected output for each row. This is the slow part, and it's slow because it forces you to decide what 'correct' means. If you can't write it down, your agent can't hit it.
3. Add the Evaluation Trigger to the live agent workflow and point it at the dataset.
4. Drop an Evaluation node set to Check If Evaluating in front of your first side-effecting node, and route the evaluation branch around it.
5. Add an Evaluation node with Set Metrics at the end, pick your two metrics, and map the agent's output plus the expected output into it.
6. Run the suite against today's agent to get a baseline. Whatever it scores is your starting line. Set the pass bar just under it and gate every future change on that number.

The baseline step matters more than it sounds. People skip it, run the suite for the first time, see 74%, and panic-tune the prompt for an afternoon. 74% was your Tuesday. You just never had a number for it before.

Node-level specifics live in the Evaluation Trigger docs. If you're on a self-hosted instance running a lot of these, note that eval runs are still executions and they compete with production for workers — the same queue-mode math I covered in scaling n8n with queue mode applies.

Every production incident becomes a test case, same day, before you close the ticket. Agent misrouted a lead? That exact input goes in the sheet with the correct answer next to it. Hallucinated a price? In the sheet. Called the wrong tool? In the sheet.

My first suite was 22 hand-picked cases and it caught almost nothing for a month. It got useful around case 40, and every one of those extra rows came from something that actually broke. That's the difference between a suite that scores your agent and a suite that protects it: the second one is built out of scar tissue.

The corollary is that suite score is not a vanity metric. When a client asks “is the agent getting better?” the honest answer used to be a shrug. Now it's a number with a date on it, and the number went from 74% to 91% over six weeks of changes I could actually verify. That's a much better conversation.

Less than people assume, if you keep the judge off the hot path. A 30-case suite with two judged metrics is 60 extra model calls per run. On a small judge model that's cents. Run it on every prompt change and once nightly and you're still under a coffee per month.

Where cost actually explodes is scoring live traffic. An LLM judge on every production request doubles your model spend and adds latency to a user-facing path for a number nobody reads until Friday. The pattern that works: score offline runs fully, sample five to ten percent of production, and run the free deterministic metrics on everything. Same discipline I apply to agent spend generally in cost controls for autopilot agents.

One more thing worth knowing: judge models are not neutral referees. A naive judge prompt with no examples and no forced structure is unreliable enough that you'll chase phantom regressions. Give the judge two or three scored examples, make it output structured JSON, and spot-check twenty of its verdicts by hand before you trust the aggregate. The same care you'd put into writing a good tool definition applies to writing a good judge.

• Testing a duplicate workflow: I cloned the agent into an 'AGENT — TEST' workflow so evals couldn't touch anything real. Two weeks later the clone was three prompt versions behind and my green suite meant nothing. Check If Evaluating exists precisely so you never do this.
• Generating test cases with an LLM: Fast, and almost useless. Generated inputs cluster around the same tidy shape and your score climbs while real accuracy doesn't budge. Mine your execution logs instead.
• Only testing the happy path: Roughly a third of my cases are now ambiguous, hostile, or missing-data inputs. That third finds nearly all the regressions.
• Chasing 100%: A suite that always passes has stopped telling you anything. If you're at 100%, your cases are too easy — go add the ones that scare you.

Worth saying plainly: if your agent is one prompt and no tools, you probably don't need this yet. Evals earn their keep the moment an agent has branches, tools, or more than one person editing it — which is also the moment the n8n-versus-code question stops being theoretical.

• An eval is four pieces: dataset, Evaluation Trigger, a metric, and a threshold you refuse to ship below.
• Test the live workflow, never a copy — Check If Evaluating branches side effects out of the run.
• Two metrics per agent: one for whether the answer is right, one for whether the agent behaved right.
• Every production incident becomes a test case the same day. That's what turns a scorecard into a safety net.
• Score offline runs fully, sample production, and keep the LLM judge off the user-facing path.

If you want to store your test cases natively rather than in a spreadsheet, n8n's Data Tables are the cleanest place to put them.