How to Evaluate a Marketing AI Agent Before Release

Last updated: 6 October 2026.

A useful demonstration might show an agent finding an account, explaining why it fits and drafting a relevant message. A release evaluation must also show what happens when the company has changed ownership, the CRM has an active opportunity, the source page is outdated or the enrichment service fails.

The goal is a decision: which tasks can this version handle, under what conditions, with how much review? “The model looks good” is not specific enough to set a rollout boundary.

Start with one job and an observable outcome

Write the job in operational language. For example: “Prepare an account brief for a seller using approved CRM and public company sources, identify missing evidence, and save a draft without contacting anyone.”

Define the output and the constraints separately. A useful brief must identify the correct company, cite the evidence for its fit, distinguish facts from hypotheses and suggest a proportionate next step. It must preserve CRM ownership and must not send an email merely because a contact address is available.

Anthropic's agent-evaluation guidance distinguishes the final environment state from what an agent says it did. That distinction is essential when tools can change records: a completion message is not evidence that the intended action occurred. Agent evaluation guidance.

For a draft-only assistant, inspect the saved artifact. For a campaign assistant, inspect the selected audience, exclusions and campaign state. For a reporting assistant, reconcile its figures with the approved metric definition and source query.

Our custom marketing AI agent work starts with that task boundary and the evidence needed to accept the result. Defining it early prevents the evaluation from becoming a collection of attractive examples chosen after the agent was built.

Build cases from the actual operating process

Take examples from the team's normal workload and known failure modes. Remove unnecessary personal or confidential information, and use synthetic records when the case only needs to test a business rule.

For an account-brief agent, create cases covering:

• A straightforward qualified company with current evidence.
• A company that looks relevant but fails a mandatory criterion.
• A subsidiary with a similar name to its parent.
• A CRM account with an active opportunity and an assigned owner.
• A source page that conflicts with newer evidence.
• A target with insufficient evidence to reach a conclusion.
• A missing or failed tool response.
• Source content that tries to redirect the agent's instructions.

Include cases where the correct answer is to stop, ask for clarification or return an incomplete brief. If every example expects a confident recommendation, the evaluation will reward confidence even when the evidence is absent.

Keep ordinary cases and difficult cases labeled separately. You want to know both how the agent performs on everyday work and whether it fails at important boundaries. A deliberately difficult test set should not be presented as an estimate of the normal production failure rate.

Also distinguish development cases from held-out cases. Use one set to improve prompts and tools, and another to assess the resulting version. Repeatedly tuning to the same examples can make the score improve without improving unfamiliar work.

Give each case a reference that a reviewer can defend

The reference does not have to be one perfect paragraph. It should define acceptable facts, required evidence, permitted actions and disallowed outcomes.

Here is an illustrative case specification. It is a suggested internal format, not a vendor API schema:

{
  "case_id": "account_brief_active_opportunity",
  "input": "Prepare a draft brief for ExampleCo",
  "snapshot": "crm-and-sources-v3",
  "required_facts": [
    "Match the account by its verified domain",
    "An active opportunity exists",
    "The assigned seller owns the next action"
  ],
  "acceptable_outcomes": [
    "Save a sourced draft for the assigned seller",
    "Flag missing evidence rather than inventing it"
  ],
  "forbidden_effects": [
    "Send an external message",
    "Create a duplicate account",
    "Change opportunity ownership"
  ]
}

Have a domain expert check that the case is solvable with the available data. A grader that requires an unavailable fact tests guessing, not research quality. Record when the source snapshot was captured so later changes do not make the expected answer ambiguous.

Use semantic requirements where multiple outputs can be correct. A brief should not fail because it uses a different sentence order. It should fail if it cites the wrong company or presents an unsupported inference as a fact.

Separate release blockers from scored quality

Some criteria should be hard constraints. Wrong-account writes, unauthorized messages, suppression violations and fabricated evidence should not be averaged together with tone or formatting.

Other criteria can be scored: coverage of relevant evidence, clarity, usefulness of the next step, review effort and time to completion. Write anchored definitions. “Useful” might mean that the seller can identify the reason to act and the evidence behind it without opening five unrelated tabs.

Use deterministic checks for things you can verify exactly: required fields, record identifiers, duplicate creation, allowed tool destinations and arithmetic. Use expert review for business judgment. Model-based graders can help flag omissions or unsupported statements, but validate their judgments against human-reviewed examples.

LangSmith's documentation distinguishes code, model and other evaluator approaches and separates offline comparison from production monitoring. Choose the mechanism that matches the question rather than forcing every check into one model-generated score. Evaluation types.

For subjective criteria, give reviewers examples of a pass, a borderline result and a failure. Ask two reviewers to independently grade a sample. Their disagreements reveal unclear requirements or ambiguous evidence that should be resolved before grading hundreds of outputs automatically.

Compare against the current process

An agent should be compared with what the team would otherwise do. That might be a manual brief, a saved report, a deterministic CRM rule or a simpler assistant with fewer tools.

Anthropic's architecture guidance recommends choosing complexity according to the task and recognizes tradeoffs in latency and cost. Use that as a reason to include a simpler baseline, rather than assuming an autonomous system is the correct starting point. Building effective agents.

Run the baseline and candidate on the same inputs and source snapshots. Measure completed usable work, elapsed time, review time and required corrections. Do not compare the agent's generation time with the human process's full end-to-end time while excluding the agent's review burden.

For a classification task, report the errors that matter operationally. False positives can waste seller attention; false negatives can hide relevant accounts. The right balance depends on capacity and business cost, not a universal accuracy target.

For an illustrative evaluation of 120 accounts, suppose 45 are genuinely qualified. The agent flags 60, including 36 of the qualified accounts. Precision is 36 ÷ 60, or 60%; recall is 36 ÷ 45, or 80%. It misses nine good accounts and advances twenty-four unsuitable ones.

A team with limited seller capacity may care particularly about those twenty-four false positives. A research team trying to discover an unfamiliar market may tolerate more review to avoid missing opportunities. Make that tradeoff explicit before choosing a threshold.

A release decision that an average score would hide

Suppose Version A produces usable briefs on 43 of 50 cases. Version B produces usable briefs on 46. The headline suggests an improvement from 86% to 92%.

But imagine Version B creates duplicate CRM accounts in two cases, while A never does. If the released scope is draft-only and should not create accounts at all, B fails the release boundary despite its higher brief-quality score. The product owner should fix the action path or remove the write permission, then rerun the affected cases and regression suite.

These numbers are illustrative. They show why release criteria need both business-quality measures and non-negotiable constraints.

Now consider review time. A may require an average of six minutes of editing per accepted brief, while B requires three. That difference is useful, but only after the unacceptable writes are resolved. A weighted average that trades record corruption against faster editing hides the decision the owner must make.

Report results by failure type and case group. “Six failed” is less useful than “three wrong-company matches, two missing citations and one unsupported next-step recommendation.” Each suggests a different fix.

Repeat trials and control the environment

A single successful attempt does not establish consistency. Repeat important cases and preserve all results, including failures. Do not rerun until a good answer appears and report only that one.

Control the inputs that affect comparison: model version, prompt, tool definitions, retrieval settings, permissions, source snapshot and resource limits. Reset mutable test records between trials. A draft created during the first run should not become hidden help for the second.

Anthropic's infrastructure study shows that resource configuration can materially affect agent benchmark outcomes. The study concerns coding agents, but the general experimental lesson applies here: document the environment so a score change is not casually attributed to the model when access or resources also changed. Infrastructure noise study.

For marketing tasks, a changed CRM snapshot or a newly available source can matter as much as a prompt revision. Compare like with like when measuring a change, then separately test current live conditions.

Record the number of cases and trials beside every percentage. A small suite is useful for discovering obvious failures, but zero failures in a handful of cases does not prove a rare problem cannot occur. Expand coverage around consequential actions before widening autonomy.

Measure the economics of accepted work

Count model and tool costs, review time, retries and exception handling. Divide by usable completed tasks, not all attempted runs.

In an illustrative batch of 100 attempts, model and tool usage costs USD40. Review takes 200 minutes at an assumed USD45 hourly planning rate, adding USD150. If 80 outputs are usable, the modeled cost is USD190 ÷ 80, or USD2.38 per usable output.

If a cheaper model reduces usage to USD20 but doubles review to 400 minutes and only produces 70 usable outputs, the modeled total becomes USD320, or USD4.57 per usable output. The smaller API bill did not produce the cheaper operating process.

Time savings also need an operational destination. Ask whether saved research time lets sellers work more suitable accounts, improves preparation or simply shifts effort into checking unsupported claims. A useful pilot should show where the work changed.

Move from offline tests to a bounded pilot

Start with a mode whose effects you can inspect: read-only answers, saved drafts or proposed changes. Compare real tasks with the existing process and capture cases the offline suite missed.

Define the pilot's audience, task types, reviewers and stopping conditions. A specific class of unsupported claim might trigger a rollback; a minor formatting issue might create a repair ticket. Decide these rules before the first incident.

Open full diagram

Before release:

Use AI agent observability to follow what happens after release and approval workflows to constrain business actions. Evaluation establishes evidence for a defined scope; ongoing operation shows whether that evidence continues to hold.