Home / Claude Certified Architect – Professional / Objective 4.2

Domain 4: Evaluation, Testing & Optimization · 16% of the exam · Full exam outline

4.2 Design evaluation datasets and test frameworks using mixed methodologies

Practice questions

Question 1easy

A team grades its summarizer with an LLM judge that runs on the same model as the summarizer. Reviewers suspect the judge rates that model's own phrasing too kindly. Which change targets this specific concern?

Included in Sample: sample every domain.See plans

Choose an answer to check it. Afterward, select any option to review its explanation.

Constrained verdicts make judging more rigorous and consistent than free-form scores. They don't change who is grading, so a model favoring its own phrasing would still do so with fewer labels.

The concern is self-preference: a model grading its own outputs. The grading ladder calls for judging with a different model than the one being evaluated, which removes that bias at the source.

Summary quality needs interpretation, which a regex can't supply. This swaps a biased grader for one that can't measure the behavior at all.

A smaller set trades away coverage, and the judge would still be grading its own model. Volume of cheap, automated cases is preferred, not fewer.

Question 2easy

A team builds its golden dataset from the 200 cleanest, best-formatted invoices in its archive. What is the main risk of this choice?

Included in Master: master every domain.See plans