Home / AIF-C01 / Objective 3.4

Domain 3: Applications of Foundation Models · 28% of the exam · Full exam outline

3.4 Describe methods to evaluate FM performance

What this objective covers · 7% of the exam

Determine approaches to evaluate FM performance (for example, human-in-the-loop evaluation, benchmark datasets, Amazon Bedrock Model Evaluation).; Identify relevant metrics to assess FM performance (for example, Recall-Oriented Understudy for Gisting Evaluation [ROUGE], Bilingual Evaluation Understudy [BLEU], BERTScore, LLM-as-a-judge).; Determine whether an FM effectively meets business objectives (for example, productivity, user engagement, task engineering).; Identify approaches to evaluate the performance of applications built with FM (for example, RAG, agents, workflows).; Identify business objective alignment metrics for AI applications (for example, task completion rate, user satisfaction, cost per interaction).

Practice questions

Question 1easy

A tax-help assistant is benchmarked every month against an internal golden dataset. This season, customers have started asking many questions about a newly introduced tax credit that the dataset does not cover. What should the team do with the dataset?

Included in Master: master every domain.See plans
Question 2easy

A team wants to compare two Amazon Bedrock Knowledge Bases only on how well they find relevant passages, without judging any generated answers. Which RAG evaluation job type fits?

Included in Master: master every domain.See plans