Evaluation, Safety, Bias and Responsible AI

LLM metrics, LLM-as-a-judge, golden datasets, benchmark pitfalls, A/B testing, hallucination control, red teaming, fairness, privacy, the EU AI Act and guardrails.

• Advanced
• 18 min read
• 11 questions

Before this lessonLesson 8: Inference, Serving Performance and LLMOps

If there is one topic that separates experienced AI engineers from enthusiasts, it is evaluation. Anyone can build a demo; the people companies hire can prove a system works, notice when it stops working, and show it is safe and fair enough to ship.

Interviewers also probe responsibility: bias, privacy, misuse and regulation. Answer concretely (a metric, a control, a process) rather than with values alone.

The 11 questions in this lesson

89. How do you evaluate the output of an LLM? Explain BLEU, ROUGE, BERTScore, perplexity and pass@k, and when each is appropriate.
90. What is LLM-as-a-judge, and what are its limitations and how do you calibrate it?
91. You have to evaluate a new LLM application but there is no labelled data and experts are expensive. How do you build an evaluation set?
92. What are MMLU, HumanEval and GSM8K? What is benchmark contamination, and why can a model with higher benchmark scores feel worse to users?
93. How do you compare two prompts or models in a statistically sound way? Offline versus online evaluation?
94. How do you detect, measure and reduce hallucinations?
95. What is red teaming, and how would you red team an LLM chatbot (including a multimodal one) before launch?
96. How do you detect and mitigate bias in an AI system? Explain fairness metrics and proxy discrimination.
97. How do you protect privacy in AI systems: PII handling, GDPR, the right to be forgotten, membership inference and differential privacy?
98. What is the EU AI Act, the NIST AI Risk Management Framework, and model cards? How do you make AI decisions explainable and auditable?
99. How do you implement guardrails, content moderation and AI-content provenance, and how do you respond when an AI system causes harm?

89. How do you evaluate the output of an LLM? Explain BLEU, ROUGE, BERTScore, perplexity and pass@k, and when each is appropriate.

Core

Pick the metric by what a good answer is for the task, and prefer checks that can be automated against a ground truth when one exists.

The unbiased estimator for pass@k from n samples with c correct is 1 - C(n-c, k) / C(n, k):

from math import comb

def pass_at_k(n, c, k):
    # probability that at least one of k samples (drawn from n, c correct) is correct
    if n - c < k:
        return 1.0
    return 1.0 - comb(n - c, k) / comb(n, k)

n, c = 20, 5                        # 20 generations for one problem, 5 pass the tests
for k in (1, 5, 10):
    print("pass@%-2d = %.3f" % (k, pass_at_k(n, c, k)))

In practice combine several: automatic checks (format, exact facts, tests), an LLM judge for quality, and periodic human review, and always keep metrics that reflect the user's goal.

Follow-up: Why is ROUGE a poor primary metric for an abstractive summariser?

90. What is LLM-as-a-judge, and what are its limitations and how do you calibrate it?

Core

An LLM-as-a-judge scores or ranks outputs against a rubric ("is every claim supported by the context?", "which answer is more helpful, A or B?"). It scales far beyond human review and correlates well with human ratings when done carefully, which is why frameworks such as G-Eval, MT-Bench-style pairwise judging and RAG evaluators use it.

• Position bias: judges favour the first (or last) answer. Swap the order and count a win only if it is consistent.
• Verbosity bias: longer answers get higher scores regardless of quality. Control for length or instruct against it.
• Self-preference: a model tends to prefer its own family's style. Use a different, stronger model as the judge.
• Leniency, inconsistency and prompt sensitivity: scores vary between runs and phrasings. Use a clear rubric with examples, low temperature, and integer scales with definitions.
• Shared blind spots: the judge may miss the same factual errors the generator makes. Give it the source or reference, or use retrieval and tools to verify.

import random

def biased_judge(a, b, rng):
    # a judge that says 'first is better' 65% of the time regardless of quality
    return "A" if rng.random() < 0.65 else "B"

rng = random.Random(0)
consistent = 0
trials = 2000
for _ in range(trials):
    r1 = biased_judge("resp1", "resp2", rng)            # order 1
    r2 = biased_judge("resp2", "resp1", rng)            # swapped order
    # consistent only if the same response wins in both orders
    consistent += (r1 == "A" and r2 == "B") or (r1 == "B" and r2 == "A")
print("verdicts that survive swapping the order: %.0f%%" % (100 * consistent / trials))
print("a truly quality-driven judge would be consistent nearly always")

Follow-up: How would you detect that your judge prefers longer answers?

91. You have to evaluate a new LLM application but there is no labelled data and experts are expensive. How do you build an evaluation set?

Core

Start small and grow it deliberately. A useful first set is 50 to 200 examples that are representative and hard, not thousands of easy ones.

1. Harvest real inputs: logs, support tickets, search queries, pilot-user questions (with privacy review). If none exist, have domain colleagues write realistic ones.
2. Cover the space: tag each example by topic, difficulty and type, and make sure the set includes edge cases, ambiguous queries, unanswerable questions (the right answer is a refusal), adversarial prompts, multilingual input and long inputs.
3. Get labels cheaply: have a strong model draft reference answers or rubrics, then have an expert review and correct them (reviewing is far faster than writing); label only a stratified sample if budget is tight.
4. Prefer checkable criteria: exact facts, required fields, must-include or must-not-include phrases, schema validity, unit tests, so much of the score is automatic.
5. Calibrate the judge on the human-reviewed subset for the fuzzy criteria.
6. Keep it alive: add every production failure as a new test case, version the set, keep a frozen held-out portion that you never tune against, and re-run on every prompt, model or retrieval change (evaluation-driven development).

Define "done" before building: an agreed metric and threshold (for example 90 percent faithfulness and under 2 percent unsafe outputs) makes the evaluation a decision tool rather than a report.

Follow-up: How do you keep the evaluation set from leaking into your prompts or fine-tuning data?

92. What are MMLU, HumanEval and GSM8K? What is benchmark contamination, and why can a model with higher benchmark scores feel worse to users?

Core

Follow-up: A vendor claims a new state-of-the-art benchmark score. What would you check before believing it?

93. How do you compare two prompts or models in a statistically sound way? Offline versus online evaluation?

Deep dive

To compare A and B honestly: run both on the same inputs (paired design) so difficulty differences cancel; run several samples per input if the model is stochastic; report a confidence interval for the difference (a paired bootstrap works for any metric) rather than a single average; make sure you have enough examples for the effect size you care about (small differences need hundreds or thousands of cases); check subgroups for hidden regressions; and correct for multiple comparisons if you test many variants. For online tests, randomise by user, fix the sample size and metric in advance, watch guardrail metrics (safety, cost, latency), and beware novelty effects.

import random
random.seed(11)

# per-question correctness (1/0) for two prompts evaluated on the SAME 150 questions
a = [1 if random.random() < 0.72 else 0 for _ in range(150)]
b = [x if random.random() < 0.85 else (1 if random.random() < 0.8 else 0) for x in a]   # B is a bit better

def paired_bootstrap(a, b, iters=5000):
    n, diffs = len(a), []
    for _ in range(iters):
        idx = [random.randrange(n) for _ in range(n)]
        diffs.append(sum(b[i] - a[i] for i in idx) / n)
    diffs.sort()
    return diffs[int(0.025 * iters)], diffs[int(0.975 * iters)]

lo, hi = paired_bootstrap(a, b)
print("accuracy A %.3f  B %.3f  observed diff %.3f" % (sum(a) / 150, sum(b) / 150, (sum(b) - sum(a)) / 150))
print("95%% CI for the difference: [%.3f, %.3f]" % (lo, hi))
print("interval includes 0 -> not convincing yet" if lo <= 0 <= hi else "interval excludes 0 -> B is credibly better")

Follow-up: Your offline test says B is 3 points better, but the online test shows no change. What could explain it?

94. How do you detect, measure and reduce hallucinations?

Core

flowchart TB A[Model choice and grounding] --> B[Retrieval and tools for facts, maths and lookups] B --> C[Prompt: answer only from sources, allow I do not know, require citations] C --> D[Decoding: lower temperature, constrained formats] D --> E[Verification: claim-level faithfulness check, citation validation, second-model review] E --> F[Fallback: refuse, ask a clarifying question, or hand off to a human] F --> G[Monitoring: sampled review and user feedback loop]

No technique removes hallucination; you reduce and contain it. Match the control to the stakes: a marketing draft can tolerate errors, medical or legal answers need grounded sources, verification and human sign-off. Be honest with users about limits, show sources, and design the interface so mistakes are easy to spot.

Follow-up: Your summariser inserts facts that are not in the article. Name three fixes and how you would prove they worked.

95. What is red teaming, and how would you red team an LLM chatbot (including a multimodal one) before launch?

Core

Red teaming is adversarial testing by people (or tools) trying to make the system fail: produce harmful content, leak secrets, follow injected instructions, discriminate, or take unsafe actions. Automated benchmarks find known problems; red teaming finds the creative ones.

1. Define scope and harms: from the product's context (a banking bot cares about fraud and data leakage; a kids' app cares about age-inappropriate content), plus universal categories (self-harm, hate, illegal activity, privacy, misinformation, prompt injection, tool abuse).
2. Assemble diverse testers: security engineers, domain experts, and people from affected communities, plus automated attackers (LLM-generated attack prompts, fuzzing, jailbreak libraries).
3. Attack systematically: direct and indirect prompt injection, role-play and multi-turn escalation, encoding and translation tricks, system-prompt extraction, tool and permission abuse, data exfiltration, over-reliance and misuse scenarios.
4. Record everything with severity ratings and reproduction steps; convert each finding into a regression test.
5. Fix at the right layer: model or prompt changes, filters, permission changes, product design; re-test after fixes; set launch criteria (for example zero critical findings open).
6. Continue after launch: monitoring, bug bounties and periodic exercises.

Follow-up: Which findings would block your launch, and who signs off on shipping with known residual risk?

96. How do you detect and mitigate bias in an AI system? Explain fairness metrics and proxy discrimination.

Deep dive

Bias can enter through data (under-representation, historical inequities, biased labels), design (proxy features, objective choice) and deployment (feedback loops where model outputs shape future data). Detect it by evaluating performance and outcomes per group, including intersections (a model fair for gender and for race separately can still fail for a specific combination).

These cannot generally all be satisfied at once when base rates differ, so the choice is a policy decision tied to the harm you most want to avoid, made with legal and domain input. A common screening rule in US employment settings is the four-fifths rule: if a group's selection rate is below 80 percent of the highest group's, that is evidence of adverse impact worth investigating.

applicants = {                       # group: (number screened, number advanced)
    "men":   (1000, 300),
    "women": (800, 160),
}
rates = {g: adv / n for g, (n, adv) in applicants.items()}
best = max(rates.values())
for g, r in rates.items():
    print("%-6s selection rate %.2f   ratio to best %.2f%s" % (g, r, r / best, "   <-- below 0.80" if r / best < 0.8 else ""))

Follow-up: Your resume-screening model advances fewer women. You removed the gender column and nothing changed. Why?

97. How do you protect privacy in AI systems: PII handling, GDPR, the right to be forgotten, membership inference and differential privacy?

Deep dive

Treat privacy as a design constraint from the start:

• Minimise and control data: collect only what is needed, redact or pseudonymise PII before it reaches prompts, logs or training data, set retention limits, encrypt, and restrict access. Check provider terms on whether prompts are stored or used for training, and use zero-retention or private deployments for sensitive data.
• Lawful basis and rights (GDPR, CCPA and similar): consent or another lawful basis, purpose limitation, transparency, access and deletion rights, data-protection impact assessments for risky processing, and explanation of automated decisions.
• Right to be forgotten: deleting a person's data from a database is easy; removing what a model has learned from its weights is not. Practical approaches: keep personal data out of training in the first place and use RAG so deletion means removing it from the index; maintain data lineage so you know which models saw what; retrain or fine-tune without the data on a schedule; apply output filters; and consult counsel, as regulators are still working out the details for model weights. Machine unlearning is active research, not a solved tool.
• Memorisation and leakage: models can regurgitate training data. Membership-inference attacks test whether a record was in the training set; extraction attacks recover text. Mitigate with deduplication, less overfitting, output filtering and privacy-preserving training.
• Differential privacy: DP-SGD clips each example's gradient and adds calibrated noise so the trained model's behaviour barely depends on any single individual, giving a provable privacy budget (epsilon). The cost is accuracy and compute; small epsilons hurt utility, so balance and report the trade-off.
• Re-identification: removing names is not anonymisation; combinations of quasi-identifiers (age, postcode, dates) can identify people. Use aggregation, k-anonymity checks, noise or synthetic data, and test re-identification risk.

Follow-up: A user demands you delete their data, but you fine-tuned on support tickets that included it. What do you do?

98. What is the EU AI Act, the NIST AI Risk Management Framework, and model cards? How do you make AI decisions explainable and auditable?

Core

Follow-up: Is your customer-support chatbot 'high-risk' under the EU AI Act? Which duties still apply?

99. How do you implement guardrails, content moderation and AI-content provenance, and how do you respond when an AI system causes harm?

Core

• Input: authentication and rate limits; prompt-injection and jailbreak detection; PII detection and redaction; topic and scope checks (a banking bot should decline to write malware).
• Model and prompt: a clear system prompt with refusal policy; a safety-tuned model; least-privilege tools.
• Output: moderation classifiers (toxicity, self-harm, sexual content, hate, violence), schema validation, groundedness checks, PII and secret scanning, allow-lists for actions; block, rewrite or route to a human as appropriate.
• Domain-specific: a medical or legal bot must not present itself as a professional, must escalate emergencies (crisis resources for self-harm), and must show sources and disclaimers appropriate to the domain.
• Cultural context: moderation trained on one market may flag normal expressions elsewhere; evaluate and tune per locale with local reviewers, and provide appeals.

Follow-up: Your mental-health chatbot gave harmful advice to a user in crisis. What are your first 24 hours?

Sources and further reading

• Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Zhang et al., BERTScore
• Chen et al., Evaluating LLMs Trained on Code (HumanEval, pass@k)
• Manakul et al., SelfCheckGPT
• NIST AI Risk Management Framework
• EU AI Act overview (European Commission)
• Mitchell et al., Model Cards for Model Reporting
• AI Engineering interview questions (Outcome School, Apache-2.0)