Your assignment
Create an evaluation sheet with correctness, missing facts, latency and fallback outcome. Compare a deterministic baseline and a model response.
Work through it
- Start from the scenario below and write the expected behavior before changing code. If this is a design exercise, produce a diagram or decision table; for an implementation exercise, create a small reproducible project or test fixture.
- Address each of these decisions explicitly: Evaluation cases, Data minimization, Operational budgets. Record the assumptions that influence your choices.
- Reproduce the ordinary case and the failure described in the scenario. Use controlled inputs or fake dependencies where a live service would make the result unpredictable.
- Compare the observed result with your expected behavior. Fix the underlying decision or implementation when they differ; do not change the expected outcome merely to hide a failure.
- Save the evidence and explain what remains unverified. A simulated service failure demonstrates application handling, but does not establish real-device or provider compatibility.
Scenario to test
Two prompts produce equally fluent summaries, but one invents dates in three of ten notes. Fluency alone would hide the regression.
Completion criteria
Your evaluation can identify a worse prompt even when its prose sounds better.
Submit the diagram or relevant code, the inputs used, the observed result and a short explanation of the tradeoff. Include at least one ordinary case and one boundary or failure case.
Review your work
- Could another learner reproduce the result from your notes?
- Does the evidence prove the completion criterion, or only that a tool ran?
- Which assumption would change your design if it proved false?
Continue only when you can explain the result independently. The final project will combine these decisions.