Write evals before tuning prompts

The question

What needs to be done/how can you determine if your prompts are tuned well enough?

Setup

[
  {
    "question": "What was the revenue from the PowerMax tech refresh?",
    "expected": "$2.2M",
    "source": "Level"
  },
  {
    "question": "What margin did Chris achieve on the PowerMax tech refresh?",
    "expected": "27 points",
    "source": "Level"
  },
  {
    "question": "What was the revenue from the net-new disaster recovery solution?",
    "expected": "$2.2M",
    "source": "Level"
  },
  {
    "question": "What margin did Chris achieve on the disaster recovery solution?",
    "expected": "31 points",
    "source": "Level"
  },
...

Results

Baseline

# python evals/run_evals.py 

======================================================================
QUESTION: What was the revenue from the PowerMax tech refresh?
EXPECTED: $2.2M
ANSWER:   The PowerMax technology refresh generated $2.2M in revenue.
RESULT:   PASS

======================================================================
QUESTION: What margin did Chris achieve on the PowerMax tech refresh?
EXPECTED: 27 points
ANSWER:   Chris achieved 27 points of margin on the PowerMax technology refresh.
RESULT:   PASS

======================================================================
QUESTION: What was the revenue from the net-new disaster recovery solution?
EXPECTED: $2.2M
ANSWER:   The net-new disaster recovery solution generated $2.2M in revenue.
RESULT:   PASS

======================================================================
QUESTION: What margin did Chris achieve on the disaster recovery solution?
EXPECTED: 31 points
ANSWER:   Chris achieved a 31-point margin (31%) on the disaster recovery solution.
RESULT:   FAIL

======================================================================
QUESTION: How many engineers did Chris lead during the NetApp bank storage migration?
EXPECTED: 5
ANSWER:   He led a five-engineer team.
RESULT:   FAIL

======================================================================
QUESTION: What technologies did Chris use during the NetApp bank migration?
EXPECTED: Perl, Bash, and the NetApp SDK
ANSWER:   Chris used the following technologies during the NetApp bank migration:

- NetApp infrastructure and NetApp SDKs  
- Storage protocols: Fibre Channel (FC), iSCSI, and NAS  
- Scripting: Perl and Bash  
- ETL/data-transformation techniques and infrastructure automation
RESULT:   FAIL

======================================================================
QUESTION: What database did Chris migrate from SQL Server to?
EXPECTED: Aurora
ANSWER:   He migrated it to Amazon Aurora.
RESULT:   PASS

======================================================================
QUESTION: How much did the database migration save per month?
EXPECTED: $1,400
ANSWER:   The document states ongoing infrastructure expenses were reduced by approximately $1,400 per month through AWS infrastructure optimization, but it does not specify how much of that savings was attributable specifically to the database migration.
RESULT:   PASS

======================================================================
QUESTION: What payment platform did Chris integrate at ERC Today?
EXPECTED: Stripe
ANSWER:   He integrated Stripe — a custom subscription and payment system using Stripe APIs and webhooks.
RESULT:   PASS

======================================================================
QUESTION: What cloud platform does Chris have experience with?
EXPECTED: AWS
ANSWER:   Chris has experience with AWS (Amazon Web Services).
RESULT:   PASS

======================================================================
EVALUATION RESULTS
======================================================================
Tests:     10
Passed:    7
Failed:    3
Accuracy:  70.0%
======================================================================

gpt-5-mini guardrails

You are evaluating the quality of an AI-generated answer.

Determine whether the generated answer is factually consistent
with the expected answer.

The generated answer does NOT need to use the same words as
the expected answer.

Treat these as equivalent when appropriate:
- numbers written as words vs digits
- singular/plural variations
- minor wording differences
- additional relevant information

Mark the answer as PASS if it contains the expected fact
or an equivalent fact.

Mark it as FAIL if the answer contradicts the expected answer
or fails to provide the expected information.

Return ONLY valid JSON in this format:

{
  "result": "PASS" or "FAIL",
  "reason": "brief explanation"
}

After adding guardrails for response interpretation and adjusting the database migration test question above.

Focusing on the False Negatives in the baseline

======================================================================
QUESTION: What technologies did Chris use during the NetApp bank migration?
EXPECTED: Perl, Bash, and the NetApp SDK

RETRIEVED CHUNKS:
  Source:     migration-team-leadership.md
  Section:    Situation
  Similarity: 0.5783
  Source:     migration-team-leadership.md
  Section:    Skills Demonstrated
  Similarity: 0.5664
  Source:     migration-team-leadership.md
  Section:    Result
  Similarity: 0.5535
ANSWER:   Chris used the following technologies during the NetApp bank migration:

- NetApp infrastructure and NetApp SDKs  
- Storage protocols: Fibre Channel (FC), iSCSI, and NAS  
- Scripting: Perl and Bash  
- ETL / data-transformation techniques  
- Infrastructure automation (automation tooling/processes)
RESULT:   PASS
REASON:   The generated answer explicitly lists Perl, Bash, and the NetApp SDK (NetApp SDKs), matching the expected technologies.

======================================================================

QUESTION: How many engineers did Chris lead during the NetApp bank storage migration?
EXPECTED: 5

RETRIEVED CHUNKS:
  Source:     migration-team-leadership.md
  Section:    Situation
  Similarity: 0.6088
  Source:     migration-team-leadership.md
  Section:    Skills Demonstrated
  Similarity: 0.5952
  Source:     migration-team-leadership.md
  Section:    Action
  Similarity: 0.5937
ANSWER:   He led a five-engineer team.
RESULT:   PASS
REASON:   Generated answer states he led a five-engineer team, which matches the expected answer of 5.

======================================================================

QUESTION: What margin did Chris achieve on the disaster recovery solution?
EXPECTED: 31 points

RETRIEVED CHUNKS:
  Source:     rapid-impact.md
  Section:    Result**
  Similarity: 0.4407
  Source:     rapid-impact.md
  Section:    Action**
  Similarity: 0.4397
  Source:     rapid-impact.md
  Section:    Skills Demonstrated**
  Similarity: 0.4074
ANSWER:   He achieved 31 points of margin on the disaster recovery solution.
RESULT:   PASS
REASON:   The generated answer states he achieved 31 points of margin, matching the expected answer of 31 points.

======================================================================

What I Learned

A bad evaluator can make a good AI system look bad. All 3 of the false results in the baseline are false failures. An LLM was not used to set the baseline. I just used simple math, if the text matching accuracy was >= 70%, the result was a Pass. Also, notice the source, none of them are legitimate sources.
Evaluating with an LLM (guardrails) in the middle will help produce correct responses consistantly.

GIT Repository

https://github.com/pittperson/mini-rag

Log

2026-10-05 — Evaluating how well my prompt is working. There are some false negatives.
2026-10-06 — Included gpt-5-mini with guardrails on how to interpret the response. Fixed the chunk sources. Adjusted the false negative test questions.
2026-10-09 — Satisfies tooling and agent and mapping where it fails. In this case it failed without an LLM and guardrails in the mix.