Lab 04 / A recorded model comparison
Get a second
opinion.
The larger model earned more rubric credit. The smaller model answered faster. The routing rule missed both failures.
3 of 5 clinical rubric passes
3.87 s average response
5 of 5 clinical rubric passes
39.62 s average response
0 of 2 rubric failures flagged
0 of 5 questions escalated
Where the gap lives
The specialist matched more required keyword groups on rare-disease workup and ontology mapping. Both models received full rubric credit on the other three questions. Both cost $0 in this run.
| Clinical task | GP coverage | Specialist coverage | GP time | Specialist time |
|---|---|---|---|---|
| Oncology staging | 100% | 100% | 6.11 s | 11.42 s |
| Cardiology guideline | 100% | 100% | 4.45 s | 21.20 s |
| Drug-drug interaction | 100% | 100% | 4.81 s | 16.58 s |
| Rare-disease workup | 66.7% | 100% | 2.03 s | 5.51 s |
| Ontology mapping | 50% | 100% | 1.96 s | 143.41 s |
| Mean across five questions | 83.3% | 100% | 3.87 s | 39.62 s |
Coverage is the share of required keyword groups matched. Full credit requires every group and no banned phrase. The specialist's 143.41-second ontology response increased its average.
A rubric pass still contained an error. The GP's drug-interaction answer linked a low INR to bleeding and received full credit. In the warfarin context, a low INR can indicate increased clot risk. This false pass shows why keyword matches need independent review. MedlinePlus: interpreting INR results.
Models and scoring details
GP: liquid/lfm-2.5-2.6b:free
Specialist: nvidia/nemotron-3-ultra-550b-a55b:free
All 10 original clinical calls succeeded. Requested and responding model IDs matched in the recorded run. Both models received the same five questions and system prompt. The scorecard recorded zero banned-phrase flags and zero refusals for both models; zero flags did not mean zero errors.
Recorded cost per question and cost per rubric-pass answer were $0 for both models. These are experiment measurements, not a statement about today's model availability or pricing.
A specialist helps only
if the question reaches it
The measured cascade used the GP for every question. Its 60% rubric result did not improve. The blue curve is a separate oracle replay that uses knowledge of the scores to choose what to escalate.
2 of 5 questions sent to the specialist (40%). This assumes the rubric scores are known in advance.
Moving this slider changes only the hypothetical replay. The actual measured result remains 0% escalation and 60% rubric full credit. Both the cascade and specialist-only replay had $0 model API cost.
Routine supply questions
closed the score gap
Five fictional hospital supply exercises tested reorder points, inventory position, supplier costs, fill rate, and on-time delivery. Both models received full credit on all five under the keyword rubric.
Small model
Specialist
| Supply task | GP rubric | Specialist rubric | GP time | Specialist time |
|---|---|---|---|---|
| Reorder point | Full credit | Full credit | 1.78 s | 1.94 s |
| Inventory position | Full credit | Full credit | 3.03 s | 3.32 s |
| Supplier comparison | Full credit | Full credit | 3.85 s | 11.51 s |
| Immediate fill rate | Full credit | Full credit | 1.71 s | 13.72 s |
| On-time delivery | Full credit | Full credit | 2.58 s | 2.95 s |
Supplier comparison: both chose Supplier A
At least 850 units were needed, purchases had to use whole packs, and quality and delivery time were identical.
Supplier A
17 packs × 50 units = 850 units
17 × $70; no delivery fee
Supplier B
9 packs × 100 units = 900 units
9 × $130 + $25 delivery fee
On five fictional hospital supply questions connected to my supply systems experience, both models earned 100% under the keyword rubric, so the score gap was zero percentage points. Both correctly chose Supplier A at $1,190 over Supplier B at $1,195 after accounting for whole packs and delivery charges. The small model averaged 2.59 seconds per answer versus 6.69 seconds for the specialist, and both cost $0. I would choose the smaller model for these routine calculations and would not pay extra based on this test, but I would still verify its answers because five examples and keyword matches do not establish overall reliability.
The same prompt,
three recorded responses
The local Qwen2.5-0.5B-Instruct model had 494 million parameters and ran on the Colab GPU. At temperature 1.2 it introduced an unsupported six-month rule and stopped mid-sentence at the 120-token limit.
Prompt: A patient with type 2 diabetes asks whether they can stop metformin now that their A1c is back in range. Answer in three sentences.
4.3 seconds · 120 new tokens · 28.0 tokens/second
It's generally recommended to wait for at least six months after getting your HbA1c level under control before stopping the metformin medication if it has been approved and you have not experienced any severe adverse events such as ketoacidosis or liver damage from excessive weight loss associated with losing too much fat. Metformin can still be effective in controlling blood sugar levels over longer periods when properly prescribed and monitored by healthcare professionals. However, there are always alternative strategies depending on each patient's specific needs, medical history, lifestyle factors and doctor's judgement. It ultimately depends on individual circumstances and should be
These are unvalidated model responses, not medication instructions. A lower temperature does not establish that an answer is correct. The displayed sweep is the final recorded Part A run used in the notebook's takeaways.
My takeaways
My small model earned full credit on 60% of the five questions, compared with 100% for the specialist, a 40-percentage-point gap. Both passed the cardiology guideline rubric, suggesting that listing standard treatment classes was manageable for the smaller model in this example. Both cost $0; I would pay only if the extra cost per independently verified correct answer was less than the cost of reviewing or correcting mistakes. The cascade stayed at 60% because it escalated nothing and detected 0% of the two rubric failures. I would require external checks and human review instead of trusting the model to admit uncertainty. The drug-interaction answer also passed despite contradictory claims about anticoagulation risk, showing that keyword scores can miss errors. At temperature 1.2, the local model gave an unsupported six-month rule for stopping medication and reached its 120-token limit mid-sentence. I skipped the optional paid comparison.
Stakeholders, failure modes, and deployment limits
Stakeholders: A health system administrator could benefit from reduced staff workload, while patients could receive faster responses. However, the patient who follows incorrect medication advice carries the health risk, and clinicians must identify and correct mistakes. My GP averaged 3.87 seconds versus 39.62 seconds for the specialist, but both cost $0, so this run did not demonstrate model-fee savings. Because the GP flagged neither of its two rubric failures, I would require clinician review before delivering these clinical answers to patients.
Failure modes: My GP’s warfarin answer earned full credit while incorrectly linking low INR to bleeding, although an INR that is too low can indicate increased clot risk. MedlinePlus A false fail could occur if an otherwise correct answer says “type II diabetes” while the rubric recognizes only “type 2 diabetes” or “diabetes mellitus type 2.” A false pass is more dangerous because a passing score could encourage someone to trust harmful advice. I would first add expert review and checks against trusted references for contradictions, then expand the accepted wording to reduce false failures.
When not to build this: I would not deploy an LLM to independently give patients drug-interaction advice or change medication doses. The GP’s false pass shows why matching keywords is insufficient, and the specialist’s five rubric passes do not establish clinical safety. Before using either model as a clinician’s assistant, I would require expert evaluation on a larger, representative set of independent test cases, including uncommon and dangerous situations. I would also require reliable drug references, clear responsibility for reviewing answers, and ongoing monitoring with a way to stop the system when problems appear.
Inspect the evidence
This showcase was assembled from the completed notebook's saved outputs. The supporting notebook preserves the original code, charts, measured results, and written responses.
The results download includes displayed scores, timings, question rubrics, and answer examples printed in the notebook. It does not replace the original full-response CSV or API cache. Missing raw responses have not been reconstructed.
Clinical questions and rubric context
Oncology staging
A 62-year-old woman has a 2.5 cm invasive ductal carcinoma of the left breast, confined to the breast. Two axillary sentinel lymph nodes contain metastases. Imaging shows no distant metastases. Give the T, N and M categories and the AJCC anatomic stage group.
Rubric: 4 required keyword groups; 2 banned phrases. The full rubric is in the notebook and downloadable results.
Cardiology guideline
A 68-year-old has heart failure with an LVEF of 30%, NYHA class II, and is stable on lisinopril and furosemide. Name the four drug classes that make up guideline-directed medical therapy for HFrEF, and say what you would change first.
Rubric: 4 required keyword groups; 1 banned phrases. The full rubric is in the notebook and downloadable results.
Drug-drug interaction
A 71-year-old on warfarin has a stable INR of 2.4. She is prescribed fluconazole 200 mg daily, and asks whether she can also start over-the-counter St John's wort for low mood. Explain both interactions and what you recommend.
Rubric: 4 required keyword groups; 3 banned phrases. The full rubric is in the notebook and downloadable results.
Rare-disease workup
A 9-year-old has had four fractures from minor falls, blue-grey sclerae, dentinogenesis imperfecta, and a family history of similar fractures across three generations. Give the most likely diagnosis, the genes most often involved, and the test that confirms it.
Rubric: 3 required keyword groups; 1 banned phrases. The full rubric is in the notebook and downloadable results.
Ontology mapping (ICD-10-CM to SNOMED CT)
A hospital is moving its problem list from ICD-10-CM to SNOMED CT. Map the ICD-10-CM code E11.9 (Type 2 diabetes mellitus without complications) to its SNOMED CT concept: give the concept title and, if you know it, the SCTID. Then explain in two sentences why this kind of mapping is rarely one-to-one.
Rubric: 2 required keyword groups; 1 banned phrases. The full rubric is in the notebook and downloadable results.
AI use disclosure
I used ChatGPT/Codex to help write, debug, and explain the temperature sweep, rubric scorer, benchmark, cascade, routing function, and supply-question extension, and to draft the Takeaways and reflection responses. Verification included executing the code in Colab and reviewing my actual outputs with the assistant to check the reported scores, timings, and costs. The assistant also consulted MedlinePlus to assess the warfarin response and identified an incorrect clinical claim that the keyword rubric had passed. Changes made with AI assistance included correcting the cascade cost calculation to count both calls when escalation occurs and revising a chart caption that misrepresented the 0% escalation result. These checks support the notebook’s calculations but do not constitute clinical validation. My own work included setting up the accounts and secrets, running the notebook, and collecting the outputs; the code and written interpretation received substantial AI assistance.
During final review, the assistant added a missing import and removed an unused starter function.
ChatGPT/Codex also assembled this static showcase from the saved notebook, extracted the displayed measurements, and redrew its two charts using the recorded coordinates. No models were rerun to prepare the page.