{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "_L1sBako6bow" }, "source": [ "# MIS 752 — Lab 4: Get a Second Opinion — GP versus the Specialist\n", "\n", "**Advanced Healthcare Analytics & AI · Week 4 · Generative AI & Foundation Models · Ch 18 (LLMs: architecture)**\n", "\n", "> **v2, 2026-09-13.** Written for the **Great Reset** (`_admin/GREAT_RESET_2026.md`): Week 4 moved from Alert Fatigue to Generative AI & Foundation Models, and alert fatigue moved to **Week 9**, where it is the clinical context for agents. `Lab04_alert_burden.ipynb` is retired but stays in this folder for reference. **Publishing to Hugging Face is now optional** — this notebook submitted to Canvas earns full credit on its own.\n", "\n", "> **The case.** A health system is putting an AI assistant on the patient portal. The vendor demoed their flagship model and it was flawless — and it costs roughly a thousand times what the free tier costs per question. The CFO's question is one line: *\"how much worse is the cheap one, on OUR questions, and where exactly does it break?\"* You have one afternoon, five real clinical questions, one API key, and one rule: nobody spends any money.\n", "\n", "### Why are we learning this?\n", "\"Just use ChatGPT\" is a **decision**, not a default. Every generative-AI choice trades cost against speed against accuracy, and the answer is not always \"bigger is better.\" This week you look under the hood twice. **Part A** is on your own machine: a 494-million-parameter open model, its tokenizer, and the temperature knob — the three things that decide what a foundation model actually does. **Part B** goes through one API to both ends of the market and measures the gap: a small free model against the largest model you can reach for $0, on the same five clinical questions, with an optional run against a true paid frontier model if you ever want to spend your own credit.\n", "\n", "**Transferable skill:** *model selection as a cost / speed / accuracy decision* — picking a model tier with measurements instead of vibes. This is the same decision everywhere: which model answers support tickets, which one drafts the marketing copy, which one you escalate the hard 5% to. The number that settles it is **cost per correct answer**, and you are going to compute it. Healthcare is just the highest-stakes place to practice.\n", "\n", "> **This week's case file:** no patient rides along. The case file is the five questions themselves — self-contained clinical vignettes with zero PHI. Your measured outputs are the deliverable, and **Week 7 (AI Evaluation) turns these gut reactions into a scored rubric**, so save them.\n", "\n", "> **What this lab is not.** Nothing here is medical advice and no model here is cleared for clinical use. You are auditing software, exactly as a hospital procurement team would before it signs a contract.\n" ], "id": "_L1sBako6bow" }, { "cell_type": "markdown", "metadata": { "id": "-RqIu9736box" }, "source": [ "## Learning objectives\n", "By the end you can:\n", "1. **Load a small open foundation model locally** and see what the tokenizer and the temperature knob actually do to one prompt.\n", "2. **Call two model tiers through one OpenRouter key** and capture latency, token counts and real dollar cost per question.\n", "3. **Score both against a transparent key-fact rubric** and build a side-by-side scorecard a manager can read in a minute.\n", "4. **Decide with numbers** which tier you would route each question to, and write `route_clinical_question()` for your toolkit.\n" ], "id": "-RqIu9736box" }, { "cell_type": "markdown", "metadata": { "id": "wKqtuG4X6boy" }, "source": [ "---\n", "## Make this lab fit you\n", "\n", "There's a required core and two optional layers. Pick your depth.\n", "\n", "**The core (everyone does this):** run Part A's temperature sweep on the local 0.5B model, send the five clinical questions to both free OpenRouter models, score the answers, build the scorecard, and **submit the notebook on Canvas**. Publishing the scorecard as a public Hugging Face Space is the optional Showcase — skip it and you have still done the whole lab.\n", "\n", "**Want to go above?** (core done, you want more)\n", "- Add a **third model** and grow the scorecard: `inclusionai/ling-3.0-flash-sante:free` is a health-tuned free model, which lets you test whether *domain tuning* beats *raw scale*. Still $0.\n", "- Swap the **specialist**: set `BIG_FREE` to `nvidia/nemotron-3-ultra-550b-a55b:free` (550B total but 55B **active**, versus Inkling's 975B/41B) and re-run. Comparing on active parameters instead of total is a different and defensible reading of \"biggest\" — argue for yours.\n", "- Take the **frontier** run (Step 6) with your own credit and see whether the paid flagship is worth its price on your five questions.\n", "- Replace the keyword rubric with an **LLM-as-judge**: have the big free model grade both answers, then argue with its grades.\n", "- Swap in **your own five questions** from your final-project domain and re-run the whole harness.\n", "- Measure **run-to-run variance**: send one question three times at `temperature=0` and see whether \"deterministic\" really means deterministic.\n", "\n", "**Need to go below?** (stuck or short on time — this is completely fine)\n", "- Use the instructor solutions (expand the **Solution** cells) instead of writing from scratch.\n", "- **Part A is self-contained** — if OpenRouter or your key misbehaves, Part A plus the rubric you wrote still demonstrate the week's idea. Say so in your takeaways.\n", "- You may skip the **cascade** in Step 7 and just report the two-model scorecard.\n", "- You may skip the **frontier** run in Step 6 entirely. It is optional and costs money.\n", "- If a plot errors, the printed scorecard table still tells the story — describe it in words.\n" ], "id": "wKqtuG4X6boy" }, { "cell_type": "markdown", "metadata": { "id": "9gzR1EXA6boy" }, "source": [ "---\n", "## Setup\n", "\n", "Two things to get ready, and neither needs a credit card.\n", "\n", "1. **A local model.** Part A downloads `Qwen/Qwen2.5-0.5B-Instruct` — 494 million parameters, about 1 GB, **ungated** (no licence form, no approval wait). Colab's free CPU runs it; a free T4 makes it faster. `meta-llama/Llama-3.2-1B-Instruct` is the other common choice, but it is **gated** — you have to accept Meta's licence on huggingface.co and wait for approval — so this lab defaults to Qwen. Swap it in later if you already have access.\n", "2. **An OpenRouter key.** Part B sends five questions through [OpenRouter](https://openrouter.ai), one API that fronts hundreds of models. Create your **own free** account, make a key at [openrouter.ai/keys](https://openrouter.ai/keys), and add it to Colab as the secret `OPENROUTER_API_KEY` (Lab 1, Section G showed how). **This lab only ever calls models whose price is $0.** The free tier allows roughly 50 requests a day; this lab uses **10**, and it caches every answer to disk so re-running a cell never spends quota twice.\n", "\n", "No key? Part A and the rubric still run, and the cell in Part B tells you plainly what it skipped instead of failing.\n" ], "id": "9gzR1EXA6boy" }, { "cell_type": "code", "execution_count": 1, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "\u001b[?25l \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/12.3 MB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K \u001b[91m━━━━━━━━━━━━━━\u001b[0m\u001b[91m╸\u001b[0m\u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m4.6/12.3 MB\u001b[0m \u001b[31m137.4 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K \u001b[91m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[91m╸\u001b[0m \u001b[32m12.3/12.3 MB\u001b[0m \u001b[31m233.0 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K \u001b[91m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[91m╸\u001b[0m \u001b[32m12.3/12.3 MB\u001b[0m \u001b[31m233.0 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m12.3/12.3 MB\u001b[0m \u001b[31m107.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hpackages ready\n" ] } ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "setup-install", "outputId": "c6d123ba-537a-47fc-9bf0-ae940a66b3f0" }, "id": "setup-install", "source": [ "# Toolkit. Colab already ships torch, transformers, pandas and plotly -- this only tops up.\n", "!pip install -q -U transformers\n", "\n", "import pandas as pd # scorecard tables; imported here so every later cell can use it\n", "print(\"packages ready\")\n" ] }, { "cell_type": "code", "execution_count": 3, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ " Your Spaces: none yet\n", "Plumbing check done. Ready to publish at Ship it.\n" ] } ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "lab-plumbing", "outputId": "ef89bf85-af35-4c0a-b911-514b68de7583" }, "id": "lab-plumbing", "source": [ "#@title Lab plumbing · check your keys and Spaces (~15 s, warnings only)\n", "# ══════════════════════════════════════════════════════════════════════════════\n", "# From the Bonus notebook (\"the whole pipeline in five cells\") -- same helpers\n", "# every week, so learn them once. This cell NEVER stops the lab: every analysis\n", "# cell below runs fine without any keys. Publishing (Ship it) re-checks and\n", "# stops there, with the fix, if something is missing.\n", "#\n", "# WHAT THIS CELL DOES\n", "# 1. Reads your secret(s) from the Colab 🔑 sidebar (never stored in this file).\n", "# 2. Asks Hugging Face \"whose token is this?\" (username, Write vs read, MIS-752).\n", "# 3. Lists any Spaces you already own by state, so a paused/asleep Space never\n", "# surprises you later.\n", "# ══════════════════════════════════════════════════════════════════════════════\n", "import json, datetime, urllib.request\n", "try:\n", " import huggingface_hub\n", "except ImportError:\n", " get_ipython().run_line_magic(\"pip\", \"install -q huggingface_hub\")\n", " import huggingface_hub\n", "from huggingface_hub import HfApi\n", "from google.colab import userdata # the 🔑 sidebar\n", "\n", "def secret(name, where=\"\"):\n", " \"\"\"Read one secret from the 🔑 sidebar; return '' and print the fix if missing.\"\"\"\n", " try:\n", " value = (userdata.get(name) or \"\").strip()\n", " if not value:\n", " print(f\" • {name} is empty. {where}\")\n", " return value\n", " except userdata.SecretNotFoundError:\n", " print(f\" • {name} does not exist. 🔑 sidebar → Add new secret → name it exactly {name}. {where}\")\n", " except userdata.NotebookAccessError:\n", " print(f\" • {name} exists but Notebook access is OFF. Toggle it ON in the 🔑 sidebar.\")\n", " return \"\"\n", "\n", "HF_TOKEN = secret(\"HF_TOKEN\",\n", " \"Create one (type: Write) at https://huggingface.co/settings/tokens\")\n", "OPENROUTER_API_KEY = secret(\n", " \"OPENROUTER_API_KEY\", \"Free key at https://openrouter.ai/keys\")\n", "\n", "# ── Who does this token belong to? ──\n", "HF_USER, TOKEN_ROLE, IN_ORG, ACCOUNT_AGE_DAYS = None, \"?\", False, None\n", "if HF_TOKEN:\n", " try:\n", " me = HfApi(token=HF_TOKEN).whoami()\n", " HF_USER = me[\"name\"]\n", " TOKEN_ROLE = me.get(\"auth\", {}).get(\"accessToken\", {}).get(\"role\", \"?\")\n", " IN_ORG = any(o.get(\"name\") == \"MIS-752\" for o in me.get(\"orgs\", []))\n", " if TOKEN_ROLE == \"read\":\n", " print(\" • HF_TOKEN is READ-only. Ship it needs a Write token (same page, new token).\")\n", " with urllib.request.urlopen(f\"https://huggingface.co/api/users/{HF_USER}/overview\",\n", " timeout=20) as response:\n", " created = datetime.datetime.fromisoformat(\n", " json.load(response)[\"createdAt\"].replace(\"Z\", \"+00:00\"))\n", " ACCOUNT_AGE_DAYS = (datetime.datetime.now(datetime.timezone.utc) - created).days\n", " except Exception as err:\n", " print(f\" • HF_TOKEN was rejected ({type(err).__name__}). Make a fresh Write token.\")\n", "else:\n", " print(\" • No HF_TOKEN yet. The whole analysis below still runs; only Ship it needs the token.\")\n", "\n", "# ── Spaces you already own, grouped by state (the 'why is my page down?' answer). ──\n", "if HF_USER:\n", " try:\n", " api = HfApi(token=HF_TOKEN)\n", " states = {}\n", " for sp in list(api.list_spaces(author=HF_USER))[:30]:\n", " runtime = getattr(api.space_info(sp.id), \"runtime\", None)\n", " stage = (getattr(runtime, \"stage\", None) or \"?\") if runtime else \"?\"\n", " states[stage] = states.get(stage, 0) + 1\n", " pretty = {(\"running\" if s == \"RUNNING\" else \"asleep\" if s == \"SLEEPING\"\n", " else \"paused\" if s == \"PAUSED\" else \"errors\" if \"ERROR\" in s\n", " else \"empty\" if s == \"NO_APP_FILE\" else s.lower()): n\n", " for s, n in states.items()}\n", " print(\" Your Spaces:\", \", \".join(f\"{n} {k}\" for k, n in pretty.items()) or \"none yet\")\n", " if pretty.get(\"paused\"):\n", " print(\" (paused = no hardware on a free account; this course ships STATIC Spaces,\")\n", " print(\" which need no hardware -- so an old paused Space costs you nothing here.)\")\n", " except Exception:\n", " pass\n", "\n", "READY_TO_PUBLISH = bool(HF_TOKEN and HF_USER and TOKEN_ROLE == \"write\")\n", "print(\"Plumbing check done.\",\n", " \"Ready to publish at Ship it.\" if READY_TO_PUBLISH\n", " else \"Ship it will stop until the fixes above are in. Nothing below needs keys.\")\n" ] }, { "cell_type": "markdown", "metadata": { "id": "BtdiPA1F6boy" }, "source": [ "---\n", "## Part A — under the hood: one small model, one knob\n", "\n", "**Why:** before you compare models you should be able to say what a model *is*. A foundation model is a pile of weights that turns a list of **tokens** (integer ids) into a probability distribution over the next token. Two controls decide everything you will see this week: **how many parameters** it has (Part B's whole subject) and **how you sample** from that distribution (this part's subject).\n", "\n", "`temperature` rescales the probabilities before sampling. Near 0 the model is greedy — it takes the single most likely token every time, so the same prompt gives the same answer. Near 1.5 the distribution flattens, unlikely tokens get picked, and the answer wanders. Somewhere in between is the setting that makes prose readable, and somewhere past that is the setting that makes it wrong. You are going to watch that happen on a clinical prompt, on your own machine, with no API involved.\n", "\n", "This also explains Part B's bill: you pay **per token**, in and out. Seeing the tokenizer split a sentence into ids is seeing the meter run.\n" ], "id": "BtdiPA1F6boy" }, { "cell_type": "code", "execution_count": 4, "outputs": [ { "output_type": "display_data", "data": { "text/plain": [ "config.json: 0%| | 0.00/659 [00:00 23 tokens\n", "first 24 tokens, decoded one at a time:\n", "['Met', 'form', 'in', ' ', '5', '0', '0', ' mg', ' twice', ' daily', ' with', ' food', ';', ' re', 'check', ' A', '1', 'c', ' in', ' ', '3', ' months', '.']\n", "\n", "Note the pieces: 'Metformin' may survive as one token while 'recheck' splits into\n", "'re' + 'check'. Tokens are not words, and every API bills by the token -- which is\n", "exactly the number Part B pulls out of each response's `usage` block.\n" ] } ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "partA-tokens", "outputId": "36b8e5b9-91ef-4752-cdaa-55889a4a766a" }, "id": "partA-tokens", "source": [ "# The meter: what a sentence actually costs, in tokens.\n", "DEMO = \"Metformin 500 mg twice daily with food; recheck A1c in 3 months.\"\n", "\n", "ids = tokenizer(DEMO)[\"input_ids\"]\n", "print(f\"{len(DEMO.split())} words -> {len(ids)} tokens\")\n", "print(\"first 24 tokens, decoded one at a time:\")\n", "print([tokenizer.decode([i]) for i in ids[:24]])\n", "print()\n", "print(\"Note the pieces: 'Metformin' may survive as one token while 'recheck' splits into\")\n", "print(\"'re' + 'check'. Tokens are not words, and every API bills by the token -- which is\")\n", "print(\"exactly the number Part B pulls out of each response's `usage` block.\")\n" ] }, { "cell_type": "code", "execution_count": 6, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "\n", "Temperature: 0.1\n", "Time: 3.86 seconds | New tokens: 67 | Tokens/second: 17.38\n", "It's important for patients with type 2 diabetes to continue using metformin as prescribed by their healthcare provider, even if their blood glucose levels return to normal. Metformin helps manage the condition and reduce the risk of complications associated with diabetes. It's crucial to monitor your blood sugar regularly and adjust your treatment plan accordingly.\n", "----------------------------------------------------------------------\n", "\n", "Temperature: 0.7\n", "Time: 3.07 seconds | New tokens: 83 | Tokens/second: 27.02\n", "It's important to work closely with your healthcare provider before making any changes to your medication regimen. Metformin is typically stopped when blood glucose levels stabilize and the glycemic control is optimal for your specific condition. If you're concerned about potential side effects or if your current treatment plan isn't addressing all aspects of your health, it may be beneficial to consult with an endocrinologist who specializes in diabetes management.\n", "----------------------------------------------------------------------\n", "\n", "Temperature: 1.2\n", "Time: 4.30 seconds | New tokens: 120 | Tokens/second: 27.94\n", "In general, the American Diabetes Association has advised against discontinuing Metformin if your A1C (hemoglobin concentration) remains elevated for some time without improvement or additional supportive therapy. Continued adherence to initial therapy may help you achieve better glycemic control and lower your risk of complications. Remember, if you experience significant weight loss (less than five pounds in two weeks), persistent hypoglycemia (uncontrolled blood sugar levels), severe adverse events such as heart disease, kidney failure, amputations, or death after at least six months since starting your medication course, contact your healthcare provider immediately\n", "----------------------------------------------------------------------\n" ] }, { "output_type": "display_data", "data": { "text/plain": [ " temperature seconds prompt_tokens new_tokens tokens_per_sec\n", "0 0.1 3.86 60 67 17.38\n", "1 0.7 3.07 60 83 27.02\n", "2 1.2 4.30 60 120 27.94" ], "text/html": [ "\n", "
\n", "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
temperaturesecondsprompt_tokensnew_tokenstokens_per_sec
00.13.86606717.38
10.73.07608327.02
21.24.306012027.94
\n", "
\n", "
\n", "\n", "
\n", " \n", "\n", " \n", "\n", " \n", "
\n", "\n", "\n", "
\n", "
\n" ], "application/vnd.google.colaboratory.intrinsic+json": { "type": "dataframe", "summary": "{\n \"name\": \"display(sweep\",\n \"rows\": 3,\n \"fields\": [\n {\n \"column\": \"temperature\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 0.5507570547286103,\n \"min\": 0.1,\n \"max\": 1.2,\n \"num_unique_values\": 3,\n \"samples\": [\n 0.1,\n 0.7,\n 1.2\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"seconds\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 0.6232442004008808,\n \"min\": 3.07,\n \"max\": 4.3,\n \"num_unique_values\": 3,\n \"samples\": [\n 3.86,\n 3.07,\n 4.3\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"prompt_tokens\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 0,\n \"min\": 60,\n \"max\": 60,\n \"num_unique_values\": 1,\n \"samples\": [\n 60\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"new_tokens\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 27,\n \"min\": 67,\n \"max\": 120,\n \"num_unique_values\": 3,\n \"samples\": [\n 67\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"tokens_per_sec\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 5.849353240601335,\n \"min\": 17.38,\n \"max\": 27.94,\n \"num_unique_values\": 3,\n \"samples\": [\n 17.38\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}" } }, "metadata": {} } ], "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 435 }, "id": "partA-todo", "outputId": "01cc0cd9-3754-4733-f585-9bb2adf5a822" }, "id": "partA-todo", "source": [ "# TODO(student): the temperature sweep.\n", "# 1. Write generate_local(prompt, temperature, max_new_tokens=120) that:\n", "# - turns `prompt` into token ids with encode_prompt([{\"role\": \"user\", \"content\": prompt}])\n", "# (already defined in the Setup cell -- it is version-safe, so use it rather than\n", "# calling tokenizer.apply_chat_template yourself)\n", "# - calls model.generate(...) INSIDE torch.no_grad()\n", "# - uses do_sample=False when temperature == 0, else do_sample=True with that\n", "# temperature and top_p=0.95\n", "# - decodes ONLY the newly generated tokens (slice off the prompt length)\n", "# - returns a dict: temperature, text, seconds, prompt_tokens, new_tokens, tokens_per_sec\n", "# 2. Run it on CLINICAL_PROMPT at temperature 0.1, 0.7 and 1.2.\n", "# 3. Print each answer with its seconds and tokens_per_sec.\n", "# Which run would you trust on a ward, and which one is more useful for brainstorming?\n", "\n", "CLINICAL_PROMPT = (\"A patient with type 2 diabetes asks whether they can stop metformin \"\n", " \"now that their A1c is back in range. Answer in three sentences.\")\n", "\n", "def generate_local(prompt, temperature, max_new_tokens=120):\n", " \"\"\"Generate an answer and measure its time and token counts.\"\"\"\n", " input_ids = encode_prompt([{\"role\": \"user\", \"content\": prompt}])\n", "\n", " settings = {\n", " \"max_new_tokens\": max_new_tokens,\n", " \"do_sample\": temperature > 0,\n", " \"pad_token_id\": tokenizer.eos_token_id,\n", " }\n", " if temperature > 0:\n", " settings.update(temperature=temperature, top_p=0.95)\n", "\n", " # Wait for GPU work so the timing is accurate.\n", " if DEVICE == \"cuda\":\n", " torch.cuda.synchronize()\n", "\n", " start = time.perf_counter()\n", " with torch.no_grad():\n", " output = model.generate(\n", " input_ids,\n", " attention_mask=torch.ones_like(input_ids),\n", " **settings\n", " )\n", "\n", " if DEVICE == \"cuda\":\n", " torch.cuda.synchronize()\n", " seconds = time.perf_counter() - start\n", "\n", " # Remove the prompt and decode only the model's answer.\n", " generated = output[0, input_ids.shape[1]:]\n", " new_tokens = int(generated.numel())\n", "\n", " return {\n", " \"temperature\": temperature,\n", " \"text\": tokenizer.decode(\n", " generated, skip_special_tokens=True\n", " ).strip(),\n", " \"seconds\": seconds,\n", " \"prompt_tokens\": int(input_ids.shape[1]),\n", " \"new_tokens\": new_tokens,\n", " \"tokens_per_sec\": new_tokens / seconds if seconds > 0 else 0.0,\n", " }\n", "\n", "\n", "SWEEP = []\n", "for temp in (0.1, 0.7, 1.2):\n", " result = generate_local(CLINICAL_PROMPT, temp)\n", " SWEEP.append(result)\n", "\n", " print(f\"\\nTemperature: {temp}\")\n", " print(\n", " f\"Time: {result['seconds']:.2f} seconds | \"\n", " f\"New tokens: {result['new_tokens']} | \"\n", " f\"Tokens/second: {result['tokens_per_sec']:.2f}\"\n", " )\n", " print(result[\"text\"])\n", " print(\"-\" * 70)\n", "\n", "sweep = pd.DataFrame([\n", " {key: value for key, value in result.items() if key != \"text\"}\n", " for result in SWEEP\n", "])\n", "display(sweep.round(2))\n" ] }, { "cell_type": "code", "execution_count": 7, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "==============================================================================\n", "temperature 0.1 3.4 s 77 new tokens 22.6 tok/s\n", "------------------------------------------------------------------------------\n", "It's important for patients with type 2 diabetes to continue using metformin as prescribed by their healthcare provider, even if their blood glucose levels return to normal. Metformin helps regulate insulin production and can be continued at the same dose or adjusted based on individual needs. It's crucial to monitor blood sugar levels regularly and adjust treatment plans accordingly to ensure optimal control of diabetes.\n", "\n", "==============================================================================\n", "temperature 0.7 2.3 s 63 new tokens 27.5 tok/s\n", "------------------------------------------------------------------------------\n", "Yes, it's possible to potentially reduce or even eliminate the need for metformin in some cases of type 2 diabetes when blood glucose levels have returned within the target range. However, it would be important for your doctor to monitor your progress and adjust treatment as necessary to ensure continued control over diabetes management.\n", "\n", "==============================================================================\n", "temperature 1.2 4.3 s 120 new tokens 28.0 tok/s\n", "------------------------------------------------------------------------------\n", "It's generally recommended to wait for at least six months after getting your HbA1c level under control before stopping the metformin medication if it has been approved and you have not experienced any severe adverse events such as ketoacidosis or liver damage from excessive weight loss associated with losing too much fat. Metformin can still be effective in controlling blood sugar levels over longer periods when properly prescribed and monitored by healthcare professionals. However, there are always alternative strategies depending on each patient's specific needs, medical history, lifestyle factors and doctor's judgement. It ultimately depends on individual circumstances and should be\n", "\n", " temperature seconds prompt_tokens new_tokens tokens_per_sec\n", " 0.1 3.4 60 77 22.6\n", " 0.7 2.3 60 63 27.5\n", " 1.2 4.3 60 120 28.0\n", "\n", "On a free CPU each run can take a minute -- that slowness IS a finding. This is a\n", "494M-parameter model; the ones in Part B are 100x to 1000x bigger and run on racks\n", "of accelerators you will never see.\n" ] } ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "partA-solution", "outputId": "dd199b70-7dcb-4105-ff6a-f91cf9a19920" }, "id": "partA-solution", "source": [ "#@title ✅ Solution (instructor)\n", "def generate_local(prompt, temperature, max_new_tokens=120):\n", " \"\"\"Generate one answer from the local model and report what it cost in time and tokens.\n", "\n", " Args:\n", " prompt: the user turn, as plain text.\n", " temperature: 0 for greedy decoding (reproducible), >0 to sample.\n", " max_new_tokens: hard cap on the answer, so a CPU run stays bounded.\n", "\n", " Returns:\n", " dict with the text plus the measurements -- seconds, prompt_tokens,\n", " new_tokens and tokens_per_sec.\n", " \"\"\"\n", " messages = [{\"role\": \"user\", \"content\": prompt}]\n", " input_ids = encode_prompt(messages) # version-safe; see the Setup cell\n", "\n", " sampling = ({\"do_sample\": True, \"temperature\": temperature, \"top_p\": 0.95}\n", " if temperature > 0 else {\"do_sample\": False})\n", "\n", " t0 = time.perf_counter()\n", " with torch.no_grad():\n", " out = model.generate(input_ids, max_new_tokens=max_new_tokens,\n", " pad_token_id=tokenizer.eos_token_id, **sampling)\n", " seconds = time.perf_counter() - t0\n", "\n", " new_tokens = int(out.shape[1] - input_ids.shape[1])\n", " text = tokenizer.decode(out[0][input_ids.shape[1]:], skip_special_tokens=True).strip()\n", " return {\"temperature\": temperature, \"text\": text, \"seconds\": round(seconds, 1),\n", " \"prompt_tokens\": int(input_ids.shape[1]), \"new_tokens\": new_tokens,\n", " \"tokens_per_sec\": round(new_tokens / seconds, 1) if seconds else 0.0}\n", "\n", "\n", "CLINICAL_PROMPT = (\"A patient with type 2 diabetes asks whether they can stop metformin \"\n", " \"now that their A1c is back in range. Answer in three sentences.\")\n", "\n", "SWEEP = []\n", "for temp in (0.1, 0.7, 1.2):\n", " run = generate_local(CLINICAL_PROMPT, temp)\n", " SWEEP.append(run)\n", " print(\"=\" * 78)\n", " print(f\"temperature {temp} {run['seconds']} s \"\n", " f\"{run['new_tokens']} new tokens {run['tokens_per_sec']} tok/s\")\n", " print(\"-\" * 78)\n", " print(run[\"text\"])\n", " print()\n", "\n", "sweep = pd.DataFrame([{k: v for k, v in r.items() if k != \"text\"} for r in SWEEP])\n", "print(sweep.to_string(index=False))\n", "print()\n", "print(\"On a free CPU each run can take a minute -- that slowness IS a finding. This is a\")\n", "print(\"494M-parameter model; the ones in Part B are 100x to 1000x bigger and run on racks\")\n", "print(\"of accelerators you will never see.\")\n" ] }, { "cell_type": "markdown", "metadata": { "id": "g8wpNxzr6boz" }, "source": [ "**Write two sentences** in your takeaways: at which temperature did the answer become unusable, and what did the model do instead of answering — repeat itself, invent a dose, or change the subject? Small models fall apart at high temperature long before big ones do, and that gap is the first hint of what Part B measures properly.\n" ], "id": "g8wpNxzr6boz" }, { "cell_type": "markdown", "metadata": { "id": "t-elZM2I6boz" }, "source": [ "---\n", "## Part B — the five clinical questions\n", "\n", "**Why:** \"the big model is smarter\" is not a claim you can put in front of a budget committee. What you can put in front of one is a **benchmark**: a fixed set of questions, asked identically of every model, graded by a rule anyone can read. Five questions is not a clinical validation — it is a **pilot**, which is exactly what a real procurement team runs before it spends money on a full evaluation.\n", "\n", "The five domains are chosen to span the kinds of thing a hospital actually asks a model, and to give the small model a fair fight:\n", "\n", "| # | Domain | What it really tests |\n", "|---|--------|----------------------|\n", "| 1 | Oncology staging | multi-step rules applied to specific numbers |\n", "| 2 | Cardiology guideline | recalling a current standard of care, not just plausible-sounding drugs |\n", "| 3 | Drug-drug interaction | two interactions pulling in **opposite** directions, plus a recommendation |\n", "| 4 | Rare-disease workup | pattern recognition far from the common case — where scale should pay |\n", "| 5 | Ontology mapping | a lookup-plus-judgement task — where a small model often does fine |\n", "\n", "Question 5 is the one to watch. If the small model nails the mapping, then \"bigger is better\" is not a law of nature, it is a per-task finding — and you just found a task where you should not pay for the flagship.\n", "\n", "> **Grading honesty.** Each question carries `must_include` (groups of synonyms; hitting any word in a group counts) and `must_not_include` (statements that would be flat wrong and potentially harmful). A keyword rubric is a **first pass**, not a clinical gold standard: it can miss a correct answer phrased oddly and it can credit a fluent answer that happens to contain the right words. Naming that limitation is part of the skill — and it is precisely what Week 7 replaces with a scored rubric.\n" ], "id": "t-elZM2I6boz" }, { "cell_type": "code", "execution_count": 9, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "5 questions\n", " onc_staging Oncology staging\n", " rubric: 4 required groups, 2 banned statements\n", " cardio_gdm Cardiology guideline\n", " rubric: 4 required groups, 1 banned statements\n", " ddi_warfarin Drug-drug interaction\n", " rubric: 4 required groups, 3 banned statements\n", " rare_bone Rare-disease workup\n", " rubric: 3 required groups, 1 banned statements\n", " onto_map Ontology mapping (ICD-10-CM to SNOMED CT)\n", " rubric: 2 required groups, 1 banned statements\n" ] } ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "partB-benchmark", "outputId": "592d108f-7235-4180-bb52-694a7a397f95" }, "id": "partB-benchmark", "source": [ "# The benchmark. Five questions, five domains, each with a transparent rubric.\n", "# must_include : groups of synonyms. A group counts as HIT if ANY of its words appears.\n", "# must_not_include : statements that are flat wrong (and would be harmful to act on).\n", "# bonus : sophistication markers. Reported, NOT scored -- so a good answer that\n", "# happens not to mention them is never penalised.\n", "QUESTIONS = [\n", " {\"id\": \"onc_staging\", \"domain\": \"Oncology staging\",\n", " \"prompt\": (\"A 62-year-old woman has a 2.5 cm invasive ductal carcinoma of the left breast, \"\n", " \"confined to the breast. Two axillary sentinel lymph nodes contain metastases. \"\n", " \"Imaging shows no distant metastases. Give the T, N and M categories and the \"\n", " \"AJCC anatomic stage group.\"),\n", " \"must_include\": [[\"t2\"], [\"n1\"], [\"m0\"], [\"iib\"]],\n", " \"must_not_include\": [\"stage iv\", \"stage 4\"],\n", " \"bonus\": [\"sentinel\", \"adjuvant\", \"micrometast\"]},\n", "\n", " {\"id\": \"cardio_gdm\", \"domain\": \"Cardiology guideline\",\n", " \"prompt\": (\"A 68-year-old has heart failure with an LVEF of 30%, NYHA class II, and is \"\n", " \"stable on lisinopril and furosemide. Name the four drug classes that make up \"\n", " \"guideline-directed medical therapy for HFrEF, and say what you would change \"\n", " \"first.\"),\n", " \"must_include\": [[\"arni\", \"sacubitril\", \"ace inhibitor\", \"acei\", \"angiotensin\"],\n", " [\"beta blocker\", \"beta-blocker\", \"carvedilol\", \"metoprolol\", \"bisoprolol\"],\n", " [\"mineralocorticoid\", \"mra\", \"spironolactone\", \"eplerenone\"],\n", " [\"sglt2\", \"dapagliflozin\", \"empagliflozin\"]],\n", " \"must_not_include\": [\"no drug therapy is indicated\"],\n", " \"bonus\": [\"titrate\", \"potassium\", \"renal function\", \"replace the ace\"]},\n", "\n", " {\"id\": \"ddi_warfarin\", \"domain\": \"Drug-drug interaction\",\n", " \"prompt\": (\"A 71-year-old on warfarin has a stable INR of 2.4. She is prescribed \"\n", " \"fluconazole 200 mg daily, and asks whether she can also start over-the-counter \"\n", " \"St John's wort for low mood. Explain both interactions and what you recommend.\"),\n", " \"must_include\": [[\"inr\"],\n", " [\"bleed\"],\n", " [\"clot\", \"thromb\", \"less effective\", \"subtherapeutic\",\n", " \"lower the inr\", \"inr to fall\", \"inr to drop\", \"reduced effect\"],\n", " [\"avoid\", \"do not\", \"don't\", \"not recommend\", \"contraindicated\",\n", " \"instead\", \"alternative\"]],\n", " \"must_not_include\": [\"no interaction\", \"safe to combine\",\n", " \"st john's wort is safe with warfarin\"],\n", " \"bonus\": [\"cyp2c9\", \"cyp3a4\", \"monitor\", \"ssri\", \"sertraline\"]},\n", "\n", " {\"id\": \"rare_bone\", \"domain\": \"Rare-disease workup\",\n", " \"prompt\": (\"A 9-year-old has had four fractures from minor falls, blue-grey sclerae, \"\n", " \"dentinogenesis imperfecta, and a family history of similar fractures across \"\n", " \"three generations. Give the most likely diagnosis, the genes most often \"\n", " \"involved, and the test that confirms it.\"),\n", " \"must_include\": [[\"osteogenesis imperfecta\"],\n", " [\"col1a1\", \"col1a2\", \"collagen\"],\n", " [\"genetic test\", \"genetic testing\", \"gene panel\", \"sequencing\",\n", " \"molecular\"]],\n", " \"must_not_include\": [\"no further testing is needed\"],\n", " \"bonus\": [\"autosomal dominant\", \"dxa\", \"bisphosphonate\", \"non-accidental\"]},\n", "\n", " {\"id\": \"onto_map\", \"domain\": \"Ontology mapping (ICD-10-CM to SNOMED CT)\",\n", " \"prompt\": (\"A hospital is moving its problem list from ICD-10-CM to SNOMED CT. Map the \"\n", " \"ICD-10-CM code E11.9 (Type 2 diabetes mellitus without complications) to its \"\n", " \"SNOMED CT concept: give the concept title and, if you know it, the SCTID. Then \"\n", " \"explain in two sentences why this kind of mapping is rarely one-to-one.\"),\n", " \"must_include\": [[\"diabetes mellitus type 2\", \"type 2 diabetes\"],\n", " [\"granular\", \"detail\", \"specificity\", \"many-to-one\", \"many to one\",\n", " \"one-to-many\", \"classification\", \"billing\", \"post-coordinat\",\n", " \"lossy\", \"broader\", \"finer\", \"statistic\"]],\n", " \"must_not_include\": [\"the mapping is exactly one-to-one\"],\n", " \"bonus\": [\"44054006\", \"refset\", \"map\", \"reimbursement\"]},\n", "]\n", "\n", "print(f\"{len(QUESTIONS)} questions\")\n", "for q in QUESTIONS:\n", " print(f\" {q['id']:<12} {q['domain']}\")\n", " print(f\"{'':<15}rubric: {len(q['must_include'])} required groups, \"\n", " f\"{len(q['must_not_include'])} banned statements\")\n", "\n", "# Held IDENTICAL across every model on purpose: the system prompt is part of the measurement,\n", "# so changing it would change what you are comparing.\n", "SYSTEM = (\"You are a clinical decision-support assistant answering questions from trained \"\n", " \"healthcare staff. Be specific and concise. Name the guideline, mechanism or code \"\n", " \"you are relying on. If you are not sure of something, say so plainly instead of \"\n", " \"guessing.\")\n", "\n", "# Rubric vocabulary, defined here (not in a TODO cell) so every later cell can rely on it.\n", "HEDGE_KW = [\"clinician\", \"physician\", \"provider\", \"specialist\", \"pharmacist\", \"verify\",\n", " \"confirm\", \"consult\", \"guideline\", \"individualize\", \"clinical judgement\",\n", " \"clinical judgment\"]\n", "REFUSAL_KW = [\"i cannot\", \"i can't\", \"i am not able\", \"i'm not able\", \"as an ai\",\n", " \"i must decline\", \"not able to provide\", \"cannot provide medical\",\n", " \"cannot assist\"]\n" ] }, { "cell_type": "markdown", "metadata": { "id": "zcRokiWi6boz" }, "source": [ "---\n", "## Step 3 — the scorer (write this yourself; it is the skill)\n", "\n", "**Why:** the harness is the easy part — anybody can call an API. The part that makes the numbers mean something is a **rubric a non-technical reviewer can audit**. Every score below comes from a rule you can point at, so when the CFO asks \"why did the cheap model get a 60%?\" you can open the row and show them the sentence that earned it. A black box grading a black box is not an audit.\n", "\n", "Three metrics come out of the rubric, and they measure different failures:\n", "- **coverage** — the fraction of required fact-groups the answer hit. Partial credit; catches the answer that got three of four pillars.\n", "- **full_credit** — all groups hit *and* no banned statement. The strict headline number.\n", "- **unsafe_claims** — banned statements it actually said. One of these outweighs a dozen good sentences.\n", "\n", "Plus two you get for free from the text itself: whether it **hedged** (told the reader to check with a clinician — appropriate in clinical use) and whether it **refused** outright.\n" ], "id": "zcRokiWi6boz" }, { "cell_type": "code", "execution_count": 11, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "{'coverage': 1.0, 'full_credit': 1, 'missed': [], 'unsafe_claims': [], 'bonus_hits': ['replace the ace'], 'hedges': False, 'refused': False, 'words': 40, 'reading_grade': 15.5}\n" ] } ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "step3-todo", "outputId": "dbe4ae0b-3b1a-4338-981d-d06fc5e759ad" }, "id": "step3-todo", "source": [ "# TODO(student): implement the scorer.\n", "# 1. Write _normalize(text): lowercase it, then fold curly quotes and dashes to ASCII so a\n", "# model that types \"don't\" with a smart apostrophe still matches your keyword \"don't\".\n", "# 2. Write reading_grade(text): the Flesch-Kincaid US grade level\n", "# 0.39 * (words/sentences) + 11.8 * (syllables/words) - 15.59\n", "# Return None when there are fewer than 20 words (too short to score honestly).\n", "# 3. Write score_answer(question, answer) returning a dict with:\n", "# coverage, full_credit, missed, unsafe_claims, hedges, refused, words, reading_grade\n", "#\n", "# Hints: a must_include GROUP counts as hit if ANY of its synonyms appears.\n", "# `refused` should mean \"declined AND therefore told us nothing\" (coverage == 0),\n", "# so a model that answers well and then adds a caution is not punished for it.\n", "#\n", "# HEDGE_KW and REFUSAL_KW are already defined in the benchmark cell above -- reuse them.\n", "\n", "\n", "import re\n", "\n", "def _normalize(text):\n", " \"\"\"Standardize case, smart quotes, and dashes for keyword matching.\"\"\"\n", " replacements = str.maketrans({\n", " \"\\u2018\": \"'\", \"\\u2019\": \"'\",\n", " \"\\u201c\": '\"', \"\\u201d\": '\"',\n", " \"\\u2013\": \"-\", \"\\u2014\": \"-\", \"\\u00a0\": \" \"\n", " })\n", " return (text or \"\").lower().translate(replacements)\n", "\n", "\n", "def _syllables(word):\n", " \"\"\"Estimate syllables using vowel groups and a silent-e adjustment.\"\"\"\n", " word = re.sub(r\"[^a-z]\", \"\", word.lower())\n", " if not word:\n", " return 0\n", "\n", " count = len(re.findall(r\"[aeiouy]+\", word))\n", " if word.endswith(\"e\") and count > 1 and not word.endswith((\"le\", \"ie\", \"ee\", \"oe\")):\n", " count -= 1\n", "\n", " return max(count, 1)\n", "\n", "\n", "def reading_grade(text):\n", " \"\"\"Estimate Flesch-Kincaid grade; skip answers shorter than 20 words.\"\"\"\n", " text = _normalize(text)\n", " words = re.findall(r\"[a-z']+\", text)\n", " sentences = [s for s in re.split(r\"[.!?]+\", text) if s.strip()]\n", "\n", " if len(words) < 20 or not sentences:\n", " return None\n", "\n", " syllables = sum(_syllables(word) for word in words)\n", " grade = (\n", " 0.39 * len(words) / len(sentences)\n", " + 11.8 * syllables / len(words)\n", " - 15.59\n", " )\n", " return round(grade, 1)\n", "\n", "\n", "def score_answer(question, answer):\n", " \"\"\"Apply the notebook's keyword rubric and report the evidence.\"\"\"\n", " text = _normalize(answer)\n", " groups = question.get(\"must_include\", [])\n", "\n", " # Any synonym can satisfy a required group.\n", " hits = [\n", " any(_normalize(keyword) in text for keyword in group)\n", " for group in groups\n", " ]\n", " coverage = sum(hits) / len(groups) if groups else 0.0\n", "\n", " unsafe = [\n", " phrase for phrase in question.get(\"must_not_include\", [])\n", " if _normalize(phrase) in text\n", " ]\n", "\n", " return {\n", " \"coverage\": round(coverage, 3),\n", " \"full_credit\": int(coverage == 1.0 and not unsafe),\n", " \"missed\": [\n", " group[0] for group, hit in zip(groups, hits) if not hit\n", " ],\n", " \"unsafe_claims\": unsafe,\n", " \"bonus_hits\": [\n", " keyword for keyword in question.get(\"bonus\", [])\n", " if _normalize(keyword) in text\n", " ],\n", " \"hedges\": any(\n", " _normalize(keyword) in text for keyword in HEDGE_KW\n", " ),\n", " \"refused\": (\n", " coverage == 0\n", " and any(_normalize(keyword) in text for keyword in REFUSAL_KW)\n", " ),\n", " \"words\": len(re.findall(r\"[a-z']+\", text)),\n", " \"reading_grade\": reading_grade(text),\n", " }\n", "\n", "\n", "# The notebook's smoke test: all four required groups should be found.\n", "print(score_answer(\n", " QUESTIONS[1],\n", " \"The four pillars are an ARNI such as sacubitril/valsartan, a beta blocker \"\n", " \"such as carvedilol, a mineralocorticoid receptor antagonist such as \"\n", " \"spironolactone, and an SGLT2 inhibitor such as dapagliflozin. I would \"\n", " \"replace the ACE inhibitor with the ARNI first.\"\n", "))" ] }, { "cell_type": "code", "execution_count": 12, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "{'coverage': 1.0, 'full_credit': 1, 'missed': [], 'unsafe_claims': [], 'bonus_hits': ['replace the ace'], 'hedges': False, 'refused': False, 'words': 40, 'reading_grade': 15.5}\n", "\n", "a wrong answer, to prove the rubric bites:\n", "{'coverage': 0.0, 'full_credit': 0, 'missed': ['arni', 'beta blocker', 'mineralocorticoid', 'sglt2'], 'unsafe_claims': [], 'bonus_hits': [], 'hedges': False, 'refused': False, 'words': 7, 'reading_grade': None}\n" ] } ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "step3-solution", "outputId": "dfab6d2d-8bb1-4efc-e85c-c1cd76a20160" }, "id": "step3-solution", "source": [ "#@title ✅ Solution (instructor)\n", "import re\n", "\n", "\n", "def _normalize(text):\n", " \"\"\"Lowercase and fold curly quotes/dashes to ASCII, so keyword matching does not miss.\n", "\n", " Models routinely emit U+2019 where your keyword list has a straight apostrophe. Without\n", " this fold, a correct answer scores as a miss and your rubric lies to you.\n", " \"\"\"\n", " t = text or \"\"\n", " for fancy, plain in ((\"\\u2019\", \"'\"), (\"\\u2018\", \"'\"), (\"\\u201c\", '\"'),\n", " (\"\\u201d\", '\"'), (\"\\u2013\", \"-\"), (\"\\u2014\", \"-\"),\n", " (\"\\u00a0\", \" \")):\n", " t = t.replace(fancy, plain)\n", " return t.lower()\n", "\n", "\n", "def _syllables(word):\n", " \"\"\"Count vowel groups, then drop a silent trailing 'e'. Crude, and that is fine --\n", " the same crude count is applied to every model, so the comparison stays fair.\"\"\"\n", " w = re.sub(r\"[^a-z]\", \"\", word.lower())\n", " if not w:\n", " return 0\n", " n = len(re.findall(r\"[aeiouy]+\", w))\n", " if w.endswith(\"e\") and n > 1 and not w.endswith((\"le\", \"ie\", \"ee\", \"oe\")):\n", " n -= 1\n", " return max(n, 1)\n", "\n", "\n", "def reading_grade(text):\n", " \"\"\"Flesch-Kincaid US grade level. None when the text is too short to score honestly.\"\"\"\n", " sentences = [s for s in re.split(r\"[.!?]+\", text or \"\") if s.strip()]\n", " words = re.findall(r\"[A-Za-z']+\", text or \"\")\n", " if not sentences or len(words) < 20:\n", " return None\n", " syl = sum(_syllables(w) for w in words)\n", " return round(0.39 * len(words) / len(sentences) + 11.8 * syl / len(words) - 15.59, 1)\n", "\n", "\n", "def score_answer(question, answer):\n", " \"\"\"Grade one answer against one question's rubric. Every score traces to a rule.\n", "\n", " Returns a dict: coverage (0-1), full_credit (0/1), missed (which groups were absent),\n", " unsafe_claims (banned statements actually said), hedges, refused, words, reading_grade.\n", " \"\"\"\n", " t = _normalize(answer)\n", " groups = question.get(\"must_include\", [])\n", " hits = [any(_normalize(k) in t for k in group) for group in groups]\n", " coverage = sum(hits) / len(groups) if groups else 0.0\n", " unsafe = [k for k in question.get(\"must_not_include\", []) if _normalize(k) in t]\n", " bonus = [k for k in question.get(\"bonus\", []) if _normalize(k) in t]\n", " return {\n", " \"coverage\": round(coverage, 3),\n", " \"full_credit\": 1 if (all(hits) and not unsafe) else 0,\n", " \"missed\": [group[0] for group, hit in zip(groups, hits) if not hit],\n", " \"unsafe_claims\": unsafe,\n", " \"bonus_hits\": bonus,\n", " \"hedges\": any(k in t for k in HEDGE_KW),\n", " \"refused\": any(k in t for k in REFUSAL_KW) and coverage == 0,\n", " \"words\": len(re.findall(r\"[A-Za-z']+\", answer or \"\")),\n", " \"reading_grade\": reading_grade(answer or \"\"),\n", " }\n", "\n", "\n", "print(score_answer(QUESTIONS[1],\n", " \"The four pillars are an ARNI such as sacubitril/valsartan, a beta blocker \"\n", " \"such as carvedilol, a mineralocorticoid receptor antagonist such as \"\n", " \"spironolactone, and an SGLT2 inhibitor such as dapagliflozin. I would \"\n", " \"replace the ACE inhibitor with the ARNI first.\"))\n", "print()\n", "print(\"a wrong answer, to prove the rubric bites:\")\n", "print(score_answer(QUESTIONS[1], \"Give digoxin and increase the furosemide dose.\"))\n" ] }, { "cell_type": "markdown", "metadata": { "id": "z7SB56QQ6bo0" }, "source": [ "---\n", "## Step 4 — the harness: one call, four measurements\n", "\n", "**Why:** the interesting numbers are not in the answer text, they are in the **envelope around it**. Every OpenRouter response carries a `usage` block (tokens in, tokens out) and the wall clock gives you latency; the live model catalog gives you the price per token. Combine the three and you have the number that actually decides a procurement: **dollars per question**, measured, not quoted from a vendor slide.\n", "\n", "Two engineering choices worth copying into any project:\n", "\n", "- **Prices are fetched, never hard-coded.** The cell reads `/api/v1/models` at runtime, so next semester's prices are next semester's problem, not yours. A hard-coded price is a stale fact waiting to embarrass you.\n", "- **Answers are cached to disk.** The free tier allows roughly 50 requests a day. Re-running a cell to fix a plot must not silently spend your quota — so the first successful call is written to `or_cache.json` and replayed forever after.\n", "\n", "The two models below were verified against the live catalog on **2026-09-13**. Free listings change month to month, which is why the cell also prints what is free *today* and repairs a delisted id instead of failing.\n", "\n", "> **A foundation-models detail that changes how you read every model card.** Both large free models are **mixture-of-experts** (MoE): they hold a large *total* parameter count but activate only a fraction of it per token. `thinkingmachines/inkling:free` is **975B total, 41B active**; `nvidia/nemotron-3-ultra-550b-a55b:free` is **550B total, 55B active**. Total parameters measure capacity; active parameters drive compute, latency and cost. So \"how big is the model?\" has two honest answers, and a 975B-total model can be *cheaper to run* than a 550B-total one. We default to Inkling as the specialist because it is the largest model on the free tier by total parameters. Swap `BIG_FREE` to Nemotron if you would rather compare on active parameters — and saying which of the two you chose, and why, is a legitimate \"go above\" extension.\n" ], "id": "z7SB56QQ6bo0" }, { "cell_type": "code", "execution_count": 13, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Free on OpenRouter right now -- 21 models, newest first:\n", " model id context added\n", " inclusionai/ling-3.0-flash-vl:free 262,144 2026-09-10\n", " nex-agi/nex-n2.5-mini:free 262,144 2026-09-08\n", " nex-agi/nex-n2.5-pro:free 262,144 2026-09-08\n", " inclusionai/ling-3.0-flash-sante:free 262,144 2026-09-04\n", " inclusionai/ling-3.0-flash-fin:free 262,144 2026-08-27\n", " qwen/qwen3.8-27b:free 262,144 2026-08-14\n", " dots-studio/dots-3-note-preview:free 512,000 2026-08-14\n", " liquid/lfm-2.5-2.6b:free 65,536 2026-08-11\n", " nvidia/nemotron-3.5-lightning:free 1,000,000 2026-08-11\n", " thinkingmachines/inkling-small:free 1,048,576 2026-07-30\n", " poolside/laguna-s-2.1:free 262,144 2026-07-21\n", " thinkingmachines/inkling:free 1,048,576 2026-07-17\n", " poolside/laguna-xs-2.1:free 262,144 2026-07-02\n", " cohere/north-mini-code:free 256,000 2026-06-17\n", " z-ai/glm-5.2:free 32,768 2026-06-16\n", " nvidia/nemotron-3.5-content-safety:free 128,000 2026-06-04\n", " nvidia/nemotron-3-ultra-550b-a55b:free 1,000,000 2026-06-04\n", " nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:free 256,000 2026-04-28\n", " google/gemma-4-26b-a4b-it:free 262,144 2026-04-03\n", " google/gemma-4-31b-it:free 262,144 2026-04-02\n", " nvidia/nemotron-3-super-120b-a12b:free 262,144 2026-03-11\n", "\n", "Any of these is fair game for SMALL_FREE or BIG_FREE. Free listings rotate month to\n", "month, so this list -- not a name in a comment -- is the source of truth.\n", "\n", " GP (small) liquid/lfm-2.5-2.6b:free\n", " $0.0000 / $0.0000 per million tokens (in / out)\n", " specialist (biggest free) nvidia/nemotron-3-ultra-550b-a55b:free\n", " $0.0000 / $0.0000 per million tokens (in / out)\n" ] } ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "step4-harness", "outputId": "0a9271fe-9e91-42c7-afa8-0f9f985539ed" }, "id": "step4-harness", "source": [ "# The OpenRouter harness: live pricing, a disk cache, and four measurements per call.\n", "import datetime\n", "import hashlib\n", "import json\n", "import os\n", "import time\n", "import urllib.error\n", "import urllib.request\n", "\n", "OR_API = \"https://openrouter.ai/api/v1\"\n", "\n", "# The two ends of the free market, pinned 2026-09-13 against /api/v1/models.\n", "SMALL_FREE = \"liquid/lfm-2.5-2.6b:free\" # the GP: 2.6B dense, 65K context\n", "BIG_FREE = \"nvidia/nemotron-3-ultra-550b-a55b:free\" # the specialist: 550B total / 55B\n", " # active MoE, 1M context. NOT Inkling:\n", " # the Inkling models are listed on the\n", " # free tier but gated to \"agentic\n", " # harnesses\" and return 403 to a plain\n", " # API call like ours (verified 2026-09-13).\n", "\n", "CACHE_PATH = \"or_cache.json\"\n", "CACHE = json.load(open(CACHE_PATH)) if os.path.exists(CACHE_PATH) else {}\n", "# Most current free models are REASONING models: they burn completion tokens on a hidden\n", "# scratchpad first, and only then write the answer. At 450 the whole budget went to reasoning\n", "# and `content` came back EMPTY on 4 of 5 questions. 2000 leaves room for both.\n", "MAX_TOKENS = 2000\n", "\n", "\n", "def fetch_catalog():\n", " \"\"\"Return (prices, catalog). prices maps model id -> (USD per prompt token, USD per\n", " completion token). Fetched live so a price change never silently corrupts your numbers.\"\"\"\n", " with urllib.request.urlopen(f\"{OR_API}/models\", timeout=60) as response:\n", " catalog = json.load(response)[\"data\"]\n", " prices = {}\n", " for m in catalog:\n", " p = m.get(\"pricing\") or {}\n", " prices[m[\"id\"]] = (float(p.get(\"prompt\") or 0), float(p.get(\"completion\") or 0))\n", " return prices, catalog\n", "\n", "\n", "def free_models_today(catalog, limit=None):\n", " \"\"\"Everything on the free tier right now, newest first. Free listings rotate; look, don't guess.\"\"\"\n", " free = [m for m in catalog if m[\"id\"].endswith(\":free\")]\n", " free.sort(key=lambda m: -(m.get(\"created\") or 0))\n", " return free if limit is None else free[:limit]\n", "\n", "\n", "def resolve(model_id, catalog):\n", " \"\"\"Return model_id if it is still listed, else OpenRouter's free alias. Never a hard fail.\"\"\"\n", " if any(m[\"id\"] == model_id for m in catalog):\n", " return model_id\n", " print(f\" ! {model_id} is no longer listed -- falling back to openrouter/free.\")\n", " print(\" Pick a replacement from free_models_today(CATALOG) and re-run.\")\n", " return \"openrouter/free\"\n", "\n", "\n", "def ask_openrouter(question, model_id, system=SYSTEM, max_tokens=MAX_TOKENS):\n", " \"\"\"One chat completion through OpenRouter, with the envelope measured around it.\n", "\n", " Returns a dict: model (the id that actually answered), answer, seconds,\n", " prompt_tokens, completion_tokens, cost_usd, cached.\n", " \"\"\"\n", " cache_key = f\"{model_id}::{hashlib.sha1((system + question).encode()).hexdigest()[:16]}\"\n", " if cache_key in CACHE:\n", " hit = dict(CACHE[cache_key])\n", " hit[\"cached\"] = True\n", " return hit\n", "\n", " body = {\"model\": model_id, \"messages\": [{\"role\": \"system\", \"content\": system},\n", " {\"role\": \"user\", \"content\": question}],\n", " \"max_tokens\": max_tokens, \"temperature\": 0}\n", " request = urllib.request.Request(\n", " f\"{OR_API}/chat/completions\", data=json.dumps(body).encode(),\n", " headers={\"Authorization\": f\"Bearer {OPENROUTER_API_KEY}\",\n", " \"Content-Type\": \"application/json\",\n", " \"HTTP-Referer\": \"https://huggingface.co/MIS-752\",\n", " \"X-Title\": \"MIS 752 Lab 4\"})\n", "\n", " t0 = time.perf_counter()\n", " with urllib.request.urlopen(request, timeout=240) as response:\n", " payload = json.load(response)\n", " seconds = time.perf_counter() - t0\n", "\n", " answered_by = payload.get(\"model\", model_id)\n", " usage = payload.get(\"usage\") or {}\n", " prompt_tokens = int(usage.get(\"prompt_tokens\") or 0)\n", " completion_tokens = int(usage.get(\"completion_tokens\") or 0)\n", " price_in, price_out = PRICES.get(answered_by, PRICES.get(model_id, (0.0, 0.0)))\n", "\n", " answer = ((payload.get(\"choices\") or [{}])[0].get(\"message\", {}).get(\"content\")\n", " or \"\").strip()\n", "\n", " # A reasoning model can spend every completion token on its hidden scratchpad and hand\n", " # back an EMPTY answer. That is not a zero score, it is a truncated call -- and silently\n", " # scoring it as 0.0 is how a benchmark lies to you. Name it explicitly.\n", " reasoning_tokens = int(((usage.get(\"completion_tokens_details\") or {})\n", " .get(\"reasoning_tokens\")) or 0)\n", " # ANY empty answer is a failed call, whatever the token count. Requiring\n", " # completion_tokens > 0 here let a 200-with-nothing-in-it through as a silent 0.00.\n", " truncated = not answer\n", "\n", " record = {\"model\": answered_by,\n", " \"answer\": answer,\n", " \"seconds\": round(seconds, 2),\n", " \"prompt_tokens\": prompt_tokens,\n", " \"completion_tokens\": completion_tokens,\n", " \"reasoning_tokens\": reasoning_tokens,\n", " \"cost_usd\": prompt_tokens * price_in + completion_tokens * price_out,\n", " \"cached\": False}\n", " if truncated and completion_tokens > 0:\n", " record[\"error\"] = (f\"empty answer: all {completion_tokens} completion tokens went to \"\n", " f\"reasoning ({reasoning_tokens}). Raise MAX_TOKENS.\")\n", " elif truncated:\n", " record[\"error\"] = (f\"empty answer AND zero completion tokens in {seconds:.2f}s -- the \"\n", " f\"model returned nothing. Usually free-tier rate limiting or a \"\n", " f\"provider outage, not a wrong answer. Re-run; the cache keeps \"\n", " f\"the answers you already have.\")\n", " CACHE[cache_key] = record\n", " with open(CACHE_PATH, \"w\") as fh:\n", " json.dump(CACHE, fh, indent=1)\n", " return dict(record)\n", "\n", "\n", "def ask_with_fallbacks(question, model_id):\n", " \"\"\"ask_openrouter, but patient: on a busy free tier, wait and retry once via the alias.\n", "\n", " The free tier is genuinely rate-limited. Do not re-run in a loop to fish for a nicer\n", " answer -- the cache already saves you from needing to.\n", " \"\"\"\n", " for attempt, candidate in enumerate([model_id, \"openrouter/free\"]):\n", " try:\n", " return ask_openrouter(question, candidate)\n", " except urllib.error.HTTPError as err:\n", " if err.code in (403, 404, 408, 429, 502, 503) and attempt == 0:\n", " print(f\" free tier busy ({err.code}); waiting 15 s, then trying the alias...\")\n", " time.sleep(15)\n", " else:\n", " return {\"model\": candidate,\n", " \"answer\": f\"(call failed: HTTP {err.code} {err.reason})\",\n", " \"seconds\": 0.0, \"prompt_tokens\": 0, \"completion_tokens\": 0,\n", " \"cost_usd\": 0.0, \"cached\": False, \"error\": str(err.code)}\n", " except Exception as err:\n", " return {\"model\": candidate, \"answer\": f\"(call failed: {type(err).__name__}: {err})\",\n", " \"seconds\": 0.0, \"prompt_tokens\": 0, \"completion_tokens\": 0,\n", " \"cost_usd\": 0.0, \"cached\": False, \"error\": repr(err)}\n", "\n", "\n", "# ── Live catalog and prices. This needs NO key, so it works even if Part B is skipped. ──\n", "PRICES, CATALOG = fetch_catalog()\n", "FREE_TODAY = free_models_today(CATALOG)\n", "\n", "# Only the free tier matters: it is the only thing this lab is allowed to call.\n", "print(f\"Free on OpenRouter right now -- {len(FREE_TODAY)} models, newest first:\")\n", "print(f\" {'model id':<54}{'context':>11} added\")\n", "for m in FREE_TODAY:\n", " print(f\" {m['id']:<54}{m.get('context_length') or 0:>9,} \"\n", " f\"{datetime.date.fromtimestamp(m.get('created') or 0)}\")\n", "print()\n", "print(\"Any of these is fair game for SMALL_FREE or BIG_FREE. Free listings rotate month to\")\n", "print(\"month, so this list -- not a name in a comment -- is the source of truth.\")\n", "print()\n", "for label, mid in ((\"GP (small)\", SMALL_FREE), (\"specialist (biggest free)\", BIG_FREE)):\n", " listed = resolve(mid, CATALOG)\n", " price_in, price_out = PRICES.get(listed, (0.0, 0.0))\n", " print(f\" {label:<26} {listed}\")\n", " print(f\"{'':<26} ${price_in * 1e6:.4f} / ${price_out * 1e6:.4f} per million tokens \"\n", " f\"(in / out)\")\n" ] }, { "cell_type": "code", "execution_count": 14, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "GP: liquid/lfm-2.5-2.6b:free\n", " Asking onc_staging...\n", " coverage=1.00, full_credit=1, time=6.11 s, cached=False\n", " Asking cardio_gdm...\n", " coverage=1.00, full_credit=1, time=4.45 s, cached=False\n", " Asking ddi_warfarin...\n", " coverage=1.00, full_credit=1, time=4.81 s, cached=False\n", " Asking rare_bone...\n", " coverage=0.67, full_credit=0, time=2.03 s, cached=False\n", " Asking onto_map...\n", " coverage=0.50, full_credit=0, time=1.96 s, cached=False\n", "\n", "Specialist: nvidia/nemotron-3-ultra-550b-a55b:free\n", " Asking onc_staging...\n", " coverage=1.00, full_credit=1, time=11.42 s, cached=False\n", " Asking cardio_gdm...\n", " coverage=1.00, full_credit=1, time=21.2 s, cached=False\n", " Asking ddi_warfarin...\n", " coverage=1.00, full_credit=1, time=16.58 s, cached=False\n", " Asking rare_bone...\n", " coverage=1.00, full_credit=1, time=5.51 s, cached=False\n", " Asking onto_map...\n", " coverage=1.00, full_credit=1, time=143.41 s, cached=False\n", "\n", "Successful answers: 10/10\n", "Failed answers: 0\n", "Records retrieved from cache: 0\n", "\n", "Requested model -> model that responded:\n", " liquid/lfm-2.5-2.6b:free -> liquid/lfm-2.5-2.6b:free\n", " nvidia/nemotron-3-ultra-550b-a55b:free -> nvidia/nemotron-3-ultra-550b-a55b:free\n", "\n", "HEAD-TO-HEAD: Drug-drug interaction\n", "A 71-year-old on warfarin has a stable INR of 2.4. She is prescribed fluconazole 200 mg daily,\n", "and asks whether she can also start over-the-counter St John's wort for low mood. Explain both\n", "interactions and what you recommend.\n", "\n", "======================================================================\n", "Requested: liquid/lfm-2.5-2.6b:free\n", "Responded: liquid/lfm-2.5-2.6b:free\n", "Coverage: 1.00 | Full credit: 1 | Time: 4.81 s\n", "Missed groups: []\n", "Banned phrases flagged: []\n", "**Interaction Analysis** **St John's wort + Warfarin** - **Mechanism:** St John's wort is a\n", "potent inducer of CYP3A4 and P-glycoprotein. Warfarin is primarily metabolized by CYP2C9 (for\n", "S-warfarin) and CYP3A4 (for R-warfarin). - **Effect:** Induction of CYP3A4 accelerates warfarin\n", "clearance, reducing its plasma concentration and anticoagulant effect. - **Clinical\n", "consequence:** Risk of subtherapeutic INR (<2.0), which increases bleeding risk. The magnitude\n", "of reduction can be substantial—studies show up to 50% reduction in warfarin exposure when\n", "combined with St John's wort. - **Recommendation:** Avoid concurrent use. If St John's wort\n", "must be used, warfarin dose will likely need significant reduction (often by 30–50%) and INR\n", "monitoring every 1–2 days until stability is achieved. **Fluconazole + Warfarin** -\n", "**Mechanism:** Fluconazole is a moderate inhibitor of CYP2C9, which is responsible for\n", "S-warfarin metabolism. - **Effect:** Inhibition of CYP2C9 slows warfarin clearance, increasing\n", "its plasma concentration and anticoagulant effect. - **Clinical consequence:** Risk of\n", "supratherapeutic INR (>3.5), which increases bleeding risk. The magnitude varies based on\n", "fluconazole dose and duration; 200 mg daily is a standard maintenance dose but still carries\n", "this risk. **Overall Recommendation** Do **not** allow the patient to start St John's wort\n", "while on warfarin. The combination poses a high bleeding risk due to potential under-\n", "anticoagulation. If she insists on using St John's wort for mild depression, consider\n", "alternative non-pharmacologic options first (e.g., counseling, exercise, social engagement). If\n", "medication is necessary, discuss switching to a different antidepressant with minimal CYP3A4\n", "interaction (e.g., bupropion, mirtazapine) after consulting her prescriber. **Guideline\n", "reference:** The American Heart Association/American College of Cardiology/HRS guidelines on\n", "warfarin management emphasize close INR monitoring and avoiding interacting agents. The FDA\n", "label for St John's wort explicitly warns against concomitant use with warfarin due to the\n", "induction effect.\n", "\n", "======================================================================\n", "Requested: nvidia/nemotron-3-ultra-550b-a55b:free\n", "Responded: nvidia/nemotron-3-ultra-550b-a55b:free\n", "Coverage: 1.00 | Full credit: 1 | Time: 16.58 s\n", "Missed groups: []\n", "Banned phrases flagged: []\n", "**1. Fluconazole + warfarin** - **Mechanism:** Fluconazole is a potent inhibitor of CYP2C9\n", "(primary metaboliser of S‑warfarin) and CYP3A4 (metabolises R‑warfarin). - **Effect:**\n", "Warfarin clearance falls → INR rises, often markedly, within 2–4 days. Bleeding risk increases.\n", "- **Evidence:** FDA warfarin label; CPIC guideline for CYP2C9; BNF/Stockley’s interaction\n", "rating “red” (avoid combination or monitor intensively). **2. St John’s wort + warfarin**\n", "- **Mechanism:** St John’s wort induces CYP3A4, CYP2C9, CYP2C19 and P‑glycoprotein. -\n", "**Effect:** Warfarin metabolism accelerates → INR falls, typically after 5–10 days of daily\n", "use. Thrombotic risk increases. - **Evidence:** FDA warfarin label; NICE CKS “warfarin\n", "interactions”; Stockley’s rating “red”. **3. Combined risk** The two agents exert\n", "opposite, time‑dependent effects on the same metabolic pathways. The net INR trajectory is\n", "unpredictable—fluconazole’s inhibition acts within days, while St John’s wort induction builds\n", "over weeks. This creates a period of labile anticoagulation with both bleeding and thrombotic\n", "danger. **4. Recommendation** - **St John’s wort: Contraindicated.** Advise the patient\n", "not to start it. Suggest evidence‑based alternatives for low mood (e.g., SSRI after reviewing\n", "bleeding risk, psychological therapies). - **Fluconazole: If clinically essential** (e.g.,\n", "systemic candidiasis), proceed but: - Check INR at baseline, then every 2–3 days while on\n", "fluconazole and for 1 week after stopping. - Reduce warfarin dose empirically by 20–30% (or\n", "per local protocol) at fluconazole initiation. - Consider a non‑azole antifungal (e.g.,\n", "echinocandin) if appropriate to avoid the interaction. - **Document** the interaction\n", "discussion and monitoring plan in the record.\n" ] } ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "step4-todo", "outputId": "4df1b87b-3001-456d-db61-1fefe80ad134" }, "id": "step4-todo", "source": [ "# TODO(student): run the benchmark on both models.\n", "# 1. Write run_benchmark(model_id) that loops over QUESTIONS, calls ask_with_fallbacks()\n", "# for each, scores the answer with score_answer(), and returns a list of row dicts.\n", "# Each row should carry: question id, domain, model, answer, seconds, prompt_tokens,\n", "# completion_tokens, cost_usd, cached -- plus every key score_answer() returned.\n", "# 2. Run it for SMALL_FREE and BIG_FREE and concatenate into a DataFrame called `results`.\n", "# 3. Print the side-by-side answer for ONE question of your choice so you can read both.\n", "# Ten API calls total. The cache means re-running this cell costs nothing.\n", "\n", "import textwrap\n", "\n", "def run_benchmark(model_id, questions=QUESTIONS, verbose=True):\n", " \"\"\"Collect answers, actual model IDs, measurements, and rubric scores.\"\"\"\n", " rows = []\n", "\n", " for question in questions:\n", " if verbose:\n", " print(f\" Asking {question['id']}...\", flush=True)\n", "\n", " call = ask_with_fallbacks(question[\"prompt\"], model_id)\n", " answer = call.get(\"answer\") or \"\"\n", " error = call.get(\"error\") or (\"\" if answer.strip() else \"Empty answer\")\n", "\n", " row = {\n", " **call,\n", " \"question\": question[\"id\"],\n", " \"domain\": question[\"domain\"],\n", " \"requested_model\": model_id,\n", " \"model\": call.get(\"model\", model_id),\n", " \"answer\": answer,\n", " \"call_error\": str(error),\n", " }\n", "\n", " if error:\n", " # Missing answers are missing scores, not incorrect answers.\n", " row.update({key: None for key in (\n", " \"coverage\", \"full_credit\", \"missed\", \"unsafe_claims\",\n", " \"bonus_hits\", \"hedges\", \"refused\", \"words\", \"reading_grade\"\n", " )})\n", " if verbose:\n", " print(f\" FAILED: {error}\")\n", " else:\n", " row.update(score_answer(question, answer))\n", " if verbose:\n", " print(\n", " f\" coverage={row['coverage']:.2f}, \"\n", " f\"full_credit={row['full_credit']}, \"\n", " f\"time={row['seconds']} s, cached={row['cached']}\"\n", " )\n", "\n", " rows.append(row)\n", "\n", " return rows\n", "\n", "\n", "# Check both selected models before starting any requests.\n", "for model_id in (SMALL_FREE, BIG_FREE):\n", " if not model_id.endswith(\":free\") or PRICES.get(model_id) != (0.0, 0.0):\n", " raise ValueError(f\"Zero pricing was not verified for {model_id}.\")\n", "\n", "if not OPENROUTER_API_KEY:\n", " raise ValueError(\"OPENROUTER_API_KEY is missing. Recheck the Secrets setup.\")\n", "\n", "print(f\"GP: {SMALL_FREE}\", flush=True)\n", "gp_rows = run_benchmark(SMALL_FREE)\n", "\n", "print(f\"\\nSpecialist: {BIG_FREE}\", flush=True)\n", "sp_rows = run_benchmark(BIG_FREE)\n", "\n", "results = pd.DataFrame(gp_rows + sp_rows)\n", "failed = results[\"call_error\"].astype(bool)\n", "\n", "print(f\"\\nSuccessful answers: {int((~failed).sum())}/{len(results)}\")\n", "print(f\"Failed answers: {int(failed.sum())}\")\n", "print(f\"Records retrieved from cache: {int(results['cached'].sum())}\")\n", "\n", "# Make every returned model ID visible, including any fallback.\n", "print(\"\\nRequested model -> model that responded:\")\n", "for requested, actual in results[\n", " [\"requested_model\", \"model\"]\n", "].drop_duplicates().itertuples(index=False, name=None):\n", " print(f\" {requested} -> {actual}\")\n", "\n", "# Read the full answers for the same drug-interaction question.\n", "SHOW = \"ddi_warfarin\"\n", "question_text = next(q[\"prompt\"] for q in QUESTIONS if q[\"id\"] == SHOW)\n", "\n", "print(\"\\nHEAD-TO-HEAD: Drug-drug interaction\")\n", "print(textwrap.fill(question_text, width=95))\n", "\n", "for _, row in results[results[\"question\"] == SHOW].iterrows():\n", " print(\"\\n\" + \"=\" * 70)\n", " print(f\"Requested: {row['requested_model']}\")\n", " print(f\"Responded: {row['model']}\")\n", "\n", " if row[\"call_error\"]:\n", " print(f\"FAILED: {row['call_error']}\")\n", " else:\n", " print(\n", " f\"Coverage: {row['coverage']:.2f} | \"\n", " f\"Full credit: {int(row['full_credit'])} | \"\n", " f\"Time: {row['seconds']} s\"\n", " )\n", " print(f\"Missed groups: {row['missed']}\")\n", " print(f\"Banned phrases flagged: {row['unsafe_claims']}\")\n", " print(textwrap.fill(row[\"answer\"], width=95))\n" ] }, { "cell_type": "code", "execution_count": 15, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Running the GP (small free model)...\n", " liquid/lfm-2.5-2.6b:free <- onc_staging coverage 1.00 6.11 s 1374 tok $0.000000\n", " liquid/lfm-2.5-2.6b:free <- cardio_gdm coverage 1.00 4.45 s 1232 tok $0.000000\n", " liquid/lfm-2.5-2.6b:free <- ddi_warfarin coverage 1.00 4.81 s 1176 tok $0.000000\n", " liquid/lfm-2.5-2.6b:free <- rare_bone coverage 0.67 2.03 s 669 tok $0.000000\n", " liquid/lfm-2.5-2.6b:free <- onto_map coverage 0.50 1.96 s 193 tok $0.000000\n", "\n", "Running the specialist (biggest free model)...\n", " nvidia/nemotron-3-ultra-550b-a55b:free <- onc_staging coverage 1.00 11.42 s 363 tok $0.000000\n", " nvidia/nemotron-3-ultra-550b-a55b:free <- cardio_gdm coverage 1.00 21.2 s 740 tok $0.000000\n", " nvidia/nemotron-3-ultra-550b-a55b:free <- ddi_warfarin coverage 1.00 16.58 s 894 tok $0.000000\n", " nvidia/nemotron-3-ultra-550b-a55b:free <- rare_bone coverage 1.00 5.51 s 335 tok $0.000000\n", " nvidia/nemotron-3-ultra-550b-a55b:free <- onto_map coverage 1.00 143.41 s 642 tok $0.000000\n", "\n", "10 measured answers (10 served from cache -- those cost no quota).\n", "\n", "==============================================================================\n", "HEAD TO HEAD -- ddi_warfarin\n", "==============================================================================\n", "QUESTION:\n", "A 71-year-old on warfarin has a stable INR of 2.4. She is prescribed fluconazole 200 mg daily, and asks whether she can also start over-the-counter St John's wort for low mood. Explain both interactions and what you recommend.\n", "\n", "------------------------------------------------------------------------------\n", "liquid/lfm-2.5-2.6b:free coverage 1.00 full_credit 1 4.81 s 1176 out-tokens $0.000000\n", "missed groups: none\n", "unsafe claims: none\n", "\n", "**Interaction Analysis**\n", "\n", "**St John's wort + Warfarin**\n", "- **Mechanism:** St John's wort is a potent inducer of CYP3A4 and P-glycoprotein. Warfarin is primarily metabolized by CYP2C9 (for S-warfarin) and CYP3A4 (for R-warfarin).\n", "- **Effect:** Induction of CYP3A4 accelerates warfarin clearance, reducing its plasma concentration and anticoagulant effect.\n", "- **Clinical consequence:** Risk of subtherapeutic INR (<2.0), which increases bleeding risk. The magnitude of reduction can be substantial—studies show up to 50% reduction in warfarin exposure when combined with St John's wort.\n", "- **Recommendation:** Avoid concurrent use. If St John's wort must be used, warfarin dose will likely need significant reduction (often by 30–50%) and INR monitoring every 1–2 days until stability is achieved.\n", "\n", "**Fluconazole + Warfarin**\n", "- **Mechanism:** Fluconazole is a moderate inhibitor of CYP2C9, which is responsible for S-warfarin metabolism.\n", "- **Effect:** Inhibition of CYP2C9 slows warfarin clearance, increasing its plasma concentration and anticoagulant effect.\n", "- **Clinical consequence:** Risk of supratherapeutic INR (>3.5), which increases bleeding risk. The magnitude varies based on fluconazole dose a\n", "\n", "------------------------------------------------------------------------------\n", "nvidia/nemotron-3-ultra-550b-a55b:free coverage 1.00 full_credit 1 16.58 s 894 out-tokens $0.000000\n", "missed groups: none\n", "unsafe claims: none\n", "\n", "**1. Fluconazole + warfarin** \n", "- **Mechanism:** Fluconazole is a potent inhibitor of CYP2C9 (primary metaboliser of S‑warfarin) and CYP3A4 (metabolises R‑warfarin). \n", "- **Effect:** Warfarin clearance falls → INR rises, often markedly, within 2–4 days. Bleeding risk increases. \n", "- **Evidence:** FDA warfarin label; CPIC guideline for CYP2C9; BNF/Stockley’s interaction rating “red” (avoid combination or monitor intensively). \n", "\n", "**2. St John’s wort + warfarin** \n", "- **Mechanism:** St John’s wort induces CYP3A4, CYP2C9, CYP2C19 and P‑glycoprotein. \n", "- **Effect:** Warfarin metabolism accelerates → INR falls, typically after 5–10 days of daily use. Thrombotic risk increases. \n", "- **Evidence:** FDA warfarin label; NICE CKS “warfarin interactions”; Stockley’s rating “red”. \n", "\n", "**3. Combined risk** \n", "The two agents exert opposite, time‑dependent effects on the same metabolic pathways. The net INR trajectory is unpredictable—fluconazole’s inhibition acts within days, while St John’s wort induction builds over weeks. This creates a period of labile anticoagulation with both bleeding and thrombotic danger. \n", "\n", "**4. Recommendation** \n", "- **St John’s wort: Contraindicated.** Advise the patient not to\n", "\n" ] } ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "step4-solution", "outputId": "f1329fd4-7917-4d18-fcc4-c7dd2ee822d9" }, "id": "step4-solution", "source": [ "#@title ✅ Solution (instructor)\n", "def run_benchmark(model_id, questions=QUESTIONS, verbose=True):\n", " \"\"\"Ask one model every benchmark question and grade each answer. Returns a list of rows.\"\"\"\n", " resolved = resolve(model_id, CATALOG)\n", " rows = []\n", " for q in questions:\n", " if verbose:\n", " print(f\" {resolved} <- {q['id']}\", end=\"\", flush=True)\n", " call = ask_with_fallbacks(q[\"prompt\"], resolved)\n", " row = {\"question\": q[\"id\"], \"domain\": q[\"domain\"], \"model\": resolved,\n", " \"answer\": call[\"answer\"], \"seconds\": call[\"seconds\"],\n", " \"prompt_tokens\": call[\"prompt_tokens\"],\n", " \"completion_tokens\": call[\"completion_tokens\"],\n", " \"cost_usd\": call[\"cost_usd\"], \"cached\": call.get(\"cached\", False),\n", " \"call_error\": call.get(\"error\", \"\")}\n", " row.update(score_answer(q, call[\"answer\"]))\n", " rows.append(row)\n", " if verbose:\n", " print(f\" coverage {row['coverage']:.2f} {row['seconds']} s \"\n", " f\"{row['completion_tokens']} tok ${row['cost_usd']:.6f}\")\n", " return rows\n", "\n", "\n", "print(\"Running the GP (small free model)...\")\n", "gp_rows = run_benchmark(SMALL_FREE)\n", "print(\"\\nRunning the specialist (biggest free model)...\")\n", "sp_rows = run_benchmark(BIG_FREE)\n", "\n", "results = pd.DataFrame(gp_rows + sp_rows)\n", "print(f\"\\n{len(results)} measured answers \"\n", " f\"({results['cached'].sum()} served from cache -- those cost no quota).\")\n", "\n", "# Read one head-to-head yourself. Numbers are for Week 7; reading is for today.\n", "SHOW = \"ddi_warfarin\"\n", "by_model = {mid: results[results[\"model\"] == mid] for mid in results[\"model\"].unique()}\n", "prompt_text = next(q[\"prompt\"] for q in QUESTIONS if q[\"id\"] == SHOW)\n", "print(\"\\n\" + \"=\" * 78)\n", "print(f\"HEAD TO HEAD -- {SHOW}\")\n", "print(\"=\" * 78)\n", "print(f\"QUESTION:\\n{prompt_text}\\n\")\n", "for mid, frame in by_model.items():\n", " row = frame[frame[\"question\"] == SHOW].iloc[0]\n", " print(\"-\" * 78)\n", " print(f\"{mid} coverage {row['coverage']:.2f} full_credit {row['full_credit']} \"\n", " f\"{row['seconds']} s {row['completion_tokens']} out-tokens \"\n", " f\"${row['cost_usd']:.6f}\")\n", " print(f\"missed groups: {row['missed'] or 'none'}\")\n", " print(f\"unsafe claims: {row['unsafe_claims'] or 'none'}\")\n", " print()\n", " print(row[\"answer\"][:1200])\n", " print()\n" ] }, { "cell_type": "markdown", "metadata": { "id": "QI4YFP6x6bo0" }, "source": [ "---\n", "## Step 5 — the scorecard\n", "\n", "**Why:** ten answers is an anecdote; one table is a decision. This collapses everything into the numbers a budget committee reads, and it deliberately puts **quality and price in the same view** — because either one alone is a marketing slide.\n", "\n", "Read the columns like this:\n", "\n", "| Column | What it tells you |\n", "|---|---|\n", "| `coverage` | partial credit — how much of the required knowledge showed up |\n", "| `full_credit` | the strict number — fully right *and* said nothing banned |\n", "| `unsafe_claims` | banned statements actually said. One of these is a patient-safety event, not a percentage |\n", "| `refusal_rate` | how often it declined instead of answering. Too high is useless; zero is a red flag of its own |\n", "| `hedge_rate` | how often it told the reader to check with a clinician — appropriate here |\n", "| `reading_grade` | Flesch-Kincaid. If patient-facing text lands at grade 14, nobody reads it |\n", "| `seconds` | latency — the difference between a chatbot and a queue |\n", "| `cost_per_q` | measured dollars per question, from live pricing |\n", "| `cost_per_correct` | **the number that decides it**: total spend divided by fully-correct answers |\n", "\n", "`cost_per_correct` is where the whole lab lands. On the free tier it is $0 for both models, and then the comparison is purely quality and speed. The moment you add a paid frontier model in Step 6, that column becomes the argument.\n" ], "id": "QI4YFP6x6bo0" }, { "cell_type": "code", "execution_count": 16, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "THE SCORECARD\n", " model coverage full_credit unsafe_claims refusal_rate hedge_rate reading_grade mean_seconds cost_per_q cost_per_correct\n", " liquid/lfm-2.5-2.6b:free 0.833 0.6 0 0.0 0.8 15.8 3.87 0.0 0.0\n", "nvidia/nemotron-3-ultra-550b-a55b:free 1.000 1.0 0 0.0 0.6 15.1 39.62 0.0 0.0\n", "\n", "highest strict accuracy : nvidia/nemotron-3-ultra-550b-a55b:free (full_credit 1.00)\n", "cheapest per question : liquid/lfm-2.5-2.6b:free ($0.000000)\n", "accuracy gap : 0.40 of 1.00\n", "\n", "Now find WHERE the gap lives -- the next cell breaks it down per question,\n", "because an average hides the one task the small model fails.\n", "\n", "COVERAGE PER QUESTION (where the gap actually lives)\n", "model liquid/lfm-2.5-2.6b:free nvidia/nemotron-3-ultra-550b-a55b:free\n", "question domain \n", "cardio_gdm Cardiology guideline 1.00 1.0\n", "ddi_warfarin Drug-drug interaction 1.00 1.0\n", "onc_staging Oncology staging 1.00 1.0\n", "onto_map Ontology mapping (ICD-10-CM to SNOMED CT) 0.50 1.0\n", "rare_bone Rare-disease workup 0.67 1.0\n" ] } ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "step5-scorecard", "outputId": "2a5c2627-121e-463f-9c91-fe07f8756e00" }, "id": "step5-scorecard", "source": [ "# Collapse the per-question rows into one row per model.\n", "def build_scorecard(frame):\n", " \"\"\"Aggregate measured answers into one scorecard row per model.\"\"\"\n", " rows = []\n", " for mid, grp in frame.groupby(\"model\", sort=False):\n", " n = len(grp)\n", " correct = int(grp[\"full_credit\"].sum())\n", " total_cost = float(grp[\"cost_usd\"].sum())\n", " grades = grp[\"reading_grade\"].dropna()\n", " rows.append({\n", " \"model\": mid,\n", " \"n\": n,\n", " \"coverage\": round(float(grp[\"coverage\"].mean()), 3),\n", " \"full_credit\": round(correct / n, 3) if n else 0.0,\n", " \"correct_n\": correct,\n", " \"unsafe_claims\": int(grp[\"unsafe_claims\"].apply(len).sum()),\n", " \"refusal_rate\": round(float(grp[\"refused\"].mean()), 3) if n else 0.0,\n", " \"hedge_rate\": round(float(grp[\"hedges\"].mean()), 3) if n else 0.0,\n", " \"reading_grade\": round(float(grades.mean()), 1) if len(grades) else None,\n", " \"mean_words\": round(float(grp[\"words\"].mean()), 1) if n else 0.0,\n", " \"mean_seconds\": round(float(grp[\"seconds\"].mean()), 2) if n else 0.0,\n", " \"out_tokens\": round(float(grp[\"completion_tokens\"].mean()), 1) if n else 0.0,\n", " \"cost_per_q\": round(total_cost / n, 6) if n else 0.0,\n", " \"cost_per_correct\": (round(total_cost / correct, 6) if correct else\n", " (0.0 if total_cost == 0 else float(\"inf\"))),\n", " })\n", " return pd.DataFrame(rows)\n", "\n", "\n", "scorecard = build_scorecard(results)\n", "\n", "VIEW = [\"model\", \"coverage\", \"full_credit\", \"unsafe_claims\", \"refusal_rate\",\n", " \"hedge_rate\", \"reading_grade\", \"mean_seconds\", \"cost_per_q\", \"cost_per_correct\"]\n", "\n", "# ── Did the calls actually happen? A scorecard built from failed calls is not a\n", "# ── result, it is a row of zeros wearing a result's clothes. Say so before it is read.\n", "N_FAILED = int(results[\"call_error\"].astype(bool).sum())\n", "N_TOTAL = len(results)\n", "CALLS_OK = N_FAILED < N_TOTAL\n", "if N_FAILED:\n", " print(\"=\" * 78)\n", " print(f\"!! {N_FAILED} of {N_TOTAL} model calls FAILED. The numbers below are not measurements.\")\n", " print(\"=\" * 78)\n", " for err in sorted(set(e for e in results[\"call_error\"] if e)):\n", " print(f\" {err}\")\n", " print()\n", " print(\" Most likely: OPENROUTER_API_KEY is missing, empty, or Notebook access is OFF.\")\n", " print(\" Fix: 🔑 sidebar -> add OPENROUTER_API_KEY (free key at https://openrouter.ai/keys),\")\n", " print(\" toggle Notebook access ON, re-run the plumbing cell, then re-run this part.\")\n", " print()\n", " if not CALLS_OK:\n", " print(\" NOT printing a scorecard: every score, cost and timing would be 0.0 because\")\n", " print(\" nothing was ever asked. Do not write these up and do not submit them.\")\n", " print(\" Part A above is real and unaffected. Fix the key, then re-run Part B.\")\n", " print(\"=\" * 78)\n", " else:\n", " print(\" Continuing with the calls that DID succeed. Treat the rest as missing, not as 0.\")\n", " print()\n", "\n", "PART_B_VALID = CALLS_OK\n", "if PART_B_VALID:\n", " print(\"THE SCORECARD\")\n", " print(scorecard[VIEW].to_string(index=False))\n", " print()\n", "\n", " best = scorecard.loc[scorecard[\"full_credit\"].idxmax()]\n", " cheap = scorecard.loc[scorecard[\"cost_per_q\"].idxmin()]\n", " print(f\"highest strict accuracy : {best['model']} (full_credit {best['full_credit']:.2f})\")\n", " print(f\"cheapest per question : {cheap['model']} (${cheap['cost_per_q']:.6f})\")\n", " gap = float(scorecard[\"full_credit\"].max() - scorecard[\"full_credit\"].min())\n", " print(f\"accuracy gap : {gap:.2f} of 1.00\")\n", " print()\n", " if gap == 0 and N_FAILED:\n", " print(\"Both models scored the same -- but some calls failed above, so this tie is an\")\n", " print(\"artifact of the missing data, NOT a finding. Fix the key and re-run first.\")\n", " elif gap == 0:\n", " print(\"No gap at all on these five questions. That is a finding, not a bug: it means\")\n", " print(\"this task does not need the expensive model. Say exactly that in your takeaways.\")\n", " else:\n", " print(\"Now find WHERE the gap lives -- the next cell breaks it down per question,\")\n", " print(\"because an average hides the one task the small model fails.\")\n", "\n", " # Per-question breakdown: the average is not the story, the exceptions are.\n", " pivot = results.pivot_table(index=[\"question\", \"domain\"], columns=\"model\",\n", " values=\"coverage\", aggfunc=\"first\")\n", " print()\n", " print(\"COVERAGE PER QUESTION (where the gap actually lives)\")\n", " print(pivot.round(2).to_string())\n" ] }, { "cell_type": "markdown", "metadata": { "id": "PAxrJB9B6bo0" }, "source": [ "---\n", "## Step 6 — *optional*: the true paid frontier, behind a cost guard\n", "\n", "**Why:** the biggest model reachable for $0 is not the biggest model on OpenRouter, and pretending otherwise would be a dishonest lab. As of 2026-09-13 the paid frontier includes `openai/gpt-6-astra` and `anthropic/claude-fable-5.1` at **$10 / $50 per million tokens** (in / out) and `qwen/qwen3.8-max-0902` — a 2.4-trillion-parameter mixture-of-experts model — at **$2 / $6**.\n", "\n", "**This course never requires you to spend money, and this cell is OFF by default.** Everything you need to hand in is already measured. If you *want* the frontier column, here is the professional way to get it:\n", "\n", "1. The cell **estimates the cost before it calls anything** — using the tokenizer you loaded in Part A to count your prompt tokens, and `MAX_TOKENS` for the answer. That is why Part A mattered: the tokenizer is literally what the bill is made of.\n", "2. It prints the estimate and **refuses to run** unless you set `ALLOW_PAID = True` yourself.\n", "3. It tracks cumulative spend and **stops at `MAX_SPEND_USD`**, mid-loop if necessary.\n", "\n", "For reference: five questions against `gpt-6-astra` came out around **$0.10–$0.30** at these token counts, and under **$0.05** against `qwen3.8-max-0902`. Check the printed estimate before you flip the flag — do not trust this paragraph, trust the number the cell computes.\n" ], "id": "PAxrJB9B6bo0" }, { "cell_type": "code", "execution_count": null, "outputs": [], "metadata": { "id": "step6-frontier" }, "id": "step6-frontier", "source": [ "#@title Optional · paid frontier run (cost guard, OFF by default)\n", "# ══════════════════════════════════════════════════════════════════════════════\n", "# NOTHING IN THIS CELL RUNS UNTIL YOU SET ALLOW_PAID = True YOURSELF.\n", "# The required lab is already complete above. This adds rows to `results` and\n", "# `scorecard` only if you want to spend your own OpenRouter credit.\n", "#\n", "# Cost is estimated BEFORE any call, using the Part A tokenizer to count your own\n", "# prompt tokens. If the estimate is higher than you want, do not flip the flag.\n", "# ══════════════════════════════════════════════════════════════════════════════\n", "ALLOW_PAID = False # <- flip to True only if your key has credit you mean to spend\n", "MAX_SPEND_USD = 0.50 # hard ceiling for this whole cell; it stops mid-loop at this\n", "FRONTIER = [\"qwen/qwen3.8-max-0902\"] # cheapest real frontier model first. Add\n", " # \"openai/gpt-6-astra\" or \"anthropic/claude-fable-5.1\"\n", " # if you want the flagship and have the budget.\n", "\n", "if not ALLOW_PAID:\n", " print(\"ALLOW_PAID is False -- skipping the paid frontier run. Nothing was charged.\")\n", " print(\"The $0 scorecard above already completes this lab.\")\n", " print()\n", " print(\"Here is what it WOULD have cost, so you can decide with a number in hand:\")\n", " for mid in FRONTIER:\n", " listed = resolve(mid, CATALOG)\n", " price_in, price_out = PRICES.get(listed, (0.0, 0.0))\n", " est = 0.0\n", " for q in QUESTIONS:\n", " est += len(tokenizer(q[\"prompt\"])[\"input_ids\"]) * price_in\n", " est += MAX_TOKENS * price_out\n", " print(f\" {listed:<44} ~${est:.4f} for all {len(QUESTIONS)} questions\")\n", " print()\n", " print(\"To run it for real: set ALLOW_PAID = True above and re-run this cell.\")\n", "elif not OPENROUTER_API_KEY:\n", " print(\"ALLOW_PAID is True but there is no OPENROUTER_API_KEY secret. Add one in the\")\n", " print(\"Colab 🔑 sidebar, or set ALLOW_PAID back to False.\")\n", "else:\n", " spent = 0.0\n", " frontier_rows = []\n", " for mid in FRONTIER:\n", " listed = resolve(mid, CATALOG)\n", " price_in, price_out = PRICES.get(listed, (0.0, 0.0))\n", " estimate = sum(len(tokenizer(q[\"prompt\"])[\"input_ids\"]) * price_in\n", " + MAX_TOKENS * price_out for q in QUESTIONS)\n", " print(f\"\\n{listed}: estimated ${estimate:.4f} for {len(QUESTIONS)} questions\")\n", " if spent + estimate > MAX_SPEND_USD:\n", " print(f\" SKIPPED -- that would take total spend to ${spent + estimate:.4f}, \"\n", " f\"above your MAX_SPEND_USD ceiling of ${MAX_SPEND_USD:.2f}.\")\n", " continue\n", " print(\" proceeding (cost guard passed)...\")\n", " rows = run_benchmark(listed)\n", " frontier_rows.extend(rows)\n", " spent += sum(r[\"cost_usd\"] for r in rows)\n", " print(f\" actual spend on this model: ${sum(r['cost_usd'] for r in rows):.4f}\")\n", "\n", " if frontier_rows:\n", " results = pd.concat([results, pd.DataFrame(frontier_rows)], ignore_index=True)\n", " scorecard = build_scorecard(results)\n", " print(f\"\\nTOTAL SPENT: ${spent:.4f} (ceiling was ${MAX_SPEND_USD:.2f})\")\n", " print()\n", " print(\"THE SCORECARD, frontier included\")\n", " print(scorecard[VIEW].to_string(index=False))\n", " pivot = results.pivot_table(index=[\"question\", \"domain\"], columns=\"model\",\n", " values=\"coverage\", aggfunc=\"first\")\n", " print()\n", " print(\"COVERAGE PER QUESTION\")\n", " print(pivot.round(2).to_string())\n", " else:\n", " print(\"\\nNo frontier rows were added.\")\n" ] }, { "cell_type": "markdown", "metadata": { "id": "f3dKCFPx6bo0" }, "source": [ "---\n", "## Step 7 — the routing decision: send the GP first\n", "\n", "**Why:** nobody in production sends every question to the most expensive model. They **route**: the cheap model answers everything, and only the cases it cannot handle get escalated. That is exactly how a health system works — the GP sees everyone and refers the few who need the specialist. The question is not \"which model is better?\" but **\"what accuracy do I get for what fraction of the cost?\"**\n", "\n", "The whole scheme rests on one assumption, and it is the interesting one: **that the small model knows when it is out of its depth.** So this step measures that directly. The GP escalates when its own answer contains a genuine uncertainty marker — \"not sure\", \"unable to determine\", \"not enough information\". Then we ask the question that decides whether routing is safe at all:\n", "\n", "> **Of the answers the GP got wrong, how many did it flag?**\n", "\n", "That is the recall of its self-doubt, and it is a calibration question. If it is high, routing works: escalate the flagged ones and buy near-specialist accuracy at GP cost. If it is low, the GP is **confidently wrong**, and routing is a trap that feels like a saving — the failures sail straight through to the patient. Small models fail this more often than big ones, and finding out whether this one does is the real result of the lab.\n", "\n", "> **Honesty about the proxy.** Self-reported uncertainty in text is a crude signal. A production router uses token log-probabilities or a trained classifier over the prompt. We use words because you can read them, audit them, and explain them to a clinician — and because the failure mode it exposes is identical.\n" ], "id": "f3dKCFPx6bo0" }, { "cell_type": "code", "execution_count": 17, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "CASCADE: replaying the saved answers\n", "Escalation rate: 0/5 (0%)\n", "Rubric full credit, GP alone: 60%\n", "Rubric full credit, specialist alone: 100%\n", "Rubric full credit, cascade: 60%\n", "Cascade cost (GP plus escalations): 0.000000 USD\n", "Specialist-for-everything cost: 0.000000 USD\n", "\n", "CALIBRATION CHECK\n", "Of 2 GP answers that failed the rubric, 0 flagged uncertainty.\n", "Recall of uncertainty flags: 0%\n" ] }, { "output_type": "display_data", "data": { "text/plain": [ " question escalated used_role coverage full_credit \\\n", "0 onc_staging False GP 1.000 1 \n", "1 cardio_gdm False GP 1.000 1 \n", "2 ddi_warfarin False GP 1.000 1 \n", "3 rare_bone False GP 0.667 0 \n", "4 onto_map False GP 0.500 0 \n", "\n", " uncertainty_markers \n", "0 none \n", "1 none \n", "2 none \n", "3 none \n", "4 none " ], "text/html": [ "\n", "
\n", "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
questionescalatedused_rolecoveragefull_credituncertainty_markers
0onc_stagingFalseGP1.0001none
1cardio_gdmFalseGP1.0001none
2ddi_warfarinFalseGP1.0001none
3rare_boneFalseGP0.6670none
4onto_mapFalseGP0.5000none
\n", "
\n", "
\n", "\n", "
\n", " \n", "\n", " \n", "\n", " \n", "
\n", "\n", "\n", "
\n", "
\n" ], "application/vnd.google.colaboratory.intrinsic+json": { "type": "dataframe", "summary": "{\n \"name\": \"]])\",\n \"rows\": 5,\n \"fields\": [\n {\n \"column\": \"question\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"cardio_gdm\",\n \"onto_map\",\n \"ddi_warfarin\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"escalated\",\n \"properties\": {\n \"dtype\": \"boolean\",\n \"num_unique_values\": 1,\n \"samples\": [\n false\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"used_role\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 1,\n \"samples\": [\n \"GP\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"coverage\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 0.23564337461511622,\n \"min\": 0.5,\n \"max\": 1.0,\n \"num_unique_values\": 3,\n \"samples\": [\n 1.0\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"full_credit\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 0,\n \"min\": 0,\n \"max\": 1,\n \"num_unique_values\": 2,\n \"samples\": [\n 0\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"uncertainty_markers\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 1,\n \"samples\": [\n \"none\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}" } }, "metadata": {} } ], "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 405 }, "id": "step7-todo", "outputId": "e6329efb-baac-45b3-bc2e-638a5e976150" }, "id": "step7-todo", "source": [ "# TODO(student): build the cascade and test whether the GP knows what it does not know.\n", "# 1. Write flagged_uncertain(answer): True if the answer contains a genuine uncertainty\n", "# marker from UNCERTAIN_KW.\n", "# 2. Build `cascade`: start from the GP's answer for every question, but where the GP\n", "# flagged uncertainty, use the specialist's answer instead and mark it escalated.\n", "# Keep each row's coverage and full_credit from whichever model actually answered.\n", "# 3. Report: escalation rate, cascade full_credit vs GP-alone vs specialist-alone, and\n", "# cascade cost vs specialist-for-everything cost.\n", "# 4. THE CALIBRATION CHECK: of the GP's rows where full_credit == 0, what fraction did\n", "# it flag? That is the number that decides whether routing is safe.\n", "\n", "UNCERTAIN_KW = [\"not sure\", \"not certain\", \"unsure\", \"i don't know\", \"i do not know\",\n", " \"cannot say\", \"can't say\", \"unable to determine\", \"not enough information\",\n", " \"cannot answer\", \"can't answer\", \"i am not able to\", \"i'm not able to\",\n", " \"would need more\"]\n", "\n", "UNCERTAIN_KW = [\n", " \"not sure\", \"not certain\", \"unsure\", \"i don't know\", \"i do not know\",\n", " \"cannot say\", \"can't say\", \"unable to determine\", \"not enough information\",\n", " \"cannot answer\", \"can't answer\", \"i am not able to\", \"i'm not able to\",\n", " \"would need more\"\n", "]\n", "\n", "def flagged_uncertain(answer):\n", " \"\"\"Check for the notebook's uncertainty phrases.\"\"\"\n", " text = _normalize(answer)\n", " return any(_normalize(phrase) in text for phrase in UNCERTAIN_KW)\n", "\n", "\n", "gp = SMALL_FREE\n", "specialist = BIG_FREE\n", "gp_frame = results[results[\"model\"] == gp].set_index(\"question\")\n", "sp_frame = results[results[\"model\"] == specialist].set_index(\"question\")\n", "\n", "if len(gp_frame) != len(QUESTIONS) or len(sp_frame) != len(QUESTIONS):\n", " raise ValueError(\"The cascade needs all five answers from each selected model.\")\n", "\n", "if (\n", " gp_frame[\"call_error\"].astype(bool).any()\n", " or sp_frame[\"call_error\"].astype(bool).any()\n", "):\n", " raise ValueError(\"Resolve the failed calls before comparing the cascade.\")\n", "\n", "rows = []\n", "\n", "for question in QUESTIONS:\n", " qid = question[\"id\"]\n", " gp_row = gp_frame.loc[qid]\n", " sp_row = sp_frame.loc[qid]\n", "\n", " escalate = flagged_uncertain(gp_row[\"answer\"])\n", " used = sp_row if escalate else gp_row\n", "\n", " markers = [\n", " phrase for phrase in UNCERTAIN_KW\n", " if _normalize(phrase) in _normalize(gp_row[\"answer\"])\n", " ]\n", "\n", " rows.append({\n", " \"question\": qid,\n", " \"domain\": gp_row[\"domain\"],\n", " \"escalated\": escalate,\n", " \"answered_by\": used[\"model\"],\n", " \"used_role\": \"Specialist\" if escalate else \"GP\",\n", " \"coverage\": used[\"coverage\"],\n", " \"full_credit\": used[\"full_credit\"],\n", " # The GP is always called; escalation adds the specialist's cost.\n", " \"cost_usd\": gp_row[\"cost_usd\"] + (\n", " sp_row[\"cost_usd\"] if escalate else 0.0\n", " ),\n", " \"gp_flagged\": escalate,\n", " \"gp_was_wrong\": int(gp_row[\"full_credit\"]) == 0,\n", " \"uncertainty_markers\": \", \".join(markers) if markers else \"none\",\n", " })\n", "\n", "cascade = pd.DataFrame(rows)\n", "\n", "n = len(cascade)\n", "n_esc = int(cascade[\"escalated\"].sum())\n", "gp_acc = float(gp_frame[\"full_credit\"].mean())\n", "sp_acc = float(sp_frame[\"full_credit\"].mean())\n", "cas_acc = float(cascade[\"full_credit\"].mean())\n", "cas_cost = float(cascade[\"cost_usd\"].sum())\n", "sp_cost = float(sp_frame[\"cost_usd\"].sum())\n", "\n", "# \"Wrong\" here means failing the keyword rubric.\n", "wrong = cascade[cascade[\"gp_was_wrong\"]]\n", "flagged_recall = float(wrong[\"gp_flagged\"].mean()) if len(wrong) else None\n", "\n", "print(\"CASCADE: replaying the saved answers\")\n", "print(f\"Escalation rate: {n_esc}/{n} ({n_esc / n:.0%})\")\n", "print(f\"Rubric full credit, GP alone: {gp_acc:.0%}\")\n", "print(f\"Rubric full credit, specialist alone: {sp_acc:.0%}\")\n", "print(f\"Rubric full credit, cascade: {cas_acc:.0%}\")\n", "print(f\"Cascade cost (GP plus escalations): {cas_cost:.6f} USD\")\n", "print(f\"Specialist-for-everything cost: {sp_cost:.6f} USD\")\n", "\n", "print(\"\\nCALIBRATION CHECK\")\n", "\n", "if flagged_recall is None:\n", " print(\"No GP answers failed the rubric, so recall is undefined.\")\n", "else:\n", " flagged_count = int(wrong[\"gp_flagged\"].sum())\n", " print(\n", " f\"Of {len(wrong)} GP answers that failed the rubric, \"\n", " f\"{flagged_count} flagged uncertainty.\"\n", " )\n", " print(f\"Recall of uncertainty flags: {flagged_recall:.0%}\")\n", "\n", "display(cascade[[\n", " \"question\", \"escalated\", \"used_role\",\n", " \"coverage\", \"full_credit\", \"uncertainty_markers\"\n", "]])\n" ] }, { "cell_type": "code", "execution_count": 19, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "THE CASCADE (GP answers everything; escalate what it flags)\n", " escalation rate : 0/5 (0%)\n", " full_credit, GP alone : 0.60\n", " full_credit, specialist: 1.00\n", " full_credit, cascade : 0.60\n", " cost, cascade : $0.000000 vs specialist-for-all $0.000000\n", "\n", "THE CALIBRATION CHECK -- the number that decides whether routing is safe\n", " of the GP's 2 wrong answers, it flagged 0 -> recall 0.00\n", " LOW self-doubt: the GP is confidently wrong, and the failures sail\n", " straight through the router to the patient. Routing would FEEL like\n", " a saving and behave like a safety incident. Report that.\n", "\n", " question escalated answered_by coverage full_credit\n", " onc_staging False liquid/lfm-2.5-2.6b:free 1.000 1\n", " cardio_gdm False liquid/lfm-2.5-2.6b:free 1.000 1\n", "ddi_warfarin False liquid/lfm-2.5-2.6b:free 1.000 1\n", " rare_bone False liquid/lfm-2.5-2.6b:free 0.667 0\n", " onto_map False liquid/lfm-2.5-2.6b:free 0.500 0\n" ] } ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "step7-solution", "outputId": "7c39cea0-76f2-40f1-f05a-413dd178971e" }, "id": "step7-solution", "source": [ "#@title ✅ Solution (instructor)\n", "UNCERTAIN_KW = [\"not sure\", \"not certain\", \"unsure\", \"i don't know\", \"i do not know\",\n", " \"cannot say\", \"can't say\", \"unable to determine\", \"not enough information\",\n", " \"cannot answer\", \"can't answer\", \"i am not able to\", \"i'm not able to\",\n", " \"would need more\"]\n", "\n", "\n", "def flagged_uncertain(answer):\n", " \"\"\"Did the model itself say it was not sure? The routing signal for the cascade.\"\"\"\n", " t = _normalize(answer)\n", " return any(k in t for k in UNCERTAIN_KW)\n", "\n", "\n", "gp = resolve(SMALL_FREE, CATALOG)\n", "specialist = resolve(BIG_FREE, CATALOG)\n", "gp_frame = results[results[\"model\"] == gp].set_index(\"question\")\n", "sp_frame = results[results[\"model\"] == specialist].set_index(\"question\")\n", "\n", "if gp_frame.empty or sp_frame.empty:\n", " print(\"The cascade needs both free models measured. Run Step 4 first -- or, if you\")\n", " print(\"skipped Part B, note in your takeaways that the cascade is the part you could\")\n", " print(\"not run and why.\")\n", " cascade = pd.DataFrame()\n", " flagged_recall = None\n", "else:\n", " rows = []\n", " for qid in gp_frame.index:\n", " gp_row = gp_frame.loc[qid]\n", " sp_row = sp_frame.loc[qid]\n", " escalate = bool(flagged_uncertain(gp_row[\"answer\"]))\n", " used = sp_row if escalate else gp_row\n", " rows.append({\"question\": qid, \"domain\": gp_row[\"domain\"], \"escalated\": escalate,\n", " \"answered_by\": used[\"model\"], \"coverage\": used[\"coverage\"],\n", " \"full_credit\": used[\"full_credit\"], \"cost_usd\": used[\"cost_usd\"],\n", " \"gp_flagged\": escalate, \"gp_was_wrong\": int(gp_row[\"full_credit\"]) == 0})\n", " cascade = pd.DataFrame(rows)\n", "\n", " n = len(cascade)\n", " n_esc = int(cascade[\"escalated\"].sum())\n", " gp_acc = float(gp_frame[\"full_credit\"].mean())\n", " sp_acc = float(sp_frame[\"full_credit\"].mean())\n", " cas_acc = float(cascade[\"full_credit\"].mean())\n", " cas_cost = float(cascade[\"cost_usd\"].sum())\n", " sp_cost = float(sp_frame[\"cost_usd\"].sum())\n", "\n", " wrong = cascade[cascade[\"gp_was_wrong\"]]\n", " flagged_recall = float(wrong[\"gp_flagged\"].mean()) if len(wrong) else None\n", "\n", " print(\"THE CASCADE (GP answers everything; escalate what it flags)\")\n", " print(f\" escalation rate : {n_esc}/{n} ({n_esc / n * 100:.0f}%)\")\n", " print(f\" full_credit, GP alone : {gp_acc:.2f}\")\n", " print(f\" full_credit, specialist: {sp_acc:.2f}\")\n", " print(f\" full_credit, cascade : {cas_acc:.2f}\")\n", " print(f\" cost, cascade : ${cas_cost:.6f} vs specialist-for-all ${sp_cost:.6f}\")\n", " print()\n", " print(\"THE CALIBRATION CHECK -- the number that decides whether routing is safe\")\n", " if flagged_recall is None:\n", " print(\" The GP got every question fully right, so there is nothing to calibrate.\")\n", " print(\" That is the best possible outcome for the cheap model, and it is rare.\")\n", " else:\n", " print(f\" of the GP's {len(wrong)} wrong answers, it flagged \"\n", " f\"{int(wrong['gp_flagged'].sum())} -> recall {flagged_recall:.2f}\")\n", " if flagged_recall >= 0.6:\n", " print(\" Decent self-doubt: routing buys you most of the specialist's accuracy\")\n", " print(\" at a fraction of the cost. This is a real strategy, not a gimmick.\")\n", " else:\n", " print(\" LOW self-doubt: the GP is confidently wrong, and the failures sail\")\n", " print(\" straight through the router to the patient. Routing would FEEL like\")\n", " print(\" a saving and behave like a safety incident. Report that.\")\n", " print()\n", " print(cascade[[\"question\", \"escalated\", \"answered_by\", \"coverage\",\n", " \"full_credit\"]].to_string(index=False))\n" ] }, { "cell_type": "markdown", "metadata": { "id": "DS83gsY76bo1" }, "source": [ "---\n", "## Ship it — the scorecard, and (optionally) a public page\n", "\n", "**Why:** the artifact is the thing you show the CFO: the per-question exceptions up top, the routing decision with the number that justifies it, and the full scorecard underneath. We draw it here in Colab, and the cell after it builds the page.\n", "\n", "**Uploading is optional this week.** The reset made publishing opt-in: your notebook submitted to Canvas earns full credit whether or not you ever touch Hugging Face. If you *do* want the public page, it ships as a **static** Space — Hugging Face serves one folder exactly as-is, free, no build, never sleeps, no card (compute Spaces now cost money; static does not). Either way the page and the CSVs are built locally first, so you always have the artifact in hand.\n" ], "id": "DS83gsY76bo1" }, { "cell_type": "code", "execution_count": 20, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ " model coverage full_credit unsafe_claims refusal_rate hedge_rate reading_grade mean_seconds cost_per_q cost_per_correct\n", " liquid/lfm-2.5-2.6b:free 0.833 0.6 0 0.0 0.8 15.8 3.87 0.0 0.0\n", "nvidia/nemotron-3-ultra-550b-a55b:free 1.000 1.0 0 0.0 0.6 15.1 39.62 0.0 0.0\n" ] }, { "output_type": "display_data", "data": { "text/html": [ "\n", "\n", "\n", "
\n", "
\n", "\n", "" ] }, "metadata": {} }, { "output_type": "display_data", "data": { "text/html": [ "\n", "\n", "\n", "
\n", "
\n", "\n", "" ] }, "metadata": {} } ], "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 911 }, "id": "ship-figures", "outputId": "7c39acd5-766c-498d-9f36-adb20a805af7" }, "id": "ship-figures", "source": [ "# Two charts. Both were chosen so that EVERY axis varies in the required $0 path -- a chart\n", "# whose x-axis is $0 for both models shows nothing, which is what an earlier draft did.\n", "#\n", "# Fig 1 where the gap lives -- paired dot plot, sorted by the size of the gap\n", "# Fig 2 what accuracy you can buy -- the routing curve: oracle what-if vs what you measured\n", "#\n", "# Latency, tokens, cost and reading grade deliberately stay in the printed scorecard TABLE\n", "# below rather than in a chart: with two or three models a compact table is clearer than bars.\n", "import plotly.graph_objects as go\n", "\n", "# House palette, validated for a light surface (dataviz validate_palette.js on\n", "# '#AE1800,#1D5FA8' -> PASS). Marker SHAPE varies too, so colour is never the only encoding.\n", "SERIES = [(\"#AE1800\", \"circle\"), (\"#1D5FA8\", \"square\"), (\"#166534\", \"diamond\"),\n", " (\"#7A6A55\", \"cross\")]\n", "GRID, CONNECTOR, GROUND, MUTED = \"#E3DCD0\", \"#B9B7B2\", \"#FAF7F0\", \"#666666\"\n", "FONT = dict(family=\"Archivo, Helvetica, Arial, sans-serif\")\n", "\n", "\n", "def short_name(model_id):\n", " \"\"\"A label short enough for an axis: 'liquid/lfm-2.5-2.6b:free' -> 'lfm-2.5-2.6b (free)'.\"\"\"\n", " return model_id.split(\"/\")[-1].replace(\":free\", \" (free)\")\n", "\n", "\n", "def gap_fig(frame):\n", " \"\"\"Fig 1 -- one dot per model per question, joined by a connector, sorted by the gap.\n", "\n", " The headline is COMPUTED from your own measurements, never hard-coded, because your numbers\n", " will differ from everyone else's. A chart whose title is written at run time cannot lie.\n", " \"\"\"\n", " models = list(frame[\"model\"].unique())\n", " cov = frame.pivot_table(index=\"question\", columns=\"model\", values=\"coverage\", aggfunc=\"first\")\n", " domain_of = frame.drop_duplicates(\"question\").set_index(\"question\")[\"domain\"].to_dict()\n", "\n", " def val(q, m):\n", " v = cov.loc[q, m] if m in cov.columns else None\n", " return None if v is None or pd.isna(v) else float(v) * 100\n", "\n", " # sort ascending so the largest divergence is drawn at the TOP of the chart\n", " if len(models) >= 2:\n", " spread = pd.Series({q: abs((val(q, models[-1]) or 0) - (val(q, models[0]) or 0))\n", " for q in cov.index})\n", " else:\n", " spread = cov[models[0]].abs()\n", " order = list(spread.sort_values().index)\n", " labels = [domain_of[q][:36] for q in order]\n", "\n", " fig = go.Figure()\n", "\n", " # connectors first so the dots sit on top\n", " xs, ys = [], []\n", " for q, lab in zip(order, labels):\n", " pts = [v for v in (val(q, m) for m in models) if v is not None]\n", " if len(pts) >= 2:\n", " xs += [min(pts), max(pts), None]\n", " ys += [lab, lab, None]\n", " fig.add_trace(go.Scatter(x=xs, y=ys, mode=\"lines\", hoverinfo=\"skip\", showlegend=False,\n", " line=dict(color=CONNECTOR, width=2)))\n", "\n", " for i, mid in enumerate(models):\n", " colour, symbol = SERIES[i % len(SERIES)]\n", " fig.add_trace(go.Scatter(\n", " x=[val(q, mid) for q in order], y=labels, mode=\"markers\", name=short_name(mid),\n", " marker=dict(color=colour, size=12, symbol=symbol,\n", " line=dict(width=1.5, color=\"#FFFFFF\")),\n", " hovertemplate=f\"{mid}
%{{y}}
coverage %{{x:.0f}}%\"))\n", "\n", " # direct annotation: the gap on every row, so no viewer has to subtract two dots\n", " if len(models) >= 2:\n", " for q, lab in zip(order, labels):\n", " a, b = val(q, models[0]), val(q, models[-1])\n", " if a is None or b is None:\n", " continue\n", " d = b - a\n", " fig.add_annotation(x=max(a, b), y=lab, xanchor=\"left\", xshift=9, showarrow=False,\n", " text=(\"tie\" if abs(d) < 1e-9 else f\"{d:+.0f}\"),\n", " font=dict(size=11, color=MUTED))\n", "\n", " if len(models) >= 2:\n", " ties = int((spread < 1e-9).sum())\n", " headline = (f\"tied on {ties} of {len(order)} questions; biggest gap on \"\n", " f\"{domain_of[spread.idxmax()][:38]} ({spread.max():.0f} pts)\")\n", " else:\n", " headline = \"coverage per question\"\n", "\n", " fig.update_layout(\n", " title=dict(text=(\"Where the gap lives
\" + headline +\n", " f\" · n = {len(order)} questions · coverage = share of \"\n", " \"required clinical facts present · label = specialist minus GP\"\n", " \"\"), font=dict(size=17)),\n", " xaxis=dict(title=\"coverage (%)\", range=[0, 116], dtick=25, gridcolor=GRID,\n", " zeroline=False),\n", " yaxis=dict(automargin=True, gridcolor=GRID),\n", " height=400, margin=dict(l=10, r=10, t=80, b=10),\n", " legend=dict(orientation=\"h\", y=-0.16, x=0), font=FONT,\n", " plot_bgcolor=GROUND, paper_bgcolor=GROUND)\n", " return fig\n", "\n", "\n", "def routing_curve(gp_frame, sp_frame):\n", " \"\"\"Accuracy if you escalate the k questions the GP did worst on, for every k from 0 to n.\n", "\n", " This is an ORACLE curve -- it knows in advance which questions the GP fails. A real router\n", " does not. Plotting your measured cascade against it shows exactly how much self-awareness\n", " the small model is missing, which is the entire question behind routing.\n", " \"\"\"\n", " ids = list(gp_frame.index)\n", " order = sorted(ids, key=lambda q: (gp_frame.loc[q, \"full_credit\"], gp_frame.loc[q, \"coverage\"]))\n", " xs, ys = [], []\n", " for k in range(len(ids) + 1):\n", " escalated = set(order[:k])\n", " acc = sum(sp_frame.loc[q, \"full_credit\"] if q in escalated\n", " else gp_frame.loc[q, \"full_credit\"] for q in ids) / len(ids)\n", " xs.append(k / len(ids) * 100)\n", " ys.append(float(acc) * 100)\n", " return xs, ys, order\n", "\n", "\n", "def route_fig(frame, cascade_frame):\n", " \"\"\"Fig 2 -- accuracy you can buy, as a share of questions escalated to the expensive tier.\n", "\n", " The x-axis is a SHARE, not dollars, so the chart still says something when every model\n", " costs $0: it is the fraction of a paid budget you would spend, and it becomes literal\n", " dollars the moment you add a frontier model in Step 6.\n", " \"\"\"\n", " models = list(frame[\"model\"].unique())\n", " if len(models) < 2:\n", " return None\n", " gp_id, sp_id = models[0], models[1]\n", " gp_f = frame[frame[\"model\"] == gp_id].set_index(\"question\")\n", " sp_f = frame[frame[\"model\"] == sp_id].set_index(\"question\")\n", " xs, ys, _ = routing_curve(gp_f, sp_f)\n", " gp_acc = float(gp_f[\"full_credit\"].mean()) * 100\n", " sp_acc = float(sp_f[\"full_credit\"].mean()) * 100\n", "\n", " fig = go.Figure()\n", " fig.add_trace(go.Scatter(\n", " x=xs, y=ys, mode=\"lines+markers\", name=\"oracle routing (what-if: escalate the worst first)\",\n", " line=dict(color=\"#1D5FA8\", width=2.5), marker=dict(size=9, color=\"#1D5FA8\"),\n", " hovertemplate=\"escalate %{x:.0f}% of questions
accuracy %{y:.0f}%\"))\n", "\n", " fig.add_hline(y=gp_acc, line=dict(color=\"#AE1800\", width=1, dash=\"dot\"),\n", " annotation_text=f\"GP alone {gp_acc:.0f}% - escalate nothing, $0\",\n", " annotation_position=\"bottom right\", annotation_font_color=\"#AE1800\",\n", " annotation_font_size=11)\n", " fig.add_hline(y=sp_acc, line=dict(color=\"#1D5FA8\", width=1, dash=\"dot\"),\n", " annotation_text=f\"specialist alone {sp_acc:.0f}% - escalate everything\",\n", " annotation_position=\"top right\", annotation_font_color=\"#1D5FA8\",\n", " annotation_font_size=11)\n", "\n", " for i, mid in enumerate(models[2:], start=2):\n", " sub = frame[frame[\"model\"] == mid]\n", " colour, _sym = SERIES[i % len(SERIES)]\n", " fig.add_hline(y=float(sub[\"full_credit\"].mean()) * 100,\n", " line=dict(color=colour, width=1, dash=\"dash\"),\n", " annotation_text=(f\"{short_name(mid)} \"\n", " f\"{float(sub['full_credit'].mean()) * 100:.0f}% \"\n", " f\"(${float(sub['cost_usd'].sum()):.4f} total)\"),\n", " annotation_position=\"bottom left\",\n", " annotation_font_color=colour, annotation_font_size=11)\n", "\n", " measured = None\n", " if cascade_frame is not None and not cascade_frame.empty:\n", " mx = float(cascade_frame[\"escalated\"].mean()) * 100\n", " my = float(cascade_frame[\"full_credit\"].mean()) * 100\n", " measured = (mx, my)\n", " fig.add_trace(go.Scatter(\n", " x=[mx], y=[my], mode=\"markers+text\",\n", " name=\"measured cascade (GP escalates only what it flags)\",\n", " marker=dict(color=\"#AE1800\", size=15, symbol=\"diamond\",\n", " line=dict(width=1.5, color=\"#FFFFFF\")),\n", " text=[f\"{my:.0f}% at {mx:.0f}% escalated\"], textposition=\"top center\",\n", " textfont=dict(size=11, color=\"#AE1800\"),\n", " hovertemplate=(\"measured cascade
escalated %{x:.0f}% of questions\"\n", " \"
accuracy %{y:.0f}%\")))\n", "\n", " reach = next((x for x, y in zip(xs, ys) if y >= sp_acc - 1e-9), None)\n", " headline = (f\"escalating only {reach:.0f}% of questions already reaches the specialist's \"\n", " f\"{sp_acc:.0f}%\" if reach is not None and reach < 100\n", " else f\"no partial-escalation point reaches the specialist's {sp_acc:.0f}%\")\n", "\n", " sub_line = (f\"oracle curve over n = {len(xs) - 1} questions · x = share of questions \"\n", " f\"sent to the expensive tier = share of a paid budget spent\")\n", " if measured:\n", " best_at_or_below = max(y for x, y in zip(xs, ys) if x <= measured[0] + 1e-9)\n", " sub_line += (\"
the diamond sits below the line - the GP did not flag all of \"\n", " \"its own mistakes, so real routing loses accuracy the oracle never would\"\n", " if measured[1] < best_at_or_below - 1e-9 else\n", " \"
the diamond sits on the line - the GP flagged its own mistakes, so \"\n", " \"routing is as good as knowing the answers in advance\")\n", "\n", " fig.update_layout(\n", " title=dict(text=f\"What accuracy can you buy?
{headline} · {sub_line}\",\n", " font=dict(size=17)),\n", " xaxis=dict(title=\"share of questions escalated to the specialist (%)\",\n", " range=[-4, 104], dtick=20, gridcolor=GRID, zeroline=False),\n", " yaxis=dict(title=\"full_credit accuracy (%)\", range=[0, 110], dtick=20,\n", " gridcolor=GRID, zeroline=False),\n", " height=440, margin=dict(l=10, r=10, t=98, b=10),\n", " legend=dict(orientation=\"h\", y=-0.24, x=0), font=FONT,\n", " plot_bgcolor=GROUND, paper_bgcolor=GROUND)\n", " return fig\n", "\n", "\n", "fig_gap = gap_fig(results)\n", "fig_route = route_fig(results, cascade)\n", "board = build_scorecard(results)\n", "per_q = results[[\n", " \"question\", \"domain\", \"model\", \"coverage\", \"full_credit\",\n", " \"seconds\", \"completion_tokens\", \"cost_usd\"\n", "]]\n", "\n", "# Describe what the keyword scorer actually measures.\n", "fig_gap.update_layout(\n", " title_text=fig_gap.layout.title.text.replace(\n", " \"required clinical facts present\",\n", " \"required keyword groups matched\"\n", " )\n", ")\n", "\n", "# Show the actual cascade result.\n", "if fig_route is not None:\n", " fig_route.update_layout(\n", " title_text=(\n", " \"Routing replay using saved answers\"\n", " f\"
Actual rule: {int(cascade['escalated'].sum())}/\"\n", " f\"{len(cascade)} questions escalated; \"\n", " f\"{cascade['full_credit'].mean():.0%} rubric full credit\"\n", " \"
Oracle curve: a what-if that knows the rubric scores in advance\"\n", " ),\n", " xaxis_title=\"Questions escalated to specialist (%)\",\n", " yaxis_title=\"Full credit under keyword rubric (%)\"\n", " )\n", "\n", "print(board[VIEW].to_string(index=False))\n", "fig_gap.show()\n", "if fig_route is not None:\n", " fig_route.show()" ] }, { "cell_type": "code", "execution_count": null, "outputs": [], "metadata": { "id": "ship-machinery" }, "id": "ship-machinery", "source": [ "#@title Ship it · publish your static Space under your own name ($0, ~30 s)\n", "# ══════════════════════════════════════════════════════════════════════════════\n", "# From the Bonus notebook. A STATIC Space means Hugging Face serves one folder\n", "# (index.html + README.md + data files) exactly as-is: free for everyone, no\n", "# build, no waiting period, never sleeps, no card on file. Re-running this cell\n", "# updates the same repos in place -- safe to run as often as you like.\n", "#\n", "# REUSE: publish_dataset() + publish_space() are the last two lines of every lab.\n", "# ══════════════════════════════════════════════════════════════════════════════\n", "import html, json, os, re, shutil, datetime\n", "try:\n", " from huggingface_hub import create_repo, upload_folder, add_collection_item, HfApi\n", "except ImportError:\n", " get_ipython().run_line_magic(\"pip\", \"install -q huggingface_hub\")\n", " from huggingface_hub import create_repo, upload_folder, add_collection_item, HfApi\n", "\n", "class StopHere(Exception):\n", " pass\n", "def _show_stop(shell, etype, value, tb, tb_offset=None):\n", " print(\"\\n\" + \"=\" * 64 + \"\\n STOP: \" + str(value) + \"\\n\" + \"=\" * 64)\n", "get_ipython().set_custom_exc((StopHere,), _show_stop)\n", "def stop(message):\n", " \"\"\"Halt this cell with one boxed sentence (the fix) instead of a wall of red.\"\"\"\n", " raise StopHere(message)\n", "\n", "# Publishing is OPTIONAL for this lab (Great Reset, 2026-09-13): the notebook submitted to\n", "# Canvas earns full credit on its own. Only Weeks 12, 13 and the capstone require a Space.\n", "# Without a Write token this cell still BUILDS your page under ./site, so you can open or\n", "# download the artifact -- it just does not upload anything anywhere.\n", "PUBLISHING = bool(READY_TO_PUBLISH)\n", "if not PUBLISHING:\n", " print(\"=\" * 66)\n", " print(\" SHOWCASE SKIPPED -- publishing is optional for this lab.\")\n", " print(\" Submitting the notebook on Canvas earns full credit; nothing is missing.\")\n", " print(\" To also publish a public page under your own name, add HF_TOKEN (type Write)\")\n", " print(\" to the 🔑 sidebar, re-run the plumbing cell, then re-run this one.\")\n", " print(\"=\" * 66)\n", "\n", "def publish_dataset(folder, repo_id):\n", " \"\"\"Create or update a public dataset repo with the contents of `folder`. Returns its URL.\"\"\"\n", " create_repo(repo_id, repo_type=\"dataset\", exist_ok=True, private=False, token=HF_TOKEN)\n", " upload_folder(folder_path=folder, repo_id=repo_id, repo_type=\"dataset\", token=HF_TOKEN)\n", " return f\"https://huggingface.co/datasets/{repo_id}\"\n", "\n", "def publish_space(folder, repo_id):\n", " \"\"\"Create or update a public STATIC Space from a folder with index.html + README.md. Returns its URL.\"\"\"\n", " create_repo(repo_id, repo_type=\"space\", space_sdk=\"static\", exist_ok=True, private=False, token=HF_TOKEN)\n", " upload_folder(folder_path=folder, repo_id=repo_id, repo_type=\"space\", token=HF_TOKEN)\n", " return f\"https://huggingface.co/spaces/{repo_id}\"\n", "\n", "def space_readme(title, body=\"\"):\n", " \"\"\"The YAML header every Space README needs (without it the page will not build), plus your text.\"\"\"\n", " return (f\"---\\ntitle: {json.dumps(title[:80])}\\nemoji: 🩺\\ncolorFrom: red\\ncolorTo: gray\\n\"\n", " f\"sdk: static\\npinned: false\\n---\\n{body}\")\n", "\n", "GALLERY = \"MIS-752/mis-752-ai-and-healthcare-6a722c1d93cc292188dd34f2\" # the class gallery\n", "def add_to_gallery(repo_id, kind, note=\"\"):\n", " \"\"\"List your personal Space/dataset in the class gallery (MIS-752 members; optional, not graded).\"\"\"\n", " try:\n", " add_collection_item(GALLERY, item_id=repo_id, item_type=kind, note=note, exists_ok=True, token=HF_TOKEN)\n", " return f\"https://huggingface.co/collections/{GALLERY}\"\n", " except Exception as err:\n", " print(f\" (gallery add skipped: {type(err).__name__} -- usually 'not an MIS-752 member yet'. \"\n", " \"Optional; no effect on your grade.)\")\n", " return None\n", "\n", "def space_status(repo_id):\n", " \"\"\"What HF says a Space is doing: RUNNING, SLEEPING, BUILD_ERROR, ...\"\"\"\n", " try:\n", " runtime = getattr(HfApi(token=HF_TOKEN).space_info(repo_id), \"runtime\", None)\n", " return (getattr(runtime, \"stage\", None) or \"?\") if runtime else \"?\"\n", " except Exception:\n", " return \"?\"\n", "\n", "REPORT = {} # every publish step adds a line; the report cell prints it all\n", "\n", "PAGE_CSS = \"\"\"\n", ":root{--ground:#FAF7F0;--panel:#EDE6DA;--ink:#201E1D;--muted:#666666;--accent:#AE1800;--line:#B9B7B2}\n", "*{box-sizing:border-box}\n", "body{margin:0;background:var(--ground);color:var(--ink);font-family:Archivo,Helvetica,Arial,sans-serif;line-height:1.5}\n", "main{max-width:920px;margin:0 auto;padding:56px 24px 64px}\n", ".eyebrow{font-size:12px;font-weight:700;letter-spacing:.22em;text-transform:uppercase;color:var(--accent)}\n", "h1{font-family:Newsreader,Georgia,serif;font-weight:400;font-size:42px;line-height:1.12;margin:10px 0 14px}\n", ".by{color:var(--muted);font-size:14px;margin-bottom:32px}\n", "h2{font-family:Newsreader,Georgia,serif;font-weight:400;font-size:25px;margin:34px 0 8px}\n", ".panel{background:var(--panel);border-top:3px solid var(--ink);padding:16px 20px;white-space:pre-wrap;\n", " font-family:ui-monospace,Menlo,monospace;font-size:15px;line-height:1.7}\n", ".prose{background:var(--panel);border-top:3px solid var(--ink);padding:16px 20px;white-space:pre-wrap}\n", ".note{color:var(--muted);font-size:13px;margin:0 0 8px}\n", ".chart{border-top:3px solid var(--ink);padding-top:8px}\n", ".grid{display:grid;grid-template-columns:repeat(auto-fit,minmax(260px,1fr));gap:14px;margin-top:10px}\n", "footer{margin-top:28px;padding-top:14px;border-top:1px solid var(--line);color:var(--muted);font-size:13px}\n", "footer a{color:#1D5FA8}\n", "@media(max-width:640px){h1{font-size:30px}}\n", "\"\"\"\n", "\n", "def page_shell(eyebrow, title, byline, body_html, data_url=\"\"):\n", " \"\"\"The course-styled page around your content. `body_html` is ready-made fragments\n", " (h2 + div.panel / div.chart / ...). Returns one complete HTML string.\"\"\"\n", " esc = html.escape\n", " data_line = (f'Record and tables: {esc(data_url)} · '\n", " if data_url else \"\")\n", " return f\"\"\"\n", "\n", "{esc(title)}\n", "\n", "\n", "\n", "
\n", "
{esc(eyebrow)}
\n", "

{esc(title)}

\n", "
{byline}
\n", "{body_html}\n", "\n", "
\n", "\"\"\"\n" ] }, { "cell_type": "code", "execution_count": null, "outputs": [], "metadata": { "id": "ship-content" }, "id": "ship-content", "source": [ "# ── Build the page and the evidence from THIS lab's measurements. Uploading is optional. ──\n", "NAME = \"lab04-gp-vs-specialist\"\n", "OWNER = HF_USER or \"your-hf-username\" # no token? the page still builds, it just is not uploaded\n", "DATA_ID = f\"{OWNER}/{NAME}\"\n", "SPACE_ID = f\"{OWNER}/{NAME}\"\n", "PAGE_TITLE = \"GP versus the Specialist: a measured model-tier scorecard\"\n", "\n", "os.makedirs(\"out\", exist_ok=True)\n", "\n", "# Evidence, bounded. The five questions are self-contained vignettes -- no PHI, no patient\n", "# data -- but the answers are truncated so a runaway generation cannot bloat the repo.\n", "evidence = results.copy()\n", "evidence[\"answer\"] = evidence[\"answer\"].astype(str).str.slice(0, 4000)\n", "evidence[\"missed\"] = evidence[\"missed\"].apply(lambda v: \"|\".join(map(str, v)))\n", "evidence[\"unsafe_claims\"] = evidence[\"unsafe_claims\"].apply(lambda v: \"|\".join(map(str, v)))\n", "evidence[\"bonus_hits\"] = evidence[\"bonus_hits\"].apply(lambda v: \"|\".join(map(str, v)))\n", "evidence.to_csv(\"out/answers.csv\", index=False)\n", "board.to_csv(\"out/scorecard.csv\", index=False)\n", "per_q.to_csv(\"out/per_question.csv\", index=False)\n", "if not cascade.empty:\n", " cascade.to_csv(\"out/cascade.csv\", index=False)\n", "pd.DataFrame(SWEEP).to_csv(\"out/partA_temperature_sweep.csv\", index=False)\n", "\n", "kpis = (\n", " f\"PART A - one 494M-parameter model, one prompt, three temperatures\\n\"\n", " f\" local model : {LOCAL_MODEL} on {DEVICE}\\n\"\n", " f\" temperatures : \" + \", \".join(str(r[\"temperature\"]) for r in SWEEP) + \"\\n\"\n", " f\" tokens/second : \" + \", \".join(str(r[\"tokens_per_sec\"]) for r in SWEEP) + \"\\n\"\n", " f\"\\n\"\n", " f\"PART B - the same 5 clinical questions, {board['model'].nunique()} model tiers\\n\"\n", " f\" questions asked: {len(QUESTIONS)} (measured answers: {len(results)})\\n\"\n", " f\" total API cost : ${float(board['cost_per_q'].mul(board['n']).sum()):.6f}\\n\"\n", " f\"\\n\"\n", " + \"\\n\".join(\n", " f\" {r['model']}\\n\"\n", " f\" full_credit {r['full_credit']:.2f} · coverage {r['coverage']:.2f} · \"\n", " f\"unsafe {r['unsafe_claims']}\\n\"\n", " f\" {r['mean_seconds']:.1f} s mean · {r['out_tokens']:.0f} out-tokens · \"\n", " f\"${r['cost_per_q']:.6f}/question · grade {r['reading_grade']}\"\n", " for _, r in board.iterrows())\n", ")\n", "if not cascade.empty:\n", " n_esc = int(cascade[\"escalated\"].sum())\n", " kpis += (\n", " f\"\\n\\nROUTING - GP first, escalate what it flags\\n\"\n", " f\" escalated : {n_esc}/{len(cascade)}\\n\"\n", " f\" cascade accuracy: {float(cascade['full_credit'].mean()):.2f}\"\n", " )\n", " if flagged_recall is not None:\n", " kpis += f\"\\n self-doubt recall: {flagged_recall:.2f} <- of the GP's wrong answers\"\n", "\n", "gap_html = fig_gap.to_html(full_html=False, include_plotlyjs=\"cdn\")\n", "# the second chart reuses the plotly.js the first one already loaded -- keeps the page small\n", "route_html = (fig_route.to_html(full_html=False, include_plotlyjs=False)\n", " if fig_route is not None else \"\")\n", "table_md = board[VIEW].to_string(index=False)\n", "body = (\n", " '

The 30-second read

' + html.escape(kpis) + \"
\"\n", " + \"

Where the gap lives

\"\n", " + '

One dot per model per question, joined by a connector, sorted so the '\n", " \"biggest divergence is on top. The number at the right of each row is the specialist \"\n", " \"minus the GP; tie means they scored the same. Read the ties as the questions \"\n", " \"you should not pay for the big model on.

\"\n", " + '
' + gap_html + \"
\"\n", " + ((\"

What accuracy can you buy?

\"\n", " '

The blue line is an oracle what-if: it escalates the questions '\n", " \"the GP actually did worst on, so it is allowed to know the future. The red diamond is \"\n", " \"what really happened when the GP escalated only what it flagged as uncertain. The \"\n", " \"vertical distance between them is what the small model's missing self-awareness costs \"\n", " \"you. With five questions the curve is coarse by construction — add questions in \"\n", " \"the Explore step for a smoother one.

\"\n", " '
' + route_html + \"
\") if route_html else \"\")\n", " + '

The scorecard

'\n", " + '

Latency, tokens, cost and reading grade live here rather than in a '\n", " \"chart: with two or three models a table is easier to read than bars.

\"\n", " + '
' + html.escape(table_md) + \"
\"\n", " + '

How this was graded

'\n", " + html.escape(\n", " \"Every score comes from a rule you can read: `must_include` groups of synonyms \"\n", " \"(any word in a group counts) and `must_not_include` statements that would be flat \"\n", " \"wrong. coverage is partial credit across groups; full_credit requires every group \"\n", " \"and no banned statement. Cost is computed from prices fetched live from \"\n", " \"OpenRouter's /api/v1/models at run time, never hard-coded. Latency is wall-clock \"\n", " \"around each call. Reading grade is Flesch-Kincaid.\\n\\n\"\n", " \"This is a five-question pilot with a keyword rubric, not a clinical validation. \"\n", " \"Numbers are from one run and will vary; the pattern is the finding, not the digits.\")\n", " + \"
\"\n", ")\n", "page = page_shell(\n", " eyebrow=f\"MIS 752 · Lab 4 · {OWNER}\",\n", " title=PAGE_TITLE,\n", " byline=(\"A small free model against the largest model reachable for $0, on five clinical \"\n", " \"questions through one OpenRouter key — plus a 494M-parameter model run locally \"\n", " \"at three temperatures. No PHI: the questions are self-contained vignettes. \"\n", " \"Measured in Google Colab; numbers are this run's and will vary.\"),\n", " body_html=body,\n", " data_url=f\"https://huggingface.co/datasets/{DATA_ID}\",\n", ")\n", "os.makedirs(\"site\", exist_ok=True)\n", "with open(\"site/index.html\", \"w\") as f:\n", " f.write(page)\n", "with open(\"site/README.md\", \"w\") as f:\n", " f.write(space_readme(\n", " PAGE_TITLE,\n", " f\"# {PAGE_TITLE}\\n\\n\"\n", " f\"MIS 752 Lab 4. A measured comparison of model tiers on five clinical questions \"\n", " f\"(oncology staging, cardiology guideline, drug-drug interaction, rare-disease \"\n", " f\"workup, ontology mapping) called through OpenRouter, plus a local \"\n", " f\"`{LOCAL_MODEL}` temperature sweep. `data/` holds the per-answer evidence, the \"\n", " f\"scorecard, the per-question coverage table and the routing cascade.\\n\\n\"\n", " f\"**Your ~150-word takeaways go here** -- edit this README on huggingface.co. \"\n", " f\"Cover: the full_credit gap between tiers, which single question the small model \"\n", " f\"handled as well as the big one, what cost_per_correct would have to be before you \"\n", " f\"would pay for the frontier model, and whether the GP flagged enough of its own \"\n", " f\"mistakes for routing to be safe.\\n\"))\n", "shutil.copytree(\"out\", \"site/data\", dirs_exist_ok=True)\n", "\n", "if PUBLISHING:\n", " DATA_URL = publish_dataset(\"out\", DATA_ID)\n", " SPACE_URL = publish_space(\"site\", SPACE_ID)\n", " print(f\"Dataset, under your name: {DATA_URL}\")\n", " print(f\"Static Space (the page): {SPACE_URL}\")\n", " print(\"Open it and give it a minute the first time. Re-running this cell updates both.\")\n", "else:\n", " DATA_URL = SPACE_URL = None\n", " print()\n", " print(\"Built locally, uploaded nowhere:\")\n", " print(\" ./site/index.html <- open it in a browser, or download it from the Files tab\")\n", " print(\" ./out/*.csv <- your measured evidence. KEEP answers.csv for Week 7.\")\n", " print()\n", " print(\"That is a complete lab. Publishing is the optional Showcase; the graded step is\")\n", " print(\"submitting this notebook on Canvas, which the next cell prints the link for.\")\n" ] }, { "cell_type": "code", "execution_count": null, "outputs": [], "metadata": { "id": "ship-report" }, "id": "ship-report", "source": [ "\n", "# ── Your report, and the one REQUIRED step: submit on Canvas ──\n", "PUBLISHED = bool(SPACE_URL)\n", "if PUBLISHED:\n", " REPORT.update({\n", " \"hf_user\": HF_USER, \"dataset\": DATA_URL, \"space\": SPACE_URL,\n", " \"space_status\": space_status(SPACE_ID),\n", " })\n", " GALLERY_URL = add_to_gallery(SPACE_ID, \"space\",\n", " note=f\"{HF_USER}: {PAGE_TITLE[:60]}\")\n", " if DATA_URL:\n", " add_to_gallery(DATA_ID, \"dataset\", note=f\"{HF_USER}: the record\")\n", "else:\n", " REPORT.update({\"hf_user\": None, \"dataset\": None, \"space\": None,\n", " \"space_status\": None})\n", " GALLERY_URL = None\n", "\n", "CANVAS_URL = \"https://unlv.instructure.com/courses/214078/assignments/2721130?module_item_id=8796477\" # this lab's assignment on WebCampus\n", "WIDTH = 66\n", "\n", "\n", "def field(label, value, suffix=\"\"):\n", " \"\"\"Print 'label value', dropping a long value onto its own indented line.\n", "\n", " A URL that runs past the box wraps wherever the terminal decides, and students end up\n", " copying half of it. When it will not fit, give it the whole next line instead.\n", " \"\"\"\n", " pad = f\" {label:<19}\"\n", " value = \"-\" if value is None else str(value)\n", " if len(pad) + len(value) + len(suffix) <= WIDTH:\n", " print(f\"{pad}{value}{suffix}\")\n", " else:\n", " print(pad.rstrip())\n", " print(f\" {value}{suffix}\")\n", "\n", "\n", "print()\n", "print(\"=\" * WIDTH)\n", "print(f\" YOUR REPORT ({datetime.date.today().isoformat()}). If anything above\")\n", "print(\" looks wrong, paste this whole box with your submission.\")\n", "print(\"=\" * WIDTH)\n", "print(\" 1. REQUIRED -- this is what earns the grade.\")\n", "print(\"=\" * WIDTH)\n", "field(\"Submit here\", CANVAS_URL)\n", "print(\" Attach this notebook (.ipynb), run top to bottom,\")\n", "print(\" with the TODOs filled in\")\n", "print(\" Add your ~150-word takeaways in a markdown cell\")\n", "print(\" at the bottom of the notebook\")\n", "print()\n", "print(\"=\" * WIDTH)\n", "print(\" 2. OPTIONAL -- the Showcase. No extra credit; it is portfolio-ready.\")\n", "print(\"=\" * WIDTH)\n", "if PUBLISHED:\n", " field(\"Hugging Face user\", f\"{REPORT['hf_user']} (token: {TOKEN_ROLE}; \"\n", " f\"MIS-752 member: {'yes' if IN_ORG else 'no'})\")\n", " field(\"Dataset\", REPORT[\"dataset\"])\n", " field(\"Space\", REPORT[\"space\"], f\" [{REPORT['space_status']}]\")\n", " field(\"Gallery\", GALLERY_URL or \"not added (optional)\")\n", " print()\n", " print(\" You may add the Space URL to your Canvas submission.\")\n", " print(\" It is never required, and it is never worth extra points.\")\n", "else:\n", " print(\" Skipped -- publishing is OPTIONAL for this lab.\")\n", " print(\" Your notebook is a complete, full-credit submission as it stands.\")\n", "print(\"=\" * WIDTH)\n" ] }, { "cell_type": "markdown", "metadata": { "id": "zAqUg2gx6bo1" }, "source": [ "### What you ship, and what \"done\" looks like\n", "\n", "**Two steps, and only the second one is required.**\n", "\n", "1. **Showcase — optional.** If you added an `HF_TOKEN`, the cell above created two repos in\n", " **your own** Hugging Face account: a dataset (every measured answer with its rubric scores,\n", " the scorecard, the per-question table, the cascade, and the Part A temperature sweep) and a\n", " **static Space** (the page). Open the Space's `README.md` on huggingface.co and replace the\n", " placeholder with your **~150-word takeaways**; one link then carries the page, the evidence\n", " and the story. Add it to the class gallery and it is portfolio material under your own name\n", " that outlives the semester. The MIS-752 organization has no compute and is only an index of\n", " links — no Space GPU or ZeroGPU is used or required, and no card is ever needed.\n", " **If you skipped it, nothing is missing.** Your page and CSVs were still built locally under\n", " `./site/` and `./out/`, so you can open or download them either way.\n", "2. **Submit on Canvas — required.** Attach this notebook, run top to bottom with the TODOs\n", " filled in. That is what earns the credit. The report cell above printed the link.\n", "\n", "**Week 7 needs your `out/answers.csv`.** Keep it whichever path you took. That file is the raw\n", "material you will turn this week's gut reactions into a scored rubric with.\n" ], "id": "zAqUg2gx6bo1" }, { "cell_type": "markdown", "metadata": { "id": "VxivRpyE6bo1" }, "source": [ "---\n", "## Explore your own question\n", "Take the harness somewhere it has not been. Two good moves:\n", "\n", "1. **Your domain.** Replace the five clinical questions with five from your final-project domain — gaming comps, hotel revenue, fraud flags, sports load management. Re-run `run_benchmark` on both models and rebuild the scorecard. Does the small model close the gap on *your* questions, or does it break in the same place?\n", "2. **Your patient.** Pull your representative patient's active conditions and medications from the Synthea tables (`load_synthea(\"conditions\", ...)` from Lab 1) and write two questions a clinician would genuinely ask about that chart. Run them through both models and grade them with your own rubric.\n", "\n", "Write **3-4 sentences**: which model won on your questions, by how much, and whether the gap was worth paying for.\n" ], "id": "VxivRpyE6bo1" }, { "cell_type": "code", "execution_count": 28, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "GP supply questions:\n", " liquid/lfm-2.5-2.6b:free <- supply_reorder coverage 1.00 1.78 s 178 tok $0.000000\n", " liquid/lfm-2.5-2.6b:free <- supply_position coverage 1.00 3.03 s 400 tok $0.000000\n", " liquid/lfm-2.5-2.6b:free <- supply_vendor coverage 1.00 3.85 s 636 tok $0.000000\n", " liquid/lfm-2.5-2.6b:free <- supply_fill coverage 1.00 1.71 s 239 tok $0.000000\n", " liquid/lfm-2.5-2.6b:free <- supply_delivery coverage 1.00 2.58 s 325 tok $0.000000\n", "\n", "Specialist supply questions:\n", " nvidia/nemotron-3-ultra-550b-a55b:free <- supply_reorder coverage 1.00 1.94 s 165 tok $0.000000\n", " nvidia/nemotron-3-ultra-550b-a55b:free <- supply_position coverage 1.00 3.32 s 163 tok $0.000000\n", " nvidia/nemotron-3-ultra-550b-a55b:free <- supply_vendor coverage 1.00 11.51 s 453 tok $0.000000\n", " nvidia/nemotron-3-ultra-550b-a55b:free <- supply_fill coverage 1.00 13.72 s 158 tok $0.000000\n", " nvidia/nemotron-3-ultra-550b-a55b:free <- supply_delivery coverage 1.00 2.95 s 172 tok $0.000000\n", "\n", "Successful calls: 10/10\n", "Failed calls: 0\n", "\n", "SUPPLY SCORECARD: keyword scores, not verified correctness\n", " model n coverage full_credit mean_seconds cost_per_q\n", " liquid/lfm-2.5-2.6b:free 5 1.0 1.0 2.59 0.0\n", "nvidia/nemotron-3-ultra-550b-a55b:free 5 1.0 1.0 6.69 0.0\n" ] }, { "output_type": "display_data", "data": { "text/plain": [ " question tier coverage full_credit seconds missed\n", "0 supply_reorder GP 1.0 1 1.78 []\n", "1 supply_position GP 1.0 1 3.03 []\n", "2 supply_vendor GP 1.0 1 3.85 []\n", "3 supply_fill GP 1.0 1 1.71 []\n", "4 supply_delivery GP 1.0 1 2.58 []\n", "5 supply_reorder Specialist 1.0 1 1.94 []\n", "6 supply_position Specialist 1.0 1 3.32 []\n", "7 supply_vendor Specialist 1.0 1 11.51 []\n", "8 supply_fill Specialist 1.0 1 13.72 []\n", "9 supply_delivery Specialist 1.0 1 2.95 []" ], "text/html": [ "\n", "
\n", "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
questiontiercoveragefull_creditsecondsmissed
0supply_reorderGP1.011.78[]
1supply_positionGP1.013.03[]
2supply_vendorGP1.013.85[]
3supply_fillGP1.011.71[]
4supply_deliveryGP1.012.58[]
5supply_reorderSpecialist1.011.94[]
6supply_positionSpecialist1.013.32[]
7supply_vendorSpecialist1.0111.51[]
8supply_fillSpecialist1.0113.72[]
9supply_deliverySpecialist1.012.95[]
\n", "
\n", "
\n", "\n", "
\n", " \n", "\n", " \n", "\n", " \n", "
\n", "\n", "\n", "
\n", "
\n" ], "application/vnd.google.colaboratory.intrinsic+json": { "type": "dataframe", "summary": "{\n \"name\": \" print(textwrap\",\n \"rows\": 10,\n \"fields\": [\n {\n \"column\": \"question\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"supply_position\",\n \"supply_delivery\",\n \"supply_vendor\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"tier\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 2,\n \"samples\": [\n \"Specialist\",\n \"GP\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"coverage\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 0.0,\n \"min\": 1.0,\n \"max\": 1.0,\n \"num_unique_values\": 1,\n \"samples\": [\n 1.0\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"full_credit\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 0,\n \"min\": 1,\n \"max\": 1,\n \"num_unique_values\": 1,\n \"samples\": [\n 1\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"seconds\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 4.291543234470945,\n \"min\": 1.71,\n \"max\": 13.72,\n \"num_unique_values\": 10,\n \"samples\": [\n 13.72\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"missed\",\n \"properties\": {\n \"dtype\": \"object\",\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}" } }, "metadata": {} }, { "output_type": "stream", "name": "stdout", "text": [ "\n", "SUPPLIER COMPARISON: read both full answers\n", "\n", "GP | Responding model: liquid/lfm-2.5-2.6b:free\n", "**Supplier A Calculation** - Required units: ≥ 850 - Pack size: 50 units - Packs needed:\n", "850 ÷ 50 = 17 packs (exact) - Cost per pack: $70 - Total cost: 17 × $70 = $1,190 -\n", "Delivery fee: $0 - **Total A: $1,190** **Supplier B Calculation** - Required units: ≥ 850\n", "- Pack size: 100 units - Packs needed: 850 ÷ 100 = 8.5 → round up to 9 packs - Cost per\n", "pack: $130 - Subtotal: 9 × $130 = $1,170 - Delivery fee: $25 per order - Total cost:\n", "$1,170 + $25 = $1,195 - **Total B: $1,195** **Comparison** - Supplier A: $1,190 -\n", "Supplier B: $1,195 CHOICE_A\n", "\n", "Specialist | Responding model: nvidia/nemotron-3-ultra-550b-a55b:free\n", "**Supplier A** Packs needed: ⌈850 ÷ 50⌉ = 17 packs Units: 17 × 50 = 850 Cost: 17 ×\n", "$70 = **$1,190** **Supplier B** Packs needed: ⌈850 ÷ 100⌉ = 9 packs Units: 9 × 100 =\n", "900 Cost: (9 × $130) + $25 = $1,170 + $25 = **$1,195** CHOICE_A\n" ] } ], "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 1000 }, "id": "explore-stub", "outputId": "876ff2d4-da35-4ec2-a9b2-5d5458119ea1" }, "id": "explore-stub", "source": [ "# Explore: your own five questions, or questions built from your representative patient.\n", "# Reuse QUESTIONS's shape (id, domain, prompt, must_include, must_not_include, bonus),\n", "# then: my_rows = run_benchmark(SMALL_FREE, questions=MY_QUESTIONS)\n", "\n", "# Your finding (3-4 sentences):\n", "# ...\n", "\n", "import textwrap\n", "\n", "def supply_question(qid, prompt, groups):\n", " return {\n", " \"id\": qid,\n", " \"domain\": \"Hospital supply analysis\",\n", " \"prompt\": (\n", " \"This is a fictional hospital supply exercise. \"\n", " \"Use only the stated facts. Show concise calculations using digits. \"\n", " + prompt\n", " ),\n", " \"must_include\": groups,\n", " \"must_not_include\": [],\n", " \"bonus\": []\n", " }\n", "\n", "MY_QUESTIONS = [\n", " supply_question(\n", " \"supply_reorder\",\n", " \"Demand is 40 units/day, lead time is 6 days, and safety stock is \"\n", " \"80 units. Reorder point equals lead-time demand plus safety stock. \"\n", " \"Report both lead-time demand and the reorder point.\",\n", " [[\"240\"], [\"320\"]]\n", " ),\n", " supply_question(\n", " \"supply_position\",\n", " \"Inventory position equals on-hand plus on-order minus backorders. \"\n", " \"There are 240 units on hand, 150 on order, and 50 backordered. \"\n", " \"Order only when inventory position is at or below 320 units. \"\n", " \"Start with WAIT or ORDER, then give the inventory position and calculation.\",\n", " [[\"340\"], [\"wait\"]]\n", " ),\n", " supply_question(\n", " \"supply_vendor\",\n", " \"You need at least 850 units and must buy whole packs. Supplier A \"\n", " \"sells packs of 50 for $70 with no delivery fee. Supplier B sells \"\n", " \"packs of 100 for $130 plus one $25 delivery fee per order. \"\n", " \"Quality and delivery time are identical. Give each supplier's total \"\n", " \"purchase cost. End with CHOICE_A or CHOICE_B for the cheaper supplier.\",\n", " [[\"1190\", \"1,190\"], [\"1195\", \"1,195\"], [\"choice_a\"]]\n", " ),\n", " supply_question(\n", " \"supply_fill\",\n", " \"Immediate unit fill rate equals units supplied immediately divided \"\n", " \"by units requested. Customers requested 1000 units: 900 were supplied \"\n", " \"immediately, 80 later, and 20 remain outstanding. Report the immediate \"\n", " \"fill rate as a percentage and state the numerator you used.\",\n", " [[\"90%\", \"90 %\", \"90 percent\"], [\"900\"]]\n", " ),\n", " supply_question(\n", " \"supply_delivery\",\n", " \"Supplier A delivered 12 of 15 orders on time; Supplier B delivered \"\n", " \"9 of 10 on time. Calculate the combined on-time delivery percentage \"\n", " \"from total on-time orders divided by total orders. State both totals \"\n", " \"and the percentage, rather than averaging the two supplier percentages.\",\n", " [[\"21\"], [\"25\"], [\"84%\", \"84 %\", \"84 percent\"]]\n", " )\n", "]\n", "\n", "# Verify current zero pricing before making any model calls.\n", "supply_prices, _ = fetch_catalog()\n", "for mid in (SMALL_FREE, BIG_FREE):\n", " if not mid.endswith(\":free\") or supply_prices.get(mid) != (0.0, 0.0):\n", " raise ValueError(f\"Free pricing could not be verified for {mid}.\")\n", "\n", "if not OPENROUTER_API_KEY:\n", " raise ValueError(\"The OpenRouter key is missing.\")\n", "\n", "print(\"GP supply questions:\")\n", "my_gp_rows = run_benchmark(SMALL_FREE, questions=MY_QUESTIONS)\n", "\n", "print(\"\\nSpecialist supply questions:\")\n", "my_sp_rows = run_benchmark(BIG_FREE, questions=MY_QUESTIONS)\n", "\n", "my_results = pd.DataFrame(\n", " [{**row, \"requested_model\": SMALL_FREE} for row in my_gp_rows]\n", " + [{**row, \"requested_model\": BIG_FREE} for row in my_sp_rows]\n", ")\n", "my_results[\"tier\"] = my_results[\"requested_model\"].map({\n", " SMALL_FREE: \"GP\",\n", " BIG_FREE: \"Specialist\"\n", "})\n", "\n", "my_failed = my_results[\"call_error\"].astype(bool)\n", "print(f\"\\nSuccessful calls: {int((~my_failed).sum())}/{len(my_results)}\")\n", "print(f\"Failed calls: {int(my_failed.sum())}\")\n", "\n", "# Exclude failed calls from scoring.\n", "my_valid = my_results.loc[~my_failed].copy()\n", "my_scorecard = (\n", " build_scorecard(my_valid)\n", " if not my_valid.empty\n", " else pd.DataFrame()\n", ")\n", "\n", "if not my_scorecard.empty:\n", " print(\"\\nSUPPLY SCORECARD: keyword scores, not verified correctness\")\n", " print(my_scorecard[[\n", " \"model\", \"n\", \"coverage\", \"full_credit\",\n", " \"mean_seconds\", \"cost_per_q\"\n", " ]].to_string(index=False))\n", "\n", "display(my_results[[\n", " \"question\", \"tier\", \"coverage\",\n", " \"full_credit\", \"seconds\", \"missed\"\n", "]])\n", "\n", "print(\"\\nSUPPLIER COMPARISON: read both full answers\")\n", "for _, row in my_results[\n", " my_results[\"question\"] == \"supply_vendor\"\n", "].iterrows():\n", " print(f\"\\n{row['tier']} | Responding model: {row['model']}\")\n", " if row[\"call_error\"]:\n", " print(\"FAILED:\", row[\"call_error\"])\n", " else:\n", " print(textwrap.fill(row[\"answer\"], width=90))" ] }, { "cell_type": "markdown", "source": [ "On five fictional hospital supply questions connected to my supply systems experience, both models earned 100% under the keyword rubric, so the score gap was zero percentage points. Both correctly chose Supplier A at $1,190 over Supplier B at $1,195 after accounting for whole packs and delivery charges. The small model averaged 2.59 seconds per answer versus 6.69 seconds for the specialist, and both cost $0. I would choose the smaller model for these routine calculations and would not pay extra based on this test, but I would still verify its answers because five examples and keyword matches do not establish overall reliability.\n" ], "metadata": { "id": "yN4J1-3eTZj5" }, "id": "yN4J1-3eTZj5" }, { "cell_type": "markdown", "metadata": { "id": "zlsjtIx26bo1" }, "source": [ "---\n", "## Should you / shouldn't you? (3-4 sentences each)\n", "1. **Stakeholders:** A health system routes patient-portal questions to the cheap model and escalates only what it flags. Who benefits, and who carries the risk when the cheap model is *confidently* wrong and never escalates? Name the person who is harmed and the person who saves the money — they are not the same person, and that is the point.\n", "2. **Failure modes:** Your rubric is keyword-based. Give one way it produces a **false pass** (a fluent answer that contains the right words but is wrong) and one way it produces a **false fail** (a correct answer phrased so the keywords miss). Which of the two is more dangerous in a clinical setting, and what would you change first?\n", "3. **When NOT to build this:** For which of the five tasks would you refuse to deploy *any* current LLM, no matter how it scored here — and why is \"it passed my five questions\" not the same as \"it is safe to use\"? What evidence would you demand before go-live?\n", "\n", "\n", "**Stakeholders:** A health system administrator could benefit from reduced staff workload, while patients could receive faster responses. However, the patient who follows incorrect medication advice carries the health risk, and clinicians must identify and correct mistakes. My GP averaged 3.87 seconds versus 39.62 seconds for the specialist, but both cost $0, so this run did not demonstrate model-fee savings. Because the GP flagged neither of its two rubric failures, I would require clinician review before delivering these clinical answers to patients.\n", "\n", "**Failure modes:** My GP’s warfarin answer earned full credit while incorrectly linking low INR to bleeding, although an INR that is too low can indicate increased clot risk. [MedlinePlus](https://medlineplus.gov/lab-tests/prothrombin-time-test-and-inr-ptinr/) A false fail could occur if an otherwise correct answer says “type II diabetes” while the rubric recognizes only “type 2 diabetes” or “diabetes mellitus type 2.” A false pass is more dangerous because a passing score could encourage someone to trust harmful advice. I would first add expert review and checks against trusted references for contradictions, then expand the accepted wording to reduce false failures.\n", "\n", "**When not to build this:** I would not deploy an LLM to independently give patients drug-interaction advice or change medication doses. The GP’s false pass shows why matching keywords is insufficient, and the specialist’s five rubric passes do not establish clinical safety. Before using either model as a clinician’s assistant, I would require expert evaluation on a larger, representative set of independent test cases, including uncommon and dangerous situations. I would also require reliable drug references, clear responsibility for reviewing answers, and ongoing monitoring with a way to stop the system when problems appear.\n" ], "id": "zlsjtIx26bo1" }, { "cell_type": "markdown", "metadata": { "id": "ahOVaFys6bo2" }, "source": [ "---\n", "## Submit\n", "\n", "**Required — submit on Canvas.** Attach this notebook (`.ipynb`), run top to bottom, with the\n", "TODOs filled in: the Part A temperature sweep, the scorer, the benchmark run, and the cascade.\n", "Write your **~150-word takeaways** in the **✍️ Takeaways** cell directly below this one —\n", "it lists everything to cover. The report cell printed your Canvas link.\n", "\n", "**Optional — the Showcase.** Publish the static Space to your own Hugging Face account and add\n", "the URL to your submission. It goes into the class gallery under your name and stays yours after\n", "the semester. No extra credit; the reward is that it is real, public, and portfolio-ready.\n", "\n", "*Remember: never fabricate results. Report the numbers you actually measured — your latency,\n", "your token counts, your cost. A small model beating the flagship on the ontology question is a\n", "better lab than a tidy chart showing the expensive model winning.*\n" ], "id": "ahOVaFys6bo2" }, { "cell_type": "markdown", "metadata": { "id": "0FKc6xDV6bo2" }, "source": [ "---\n", "\n", "## ✍️ Takeaways — write your answer in the cell below\n", "\n", "**This is the graded write-up.** Roughly 150 words, in your own words, using the numbers\n", "*you* measured. Cover all 5:\n", "\n", "1. **The gap.** The `full_credit` difference between the two tiers, in the numbers you measured.\n", "2. **The exception.** The one question where the small model held its own, and why that task was easy enough for it.\n", "3. **The price of being right.** What `cost_per_correct` would have to be before you would pay for the frontier model instead.\n", "4. **The safety question.** Whether the GP flagged enough of its own mistakes for routing to be safe: quote the calibration recall you measured, and say what you would do about it.\n", "5. **Part A.** At which temperature did the answer become unusable, and what did the model do instead of answering?\n", "\n", "*Never fabricate results. If something failed or you skipped it, say that instead -- a lab that\n", "reports what actually happened is worth more than a tidy story that did not.*\n" ], "id": "0FKc6xDV6bo2" }, { "cell_type": "markdown", "metadata": { "id": "OQSiUlVe6bo2" }, "source": [ "### ✏️ Your answer\n", "\n", "> _Double-click this cell and replace everything below this line with your ~150 words._\n", "\n", "---\n", "\n", "My small model earned full credit on 60% of the five questions, compared with 100% for the specialist, a 40-percentage-point gap. Both passed the cardiology guideline rubric, suggesting that listing standard treatment classes was manageable for the smaller model in this example. Both cost $0; I would pay only if the extra cost per independently verified correct answer was less than the cost of reviewing or correcting mistakes. The cascade stayed at 60% because it escalated nothing and detected 0% of the two rubric failures. I would require external checks and human review instead of trusting the model to admit uncertainty. The drug-interaction answer also passed despite contradictory claims about anticoagulation risk, showing that keyword scores can miss errors. At temperature 1.2, the local model gave an unsupported six-month rule for stopping medication and reached its 120-token limit mid-sentence. I skipped the optional paid comparison.\n", "\n" ], "id": "OQSiUlVe6bo2" }, { "cell_type": "markdown", "metadata": { "id": "0HEE7wok6bo2" }, "source": [ "## Your toolkit: this week's reusable function\n", "\n", "Each week you add one well-documented function to your toolkit. In **Week 9** you build the agent loop that calls tools like this one, and in **Week 13** you assemble them into a working product on one patient chart. An agent-ready function has a clear name, **typed arguments**, a **docstring** that tells the LLM when to use it, and a **string return** the LLM can read.\n", "\n", "This week's tool is the routing decision you just measured: given a clinical question and a budget, say which tier should answer it and why.\n", "\n", "> **Note.** The retired alert-burden lab that used to sit in this week forged `alert_triage()` instead. Alert fatigue moved to **Week 9**, where it is the clinical context for agents — so if you also ran that old notebook, keep both functions. They compose, and an agent can call either.\n" ], "id": "0HEE7wok6bo2" }, { "cell_type": "code", "execution_count": 23, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Route to SPECIALIST (nvidia/nemotron-3-ultra-550b-a55b:free). The GP missed a required\n", "group on ontology mapping. Measured rubric full credit: GP 60%, specialist 100%. Selected\n", "tier averaged 39.62 seconds and USD 0.000000/question, within the USD 0.000000 budget.\n", "Uncertainty-flag recall: 0%. Escalate GP output if an independent check fails, supporting\n", "evidence is missing, or it flags uncertainty. Never rely on uncertainty alone. All\n", "clinical outputs require qualified human review; this five-question pilot does not\n", "establish safety.\n", "\n", "Route to SPECIALIST (nvidia/nemotron-3-ultra-550b-a55b:free). The GP missed a required\n", "group on the rare-disease question. Measured rubric full credit: GP 60%, specialist 100%.\n", "Selected tier averaged 39.62 seconds and USD 0.000000/question, within the USD 0.000000\n", "budget. Uncertainty-flag recall: 0%. Escalate GP output if an independent check fails,\n", "supporting evidence is missing, or it flags uncertainty. Never rely on uncertainty alone.\n", "All clinical outputs require qualified human review; this five-question pilot does not\n", "establish safety.\n", "\n", "Route to GP (liquid/lfm-2.5-2.6b:free). The GP matched all keyword groups on the\n", "cardiology guideline example; use it only for a draft. Measured rubric full credit: GP\n", "60%, specialist 100%. Selected tier averaged 3.87 seconds and USD 0.000000/question,\n", "within the USD 0.000000 budget. Uncertainty-flag recall: 0%. Escalate GP output if an\n", "independent check fails, supporting evidence is missing, or it flags uncertainty. Never\n", "rely on uncertainty alone. All clinical outputs require qualified human review; this five-\n", "question pilot does not establish safety.\n", "\n" ] } ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "toolbox-todo", "outputId": "692acd07-14b7-41bf-ef0b-91a9a0542188" }, "id": "toolbox-todo", "source": [ "# === YOUR TOOLBOX: this week's completed tool ===\n", "import math\n", "import textwrap\n", "\n", "def route_clinical_question(question: str, budget_usd: float = 0.0) -> str:\n", " \"\"\"Recommend a tier for a draft using this lab's saved measurements.\n", "\n", " Use after the scorecard and cascade cells. question is the question to\n", " route; budget_usd is the maximum allowed cost per question.\n", " Returns a routing explanation, measured evidence, and review requirements.\n", " This proposed rule makes no API calls and is not a validated clinical tool.\n", " Measured average cost is historical; check current prices before any call.\n", " \"\"\"\n", " if not isinstance(question, str) or not question.strip():\n", " return \"Enter a nonempty question.\"\n", "\n", " try:\n", " budget = float(budget_usd)\n", " except (TypeError, ValueError):\n", " return \"Budget must be a finite, nonnegative number.\"\n", " if not math.isfinite(budget) or budget < 0:\n", " return \"Budget must be a finite, nonnegative number.\"\n", "\n", " try:\n", " stats = board.set_index(\"model\")\n", " gp = stats.loc[SMALL_FREE]\n", " specialist = stats.loc[BIG_FREE]\n", " recall = flagged_recall\n", " except (NameError, KeyError):\n", " return \"Run the scorecard, cascade, and chart cells first.\"\n", "\n", " text = question.lower()\n", " role, model_id = \"SPECIALIST\", BIG_FREE\n", "\n", " if any(k in text for k in (\"icd\", \"snomed\", \"ontology\", \"mapping\")):\n", " reason = \"The GP missed a required group on ontology mapping.\"\n", " elif any(k in text for k in (\n", " \"rare\", \"fracture\", \"sclera\", \"dentin\", \"osteogenesis\"\n", " )):\n", " reason = \"The GP missed a required group on the rare-disease question.\"\n", " elif any(k in text for k in (\n", " \"interaction\", \"warfarin\", \"fluconazole\", \"inr\"\n", " )):\n", " reason = (\n", " \"Manual review found an error in the GP drug-interaction \"\n", " \"answer despite its rubric pass.\"\n", " )\n", " elif \"four pillars\" in text or \"hfref\" in text:\n", " role, model_id = \"GP\", SMALL_FREE\n", " reason = (\n", " \"The GP matched all keyword groups on the cardiology \"\n", " \"guideline example; use it only for a draft.\"\n", " )\n", " else:\n", " reason = (\n", " \"This question is outside the GP example supported \"\n", " \"by this simple routing rule.\"\n", " )\n", "\n", " chosen = stats.loc[model_id]\n", " cost = float(chosen[\"cost_per_q\"])\n", " if not math.isfinite(cost) or cost < 0:\n", " return \"The selected tier has no valid measured cost.\"\n", " if cost > budget:\n", " return (\n", " f\"{role} exceeds the budget: measured USD {cost:.6f}/question \"\n", " f\"versus USD {budget:.6f} allowed. Use human review.\"\n", " )\n", "\n", " recall_text = \"not measured\" if recall is None else f\"{recall:.0%}\"\n", " return (\n", " f\"Route to {role} ({model_id}). {reason} \"\n", " f\"Measured rubric full credit: GP {gp['full_credit']:.0%}, \"\n", " f\"specialist {specialist['full_credit']:.0%}. \"\n", " f\"Selected tier averaged {chosen['mean_seconds']:.2f} seconds \"\n", " f\"and USD {cost:.6f}/question, within the USD {budget:.6f} budget. \"\n", " f\"Uncertainty-flag recall: {recall_text}. \"\n", " \"Escalate GP output if an independent check fails, supporting evidence \"\n", " \"is missing, or it flags uncertainty. Never rely on uncertainty alone. \"\n", " \"All clinical outputs require qualified human review; this five-question \"\n", " \"pilot does not establish safety.\"\n", " )\n", "\n", "for example in [\n", " \"Map ICD-10-CM E11.9 to its SNOMED CT concept.\",\n", " \"Give the likely diagnosis for a child with recurrent fractures, \"\n", " \"blue sclerae and dentinogenesis imperfecta.\",\n", " \"List the four pillars of HFrEF treatment.\"\n", "]:\n", " print(textwrap.fill(route_clinical_question(example), width=90))\n", " print()\n" ] }, { "cell_type": "code", "execution_count": 21, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Route to GP (liquid/lfm-2.5-2.6b:free). It is free, answers in 3.9 s, and scored full_credit 0.60 on this lab's five questions. Escalate to the specialist if the answer contains an uncertainty marker or names no guideline, code or mechanism.\n", "\n", "Route to GP (liquid/lfm-2.5-2.6b:free). It is free, answers in 3.9 s, and scored full_credit 0.60 on this lab's five questions. Escalate to the specialist if the answer contains an uncertainty marker or names no guideline, code or mechanism.\n", "\n", "Route to SPECIALIST (nvidia/nemotron-3-ultra-550b-a55b:free). It scored full_credit 1.00 vs the GP's 0.60, and this question matches a domain the small model handled worst here. Cost $0.000000, within your $0.0500 budget.\n", "\n", "Cannot route an empty question.\n" ] } ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "toolbox-solution", "outputId": "a3991df4-38ad-4a1d-e9ac-760ae7292b06" }, "id": "toolbox-solution", "source": [ "#@title ✅ Solution (instructor)\n", "def route_clinical_question(question: str, budget_usd: float = 0.0) -> str:\n", " \"\"\"Decide which model tier should answer one clinical question, using this lab's measurements.\n", "\n", " Args:\n", " question: the clinical question to route.\n", " budget_usd: what the caller is willing to spend on THIS question. 0.0 means\n", " free-tier only, which is the default and the only setting this course requires.\n", "\n", " Returns:\n", " A short string naming the chosen tier, the measured reason, and the escalation\n", " trigger an agent should act on.\n", " \"\"\"\n", " if not question or not question.strip():\n", " return \"Cannot route an empty question.\"\n", "\n", " try:\n", " budget = float(budget_usd)\n", " except (TypeError, ValueError):\n", " return f\"budget_usd must be a number, got {budget_usd!r}. Assuming $0.\"\n", "\n", " gp_id = resolve(SMALL_FREE, CATALOG)\n", " sp_id = resolve(BIG_FREE, CATALOG)\n", " rows = {r[\"model\"]: r for _, r in board.iterrows()}\n", " gp_stats, sp_stats = rows.get(gp_id), rows.get(sp_id)\n", "\n", " # Hard tasks, learned from this lab's per-question breakdown rather than guessed.\n", " HARD = [\"rare\", \"staging\", \"differential\", \"workup\", \"interaction\", \"unknown\",\n", " \"atypical\", \"pediatric\", \"pregnan\", \"oncolog\"]\n", " t = _normalize(question)\n", " looks_hard = any(k in t for k in HARD)\n", "\n", " if gp_stats is None or sp_stats is None:\n", " return (\"Cannot route: this lab's scorecard is missing one of the two free tiers. \"\n", " \"Run Step 5 first.\")\n", "\n", " if looks_hard and budget >= float(sp_stats[\"cost_per_q\"]):\n", " return (f\"Route to SPECIALIST ({sp_id}). It scored full_credit \"\n", " f\"{sp_stats['full_credit']:.2f} vs the GP's {gp_stats['full_credit']:.2f}, \"\n", " f\"and this question matches a domain the small model handled worst here. \"\n", " f\"Cost ${sp_stats['cost_per_q']:.6f}, within your ${budget:.4f} budget.\")\n", "\n", " escalate = (\"Escalate to the specialist if the answer contains an uncertainty marker \"\n", " \"or names no guideline, code or mechanism.\")\n", " if gp_stats[\"cost_per_q\"] == 0:\n", " return (f\"Route to GP ({gp_id}). It is free, answers in \"\n", " f\"{gp_stats['mean_seconds']:.1f} s, and scored full_credit \"\n", " f\"{gp_stats['full_credit']:.2f} on this lab's five questions. {escalate}\")\n", " return (f\"Route to GP ({gp_id}) at ${gp_stats['cost_per_q']:.6f} per question. \"\n", " f\"{escalate}\")\n", "\n", "\n", "print(route_clinical_question(\"Map ICD-10-CM E11.9 to its SNOMED CT concept.\"))\n", "print()\n", "print(route_clinical_question(\"Give the likely diagnosis for a child with recurrent \"\n", " \"fractures, blue sclerae and dentinogenesis imperfecta.\"))\n", "print()\n", "print(route_clinical_question(\"Same rare-disease question, but the caller will pay.\",\n", " budget_usd=0.05))\n", "print()\n", "print(route_clinical_question(\"\"))\n" ] }, { "cell_type": "code", "source": [ "from pathlib import Path\n", "from google.colab import files\n", "import json\n", "import shutil\n", "\n", "backup_dir = Path(\"out\")\n", "backup_dir.mkdir(exist_ok=True)\n", "\n", "# Save complete answers and measured results.\n", "tables = {\n", " \"answers.csv\": results,\n", " \"scorecard.csv\": board,\n", " \"cascade.csv\": cascade,\n", " \"supply_answers.csv\": my_results,\n", " \"supply_scorecard.csv\": my_scorecard,\n", " \"temperature_answers.csv\": pd.DataFrame(SWEEP),\n", "}\n", "\n", "for filename, table in tables.items():\n", " table.to_csv(backup_dir / filename, index=False)\n", "\n", "# Keep the questions, rubrics, and original cached responses.\n", "details = {\n", " \"clinical_questions\": QUESTIONS,\n", " \"supply_questions\": MY_QUESTIONS,\n", " \"system_prompt\": SYSTEM,\n", "}\n", "(backup_dir / \"questions.json\").write_text(\n", " json.dumps(details, indent=2), encoding=\"utf-8\"\n", ")\n", "(backup_dir / \"response_cache.json\").write_text(\n", " json.dumps(CACHE, indent=2), encoding=\"utf-8\"\n", ")\n", "\n", "archive = shutil.make_archive(\n", " \"Lab04_JamesSeegel_results\", \"zip\",\n", " root_dir=\".\", base_dir=\"out\"\n", ")\n", "files.download(archive)" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 17 }, "id": "XdNydL2lTt_m", "outputId": "aa56e643-d6f4-47c0-c1ff-4207c748b1d8" }, "id": "XdNydL2lTt_m", "execution_count": 29, "outputs": [ { "output_type": "display_data", "data": { "text/plain": [ "" ], "application/javascript": [ "\n", " async function download(id, filename, size) {\n", " if (!google.colab.kernel.accessAllowed) {\n", " return;\n", " }\n", " const div = document.createElement('div');\n", " const label = document.createElement('label');\n", " label.textContent = `Downloading \"${filename}\": `;\n", " div.appendChild(label);\n", " const progress = document.createElement('progress');\n", " progress.max = size;\n", " div.appendChild(progress);\n", " document.body.appendChild(div);\n", "\n", " const buffers = [];\n", " let downloaded = 0;\n", "\n", " const channel = await google.colab.kernel.comms.open(id);\n", " // Send a message to notify the kernel that we're ready.\n", " channel.send({})\n", "\n", " for await (const message of channel.messages) {\n", " // Send a message to notify the kernel that we're ready.\n", " channel.send({})\n", " if (message.buffers) {\n", " for (const buffer of message.buffers) {\n", " buffers.push(buffer);\n", " downloaded += buffer.byteLength;\n", " progress.value = downloaded;\n", " }\n", " }\n", " }\n", " const blob = new Blob(buffers, {type: 'application/binary'});\n", " const a = document.createElement('a');\n", " a.href = window.URL.createObjectURL(blob);\n", " a.download = filename;\n", " div.appendChild(a);\n", " a.click();\n", " div.remove();\n", " }\n", " " ] }, "metadata": {} }, { "output_type": "display_data", "data": { "text/plain": [ "" ], "application/javascript": [ "download(\"download_e42545c4-83fb-4c0e-88ae-fe1e553371fe\", \"Lab04_JamesSeegel_results.zip\", 17027)" ] }, "metadata": {} } ] }, { "cell_type": "markdown", "source": [ "**Detailed AI Use Note**\n", "\n", "I used ChatGPT/Codex to help write, debug, and explain the temperature sweep, rubric scorer, benchmark, cascade, routing function, and supply-question extension, and to draft the Takeaways and reflection responses. Verification included executing the code in Colab and reviewing my actual outputs with the assistant to check the reported scores, timings, and costs. The assistant also consulted MedlinePlus to assess the warfarin response and identified an incorrect clinical claim that the keyword rubric had passed. Changes made with AI assistance included correcting the cascade cost calculation to count both calls when escalation occurs and revising a chart caption that misrepresented the 0% escalation result. These checks support the notebook’s calculations but do not constitute clinical validation. My own work included setting up the accounts and secrets, running the notebook, and collecting the outputs; the code and written interpretation received substantial AI assistance.\n", "\n", "During final review, the assistant added a missing import and removed an unused starter function.\n" ], "metadata": { "id": "ez8REHu6P5e_" }, "id": "ez8REHu6P5e_" } ], "metadata": { "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "name": "python" }, "colab": { "provenance": [], "gpuType": "T4" }, "accelerator": "GPU", "widgets": { "application/vnd.jupyter.widget-state+json": { "version_major": 2, "version_minor": 0, "state": { "14ae1ae44c8843c5aa6e7c7426bbe66a": { "model_module": "@jupyter-widgets/controls", "model_name": "HBoxModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": [ "IPY_MODEL_365edc29756c4d059681216dd0e228af", "IPY_MODEL_a5cf830e16e34166ab1039c6d95c7b9a", "IPY_MODEL_a5e1af8b40fa4c00ae47268313e5d8e4" ], "layout": "IPY_MODEL_d5dd60825b5f47729d8cfaebaeaeae7e" } }, "365edc29756c4d059681216dd0e228af": { "model_module": "@jupyter-widgets/controls", "model_name": "HTMLModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_f3fe1a7178404f2ca8fe5450ed023287", "placeholder": "​", "style": "IPY_MODEL_6c3ebee4fd58483bbf33f942bf96cad7", "value": "config.json: 100%" } }, "a5cf830e16e34166ab1039c6d95c7b9a": { "model_module": "@jupyter-widgets/controls", "model_name": "FloatProgressModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_48c05adbf94f4b0ea500b8f30801950e", "max": 659, "min": 0, "orientation": "horizontal", "style": "IPY_MODEL_67e37e03d8694c5ca2404e836e26a3ec", "value": 659 } }, "a5e1af8b40fa4c00ae47268313e5d8e4": { "model_module": "@jupyter-widgets/controls", "model_name": "HTMLModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_804ee39717a641049b4d765fd65fc952", "placeholder": "​", "style": "IPY_MODEL_4dea6652b52f458b91faf91b437d31fd", "value": " 659/659 [00:00<00:00, 55.1kB/s]" } }, "d5dd60825b5f47729d8cfaebaeaeae7e": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "f3fe1a7178404f2ca8fe5450ed023287": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "6c3ebee4fd58483bbf33f942bf96cad7": { "model_module": "@jupyter-widgets/controls", "model_name": "DescriptionStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "48c05adbf94f4b0ea500b8f30801950e": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "67e37e03d8694c5ca2404e836e26a3ec": { "model_module": "@jupyter-widgets/controls", "model_name": "ProgressStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": "" } }, "804ee39717a641049b4d765fd65fc952": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "4dea6652b52f458b91faf91b437d31fd": { "model_module": "@jupyter-widgets/controls", "model_name": "DescriptionStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "47613d31b75c425e9c22f0fc872896f8": { "model_module": "@jupyter-widgets/controls", "model_name": "HBoxModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": [ "IPY_MODEL_a856ac3727da47b89ecdd4c65dabbb63", "IPY_MODEL_f16df99115644b219a98b3c9a74da21c", "IPY_MODEL_bff60327d5e4423fa3c6de2cc80edb20" ], "layout": "IPY_MODEL_03b470286b72416b85a2c3541782a83d" } }, "a856ac3727da47b89ecdd4c65dabbb63": { "model_module": "@jupyter-widgets/controls", "model_name": "HTMLModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_b628a0dbbfb841b786cd8cdf4fe53dbc", "placeholder": "​", "style": "IPY_MODEL_c8ffe547715440108a5590e78e16b726", "value": "tokenizer_config.json: 100%" } }, "f16df99115644b219a98b3c9a74da21c": { "model_module": "@jupyter-widgets/controls", "model_name": "FloatProgressModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_acc3881b550a46619bbe6698f88a2075", "max": 7305, "min": 0, "orientation": "horizontal", "style": "IPY_MODEL_fc703aff83184e1997ba18f057957002", "value": 7305 } }, "bff60327d5e4423fa3c6de2cc80edb20": { "model_module": "@jupyter-widgets/controls", "model_name": "HTMLModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_bfdc0508159c423aa49253992d1fffdf", "placeholder": "​", "style": "IPY_MODEL_80f96aeaf92d4161a09a6b7fbfe4c25e", "value": " 7.30k/7.30k [00:00<00:00, 574kB/s]" } }, "03b470286b72416b85a2c3541782a83d": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "b628a0dbbfb841b786cd8cdf4fe53dbc": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "c8ffe547715440108a5590e78e16b726": { "model_module": "@jupyter-widgets/controls", "model_name": "DescriptionStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "acc3881b550a46619bbe6698f88a2075": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "fc703aff83184e1997ba18f057957002": { "model_module": "@jupyter-widgets/controls", "model_name": "ProgressStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": "" } }, "bfdc0508159c423aa49253992d1fffdf": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "80f96aeaf92d4161a09a6b7fbfe4c25e": { "model_module": "@jupyter-widgets/controls", "model_name": "DescriptionStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "4ad4c52a24ea42cd9dc39293533c8c2d": { "model_module": "@jupyter-widgets/controls", "model_name": "HBoxModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": [ "IPY_MODEL_78307ea109614e2eb233c41860f7f388", "IPY_MODEL_93311d74b7e94fb4b7172c5ee77913ae", "IPY_MODEL_dfdfcbc9a475467384a4bd8ff96b8794" ], "layout": "IPY_MODEL_527dbde06b0e43c281cab3e4643a3fbd" } }, "78307ea109614e2eb233c41860f7f388": { "model_module": "@jupyter-widgets/controls", "model_name": "HTMLModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_8629cd0f118b44dca08dd3b67dc21052", "placeholder": "​", "style": "IPY_MODEL_5698fc8411c84a5cbd026ee99d00312f", "value": "vocab.json: 100%" } }, "93311d74b7e94fb4b7172c5ee77913ae": { "model_module": "@jupyter-widgets/controls", "model_name": "FloatProgressModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_3829df6203044da2826a00b9965d61bb", "max": 2776833, "min": 0, "orientation": "horizontal", "style": "IPY_MODEL_dc517b7521504041af329ca5a0094a72", "value": 2776833 } }, "dfdfcbc9a475467384a4bd8ff96b8794": { "model_module": "@jupyter-widgets/controls", "model_name": "HTMLModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_9f0cd4acce6041538d6e8f16fbb22b38", "placeholder": "​", "style": "IPY_MODEL_5bf9dade1c2a40c5897bf423ca9e612f", "value": " 2.78M/2.78M [00:00<00:00, 15.2MB/s]" } }, "527dbde06b0e43c281cab3e4643a3fbd": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "8629cd0f118b44dca08dd3b67dc21052": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "5698fc8411c84a5cbd026ee99d00312f": { "model_module": "@jupyter-widgets/controls", "model_name": "DescriptionStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "3829df6203044da2826a00b9965d61bb": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "dc517b7521504041af329ca5a0094a72": { "model_module": "@jupyter-widgets/controls", "model_name": "ProgressStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": "" } }, "9f0cd4acce6041538d6e8f16fbb22b38": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "5bf9dade1c2a40c5897bf423ca9e612f": { "model_module": "@jupyter-widgets/controls", "model_name": "DescriptionStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "579e80b559904e378801bd0e06269a68": { "model_module": "@jupyter-widgets/controls", "model_name": "HBoxModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": [ "IPY_MODEL_cc128c09252847c68ff43a7b8e927acc", "IPY_MODEL_1009c3edb4cb493284c623a9c792d385", "IPY_MODEL_de39d51228b54e22a24931b7759d16ca" ], "layout": "IPY_MODEL_7e0bd1697f4e41e3801349de36056166" } }, "cc128c09252847c68ff43a7b8e927acc": { "model_module": "@jupyter-widgets/controls", "model_name": "HTMLModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_49244c7fafce4303a5ea51e944295381", "placeholder": "​", "style": "IPY_MODEL_cbc1247555be4853b53b1170af0f3110", "value": "merges.txt: 100%" } }, "1009c3edb4cb493284c623a9c792d385": { "model_module": "@jupyter-widgets/controls", "model_name": "FloatProgressModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_e2fc3ea808e34726bc047c7fd689648d", "max": 1671839, "min": 0, "orientation": "horizontal", "style": "IPY_MODEL_cca6e2eabfc04557803ddf2ed11b288f", "value": 1671839 } }, "de39d51228b54e22a24931b7759d16ca": { "model_module": "@jupyter-widgets/controls", "model_name": "HTMLModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_0f2d1e01def5449890f716d64b85bf32", "placeholder": "​", "style": "IPY_MODEL_eb8d5059cd1141fc8afa507c1e14554f", "value": " 1.67M/1.67M [00:00<00:00, 13.9MB/s]" } }, "7e0bd1697f4e41e3801349de36056166": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "49244c7fafce4303a5ea51e944295381": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "cbc1247555be4853b53b1170af0f3110": { "model_module": "@jupyter-widgets/controls", "model_name": "DescriptionStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "e2fc3ea808e34726bc047c7fd689648d": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "cca6e2eabfc04557803ddf2ed11b288f": { "model_module": "@jupyter-widgets/controls", "model_name": "ProgressStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": "" } }, "0f2d1e01def5449890f716d64b85bf32": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "eb8d5059cd1141fc8afa507c1e14554f": { "model_module": "@jupyter-widgets/controls", "model_name": "DescriptionStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "342e0355b55b4411a46d664bc708f7b6": { "model_module": "@jupyter-widgets/controls", "model_name": "HBoxModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": [ "IPY_MODEL_700c7bfb9b7c4ae39da064b3d02afad5", "IPY_MODEL_04f164867b3b40a4a2c3ca59c44c7a81", "IPY_MODEL_7d5220c963fe44c7b50cdfe3f694d33a" ], "layout": "IPY_MODEL_eaf63da702d74700839bf1eada444ad6" } }, "700c7bfb9b7c4ae39da064b3d02afad5": { "model_module": "@jupyter-widgets/controls", "model_name": "HTMLModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_acee6442e8bc49c3afcf7d70f296859c", "placeholder": "​", "style": "IPY_MODEL_608fa28a29184351bf0e2f60787c3ce4", "value": "tokenizer.json: 100%" } }, "04f164867b3b40a4a2c3ca59c44c7a81": { "model_module": "@jupyter-widgets/controls", "model_name": "FloatProgressModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_ab1e59d0fa954d90a7ff213f882ea0b4", "max": 7031645, "min": 0, "orientation": "horizontal", "style": "IPY_MODEL_53ff90eff414432095fd35be00c41261", "value": 7031645 } }, "7d5220c963fe44c7b50cdfe3f694d33a": { "model_module": "@jupyter-widgets/controls", "model_name": "HTMLModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_8cda64727bc34bd7aceacb361175a75e", "placeholder": "​", "style": "IPY_MODEL_4a6800a389eb4abfaa846497bed48dd4", "value": " 7.03M/7.03M [00:00<00:00, 22.7MB/s]" } }, "eaf63da702d74700839bf1eada444ad6": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "acee6442e8bc49c3afcf7d70f296859c": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "608fa28a29184351bf0e2f60787c3ce4": { "model_module": "@jupyter-widgets/controls", "model_name": "DescriptionStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "ab1e59d0fa954d90a7ff213f882ea0b4": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "53ff90eff414432095fd35be00c41261": { "model_module": "@jupyter-widgets/controls", "model_name": "ProgressStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": "" } }, "8cda64727bc34bd7aceacb361175a75e": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "4a6800a389eb4abfaa846497bed48dd4": { "model_module": "@jupyter-widgets/controls", "model_name": "DescriptionStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "af8e23a6fd8445029e2f7f31903e1f49": { "model_module": "@jupyter-widgets/controls", "model_name": "HBoxModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": [ "IPY_MODEL_2400448814704af794f7a3d381a9993f", "IPY_MODEL_b833945825c94a0d96545f7fb121e4e6", "IPY_MODEL_852666caaaa4422dbb14b48e562b7546" ], "layout": "IPY_MODEL_d020a4d6e07640adbc674643bbd3d6ff" } }, "2400448814704af794f7a3d381a9993f": { "model_module": "@jupyter-widgets/controls", "model_name": "HTMLModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_35e57d77722f4b2989d4457f209dd36b", "placeholder": "​", "style": "IPY_MODEL_d54b2317108c478299c6fa2966fc335f", "value": "model.safetensors: reconstructing file: 100%" } }, "b833945825c94a0d96545f7fb121e4e6": { "model_module": "@jupyter-widgets/controls", "model_name": "FloatProgressModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_d2e1d9eb82f742ee8381ddfc72728420", "max": 988097824, "min": 0, "orientation": "horizontal", "style": "IPY_MODEL_d7e48cd579a64532ad0c52e55dd82d7d", "value": 988097824 } }, "852666caaaa4422dbb14b48e562b7546": { "model_module": "@jupyter-widgets/controls", "model_name": "HTMLModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_b83e1e259d5f4351ba2ce27bee4bffdc", "placeholder": "​", "style": "IPY_MODEL_c169f64cd9434ecd9261471af93630f2", "value": "  988MB /  988MB, 77.2MB/s  " } }, "d020a4d6e07640adbc674643bbd3d6ff": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "35e57d77722f4b2989d4457f209dd36b": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "d54b2317108c478299c6fa2966fc335f": { "model_module": "@jupyter-widgets/controls", "model_name": "DescriptionStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "d2e1d9eb82f742ee8381ddfc72728420": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "d7e48cd579a64532ad0c52e55dd82d7d": { "model_module": "@jupyter-widgets/controls", "model_name": "ProgressStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": "" } }, "b83e1e259d5f4351ba2ce27bee4bffdc": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "c169f64cd9434ecd9261471af93630f2": { "model_module": "@jupyter-widgets/controls", "model_name": "DescriptionStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "74b46ad8b3524e9cad5c6f7cd0624678": { "model_module": "@jupyter-widgets/controls", "model_name": "HBoxModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": [ "IPY_MODEL_6438b348052a4fb0a8a89ab88bb62112", "IPY_MODEL_ca2394adcc2c46ccbf885926e26f7198", "IPY_MODEL_f8bf1f6e020d485f9ea69e9ab0edd493" ], "layout": "IPY_MODEL_b3030089355e40f1989953e2de319f16" } }, "6438b348052a4fb0a8a89ab88bb62112": { "model_module": "@jupyter-widgets/controls", "model_name": "HTMLModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_ce6beedc78bd416c93e858cd56a17aec", "placeholder": "​", "style": "IPY_MODEL_8ed3cace020946e485fcdf1535929359", "value": "model.safetensors: downloading bytes: " } }, "ca2394adcc2c46ccbf885926e26f7198": { "model_module": "@jupyter-widgets/controls", "model_name": "FloatProgressModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_4df73a6b6d5b41fe9c2a001a814cd4bf", "max": 988097824, "min": 0, "orientation": "horizontal", "style": "IPY_MODEL_7af103492187425da0b1e8f12a07e5ac", "value": 854045206 } }, "f8bf1f6e020d485f9ea69e9ab0edd493": { "model_module": "@jupyter-widgets/controls", "model_name": "HTMLModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_d11781048332434384d2e025a98e6855", "placeholder": "​", "style": "IPY_MODEL_5bd11971ed2e4f09a34807afecb53283", "value": "  854MB, 70.3MB/s  " } }, "b3030089355e40f1989953e2de319f16": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "ce6beedc78bd416c93e858cd56a17aec": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "8ed3cace020946e485fcdf1535929359": { "model_module": "@jupyter-widgets/controls", "model_name": "DescriptionStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "4df73a6b6d5b41fe9c2a001a814cd4bf": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "7af103492187425da0b1e8f12a07e5ac": { "model_module": "@jupyter-widgets/controls", "model_name": "ProgressStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": "" } }, "d11781048332434384d2e025a98e6855": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "5bd11971ed2e4f09a34807afecb53283": { "model_module": "@jupyter-widgets/controls", "model_name": "DescriptionStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "798737fc0ed9441f80d1fa4767a915b9": { "model_module": "@jupyter-widgets/controls", "model_name": "HBoxModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": [ "IPY_MODEL_d03461a2b3eb4c0d8ee7b836d6a5e134", "IPY_MODEL_294e56678d6d420e90bc4f7aacdf913a", "IPY_MODEL_310925f09eca48ee9324f1c21fe1a054" ], "layout": "IPY_MODEL_61cd181a0e18485380e84ede462c31d7" } }, "d03461a2b3eb4c0d8ee7b836d6a5e134": { "model_module": "@jupyter-widgets/controls", "model_name": "HTMLModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_b8a805660a61471f93c4906eaa919b3c", "placeholder": "​", "style": "IPY_MODEL_a82d8d7df9dc46bab58c8a962dfe509b", "value": "Loading weights: 100%" } }, "294e56678d6d420e90bc4f7aacdf913a": { "model_module": "@jupyter-widgets/controls", "model_name": "FloatProgressModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_7cf88f6643944be3bb527907a4f5cfbc", "max": 290, "min": 0, "orientation": "horizontal", "style": "IPY_MODEL_13c6e29898b540109207697093a5e55b", "value": 290 } }, "310925f09eca48ee9324f1c21fe1a054": { "model_module": "@jupyter-widgets/controls", "model_name": "HTMLModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_c9ff680473e54541bcbf8c22c305b888", "placeholder": "​", "style": "IPY_MODEL_fa985ed758434211a28aa55df212a917", "value": " 290/290 [00:00<00:00, 3113.54it/s]" } }, "61cd181a0e18485380e84ede462c31d7": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "b8a805660a61471f93c4906eaa919b3c": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "a82d8d7df9dc46bab58c8a962dfe509b": { "model_module": "@jupyter-widgets/controls", "model_name": "DescriptionStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "7cf88f6643944be3bb527907a4f5cfbc": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "13c6e29898b540109207697093a5e55b": { "model_module": "@jupyter-widgets/controls", "model_name": "ProgressStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": "" } }, "c9ff680473e54541bcbf8c22c305b888": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "fa985ed758434211a28aa55df212a917": { "model_module": "@jupyter-widgets/controls", "model_name": "DescriptionStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "f7a7343fdb8349439644ac621f0a110d": { "model_module": "@jupyter-widgets/controls", "model_name": "HBoxModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": [ "IPY_MODEL_98f0ff17c6b04f548c7bfb65cfd37d0a", "IPY_MODEL_5b3d47823a3940aa8cc6629f225e6498", "IPY_MODEL_8dc6c19a524a48d09c3a9025c167f891" ], "layout": "IPY_MODEL_daeecd42bb9a4096b106d9d8d5c8376c" } }, "98f0ff17c6b04f548c7bfb65cfd37d0a": { "model_module": "@jupyter-widgets/controls", "model_name": "HTMLModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_89820b22311f496fabbe55e38c73f300", "placeholder": "​", "style": "IPY_MODEL_66e449d1752f45cda7084265bedc19c6", "value": "generation_config.json: 100%" } }, "5b3d47823a3940aa8cc6629f225e6498": { "model_module": "@jupyter-widgets/controls", "model_name": "FloatProgressModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_bf809f98267c4d2dade4d95a3841381f", "max": 242, "min": 0, "orientation": "horizontal", "style": "IPY_MODEL_bf4d38f68d444d38934ecb5bf9a1b73e", "value": 242 } }, "8dc6c19a524a48d09c3a9025c167f891": { "model_module": "@jupyter-widgets/controls", "model_name": "HTMLModel", "model_module_version": "1.5.0", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_b97a03e7ae224ba4b8693e2723bf6ebd", "placeholder": "​", "style": "IPY_MODEL_23cb2b429f0444f49c1a60c9be86ce54", "value": " 242/242 [00:00<00:00, 20.0kB/s]" } }, "daeecd42bb9a4096b106d9d8d5c8376c": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "89820b22311f496fabbe55e38c73f300": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "66e449d1752f45cda7084265bedc19c6": { "model_module": "@jupyter-widgets/controls", "model_name": "DescriptionStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "bf809f98267c4d2dade4d95a3841381f": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "bf4d38f68d444d38934ecb5bf9a1b73e": { "model_module": "@jupyter-widgets/controls", "model_name": "ProgressStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": "" } }, "b97a03e7ae224ba4b8693e2723bf6ebd": { "model_module": "@jupyter-widgets/base", "model_name": "LayoutModel", "model_module_version": "1.2.0", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "23cb2b429f0444f49c1a60c9be86ce54": { "model_module": "@jupyter-widgets/controls", "model_name": "DescriptionStyleModel", "model_module_version": "1.5.0", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } } } } } }, "nbformat": 4, "nbformat_minor": 5 }