# Week 12: Small models and model routing

Applied AI Course — Sawan Ruparel

Public self-study edition. See https://applied-ai.sawanruparel.com/updates/ for version notes.

# The Small Model Strategy

## Right-Sizing AI for Production

Week 12 — Applied AI Course, Spring 2026

---

*Not every nail needs a frontier-model hammer.*
*Learn when a 3B parameter model outperforms — and out-earns — a trillion-parameter one.*

---

# The Cost Reality

## Frontier Model Pricing (per 1M tokens, as of early 2026)

- **GPT-4.1**: $2.00 input / $8.00 output
- **Claude Sonnet 4**: $3.00 input / $15.00 output
- **Claude Opus 4**: $15.00 input / $75.00 output
- **Gemini 2.5 Pro**: $1.25–$10.00 input / $2.50–$30.00 output
- **o3-pro**: $20.00 input / $80.00 output

## Small Model Pricing

- **GPT-4.1 mini**: $0.40 input / $1.60 output
- **GPT-4.1 nano**: $0.10 input / $0.40 output
- **Claude Haiku 3.5**: $0.80 input / $4.00 output
- **Gemini 2.0 Flash**: $0.10 input / $0.40 output
- **Self-hosted Llama 3.2 3B**: ~$0.02–0.05/M tokens (compute only)

## The Math

A pipeline processing 100M tokens/month:
- With Claude Opus 4: **~$4,500/month**
- With GPT-4.1 nano: **~$25/month**
- With self-hosted Llama 3.2 3B: **~$3/month**

That is a **180x–1,500x cost difference** for tasks where small models perform comparably.

---

# When Small Models Win

## Cost Efficiency
- High-volume, low-complexity tasks (classification, extraction, reformatting)
- Batch processing millions of documents where per-token cost dominates
- Startups and projects with tight budgets

## Latency
- Small models respond in 10–50ms vs 500ms–5s for frontier models
- Real-time applications: autocomplete, inline suggestions, chat
- Time-to-first-token matters for user experience

## Privacy and Compliance
- On-premise deployment keeps data off third-party servers
- HIPAA, GDPR, SOC2 compliance without data processing agreements
- Air-gapped environments in defense, finance, healthcare

## Edge and On-Device
- Mobile apps, IoT devices, embedded systems
- Offline capability — no network dependency
- Browser-based inference with WebGPU/WebLLM

## Specialized Tasks
- Fine-tuned small models often beat general-purpose large models on narrow tasks
- Structured output (JSON, SQL, code) with constrained generation
- Domain-specific extraction where the task is well-defined

---

# The Model Size Spectrum

```
 Parameters    Examples                    Typical Use Cases              Hardware
 ──────────    ────────                    ─────────────────              ────────

   1B          Llama 3.2 1B               Simple classification,         Phone, RPi
               Qwen 2.5 0.5B             text cleanup, routing          1–2 GB RAM
                    │
                    ▼
   3B          Llama 3.2 3B               Extraction, summarization,     Laptop CPU
               Phi-4 mini                 basic Q&A, translation         4 GB RAM
                    │
                    ▼
   7–8B        Llama 3.1 8B               General assistant, code gen,   Single GPU
               Qwen 2.5 7B               RAG, tool use                  8–16 GB VRAM
               Gemma 3 4B/12B
                    │
                    ▼
  13–14B       Qwen 2.5 14B              Complex reasoning, long-form   Single GPU
               Phi-4 14B                 generation, multi-step tasks   16–24 GB VRAM
                    │
                    ▼
  30–70B       Llama 3.3 70B             Near-frontier quality,         Multi-GPU
               Qwen 2.5 72B             complex analysis               40–80 GB VRAM
                    │
                    ▼
 Frontier      GPT-4.1, Claude Opus 4,   Hardest reasoning, creative    API only /
               Gemini 2.5 Pro            writing, novel problems        Data center
```

**Key insight:** The right model is the *smallest* one that meets your quality threshold.

## Sources

- [Introducing Meta Llama 3 (Meta, 2024)](https://ai.meta.com/blog/meta-llama-3/)
- [Qwen2.5: A Party of Foundation Models (Alibaba, 2024)](https://qwenlm.github.io/blog/qwen2.5/)
- [Introducing Phi-3: Redefining What's Possible with SLMs (Microsoft, 2024)](https://azure.microsoft.com/en-us/blog/introducing-phi-3-redefining-whats-possible-with-slms/)
- [Phi-4 Technical Report — arXiv:2412.08905 (Microsoft, 2024)](https://arxiv.org/abs/2412.08905)
- [Gemma 3 (Google, 2025)](https://blog.google/technology/developers/gemma-3/)

---

# Meta Llama 3.2 and Llama 4

## Llama 3.2 (Sept 2024) — The Open-Weight Foundation

- **1B and 3B** text models — first truly capable sub-4B models from Meta
- **11B and 90B** vision-language multimodal variants
- 128K context window across all sizes
- Trained on 9T tokens — massive data investment even for small models
- Optimized for on-device: Qualcomm, MediaTek partnerships at launch

## Llama 3.3 70B (Dec 2024)

- Matches Llama 3.1 405B performance at a fraction of the compute
- Demonstrates that distillation from larger models is highly effective

## Llama 4 (April 2025)

- **Scout**: 17B active params, 16 experts (109B total), 10M context window. Meta's release describes a single H100 with Int4 quantization. The 17B active count is not the full weight-memory requirement.
- **Maverick**: 17B active params, 128 experts (400B total)
- Mixture-of-Experts architecture — only activates a subset per token
- Early-fusion native multimodality (text, image, video)

## Why It Matters for Small Model Strategy

- Llama models use version-specific Community Licenses and acceptable-use terms. Review the actual model license for the intended use; do not treat it as Apache 2.0.
- Massive ecosystem: fine-tuning tools, quantized variants, deployment infra
- The 1B and 3B models run comfortably on a smartphone

## Sources

- [Llama 3.2: Revolutionizing Edge AI and Vision (Meta, 2024)](https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/)
- [The Llama 3 Herd of Models — arXiv:2407.21783 (Meta, 2024)](https://arxiv.org/abs/2407.21783)
- [The Llama 4 Herd: Natively Multimodal AI Innovation (Meta, 2025)](https://ai.meta.com/blog/llama-4-multimodal-intelligence/)
- [Llama model family (Meta, Hugging Face)](https://huggingface.co/meta-llama)
- [Llama 4 Community License](https://github.com/meta-llama/llama-models/blob/main/models/llama4/LICENSE)

---

# Qwen 2.5 and Qwen 3

## Qwen 2.5 (Sept 2024) — The Quiet Powerhouse

- Full size range: **0.5B, 1.5B, 3B, 7B, 14B, 32B, 72B**
- Best-in-class multilingual support (29+ languages)
- Exceptionally strong on math (GSM8K, MATH) and coding (HumanEval)
- Qwen 2.5 7B rivals Llama 3.1 70B on several benchmarks
- Apache 2.0 license for models up to 72B

## Qwen 2.5 Coder

- Dedicated coding variants at 1.5B, 7B, 14B, 32B
- Qwen 2.5 Coder 32B matches GPT-4o on code generation tasks
- Supports 90+ programming languages

## Qwen 3 / 3.6 (2025–2026)

- Hybrid thinking models: can toggle between "thinking" and "non-thinking" modes
- Sizes from **0.6B to 235B** (MoE with 22B active)
- Qwen 3 8B matches Qwen 2.5 72B on reasoning benchmarks
- **Qwen 3.6 27B (May 2026)** hits **77.2% on SWE-bench**, a new high-water mark for an open small model
- 128K context with YaRN-based extension to 1M tokens
- Apache 2.0 license across all sizes

## Why Qwen Matters

- Strongest small models for non-English languages
- Math and code performance punches well above weight class
- Fully open weights — no usage restrictions

## Sources

- [Qwen2.5: A Party of Foundation Models (Alibaba, 2024)](https://qwenlm.github.io/blog/qwen2.5/)
- [Qwen2.5 Technical Report — arXiv:2412.15115 (Qwen Team, 2024)](https://arxiv.org/abs/2412.15115)
- [Qwen2.5-Coder Technical Report — arXiv:2409.12186 (Qwen Team, 2024)](https://arxiv.org/abs/2409.12186)
- [Qwen3: Think Deeper, Act Faster (Alibaba, 2025)](https://qwenlm.github.io/blog/qwen3/)
- [Qwen3 Technical Report — arXiv:2505.09388 (Qwen Team, 2025)](https://arxiv.org/abs/2505.09388)
- [Qwen3 (QwenLM, GitHub)](https://github.com/QwenLM/Qwen3)

---

# Microsoft Phi-3 and Phi-4

## The "Textbook Quality" Thesis

- Core insight: data quality matters more than data quantity at small scale
- Phi models trained on heavily curated, synthetic, and textbook-style data
- Result: dramatically better performance per parameter than expected

## Phi-3 (April 2024)

- **Phi-3 mini (3.8B)**: rivals Mixtral 8x7B and GPT-3.5 on many benchmarks
- **Phi-3 small (7B)** and **Phi-3 medium (14B)**: competitive with much larger models
- 128K context window support
- Optimized for ONNX Runtime — fast inference on CPU and mobile

## Phi-4 (Dec 2024 – Feb 2025)

- **Phi-4 (14B)**: outperforms GPT-4o on MATH and GPQA benchmarks
- **Phi-4 mini (3.8B)**: a benchmark-beating edge model — runs at 15–20 tok/s on an M1 MacBook Air at Q4_K_M, matches GPT-4o on structured-extraction benchmarks, and fits on entry-level hardware
- Heavy use of synthetic data generated by larger models
- Particularly strong at STEM reasoning and structured tasks

## Trade-offs

- Narrower training data means weaker on open-ended creative tasks
- Less robust on adversarial/safety benchmarks compared to Llama/Qwen
- Microsoft Research license — check terms for commercial use
- Best suited for STEM, structured extraction, and reasoning pipelines

## Sources

- [Introducing Phi-3: Redefining What's Possible with SLMs (Microsoft, 2024)](https://azure.microsoft.com/en-us/blog/introducing-phi-3-redefining-whats-possible-with-slms/)
- [Phi-3 Technical Report — arXiv:2404.14219 (Microsoft, 2024)](https://arxiv.org/abs/2404.14219)
- [Introducing Phi-4: Microsoft's Newest Small Language Model (Microsoft, 2024)](https://techcommunity.microsoft.com/blog/azure-ai-foundry-blog/introducing-phi-4-microsoft%E2%80%99s-newest-small-language-model-specializing-in-comple/4357090)
- [Phi-4 Technical Report — arXiv:2412.08905 (Microsoft, 2024)](https://arxiv.org/abs/2412.08905)
- [Small Language Models: Phi-4 & Gemma On-Device 2026 (MasterPrompting)](https://masterprompting.net/blog/small-language-models-phi4-gemma-on-device-2026)

---

# Mistral and Mixtral

## Mistral 7B (Sept 2023) — The Model That Started the SLM Revolution

- Outperformed Llama 2 13B on every benchmark at half the size
- Introduced Sliding Window Attention (SWA) for efficient long contexts
- Grouped Query Attention (GQA) for faster inference
- Apache 2.0 license — fully permissive

## Mixtral 8x7B (Dec 2023) — Mixture of Experts Goes Mainstream

- 8 expert networks, 2 active per token — 46.7B total, ~13B active
- Matches or beats Llama 2 70B while using 6x less compute per token
- Proved MoE is practical for production deployment

## Mistral Small / Nemo (2024–2026)

- **Mistral Nemo 12B**: collaboration with NVIDIA, strong general-purpose model
- **Mistral Small 3.1 24B**: optimized for fast API-served inference
- **Mistral Small 4 (March 16, 2026)**: a hybrid model that unifies instruct, reasoning, multimodal, and coding capabilities in a single checkpoint
- Function calling and JSON mode built in
- Tekken tokenizer: more efficient for multilingual and code

## MoE Architecture — Why It Matters

- Total parameters determine knowledge capacity (what the model knows)
- Active parameters determine inference cost (what you pay per token)
- MoE gives you big-model knowledge at small-model cost
- Trade-off: higher memory footprint (all experts must be loaded)

## Sources

- [Mistral 7B (Mistral AI, 2023)](https://mistral.ai/news/announcing-mistral-7b)
- [Mistral 7B — arXiv:2310.06825 (Jiang et al., 2023)](https://arxiv.org/abs/2310.06825)
- [Mixtral of Experts (Mistral AI, 2023)](https://mistral.ai/news/mixtral-of-experts)
- [Mixtral of Experts — arXiv:2401.04088 (Jiang et al., 2024)](https://arxiv.org/abs/2401.04088)
- [Mistral NeMo (Mistral AI & NVIDIA, 2024)](https://mistral.ai/news/mistral-nemo)
- [Mistral Small 3.1 (Mistral AI, 2025)](https://mistral.ai/news/mistral-small-3-1)
- [Mistral Small (latest) Model Stats (LLM-Stats)](https://llm-stats.com/models/mistral-small-latest)

---

# Google Gemma 2 and Gemma 3

## Gemma 2 (June 2024)

- Sizes: **2B, 9B, 27B**
- Knowledge distillation from larger Gemini models
- 9B model rivals Llama 3 8B and Mistral 7B across benchmarks
- Optimized for TPU, GPU, and CPU inference

## Gemma 3 (March 2025) — Built for On-Device

- Sizes: **1B, 4B, 12B, 27B**
- Native multimodal: text + image + short video understanding
- 128K context window (up from 8K in Gemma 2)
- **ShieldGemma** built-in safety classifier
- Runs on a single NVIDIA H100, RTX 3090, or even TPU v5e

## Key Differentiators

- Tight integration with Google ecosystem: Vertex AI, Colab, Android
- Gemma 3 4B with quantization runs on Pixel phones
- Strong structured output and function calling support
- "Gemma terms of use" license — permissive for most commercial use

## Performance Highlights

- Gemma 3 27B competes with Llama 3.3 70B on reasoning tasks
- Gemma 3 12B beats Qwen 2.5 14B on multimodal understanding
- Designed from the ground up for efficient on-device inference

## Sources

- [Gemma 2: Improving Open Language Models at a Practical Size — arXiv:2408.00118 (Google DeepMind, 2024)](https://arxiv.org/abs/2408.00118)
- [Google Launches Gemma 2 (Google, 2024)](https://blog.google/technology/developers/google-gemma-2/)
- [Gemma 3: Google's New Open Model Based on Gemini 2.0 (Google, 2025)](https://blog.google/technology/developers/gemma-3/)
- [Gemma 3 Technical Report (Google DeepMind, 2025)](https://arxiv.org/abs/2503.19786)

---

# Head-to-Head: Small Model Benchmarks

## ~3B Parameter Class

| Benchmark        | Llama 3.2 3B | Qwen 2.5 3B | Phi-4 mini 3.8B | Gemma 3 4B |
|------------------|:------------:|:------------:|:----------------:|:----------:|
| MMLU (5-shot)    | 63.4         | 65.6         | 68.6             | 64.2       |
| GSM8K (math)     | 77.7         | 79.1         | 84.8             | 73.5       |
| HumanEval (code) | 44.5         | 49.4         | 52.1             | 42.7       |
| ARC-C (reasoning)| 78.6         | 77.7         | 80.2             | 76.5       |
| GPQA (science)   | 28.1         | 30.5         | 36.4             | 29.8       |

## ~7–14B Parameter Class

| Benchmark        | Llama 3.1 8B | Qwen 2.5 7B | Qwen 3 8B | Phi-4 14B | Gemma 3 12B |
|------------------|:------------:|:------------:|:----------:|:---------:|:-----------:|
| MMLU (5-shot)    | 73.0         | 74.2         | 80.1       | 84.8      | 79.4        |
| GSM8K (math)     | 84.5         | 85.7         | 92.3       | 95.2      | 86.1        |
| HumanEval (code) | 62.2         | 66.5         | 72.3       | 73.8      | 64.3        |
| ARC-C (reasoning)| 83.4         | 84.1         | 87.5       | 89.2      | 85.8        |
| GPQA (science)   | 32.8         | 35.2         | 43.1       | 52.6      | 40.7        |

*Scores are approximate composites from published benchmarks and independent evaluations. Always validate on your own data.*

## Key Takeaway

The gap between 7B models and frontier models has narrowed dramatically. On focused tasks, the difference can be negligible.

## Sources

- [MMLU: Measuring Massive Multitask Language Understanding — arXiv:2009.03300 (Hendrycks et al., 2020)](https://arxiv.org/abs/2009.03300)
- [HumanEval: Evaluating Large Language Models Trained on Code — arXiv:2107.03374 (Chen et al., 2021)](https://arxiv.org/abs/2107.03374)
- [GSM8K: Training Verifiers to Solve Math Word Problems — arXiv:2110.14168 (Cobbe et al., 2021)](https://arxiv.org/abs/2110.14168)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark — arXiv:2311.12022 (Rein et al., 2023)](https://arxiv.org/abs/2311.12022)

---

# When Small Beats Big

## The Counterintuitive Results

Fine-tuned small models regularly outperform general-purpose frontier models on specific tasks. This is not a fluke — it is a repeatable pattern.

## Documented Examples

- **Medical NER**: Fine-tuned Llama 3.1 8B outperformed GPT-4 on clinical entity extraction by 8% F1 (Microsoft Research, 2024)
- **Legal clause classification**: Fine-tuned Phi-3 mini beat GPT-4o by 12% accuracy on contract review tasks
- **SQL generation**: Fine-tuned Qwen 2.5 Coder 7B matched GPT-4.1 on Spider benchmark while costing 100x less
- **Sentiment analysis**: A fine-tuned 3B model achieves 95%+ accuracy — no frontier model needed
- **Structured extraction**: Constrained decoding with a 7B model produces valid JSON 99.9% of the time

## Why Does This Happen?

1. **Task narrowness**: Small models have enough capacity for focused tasks
2. **Fine-tuning efficiency**: Adapting all parameters of a small model beats prompting a large one
3. **Output constraints**: Structured tasks reduce the generation space — smaller models excel
4. **Data quality**: Domain-specific fine-tuning data teaches exactly what is needed

## The Rule of Thumb

If your task can be described in a 1-page annotation guide and you have 1,000+ labeled examples, a fine-tuned 7B model will likely match frontier performance.

## Sources

- [The Llama 3 Herd of Models — arXiv:2407.21783 (Meta, 2024)](https://arxiv.org/abs/2407.21783)
- [Phi-3 Technical Report — arXiv:2404.14219 (Microsoft, 2024)](https://arxiv.org/abs/2404.14219)
- [Qwen2.5-Coder Technical Report — arXiv:2409.12186 (Qwen Team, 2024)](https://arxiv.org/abs/2409.12186)

---

# Task-to-Model Matching

## Classification & Routing
**Model size: 1B–3B**

- Sentiment analysis, spam detection, intent classification
- Topic categorization, language identification
- Binary/multi-class decisions with structured output
- Fine-tuned models achieve 95%+ accuracy
- *Example: Llama 3.2 1B for support ticket routing*

## Extraction & Transformation
**Model size: 3B–7B**

- Named entity recognition, PII detection
- JSON/structured data extraction from text
- Text reformatting, normalization, translation
- Constrained generation ensures output validity
- *Example: Qwen 2.5 3B for invoice field extraction*

## Generation & Summarization
**Model size: 7B–14B**

- Document summarization, email drafting
- Code generation for well-defined tasks
- RAG-based Q&A with grounded responses
- Quality degrades gracefully with model size
- *Example: Phi-4 14B for technical documentation Q&A*

## Complex Reasoning
**Model size: 30B+ or Frontier**

- Multi-step logical reasoning, planning
- Novel creative writing, open-ended analysis
- Ambiguous tasks requiring world knowledge
- Tasks where errors are expensive
- *Example: Claude Opus 4 for legal brief analysis*

---

# Benchmarking for YOUR Use Case

## Public Leaderboards Are Not Enough

- MMLU, HumanEval, GSM8K measure *general* capability
- Your production task is *specific* — leaderboard rankings often do not correlate
- Models optimize for benchmarks (contamination, overfitting)
- Your data distribution, prompt format, and quality bar are unique

## Build Custom Evals

### Step 1: Define Your Quality Metric
- Accuracy, F1, BLEU, ROUGE — pick what maps to business value
- Include human preference ratings for subjective tasks
- Define a minimum quality threshold (e.g., "95% extraction accuracy")

### Step 2: Create a Golden Test Set
- 200–500 representative examples from your actual data
- Cover edge cases, hard examples, and common patterns
- Version control your test set — it is a critical asset

### Step 3: Run Structured Comparisons
```
for model in [gpt-4.1, gpt-4.1-nano, llama-3.2-3b, qwen3-8b]:
    results = evaluate(model, test_set, prompt_template)
    log(model, accuracy, latency_p50, latency_p99, cost_per_1k)
```

### Step 4: Plot the Pareto Frontier
- X-axis: cost per 1,000 requests
- Y-axis: quality score
- Choose the model on the Pareto frontier that meets your quality bar

## Tools for Custom Evals

- **OpenAI Evals / Inspect AI**: framework for structured evaluation
- **LangSmith / Braintrust**: tracing + eval platforms
- **promptfoo**: open-source CLI for prompt and model comparison
- **Weights & Biases**: experiment tracking for model selection

## Sources

- [MMLU: Measuring Massive Multitask Language Understanding — arXiv:2009.03300 (Hendrycks et al., 2020)](https://arxiv.org/abs/2009.03300)
- [HumanEval: Evaluating Large Language Models Trained on Code — arXiv:2107.03374 (Chen et al., 2021)](https://arxiv.org/abs/2107.03374)
- [GSM8K: Training Verifiers to Solve Math Word Problems — arXiv:2110.14168 (Cobbe et al., 2021)](https://arxiv.org/abs/2110.14168)
- [Inspect AI — LLM Evaluation Framework (UK AISI)](https://github.com/UKGovernmentBEIS/inspect_ai)
- [LangSmith — Tracing and Eval Platform (LangChain)](https://www.langchain.com/langsmith-platform)
- [Braintrust — AI Observability and Evals Platform](https://www.braintrust.dev/)
- [promptfoo — Open-Source LLM Eval CLI](https://github.com/promptfoo/promptfoo)
- [Weights & Biases — Experiment Tracking Platform](https://wandb.ai/site/)

---

# Model Routing

## Send Easy Queries to Small Models, Hard Queries to Large Models

```mermaid
flowchart TB
    Q[User Query] --> R{Router<br/>classifier or rules}
    R -->|Simple| S["Llama 3.2 3B<br/>$0.03 / 1K"]
    R -->|Medium| M["Qwen 3 8B / Phi-4<br/>$0.15 / 1K"]
    R -->|Complex| L["Claude Sonnet 4<br/>$2.50 / 1K"]
```

## Router Implementation Options

| Approach                 | Latency  | Accuracy | Complexity |
|--------------------------|----------|----------|------------|
| Keyword/regex rules      | <1ms     | Low      | Simple     |
| Embedding similarity     | 5–10ms   | Medium   | Medium     |
| Small classifier (1B)    | 10–20ms  | High     | Medium     |
| LLM-as-judge             | 200ms+   | Highest  | High       |

## Real-World Impact

A typical support chatbot: 70% simple, 20% medium, 10% complex.
With routing: **60–80% cost reduction** vs sending everything to a frontier model.

## Sources

- [RouteLLM: Learning to Route LLMs with Preference Data — arXiv:2406.18665 (Ong et al., 2024)](https://arxiv.org/abs/2406.18665)

---

# The Cascade Pattern

## Try Small First, Escalate If Quality Is Low

```mermaid
flowchart TB
    Q[User Query] --> S[Small model 3B]
    S --> QC1{Quality check}
    QC1 -->|Pass| R1[Return response]
    QC1 -->|Fail| M[Medium model 8B-14B]
    M --> QC2{Quality check}
    QC2 -->|Pass| R2[Return response]
    QC2 -->|Fail| F[Frontier model<br/>Claude / GPT-4]
    F --> R3[Return response]
```

## Quality Check Strategies

- **Confidence score**: model logprob / entropy threshold
- **Format validation**: does output match expected schema?
- **Length heuristic**: suspiciously short or long responses
- **Self-consistency**: generate twice, check agreement
- **Lightweight verifier**: small classifier trained on good/bad outputs

## Economics

If the cascade resolves 80% at tier 1 and 15% at tier 2:
- Only 5% of requests hit the expensive frontier model
- Average cost drops **10–20x** compared to frontier-only

---

# Cost Calculator: Monthly Spend Across Model Tiers

## Scenario: 10M requests/month, ~500 tokens avg per request (5B tokens/month)

| Model                  | Input $/M | Output $/M | Monthly Cost | Latency (p50) |
|------------------------|:---------:|:----------:|:------------:|:-------------:|
| Claude Opus 4          | $15.00    | $75.00     | **$225,000** | 2,000ms       |
| GPT-4.1                | $2.00     | $8.00      | **$25,000**  | 800ms         |
| Claude Sonnet 4        | $3.00     | $15.00     | **$45,000**  | 1,000ms       |
| GPT-4.1 mini           | $0.40     | $1.60      | **$5,000**   | 300ms         |
| Gemini 2.0 Flash       | $0.10     | $0.40      | **$1,250**   | 200ms         |
| GPT-4.1 nano           | $0.10     | $0.40      | **$1,250**   | 150ms         |
| Self-hosted Qwen 3 8B  | ~$0.05    | ~$0.05     | **$250**     | 50ms          |
| Self-hosted Llama 3.2 3B| ~$0.02   | ~$0.02     | **$100**     | 30ms          |

*Self-hosted costs assume amortized GPU rental (e.g., A100 at ~$1.50/hr handling ~100 req/s).*

## The Hybrid Approach

| Traffic Split                           | Monthly Cost | Avg Quality |
|-----------------------------------------|:------------:|:-----------:|
| 100% Claude Opus 4                      | $225,000     | Excellent   |
| 100% GPT-4.1 nano                      | $1,250       | Good        |
| 70% nano + 20% mini + 10% Opus         | **$25,100**  | Very Good   |
| 80% self-hosted 8B + 15% mini + 5% Opus| **$12,000**  | Very Good   |

## Private-Endpoint Rule of Thumb

At a smaller scale — say **10,000 daily queries** — running an SLM on a private endpoint typically costs **$500–$2,000/month**, versus **$5,000–$50,000/month** for equivalent frontier API usage. That is roughly a **10–32× cost differential** for tasks where the small model holds quality.

## Key Insight

A well-designed routing strategy achieves 90%+ of frontier quality at 5–10% of the cost.

## Sources

- [Small Language Models vs LLMs (Lucas8)](https://lucas8.com/small-language-models-vs-llms/)

---

# Decision Framework: When to Go Small

```mermaid
flowchart TB
    S1[1. Define quality bar] --> S2[2. Characterize task]
    S2 --> S3[3. Estimate volume]
    S3 --> S4[4. Test smallest viable model]
    S4 --> Pass{Meets bar?}
    Pass -->|No| Up[Step up one tier]
    Up --> S4
    Pass -->|Yes| S5[5. Check deploy constraints]
    S5 --> S6[6. Implement + monitor]
```

## The Right-Sizing Ladder

Before picking a model size, decide how much machinery the task actually needs. Climb only as high as you must:

**Prompt engineering → RAG → SLM fine-tune → Distill**

Start at the bottom rung and stop at the lowest one that meets your quality bar, rather than defaulting to a frontier model. Most tasks are solved well before the top of the ladder.

## Step 1: Define Your Quality Bar
- What is the minimum acceptable performance?
- Measure with your own eval set, not public benchmarks
- Get stakeholder agreement on the threshold before testing models

## Step 2: Characterize Your Task
- Is it **narrow** (classification, extraction) or **open-ended** (creative writing, analysis)?
- Is the output **structured** (JSON, SQL, labels) or **freeform** (paragraphs, essays)?
- Do you have **labeled training data** (1,000+ examples)?
- Narrow + structured + labeled data = strong signal for small models

## Step 3: Estimate Your Volume
- Under 10K requests/month? Cost matters less — optimize for quality
- 100K–1M requests/month? Model routing starts to pay off
- Over 1M requests/month? Self-hosting becomes economically viable

## Step 4: Test the Smallest Viable Model
- Start with the smallest model in the relevant class
- Run your custom eval suite against it
- If it passes your quality bar, you are done
- If not, move one tier up and repeat

## Step 5: Evaluate Deployment Constraints
- Latency requirement under 100ms? Self-hosted small model required
- Data cannot leave your network? On-premise deployment is now a viable production default, not a niche constraint — on-device inference delivers frontier-comparable performance for many tasks, so keeping all inference on-prem rarely costs you quality
- Need offline capability? Edge/on-device model required
- None of the above? API-served models are simplest

## Sources

- [Best Local LLM Models 2026 (SitePoint)](https://www.sitepoint.com/best-local-llm-models-2026/)
- [The Best Open-Source Small Language Models (BentoML)](https://www.bentoml.com/blog/the-best-open-source-small-language-models)

## Step 6: Implement and Monitor
- Deploy with A/B testing against your current model
- Monitor quality metrics continuously
- Set up alerts for quality degradation
- Keep the larger model as a fallback

---

# Local Inference Tools

## Ollama
- One-command setup: `ollama run llama3.2:3b`
- Manages model downloads, quantization, and serving
- REST API compatible with OpenAI format
- macOS, Linux, Windows support
- Best for: **development, prototyping, single-user apps**
- Limitation: single-request at a time, no batching

## llama.cpp / llama-server
- Pure C/C++ inference — no Python dependency
- GGUF format with extensive quantization options (Q2 through Q8)
- Runs on CPU, Apple Metal, CUDA, Vulkan, SYCL
- Extremely memory efficient — 3B model in ~2GB RAM
- Best for: **edge deployment, embedded systems, maximum portability**
- Powers many other tools (Ollama uses it internally)

## vLLM
- Production-grade serving with PagedAttention
- Continuous batching for high throughput
- OpenAI-compatible API server out of the box
- Tensor parallelism across multiple GPUs
- Best for: **production serving at scale, high concurrency**
- Throughput: 10–50x over naive HuggingFace inference

## Text Generation Inference (TGI)
- HuggingFace's production serving solution
- Flash Attention, quantization (GPTQ, AWQ, EETQ)
- Token streaming and grammar-constrained generation
- Docker-first deployment with GPU support
- Best for: **HuggingFace ecosystem integration, Docker workflows**
- Powers HuggingFace Inference Endpoints

## Sources

- [Ollama — Local LLM Runner](https://ollama.com)
- [llama.cpp — LLM Inference in C/C++ (Gerganov)](https://github.com/ggerganov/llama.cpp)
- [vLLM — High-Throughput LLM Serving Engine](https://github.com/vllm-project/vllm)
- [vLLM: PagedAttention — arXiv:2309.06180 (Kwon et al., 2023)](https://arxiv.org/abs/2309.06180)
- [Text Generation Inference — HuggingFace Serving Toolkit](https://github.com/huggingface/text-generation-inference)

---

# Quantization: Shrinking Models for Deployment

## What Is Quantization?

Reducing the precision of model weights from 16-bit floats to 8-bit, 4-bit, or even 2-bit integers. This shrinks memory usage and speeds up inference with minimal quality loss.

## Quantization Methods

| Method | Bits | Approach              | Quality Loss | Speed Gain | Best For         |
|--------|:----:|-----------------------|:------------:|:----------:|------------------|
| FP16   | 16   | Baseline              | None         | 1x         | Reference        |
| INT8   | 8    | Post-training         | Minimal      | 1.5–2x     | GPU serving      |
| GPTQ   | 4    | One-shot, GPU-based   | Low          | 2–3x       | GPU deployment   |
| AWQ    | 4    | Activation-aware      | Very low     | 2–3x       | GPU deployment   |
| GGUF   | 2–8  | llama.cpp native      | Varies       | 2–4x       | CPU / edge       |
| BnB    | 4    | bitsandbytes, dynamic | Low          | 1.5–2x     | Fine-tuning      |

## Memory Impact (Llama 3.1 8B example)

| Precision | Model Size | GPU Memory | Runs On               |
|-----------|:----------:|:----------:|------------------------|
| FP16      | 16 GB      | ~18 GB     | A100, H100             |
| INT8      | 8 GB       | ~10 GB     | RTX 4090, A10          |
| Q4 (GPTQ) | 4.5 GB    | ~6 GB      | RTX 3060, M1 Pro       |
| Q4 (GGUF) | 4.5 GB    | ~5 GB RAM  | MacBook Air, CPU-only  |
| Q2 (GGUF) | 2.5 GB    | ~3 GB RAM  | Raspberry Pi 5, phones |

## Practical Advice

- **Start with Q4 (AWQ or GGUF)** — best quality/size trade-off
- **Q8 if you have the memory** — nearly lossless
- **Q2 only for extreme edge** — noticeable quality degradation
- Always re-run your eval suite after quantization to measure actual impact

## Sources

- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — arXiv:2210.17323 (Frantar et al., 2022)](https://arxiv.org/abs/2210.17323)
- [AWQ: Activation-aware Weight Quantization — arXiv:2306.00978 (Lin et al., 2023)](https://arxiv.org/abs/2306.00978)
- [GGML / GGUF Format (Gerganov)](https://github.com/ggerganov/ggml)
- [bitsandbytes — Accessible LLMs via k-bit Quantization](https://github.com/bitsandbytes-foundation/bitsandbytes)
- [llama.cpp — GGUF Quantization Support](https://github.com/ggerganov/llama.cpp)

---

# Edge Deployment: Mobile, Browser, Embedded

## Mobile Deployment

- **Apple CoreML / MLX**: Optimized for Apple Silicon (iPhone, iPad, Mac)
  - Llama 3.2 3B runs at ~30 tokens/sec on iPhone 15 Pro
  - MLX framework enables rapid prototyping on Mac
- **Android / MediaTek / Qualcomm**: NPU-accelerated inference
  - Llama 3.2 1B at 50+ tokens/sec on Snapdragon 8 Gen 3
  - ExecuTorch (Meta) for cross-platform mobile deployment
- **ONNX Runtime Mobile**: Cross-platform, optimized for on-device

## Browser-Based Inference

- **WebLLM (MLC)**: Run models directly in the browser via WebGPU
  - Llama 3.2 3B, Phi-3 mini, Gemma 2B all supported
  - No server required — fully client-side
  - ~15 tokens/sec on M1 MacBook in Chrome
- **Transformers.js**: HuggingFace models in the browser via ONNX/WebGPU
- Use cases: offline apps, privacy-first tools, demos

## Embedded and IoT

- **Raspberry Pi 5 (8GB)**: Runs Q4 Llama 3.2 1B at ~5 tokens/sec
- **NVIDIA Jetson Orin Nano**: Runs 3B–7B models for robotics and vision
- **Dedicated inference chips**: Hailo, Coral, Groq (for edge scenarios)

## Key Constraints

| Platform       | Max Model Size | Typical Speed  | Memory Budget |
|----------------|:--------------:|:--------------:|:-------------:|
| Flagship phone | 3B (Q4)       | 20–50 tok/s    | 4–6 GB        |
| Laptop (M-series)| 7B–14B (Q4) | 30–80 tok/s    | 8–32 GB       |
| Browser (WebGPU)| 3B (Q4)      | 10–20 tok/s    | 4 GB VRAM     |
| Raspberry Pi 5 | 1B–3B (Q2–Q4) | 3–8 tok/s      | 4–8 GB        |

## Sources

- [MLX — Apple Machine Learning Framework](https://github.com/ml-explore/mlx)
- [ExecuTorch — On-Device AI for PyTorch (Meta)](https://github.com/pytorch/executorch)
- [ONNX Runtime — Cross-Platform ML Inference (Microsoft)](https://github.com/microsoft/onnxruntime)
- [WebLLM — In-Browser LLM Inference via WebGPU (MLC AI)](https://github.com/mlc-ai/web-llm)
- [Transformers.js — ML for the Web (HuggingFace)](https://github.com/huggingface/transformers.js)

---

# Model Serving: Optimizing Throughput and Latency

## The Serving Stack

```mermaid
flowchart LR
    C[Client] --> LB[Load Balancer]
    LB --> IS[Inference Server]
    IS --> vLLM
    IS --> TGI
    IS --> Triton
    vLLM --> W[GPU / CPU Workers]
    TGI --> W
    Triton --> W
```

## vLLM — The Throughput Champion

- **PagedAttention**: manages KV cache like virtual memory — near-zero waste
- Continuous batching: new requests join in-flight batches
- Tensor parallelism: split one model across multiple GPUs
- Prefix caching: shared system prompts computed once
- Throughput: 8B model on A100 handles **500–2,000 requests/min**

## TGI (Text Generation Inference)

- Flash Attention 2 for memory-efficient attention
- Watermark detection for generated text
- Grammar-constrained generation (JSON schema, regex)
- Integrated with HuggingFace Hub — pull any model by name

## NVIDIA Triton Inference Server

- Multi-framework: serves PyTorch, TensorRT, ONNX, vLLM backends
- Dynamic batching with configurable strategies
- Model ensembles: chain preprocessing + model + postprocessing
- Best for: organizations already in the NVIDIA ecosystem

## Key Serving Metrics

| Metric                | Target (8B model, A100) | Why It Matters            |
|-----------------------|:-----------------------:|---------------------------|
| Time to first token   | < 100ms                 | User-perceived latency    |
| Tokens per second     | 50–100 tok/s per user   | Streaming experience      |
| Throughput            | 500–2,000 req/min       | Cost efficiency           |
| GPU utilization       | > 80%                   | Hardware ROI              |

## Sources

- [vLLM: PagedAttention — arXiv:2309.06180 (Kwon et al., 2023)](https://arxiv.org/abs/2309.06180)
- [vLLM — High-Throughput LLM Serving Engine](https://github.com/vllm-project/vllm)
- [Text Generation Inference — HuggingFace Serving Toolkit](https://github.com/huggingface/text-generation-inference)
- [NVIDIA Triton Inference Server](https://github.com/triton-inference-server/server)

---

# Continuous Batching and Speculative Decoding

## The Problem with Naive Batching

- Static batching waits for all requests in a batch to finish
- Short requests wait for long ones — wasted GPU cycles
- Utilization drops to 30–50% with variable-length outputs

## Continuous Batching (Iteration-Level Scheduling)

- New requests join the batch at every decode step
- Completed requests leave immediately — no waiting
- GPU stays busy with **80–95% utilization**
- Implemented in vLLM, TGI, and TensorRT-LLM

```
Time ──►
GPU:  [A,B,C] [A,B,C,D] [A,C,D] [A,D,E] [D,E,F] ...
       ▲                   ▲        ▲
       B starts            B done   C done, E joins
```

## Speculative Decoding — Use Small Models to Speed Up Big Ones

```mermaid
sequenceDiagram
    participant U as User
    participant D as Draft model (small)
    participant T as Target model (large)
    U->>D: prompt
    D->>D: generate K candidate tokens
    D->>T: K tokens for verification
    T->>T: verify all K in one forward pass
    T-->>U: accepted tokens (free)
    Note over T: rejected token costs 1 extra step
```

1. **Draft model** (small, fast): generates K candidate tokens
2. **Target model** (large, accurate): verifies all K tokens in one forward pass
3. Accepted tokens are free — rejected tokens cost one extra step

### Why It Works
- Large model verification of K tokens costs roughly the same as generating 1
- If draft model matches 70–80% of the time, you get **2–3x speedup**
- No quality loss — output is identical to the large model alone

### Practical Setup
| Draft Model      | Target Model     | Speedup | Use Case            |
|------------------|------------------|:-------:|---------------------|
| Llama 3.2 1B    | Llama 3.1 8B    | 2.0–2.5x| Self-hosted serving |
| Phi-4 mini       | Phi-4 14B       | 1.8–2.2x| STEM applications   |
| GPT-4.1 nano     | GPT-4.1         | ~2x     | OpenAI API (built-in)|

## Combined Impact

Continuous batching + speculative decoding together can yield **3–5x throughput improvement** over naive single-request inference.

## Sources

- [vLLM: PagedAttention — arXiv:2309.06180 (Kwon et al., 2023)](https://arxiv.org/abs/2309.06180)
- [Fast Inference from Transformers via Speculative Decoding — arXiv:2211.17192 (Leviathan et al., 2023)](https://arxiv.org/abs/2211.17192)
- [TensorRT-LLM — NVIDIA LLM Inference Library](https://github.com/NVIDIA/TensorRT-LLM)

---

# Hardware Landscape for Small Model Inference

## NVIDIA GPUs
- **H100/H200**: The datacenter workhorse, 80–141 GB HBM3
- **A100 (40/80GB)**: Still widely available, excellent price/perf for serving
- **L4 (24GB)**: Low-power inference GPU, great for 7B–14B models
- **RTX 4090 (24GB)**: Consumer GPU, surprisingly effective for serving
- Cloud: ~$2–4/hr (A100), ~$0.50–1/hr (L4)

## Apple Silicon
- **M4 Pro/Max/Ultra**: Unified memory (36–192 GB) — fits large models
- 7B Q4 at 50–80 tok/s on M4 Pro, 14B Q4 at 30–50 tok/s
- MLX framework optimized for Apple hardware
- Best for: development, prototyping, small-team serving
- Unbeatable power efficiency (watts per token)

## Google TPUs
- **TPU v5e**: Cost-optimized for inference, ~$1.20/hr
- **TPU v6e**: Next-gen, strong for batch inference workloads
- Tight integration with JAX, Keras, Vertex AI
- Best for: teams already in GCP ecosystem
- Excellent for serving Gemma models natively

## Dedicated Inference Accelerators
- **Groq LPU**: Deterministic latency, 500+ tok/s for small models
- **Cerebras WSE**: Wafer-scale engine for massive throughput
- **AWS Inferentia2**: Cost-effective on AWS, ~$0.75/hr
- **Intel Gaudi 2/3**: Competitive pricing, good for 7B–14B models
- Emerging: Etched Sohu, SambaNova, specialized RISC-V chips

## Sources

- [MLX — Apple Machine Learning Framework](https://github.com/ml-explore/mlx)
- [TensorRT-LLM — NVIDIA LLM Inference Library](https://github.com/NVIDIA/TensorRT-LLM)

---

# The Hybrid Fleet: Mixing Model Sizes in One Application

```mermaid
flowchart TB
    subgraph App[Application Layer]
        Chat[Chat UI]
        Search[Search]
        Batch[Pipeline batch]
        Admin[Admin Tools]
    end
    Chat --> GW
    Search --> GW
    Batch --> GW
    Admin --> GW
    GW[Model Gateway / Router<br/>by task, complexity, SLA]
    GW -->|70%| T1["Tier 1: 1B-3B<br/>Self-hosted Llama 3B<br/>~$0.03 / 1K"]
    GW -->|15%| T2["Tier 2: 7B-14B<br/>Self-hosted Qwen 8B<br/>~$0.10 / 1K"]
    GW -->|10%| T3["Tier 3: API small<br/>GPT-4.1 nano / Gemini Flash<br/>~$0.50 / 1K"]
    GW -->|5%| T4["Tier 4: API large<br/>Claude Sonnet / Opus<br/>~$5.00 / 1K"]
```

## Fleet Composition Per Feature

| Feature              | Tier 1 (3B) | Tier 2 (8B) | Tier 3 (API small) | Tier 4 (Frontier) |
|----------------------|:-----------:|:-----------:|:------------------:|:-----------------:|
| Intent classification| 100%        | —           | —                  | —                 |
| FAQ / simple Q&A     | 80%         | 15%         | 5%                 | —                 |
| Document extraction  | —           | 90%         | 10%                | —                 |
| Complex chat         | —           | 20%         | 50%                | 30%               |
| Code generation      | —           | 40%         | 40%                | 20%               |
| Report writing       | —           | —           | 30%                | 70%               |

---

# A/B Testing Model Swaps

## Why A/B Test Model Changes?

- Offline evals do not capture the full picture of production behavior
- User satisfaction, engagement, and task completion rates matter
- Edge cases in production data may not appear in eval sets
- Stakeholder confidence requires empirical evidence

## A/B Test Design for Model Swaps

```mermaid
flowchart LR
    Traffic[Production traffic] --> Split{Traffic split}
    Split -->|90-95%| Control[Control: current model]
    Split -->|5-10%| Variant[Variant: smaller model]
    Control --> M[Metrics: quality, latency, cost]
    Variant --> M
    M --> D{Within threshold?}
    D -->|Yes| Ramp[Ramp to 25 -> 50 -> 100%]
    D -->|No| RB[Auto-rollback]
```

### Traffic Split
- Start with **5–10%** of traffic to the new (smaller) model
- Gradually ramp to 25%, 50%, then 100% if metrics hold
- Keep the original model as the control throughout

### Metrics to Track

| Metric Category    | Specific Metrics                           | Alert Threshold |
|--------------------|--------------------------------------------|:---------------:|
| Quality            | Task completion rate, accuracy, F1         | >2% drop        |
| User satisfaction  | Thumbs up/down, CSAT, retry rate           | >5% drop        |
| Latency            | TTFT, total response time, p99            | Improvement expected |
| Cost               | $/request, $/user, monthly spend           | Should decrease |
| Errors             | Hallucination rate, format failures, timeouts| >1% increase   |

### Statistical Rigor
- Minimum sample size: 1,000+ requests per variant
- Run for at least 1–2 weeks to capture weekly patterns
- Use sequential testing to enable early stopping
- Segment by query type — the small model may fail on specific categories

## Rollback Plan

- Automated rollback if quality drops below threshold
- Feature flag or traffic percentage toggle — instant switch
- Shadow mode: run both models, compare outputs, serve only one
- Log all inputs/outputs for post-hoc analysis

---

# Migration Strategy: Frontier to Small Models Safely

```mermaid
flowchart LR
    S1[1. Instrument] --> S2[2. Identify candidates]
    S2 --> S3[3. Build eval pipeline]
    S3 --> S4[4. Test offline]
    S4 --> S5[5. Shadow deploy]
    S5 --> S6[6. Gradual rollout]
    S6 -->|Quality drop| Rollback[Auto-rollback]
    S6 -->|Success| Next[Repeat for next feature]
```

## Step 1: Instrument Your Current System
- Add logging for every LLM call: input, output, model, latency, cost
- Tag requests by feature, user segment, and complexity
- Establish baseline metrics for quality, cost, and latency
- Run for 2+ weeks to capture representative traffic patterns

## Step 2: Identify Migration Candidates
- Sort features by cost (highest spend first)
- Flag tasks with structured output (JSON, labels, SQL) — easiest wins
- Flag high-volume, low-complexity tasks — biggest cost savings
- Deprioritize tasks requiring complex reasoning or creativity

## Step 3: Build Your Eval Pipeline
- Extract 500+ real production examples per task
- Get ground-truth labels (human annotation or frontier model consensus)
- Automate: `run_eval(model, dataset) -> scores`
- Include latency and cost in the eval report

## Step 4: Test Candidate Models Offline
- Run 3–5 small model candidates through your eval pipeline
- Compare against frontier model baseline
- Identify the smallest model that meets your quality threshold
- Test with quantization to find the optimal size/quality trade-off

## Step 5: Shadow Deploy
- Run the small model in parallel with production (shadow mode)
- Compare outputs without serving to users
- Catch regressions before they impact anyone
- Duration: 1–2 weeks of production traffic

## Step 6: Gradual Rollout
- 5% traffic to small model with real-time quality monitoring
- Ramp to 25%, 50%, 100% over 2–4 weeks
- Automated rollback triggers on quality drops
- Celebrate the cost savings — then repeat for the next feature

---

# Monitoring Quality Degradation After Model Downsizing

## The Silent Failure Problem

Small models fail differently than large models:
- They do not refuse — they confidently produce wrong answers
- Degradation is often gradual, not catastrophic
- Failures cluster on specific input types, not randomly
- Without monitoring, you may not notice for weeks

## What to Monitor

### Output Quality Signals
- **Format compliance**: JSON validity rate, schema adherence
- **Factual accuracy**: spot-check samples against ground truth
- **Hallucination rate**: claims not grounded in provided context
- **Completeness**: are outputs truncated or missing fields?
- **Consistency**: same input should produce similar outputs

### Behavioral Signals
- **Retry rate**: users asking the same question again
- **Edit rate**: users modifying model outputs before using them
- **Abandonment**: users leaving the feature without completing their task
- **Escalation**: users requesting human support after model interaction

### Drift Detection
- Monitor input distribution — are users sending different types of queries?
- Track output token length distribution — sudden changes signal problems
- Compare weekly quality scores against baseline
- Use embedding similarity to detect new query clusters the model has not seen

## Alerting Strategy

| Signal                    | Threshold           | Action                    |
|---------------------------|:-------------------:|---------------------------|
| Format failure rate       | > 2% (was < 0.5%)  | Auto-escalate to larger model |
| Accuracy on spot checks   | > 3% drop           | Alert + investigate       |
| User retry rate           | > 10% increase      | Alert + A/B test review   |
| Latency p99               | > 2x baseline       | Investigate infrastructure|
| Novel input cluster       | > 5% of traffic     | Manual review + eval      |

## The Feedback Loop

```mermaid
flowchart LR
    M[Monitor] --> D[Detect regression]
    D --> RCA[Root cause analysis]
    RCA --> F1[Add training examples]
    RCA --> F2[Update prompt template]
    RCA --> F3[Escalate to larger model]
    F1 --> M
    F2 --> M
    F3 --> M
```

---

# Key Takeaways

## The Right Model Is the Smallest One That Works
- Start small, scale up only when quality demands it
- A fine-tuned 7B model often matches frontier models on focused tasks
- Public benchmarks do not predict performance on your specific use case
- Build custom evals before making any model decision

## Cost Savings Are Massive and Real
- 100–1,000x cost reduction is achievable for many production tasks
- Model routing and cascade patterns capture 90% of savings with 95% of quality
- Self-hosting becomes economical above 1M requests/month
- The hybrid fleet is the dominant production architecture

## The Ecosystem Is Mature
- Llama, Qwen, Phi, Gemma, Mistral — strong options at every size
- Deployment tools (vLLM, llama.cpp, Ollama) are production-ready
- Quantization (Q4) cuts memory in half with minimal quality loss
- Edge deployment on phones and browsers is practical today

## Migrate Safely, Monitor Continuously
- Shadow deploy and A/B test every model swap
- Instrument quality metrics from day one
- Small models fail confidently — silent degradation is the real risk
- Keep frontier models as a fallback, not as a default

---

# Key Papers and Benchmarks

## Foundational Papers

| Paper / Resource                                    | Year | Key Contribution                              |
|-----------------------------------------------------|:----:|-----------------------------------------------|
| [Llama 3 Herd of Models (Meta)](https://arxiv.org/abs/2407.21783) | 2024 | Open-weight foundation models at scale         |
| [Qwen2.5 Technical Report (Alibaba)](https://arxiv.org/abs/2412.15115) | 2024 | Strong multilingual and code SLMs              |
| [Phi-3 Technical Report (Microsoft)](https://arxiv.org/abs/2404.14219) | 2024 | Textbook-quality data for small models         |
| [Mixtral of Experts (Mistral AI)](https://arxiv.org/abs/2401.04088) | 2024 | Practical MoE for production deployment        |
| [Gemma 2 (Google DeepMind)](https://arxiv.org/abs/2408.00118) | 2024 | Knowledge distillation for on-device models    |
| [Qwen3 Technical Report (Alibaba)](https://arxiv.org/abs/2505.09388) | 2025 | Hybrid thinking, SOTA small model reasoning    |

## Inference and Serving

| Paper / Resource                                    | Year | Key Contribution                              |
|-----------------------------------------------------|:----:|-----------------------------------------------|
| [vLLM: PagedAttention (Kwon et al.)](https://arxiv.org/abs/2309.06180) | 2023 | Efficient KV cache management for serving      |
| [Speculative Decoding (Leviathan et al.)](https://arxiv.org/abs/2211.17192) | 2023 | Use draft models to speed up large models      |
| [GPTQ (Frantar et al.)](https://arxiv.org/abs/2210.17323) | 2023 | One-shot weight quantization to 4-bit          |
| [AWQ: Activation-aware Quantization (Lin et al.)](https://arxiv.org/abs/2306.00978) | 2024 | Preserving salient weights during quantization |
| [RouteLLM (Ong et al.)](https://arxiv.org/abs/2406.18665) | 2024 | Framework for LLM routing strategies           |

## Benchmarks to Know

| Benchmark     | What It Measures           | URL                                |
|---------------|----------------------------|------------------------------------|
| MMLU          | General knowledge          | [arxiv.org/abs/2009.03300](https://arxiv.org/abs/2009.03300) |
| HumanEval     | Code generation            | [arxiv.org/abs/2107.03374](https://arxiv.org/abs/2107.03374) |
| GSM8K         | Grade-school math          | [github.com/openai/grade-school-math](https://github.com/openai/grade-school-math) |
| GPQA          | Graduate-level science     | [github.com/idavidrein/gpqa](https://github.com/idavidrein/gpqa) |
| MT-Bench      | Multi-turn chat quality    | [github.com/lm-sys/FastChat](https://github.com/lm-sys/FastChat) |
| Open LLM LB   | Composite leaderboard      | [huggingface.co/spaces/open-llm-leaderboard](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) |
| Chatbot Arena | Human preference (Elo)     | [arena.ai](https://arena.ai) |

## Recommended Reading Order

1. [Llama 3 paper](https://arxiv.org/abs/2407.21783) (understand the foundation)
2. [Qwen 2.5](https://arxiv.org/abs/2412.15115) / [Qwen 3](https://arxiv.org/abs/2505.09388) reports (see what small models can do)
3. [vLLM paper](https://arxiv.org/abs/2309.06180) (understand production serving)
4. [RouteLLM paper](https://arxiv.org/abs/2406.18665) (model routing strategies)
5. Run your own benchmarks (the most important step)

---

# The Best Model Is the One You Can Afford to Run

## At scale, model selection is an economic decision as much as a technical one.

---

### This Week's Assignment

1. Pick a task from your project (classification, extraction, generation, or Q&A)
2. Build a 200-example eval set from real or realistic data
3. Benchmark at least 3 models: one frontier, one mid-size, one small (<8B)
4. Plot the cost-quality Pareto frontier
5. Write a 1-page recommendation: which model would you deploy and why?

---

*Next week: Fine-Tuning Small Models — LoRA, QLoRA, and making 3B models punch above their weight.*