Foundations

Embrace the good, engineer around the bad

LLM capabilities are jagged

The single most important mental model for this whole day. “Jagged” comes from Ethan Mollick’s research — the idea that LLM capability isn’t a smooth gradient from easy to hard. Some tasks that feel trivial to humans are brutally hard for LLMs, and vice versa.

The strawberry problem

How many r’s are in “strawberry”?

Most LLMs confidently answer 2.

The correct answer is 3.

An “easy” task for a 5-year-old. A hard task for a trillion-parameter model.

Live demo opportunity: actually ask an LLM this in real time if you want. Recent models have gotten better at this specific example (it’s become a benchmark), but the underlying point stands: tokenization means LLMs don’t “see” individual characters the way we do. This is a symptom, not a fluke.

But also…

SCREENSHOT: an impressive Claude Code or Copilot session —
a complex coding task completed in seconds
(e.g., “build me a Shiny app that does X” → working app)

A “hard” task for most humans. Trivial for an LLM.

The juxtaposition is the point. The same system that can’t count letters can write a working web app from a paragraph of prose. The audience should feel this tension viscerally — it’s what motivates everything we build today.

The jagged frontier

IMAGE: “Jagged frontier” illustration —
a line graph with “task difficulty (human perception)” on x-axis
and “LLM performance” on y-axis, showing a jagged, non-monotonic curve
(Source: Ethan Mollick / “Co-Intelligence”, or recreate as a simple sketch)

Credit Ethan Mollick here. The frontier is jagged — there are pockets of superhuman performance next to pockets of embarrassing failure. Our job as engineers is to put LLMs to work in the high-performance pockets and build guardrails around the low ones.

The key insight

LLMs are great at

• Language → code
• Summarization
• Pattern matching
• “Fuzzy” translation

LLMs are bad at

• Precise counting / math
• Faithful recall of facts
• Consistent long-range state
• Knowing what they don’t know

Our strategy: focus LLMs on what they’re good at, and engineer around the rest.

This two-column framing is the actionable takeaway from the jagged frontier. The left column is where we’ll aim chatlas, shinychat, and querychat — we’re asking LLMs to translate user intent into code, not to be the database or be the calculator. The right column is what we’ll build harnesses around.

What is an agent?

flowchart TD
    U["👤 User"] --> |prompt| L["🧠 LLM"]
    L --> |tool call| T["🔧 Tools"]
    T --> |result| L
    L --> |response| U
    style L fill:#447099,stroke:#33567a,color:#fff
    style T fill:#ee6331,stroke:#c84e1f,color:#fff
    style U fill:#0e3b5c,stroke:#082638,color:#fff

An agent is an LLM, in a harness, that calls tools in a loop — deciding each next step from the last result.

• Read tools observe the world
• Write tools change the world

Definition from Joe Cheng’s “What is an agent?” post. Keep the diagram on screen and walk through it: the user prompts, the LLM decides what tool to call, the tool returns a result, the LLM decides what to do next. The “loop” part is what separates an agent from a single LLM call.

Read vs. write tools: reading a file, searching the web, querying a database = read tools. Writing a file, sending an email, making an API call = write tools. The risk profile is very different.

Agent harnesses are getting good

SCREENSHOT: Claude Code terminal session
(something relatable — fixing a bug, writing tests)

SCREENSHOT: OpenAI Codex or similar
(another agent IDE in action)

This has radically changed how I work in the last 6 months.

Be personal here. Talk about your actual experience — what changed in your workflow, what surprised you, what you’d never go back from. The audience is data scientists who probably use ChatGPT but haven’t let an agent loose on their codebase yet. Your honest enthusiasm (and healthy caution) is the credibility play.

YOLO mode

Let the agent run unsupervised.

Terrifying. But so useful.

✅ Viable when…

• Local dev environment
• Easy to undo (git)
• Low blast radius

❌ Dangerous when…

• Hosted / production system
• Real users, real data
• Hard to undo

“YOLO mode” is the colloquial term — Claude Code calls it “auto-accept.” The key distinction: local dev agents operate in a sandbox where the worst case is a git reset. Hosted prod agents can delete databases, email customers, or corrupt data. Same technology, completely different risk profile. This distinction — local dev vs. hosted prod — is a recurring theme today.

Hallucinations & sycophancy

Hallucination

The model invents plausible-sounding information that isn’t true.

“The pandas.DataFrame.fuzzy_merge() method…”

(This doesn’t exist.)

Sycophancy

The model agrees with you even when you’re wrong.

“You’re right, that approach is much better!”

(It wasn’t.)

Both are unsolved. Both are why we need human-in-the-loop design.

These aren’t edge cases — they’re fundamental properties of how LLMs work. Hallucination comes from next-token prediction having no ground truth anchor. Sycophancy comes from RLHF optimizing for human approval. Neither is going away soon. The practical consequence: never trust LLM output that you can’t verify.

I feel deeply conflicted about AI

I’m so much more productive.

But at what cost?

Personal — Am I losing the ability to think for myself?

Societal — Am I contributing to something harmful?

IMAGE: something contemplative —
a person at a crossroads, or a balance/scale visual
(keep it tasteful, not melodramatic)

This is the vulnerable moment. Be genuine. The audience is probably feeling some version of this too, especially at a conference like SciPy where thoughtfulness about technology is valued. Don’t resolve the tension — just name it honestly. The point is to show that using these tools doesn’t require ignoring the hard questions.

What’s the alternative?

If I practice AI abstinence, is that better than
helping people practice safe AI?

I chose to build tools that make AI safer and more transparent:

• Humans stay in the loop
• Outputs are verifiable
• The system is auditable

The sex-ed analogy is deliberately provocative but apt. The alternative to engagement isn’t purity — it’s ceding the space to people who won’t build guardrails. You’ve chosen to build tools (chatlas, querychat) that keep humans in the driver’s seat. That’s the “safe AI” framing for the rest of the day.

In data science, correctness matters

Design systems where humans can course-correct and verify.

• Show your work (code, not just answers)
• Make it reproducible
• Let the human say “no, that’s wrong”

SCREENSHOT: Posit Assistant showing
generated code alongside results —
the “show your work” pattern

This is where your DS background pays off. In stats, we learned that showing your methodology matters as much as showing your result. The same principle applies to AI-assisted analysis: an app that shows the generated code lets the user verify, reproduce, and correct. This is exactly what querychat does — but we’ll get there.

Mention your colleague’s talk on correctness/reproducibility if it’s at SciPy. TODO(Carson): add the colleague’s name and talk title.

Don’t overthink it

You probably don’t need:

Complex RAG

Multi-agent systems

Fine-tuned models

What you do need:

Good context + good tools + a human in the loop

Unless you have scale issues, agentic search of plain text files works surprisingly well.

This is the “permission to start simple” slide, and it’s the bridge to the rest of the tutorial. The three things we’ll build with today — chatlas, shinychat, querychat — are all context + tools + human-in-the-loop. No vector databases, no fine-tuning, no multi-agent orchestration. You can go far with the basics done well.

The recipe for today

1. Focus LLMs on language → code
2. Give them tools and context
3. Keep humans in the loop

That’s it. Let’s go build.

This is the thesis slide — the three principles that underpin everything we do for the rest of the day. Refer back to it at the start of each section. Quick transition to break.

☕ Break — 5 min

Remind them: next up is chatlas — we’re writing code. Make sure their API keys / LM Studio is working before we start.