Foundations

Embrace the good, engineer around the bad

Posit, open science, & the prime directive

Posit creates open-source software for data science, scientific research, and technical communication.

We build tools that prioritize correctness, transparency, and reproducibility in their output.

  • Results are often generated via Excel, SPSS, JMP, etc., which can’t be verified as trustworthy.
  • John M. Chambers coined the term “the prime directive”: the obligation for analysts to produce work that can be shown to be trustworthy.

Fulfilling the prime directive

  • ✅ Correctness: (Obviously)
  • ✅ Transparency: methods of the analysis can be inspected
  • ✅ Reproducibility: analysis can be repeated on the same data, hopefully yielding the same results

LLMs: another opaque tool, but worse

  • ❌ Correctness: LLMs are infamous for giving convincing but wrong answers.
  • ❌ Transparency: Nobody really understands how or why LLMs do what they do.
  • ❌ Reproducibility: LLMs are non-deterministic black boxes.
  • With proper context and constraints, correctness can be improved (not guaranteed).
  • LLMs are really good at coding, so they can produce verifiable results that can be vetted by humans.

LLM capabilities are jagged

You might expect performance to drop with difficulty…

LLM capabilities are jagged

… but in reality, performance is jagged.

The bad: counting / computing

How many r’s are in “strawberry”?

Most LLMs confidently answer 2, but it’s actually 3.

The bad: counting / computing

How many values are in this array [4, 8, ...]?

  • Again, most LLMs confidently answer incorrectly.
  • However, if it can execute code, correctness dramatically improves.
  • Bad at “implicit” computation, but awesome at coding!
  • Excellent way for them to learn, be precise, and productive.

How do LLMs gain coding capabilities?

Through tool calling, which provides the foundation for agents.

Agent = LLM + tools + loop

An LLM with tools (i.e., functions) running in a loop — deciding each next step from the last result.

The good: coding assistants

  • Claude Code, Codex, and Copilot etc: harnesses that allow LLMs to write and execute code.
  • Primarily designed for software engineering tasks
  • IME, YOLO mode is incredibly useful, but also terrifying.
  • Posit Assistant brings a similar experience to data science, and designs for human-in-the-loop workflows.

Posit Assistant

  • Can access and control your Python/R sessions

Posit Assistant

  • Can access and control your Python/R sessions

  • Helpful features for doing data science

Posit Assistant

  • Can access and control your Python/R sessions

  • Helpful features for doing data science

  • Tight integration with Positron, Notebooks, etc.

Posit Assistant

  • Can access and control your Python/R sessions

  • Helpful features for doing data science

  • Tight integration with Positron, Notebooks, etc.

  • Synthesize findings into reproducible reports, etc.

NOTE: Posit Assistant helps you do analysis – what about helping others leverage your work?

Querychat: ask more of your dashboards

  • Web app framework for building AI assisted data exploration apps
  • Restricted to SQL/ggsql queries for safety
  • Supplies context from data you provide
  • Limited scope can lead to more correct answers, but less creative ones

Goals for today

  1. Learn the basics of programming with LLMs (via chatlas).
    • System prompts, tool calling, extracting data, etc.
  1. Learn the basics of shinychat, which makes it easy to build LLM-powered web apps.
    • Plus some useful things for putting apps into production
  1. Form a mental model for how these things serve as the foundation for querychat.
    • Hopefully this provides inspiration to create a tailored LLM-powered app for your own work.