Tool calling and agents

LLMs don’t have access to live data

LLMs are trained on a “fixed” data set (i.e., they have a “knowledge cutoff”).


import chatlas as ctl

chat = ctl.ChatBedrockAnthropic()
chat.chat("What's the weather in San Francisco?")

I don’t have the ability to check real-time weather data.

LLMs don’t have access to live data

LLMs are trained on a “fixed” data set – providers typically refer to this as a “knowledge cutoff”.

Three main techniques:

  1. Prompting: supply knowledge to system prompt. Simple, but it doesn’t scale well.
  1. RAG: retrieve relevant information based on user prompt.
  1. Tool calling: define a tool that can access knowledge, and let the model call it.

RAG is a popular term, but the core idea is outdated. Now you would supply a tool that can retrieve relevant information, and let the model call it.

Tools = functions + metadata

Tools = functions + metadata

def get_current_weather(lat: float, lng: float):
    """Get the current weather for a location."""
    import requests
    resp = requests.get(
        "https://api.open-meteo.com/v1/forecast",
        params={"latitude": lat, "longitude": lng, "current": "temperature_2m"},
    )
    return resp.json()["current"]

Note that:

  • Type hints are required — the model uses these to know what arguments to pass.
  • Docstring is required — this is how the model understands what the tool does.

Now the LLM can get the current weather

chat.register_tool(get_current_weather)
chat.chat("What's the weather in San Francisco?")
# 🛠️ tool request
get_current_weather(37.7749, -122.4194)
# ✅ tool result
sunny

The current weather in San Francisco is sunny.

How tool calling works

The tool loop in detail

  1. User sends: prompt & tool metadata
  2. Model responds: call x tool with y arguments
  3. chatlas invokes: function with those arguments
  4. chatlas sends: result back to the model
  5. Model responds: with a natural language answer
  • Steps 2–4 can repeat — the model can call multiple tools in sequence.
  • Model can “gracefully recover” from errors: if failure occurs, the error is sent back to the model.

Exercise

  1. Open up exercises/chat-tools.py.
  2. Write a python function that reports the current date.
  3. Register it as a tool and call it from chat.
  4. Add another tool that takes Python code as input and eval() it.


Agents: LLM + tools + loop

  • Tools are a means by which LLMs both learn & alter the state of the world.
  • This “weather agent” is safe, but not very capable.
  • LLMs are very good at generating code – what if the model could write and execute code?
  • Key idea: define a tool that takes code as input and executes it.
def eval_code(code: str):
    """Execute Python code."""
    return eval(code)

⚠️ This is a illustrative example — never execute arbitrary code in production without proper sandboxing and security measures.

Guardrails for code execution

Three main strategies to mitigate risk:

  1. Use a sandbox to isolate the execution environment.
    • Warning: here be dragons — this is a hard problem.
  1. Leverage remote tools to execute on a separate process or server.
    • Major LLM providers have code execution tools.
  1. Restrict the scope of what can be executed (e.g., read-only SQL).
    • This is the approach used by querychat.
    • Can also lead to model getting “less creative”, which can be a good thing for correctness.
    • Especially relevant for web apps other people are driving.

Remote tools

  • Tools don’t have to live in the same process as the LLM client.
  • They can be:
    • Provider: the LLM provider provides tools that run on their servers
    • MCP: serve tools through a web service or local process.
  • Remote tools can be implemented in any language, not just Python.

Provider tools

chatlas comes with some “built-in” tools that use the provider’s native capabilities.

These currently include:

  1. tool_web_search() — search the web for current information
  2. tool_web_fetch() — read a specific URL

Use it like this:

chat.register_tool(ctl.tool_web_search())
chat.chat("What were the top Python packages released this month?")

MCP tools

MCP tools can be served through either:

  1. HTTP — the tool is served through a web service (e.g., DeepWiki)
await chat.register_mcp_tools_http_stream_async(
  url="https://mcp.deepwiki.com/mcp"
)

MCP tools are async, which means you must also use .chat_async() to call them.

await chat.chat_async(
  "Summarize what's in cpsievert/scipy26-tutorial on GitHub"
)

MCP tools

MCP tools can be served through either:

  1. HTTP — a web service (e.g., DeepWiki)
  2. STDIO — a local process (e.g., uvx)

If the tool must be run locally, you can serve, register and use like this:

await chat.register_mcp_tools_stdio_async(
  command="uvx",
  args=["mcp-server-fetch"],
)
await chat.chat_async(
  "What are the top 5 Python packages released this month?"
)

Is MCP dead?

  • Some argue CLIs are better and more efficient than MCP.
  • There is some truth to this, but it also assumes you’re ok with arbitrary commands running in the terminal.
  • MCP is safer since it reduces the risk surface, making it a better fit for web apps.

Exercise

  1. Go to https://www.pulsemcp.com/servers.
  2. Click on Filters -> “Remote Available”.
  3. Sort from most popular.
  4. Pick an MCP server.
  5. Open exercises/chat-mcp-tools.py. Tweak it to use the MCP server you picked.


Bonus: knowledge retrieval

  • Agentic search (i.e., tools to web search, read files, etc) gets you a long way, but not necessarily efficient.
  • Common pattern: “Agentic RAG”
    1. Pre fetch/index knowledge into an efficient storage system.
    2. Supply a tool for LLM to retrieve from it.
def search_docs(query: str, num_results: int = 5) -> str:
    "Search the documentation for relevant information."
    chunks = store.retrieve(query, top_k=num_results, deoverlap=True)
    return json.dumps(
        [{"text": chunk.text, "context": chunk.context} for chunk in chunks]
    )
  • This is using raghilda for the store. Learn more here.