Structured data extraction

Why structured data extraction?

1. LLMs are quite good at finding structure
   • Throw in large amounts of text, images, pdfs, etc., get data back.

2. You might want downstream logic to depend on LLM responses.
   • e.g., do x if the LLM says “yes”, do y if it says “no”.

3. Produce structured datasets for further analysis.

Some example use cases

1. Article summaries: Extract key points from lengthy reports or articles.

2. Image/PDF input: Extract data from images or PDFs, such as tables or forms

3. Classification: Classify text into predefined categories, such as spam detection or topic classification.

4. Entity recognition: Identify and extract entities such as names, dates, and locations

5. Sentiment analysis: Extract sentiment scores and associated entities from reviews, etc.

Basic entity recognition example

import chatlas as ctl
from pydantic import BaseModel

class Person(BaseModel):
    name: str
    age: int

chat = ctl.ChatBedrockAnthropic()
chat.chat_structured(
  "My name is Susan and I'm 13 years old", 
  data_model=Person,
)

Person(name='Susan', age=13)

Add descriptions

In addition to type hints, you can add descriptions to your fields.

class Person(BaseModel):
    """A person"""
    name: str = Field(description="Name")
    age: int = Field(description="Age, in years")
    hobbies: list[str] = Field(
        description="List of hobbies. Should be exclusive and brief."
    )

Literature mining

import chatlas as ctl
from pydantic import BaseModel

class ModelResult(BaseModel):
    model: str
    dataset: str
    metric: str
    value: float

class PaperResults(BaseModel):
    results: list[ModelResult]

abstract = """ResNet-50 achieves 93.2% top-1 accuracy on CIFAR-10 and 76.1% on
ImageNet. A fine-tuned ViT-B/16 reaches 98.1% on CIFAR-10."""

chat = ctl.ChatBedrockAnthropic()
chat.chat_structured(abstract, data_model=PaperResults)

PaperResults(results=[
    ModelResult(model='ResNet-50', dataset='CIFAR-10', metric='top-1 accuracy', value=93.2),
    ModelResult(model='ResNet-50', dataset='ImageNet', metric='top-1 accuracy', value=76.1),
    ModelResult(model='ViT-B/16',  dataset='CIFAR-10', metric='top-1 accuracy', value=98.1),
])

Image recognition example

import chatlas as ctl
from pydantic import BaseModel

class PlotInfo(BaseModel):
    chart_type: str
    x_label: str
    y_label: str
    trend: str

chat = ctl.ChatBedrockAnthropic()
chat.chat_structured(
  ctl.content_image_url("https://matplotlib.org/stable/_images/sphx_glr_simple_plot_001.png"),
  data_model=PlotInfo,
)

PlotInfo(chart_type='line', x_label='x', y_label='f(x)', trend='oscillating sine wave')

Classification example

import chatlas as ctl
from pydantic import BaseModel
from typing import Literal

class IssueLabel(BaseModel):
    label: Literal["bug", "enhancement", "docs", "question"]
    confidence: Literal["low", "medium", "high"]

chat = ctl.ChatBedrockAnthropic()
chat.chat_structured(
    "The `plt.show()` call hangs indefinitely on macOS with the Tk backend.",
    data_model=IssueLabel,
)

IssueLabel(label='bug', confidence='high')

Optional fields

import chatlas as ctl
from pydantic import BaseModel, Field

class Citation(BaseModel):
    authors: list[str]
    year: int
    title: str
    journal: str | None = Field(description="Journal name; None for preprints")
    doi: str | None = Field(description="DOI if present, otherwise None")

ref = "Harris et al. (2025). NumPy 2.0: Array API Standard Conformance. arXiv:2501.12345"

chat = ctl.ChatBedrockAnthropic()
chat.chat_structured(ref, data_model=Citation)

Citation(authors=['Harris', 'et al.'], year=2025,
         title='NumPy 2.0: Array API Standard Conformance',
         journal=None, doi=None)

Exercise

Try extracting structured data from this PDF:

https://storage.googleapis.com/generativeai-downloads/data/pdf_structured_outputs/invoice.pdf

Tips:

1. Use ctl.content_pdf_url() to pass the PDF to the LLM.
2. Note there are multiple items, as well as metadata like invoice number, date, and total.