llm-structured-confidence

Extract path-aware confidence scores from LLM structured JSON outputs using token-level log-probabilities.

Installation • Supported JSON Formats • Quick Start • Path Syntax • Documentation • Supported Providers

Designed for structured JSON outputs from OpenAI, Gemini, and compatible providers. The library aligns token logprobs to exact JSON value spans, strips structural tokens, and computes confidence only from the tokens that belong to the value itself.

Installation

pip install llm-structured-confidence

For DataFrame helpers:

pip install "llm-structured-confidence[pandas]"

Supported JSON Formats

The library works with any structured JSON output. Here are the most common patterns and how to extract confidence from each.

Single scalar field

{"category": "health and wellness"}

entries = extract_logprobs(response, field_path="category")
# entries[0].path  -> "category"
# entries[0].value -> "health and wellness"

Nested scalar field

{"classification": {"name": "Positive", "score": 0.95}}

entries = extract_logprobs(response, field_path="classification.name")
# entries[0].path  -> "classification.name"
# entries[0].value -> "Positive"

Array of strings

{"categories": ["health and wellness", "sports", "technology"]}

entries = extract_logprobs(response, field_path="categories[]")
# entries[0].path -> "categories[0]", entries[0].value -> "health and wellness"
# entries[1].path -> "categories[1]", entries[1].value -> "sports"
# entries[2].path -> "categories[2]", entries[2].value -> "technology"

Array of objects (pick one field)

{"results": [{"id": 1, "category": "deposits"}, {"id": 2, "category": "shopping"}]}

entries = extract_logprobs(response, field_path="results[].category")
# entries[0].path -> "results[0].category", entries[0].value -> "deposits"
# entries[1].path -> "results[1].category", entries[1].value -> "shopping"

Top-level array of objects (Vertex AI only)

Vertex AI supports "type": "ARRAY" at the schema root, producing a bare JSON array. OpenAI requires a top-level object, so these formats are Vertex-only.

[{"id": 1, "category": "deposits"}, {"id": 2, "category": "shopping"}]

entries = extract_logprobs(response, field_path="[].category")
# entries[0].path -> "[0].category", entries[0].value -> "deposits"
# entries[1].path -> "[1].category", entries[1].value -> "shopping"

Top-level array of strings (Vertex AI only)

The most compact multi-classification format — a flat list of category strings with no wrapper object. The enum constraint in the schema is enforced by Vertex AI.

["bars and restaurants", "transportation", "digital services"]

entries = extract_logprobs(response, field_path="[]")
# entries[0].path -> "[0]", entries[0].value -> "bars and restaurants"
# entries[1].path -> "[1]", entries[1].value -> "transportation"
# entries[2].path -> "[2]", entries[2].value -> "digital services"

Deeply nested arrays

{"groups": [{"items": [{"label": "A"}, {"label": "B"}]}]}

entries = extract_logprobs(response, field_path="groups[].items[].label")
# entries[0].path -> "groups[0].items[0].label", entries[0].value -> "A"
# entries[1].path -> "groups[0].items[1].label", entries[1].value -> "B"

Provider compatibility

Note: OpenAI's json_schema response format requires "type": "object" at the root. To get array-like output on OpenAI, wrap it in an object (e.g. {"results": [...]}). Top-level arrays (both string and object variants) are exclusive to Vertex AI.

Quick Start

import litellm
from llm_structured_confidence import extract_logprobs

response = litellm.completion(
    model="gpt-4.1-mini",
    messages=[
        {"role": "system", "content": "Classify this text."},
        {"role": "user", "content": "Morning yoga and meditation session"},
    ],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "classification",
            "strict": True,
            "schema": {
                "type": "object",
                "properties": {
                    "category": {
                        "type": "string",
                        "enum": ["sports", "health and wellness", "technology"],
                    }
                },
                "required": ["category"],
                "additionalProperties": False,
            },
        },
    },
    logprobs=True,
    top_logprobs=5,
)

entries = extract_logprobs(response, field_path="category")
entry = entries[0]

print(entry.path)                                    # category
print(entry.value)                                   # health and wellness
print(entry.field_logprob.mean_nonzero_probability)  # 0.8451

Schema auto-detection

If you pass response_schema=, the library auto-detects enum-valued paths recursively.

entries = extract_logprobs(response, response_schema=ClassificationModel)

That also enables TopAlternative.resolved_value, so token prefixes like "Pos" can resolve back to "Positive" when the match is unique.

Path Syntax

Flat/DataFrame Helpers

from llm_structured_confidence import extract_confidence, add_confidence_columns

metrics = extract_confidence(response, field_path="category")

df = add_confidence_columns(
    df,
    response_column="response",
    field_path="classifications[].name",
)

Helpers always return the first matching value and also expose its resolved path.

Documentation

• docs/USAGE.md: full public API and examples
• examples/examples.ipynb: notebook walkthrough
• AGENTS.md: compact reference for coding agents

Supported Providers

• litellm.ModelResponse
• openai.ChatCompletion
• google.genai.GenerateContentResponse
• raw OpenAI batch response dict with choices
• raw Vertex AI batch response dict with candidates

Lower-level API

Internal modules remain available for custom workflows:

from llm_structured_confidence._parser import parse_json_spans, build_token_char_ranges, tokens_for_span
from llm_structured_confidence._converter import normalize_response

These are underscore-prefixed internals and may change in minor releases.