The workflow this is for​

Improving an LLM app is an evaluation loop. You run the prompt, see where it fails, adjust the prompt or evaluator, and run it again against the examples that matter.

That loop used to be split across surfaces. The Playground is where you iterated on prompts, while evaluations and test set curation happened elsewhere.

The Playground is now an evaluation workbench. You can attach evaluators to a Playground session, see evaluator results next to each output, and keep test data connected while you debug.

What changed​

Inline evaluators​

You can now attach evaluators directly in the Playground. When you run a prompt, evaluator results appear inline next to the generated output, so you can see whether a change improved the answer while the prompt is still in front of you.

This is different from starting a full evaluation run. The evaluation button creates a persisted evaluation run over a test set. Inline evaluators are playground-scoped. They are for fast feedback while you are editing.

Connected test sets​

You can load a test set into the Playground in connected mode. Rows stay tied to the source test set, so edits you make while debugging can be synced back instead of copied by hand.

Local mode is still available when you want scratch examples. Connected mode is for the cases where your iteration should improve the dataset you already use for evaluation.

The loop​

1. Open a prompt in the Playground.
2. Load a connected test set or add local examples.
3. Attach one or more evaluators.
4. Run the prompt and inspect the inline scores.
5. Edit the prompt, evaluator, or examples.
6. Run again until the output and the score line up.

This keeps prompt changes, evaluator feedback, and test data curation in one place.

Why it matters​

Teams usually lose time moving between prompt editing, evaluation setup, and data cleanup. That makes it harder to understand whether a failure came from the prompt, the evaluator, or the examples.

Inline evaluators and connected test sets make that feedback loop shorter. Subject matter experts can stay in the Playground, inspect failures, adjust the data, and rerun the same cases without switching context.

Getting started​

Open the Playground, load a test set, and attach an evaluator from the evaluator control. Run the prompt to see evaluator results next to each output.

Agenta now has a dark theme. Every page is covered: the playground, registry, evaluations, observability, and the settings screens.

You switch themes from the top bar, next to the version label. There are three options: light, dark, and system. Pick system and Agenta follows your operating system, so it turns dark when your machine does and back to light in the morning.

Your choice is saved and stays the same the next time you open Agenta.

The workflow this is for​

The most useful thing you can do when building an LLM app is read your traces. You find the failures, label what went wrong, and turn the worst ones into test cases. Then you run those test cases against your evaluators.

Until now, that loop happened outside Agenta. Annotation queues bring it inside.

A queue holds a batch of traces or a batch of test cases. You attach a scoring schema (defined in the Evaluator Playground), assign reviewers, and watch progress as they work through it. The annotations live on the trace or test case they belong to, and a reviewed trace queue can be exported as a labeled test set in one step.

A walkthrough: error analysis on production traces​

You ship a new prompt. Over the next day, support tickets come in about hallucinated answers. You open observability, filter for the affected traces, and pick out fifty that look suspect.

Click Add to queue. Create a new queue and attach a human evaluator with two questions: "Is the answer correct?" (yes / no) and "If not, what went wrong?" (free text).

Your QA engineer opens the queue. Each scenario appears in a focused review view: the trace details on one side, your instructions on the other, the form below. They score one and move to the next.

When the queue is done, you have fifty labeled examples of how the prompt fails in production. Export the queue as a test set, and the annotations come along as columns. The next version of the prompt can be evaluated against ground truth your team produced, not against assumptions.

Three other things the same workflow does​

SME feedback. Hand a queue to a domain expert who doesn't need to learn the rest of Agenta. They see the trace, the instructions, and the form. Nothing else to figure out.

Bootstrapping a test set from real traffic. Sample production traces, review them, export the queue as a test set. You start with examples that look like your real users instead of examples you imagined.

Adding ground truth or rubrics to test sets. Put an existing test set into a queue, attach an evaluator that captures the reference answer or scoring rubric, and label them. The annotations become available to your evaluators on the next run.

Setting one up​

1. Open the Evaluator Playground and define a human evaluator with the questions you want answered.
2. Go to Annotations → Queues and create a new queue. Pick the kind (traces or test cases) and attach the evaluator.
3. Add items to the queue. From observability, select traces and click Add to queue. From a test set, add rows the same way. Or POST to the queue API from your own code.
4. Reviewers open the queue and work through scenarios.
5. (For trace queues) When you're done, export the queue as a test set.

Getting started​

A walkthrough video is above. For more context and to leave feedback, see the roadmap discussion.

Overview​

v0.96.0 restructures the REST API. Two changes affect most users who integrate with Agenta for prompt management:

1. Invoking a deployed prompt moves to a single unified endpoint: POST /services/{service}/v0/invoke, replacing /run, /generate, /generate_deployed, and /test.
2. Fetching a prompt configuration moves to POST /applications/revisions/retrieve, replacing POST /variants/configs/fetch.

The release also restructures the rest of the REST CRUD surface (apps, variants, environments, traces, annotations, invocations). If you call any of those directly, see the complete list of affected endpoints at the end of this guide.

Invoking a deployed prompt​

All invocations now go through one endpoint:

POST /services/{service}/v0/invoke

This replaces four legacy endpoints:

• POST /services/{service}/generate (draft testing)
• POST /services/{service}/test (draft testing)
• POST /services/{service}/generate_deployed (deployed config)
• POST /services/{service}/run (deployed config)

New request format​

The most common case is calling a prompt deployed to an environment with generate_deployed. Here is how that changes end-to-end.

Before (POST /services/completion/generate_deployed):

response = requests.post(
    "https://cloud.agenta.ai/services/completion/generate_deployed",
    headers={
        "Content-Type": "application/json",
        "Authorization": f"ApiKey {api_key}",
    },
    json={
        "environment": "production",
        "app": "my-app",
        "inputs": {"country": "France"},
    },
)

result = response.json()
print(result["data"])     # "The capital of France is Paris."
print(result["tree_id"])  # trace ID for observability

After (POST /services/completion/v0/invoke):

response = requests.post(
    "https://cloud.agenta.ai/services/completion/v0/invoke",
    headers={
        "Content-Type": "application/json",
        "Authorization": f"ApiKey {api_key}",
    },
    json={
        "data": {
            "inputs": {"country": "France"},
        },
        "references": {
            "application": {"slug": "my-app"},
            "environment": {"slug": "production"},
        },
    },
)

result = response.json()
print(result["data"]["outputs"])  # "The capital of France is Paris."
print(result["trace_id"])         # trace ID for observability

The new request body has two top-level sections. data carries your inputs and optional configuration parameters. references identifies which application, variant, revision, or environment to target.

The key field-by-field changes are:

• inputs moves under data.inputs.
• messages moves under data.inputs.messages (for chat applications).
• ag_config becomes data.parameters.
• The flat targeting fields (app, variant_slug, variant_version, environment) become structured entries under references.
• The ?application_id= query parameter is removed. Pass the ID in references.application.id instead.

New response format​

Before:

{
  "version": "3.0",
  "data": "The capital of France is Paris.",
  "content_type": "text/plain",
  "tree_id": "0ef1d6b7-84c3-4b8a-705b-ae5974e51954"
}

After:

{
  "version": "2025-07-14",
  "status": {"code": null, "message": null, "stacktrace": null},
  "trace_id": "0ef1d6b7-84c3-4b8a-705b-ae5974e51954",
  "span_id": "a1b2c3d4e5f6",
  "data": {
    "outputs": "The capital of France is Paris."
  }
}

The output value moves from data (a direct string) to data.outputs (nested). The tree_id field becomes trace_id, and we now return a span_id alongside it. The content_type field is removed. A new status object reports the operation code, an optional human-readable message, and a stacktrace when the call fails.

Field reference​

The following table maps every old field, endpoint, and response key to its new equivalent.

response = requests.post(
    "https://cloud.agenta.ai/services/chat/run",
    headers={
        "Content-Type": "application/json",
        "Authorization": f"ApiKey {api_key}",
    },
    json={
        "environment": "production",
        "app": "my-chat",
        "inputs": {"context": "Be helpful."},
        "messages": [{"role": "user", "content": "Hello!"}],
    },
)

response = requests.post(
    "https://cloud.agenta.ai/services/chat/v0/invoke",
    headers={
        "Content-Type": "application/json",
        "Authorization": f"ApiKey {api_key}",
    },
    json={
        "data": {
            "inputs": {
                "context": "Be helpful.",
                "messages": [{"role": "user", "content": "Hello!"}],
            },
        },
        "references": {
            "application": {"slug": "my-chat"},
            "environment": {"slug": "production"},
        },
    },
)

response = requests.post(
    "https://cloud.agenta.ai/services/completion/generate?application_id=xxx",
    headers={
        "Content-Type": "application/json",
        "Authorization": f"ApiKey {api_key}",
    },
    json={
        "inputs": {"country": "France"},
        "ag_config": {
            "prompt": {
                "messages": [{"role": "user", "content": "Capital of {{country}}?"}],
                "llm_config": {"model": "gpt-4o-mini"},
                "template_format": "curly",
            }
        },
    },
)

response = requests.post(
    "https://cloud.agenta.ai/services/completion/v0/invoke",
    headers={
        "Content-Type": "application/json",
        "Authorization": f"ApiKey {api_key}",
    },
    json={
        "data": {
            "inputs": {"country": "France"},
            "parameters": {
                "prompt": {
                    "messages": [{"role": "user", "content": "Capital of {{country}}?"}],
                    "llm_config": {"model": "gpt-4o-mini"},
                    "template_format": "curly",
                }
            },
        },
        "references": {
            "application": {"id": "xxx"},
        },
    },
)

Fetching a prompt configuration​

POST /variants/configs/fetch is deprecated and superseded by POST /applications/revisions/retrieve. The new endpoint mirrors the shape of the invocation endpoint: you pass structured references to identify the variant or environment you want to fetch, and the response carries the configuration parameters together with metadata about the revision.

Fetching by variant and version​

Before:

curl -X POST "https://cloud.agenta.ai/api/variants/configs/fetch" \
  -H "Content-Type: application/json" \
  -H "Authorization: ApiKey $AGENTA_API_KEY" \
  -d '{
    "variant_ref": {"slug": "default", "version": 1},
    "application_ref": {"slug": "my-app"}
  }'

After:

curl -X POST "https://cloud.agenta.ai/api/applications/revisions/retrieve" \
  -H "Content-Type: application/json" \
  -H "Authorization: ApiKey $AGENTA_API_KEY" \
  -d '{
    "application_ref": {"slug": "my-app"},
    "application_variant_ref": {"slug": "default"},
    "application_revision_ref": {"version": "1"}
  }'

Fetching by environment​

To fetch the configuration deployed to an environment, pass the application_ref and the environment_ref together:

Before:

curl -X POST "https://cloud.agenta.ai/api/variants/configs/fetch" \
  -H "Content-Type: application/json" \
  -H "Authorization: ApiKey $AGENTA_API_KEY" \
  -d '{
    "environment_ref": {"slug": "production"},
    "application_ref": {"slug": "my-app"}
  }'

After:

curl -X POST "https://cloud.agenta.ai/api/applications/revisions/retrieve" \
  -H "Content-Type: application/json" \
  -H "Authorization: ApiKey $AGENTA_API_KEY" \
  -d '{
    "application_ref": {"slug": "my-app"},
    "environment_ref": {"slug": "production"}
  }'

Response shape​

The response shape changed too. The legacy endpoint returned a flat envelope with the configuration under params and reference IDs as siblings. The new endpoint wraps the revision in a query envelope, with the configuration nested under application_revision.data.parameters.

Before:

{
  "params": {
    "prompt": {"...": "..."}
  },
  "url": "https://cloud.agenta.ai/services/completion/v0",
  "application_ref": {
    "slug": "my-app",
    "id": "019ce1cd-ccec-76f1-a803-eb1dc9187d57"
  },
  "variant_ref": {
    "slug": "default",
    "version": 1,
    "id": "019ce1cd-cd98-7270-b51b-550c58e51e41"
  },
  "environment_ref": {
    "slug": "production",
    "version": 1,
    "id": "..."
  }
}

After:

{
  "count": 1,
  "application_revision": {
    "id": "019ce1d0-7920-76a0-84b3-bb5a29750634",
    "slug": "default",
    "name": "default",
    "version": "1",
    "application_id": "019ce1cd-ccec-76f1-a803-eb1dc9187d57",
    "application_variant_id": "019ce1cd-cd98-7270-b51b-550c58e51e41",
    "data": {
      "parameters": {
        "prompt": {"...": "..."}
      },
      "url": "https://cloud.agenta.ai/services/completion/v0",
      "uri": "agenta:builtin:completion:v0",
      "schemas": {"...": "..."}
    },
    "flags": {"is_application": true, "is_chat": false, "...": "..."},
    "created_at": "2026-03-12T11:31:02.045131Z"
  }
}

If you read fields from the response, here is where each one moved.

The new response also exposes the input/output schemas under data.schemas and a richer set of flags (including is_chat, has_url, etc.), which were not available in the legacy response.

Python SDK unchanged​

If you fetch configurations through the SDK, you don't need to change anything:

config = ag.ConfigManager.get_from_registry(
    app_slug="my-app",
    environment_slug="production",
)

The SDK calls the new endpoint internally and returns the same shape it always has.

All affected endpoints​

Below is the complete list of REST endpoints moved in v0.96.0, grouped by domain.

Application invocation​

Apps CRUD​

Variants CRUD​

Variant configurations​

The git-style operations on variant configurations (commit, fork, history) are now exposed under the /applications/revisions/* namespace.

Environments and deployments​

Containers​

Annotations​

Annotations are observability records you attach to a trace or span to capture things like a human rating, an evaluator score, or a correction. They previously lived under /annotations/* and /preview/annotations/*. Both namespaces were removed. Annotations now share the unified /simple/traces/* API with other observability records.

Invocations​

Invocations are observability records that capture a single LLM call: the input, the output, latency, cost, and the model used. They previously lived under /invocations/*. They now share the unified /simple/traces/* API with annotations, distinguished by the kind field on each record.

Spans​

Need help?​

If you have questions about migrating, reach out on Slack or reply to the migration email. We are happy to help.

The Problem​

Deploying a prompt often needs follow-up work outside Agenta. You may want to sync prompt files into a repository, trigger CI, open a pull request, or notify an internal platform. Before this change, that required custom glue code.

The Solution​

You can now trigger webhooks and GitHub automations directly from Agenta when a deployment event happens. Point Agenta at your own HTTPS endpoint, or send the event straight to GitHub with repository_dispatch or workflow_dispatch.

This gives you a simple way to connect prompt deployments to the rest of your delivery flow. You can keep prompt changes, infrastructure checks, and repository updates in sync without building a separate integration service.

What You Can Do​

• Send deployment events to any HTTPS endpoint
• Verify deliveries with HMAC signatures or use a bearer token
• Trigger GitHub repository_dispatch with a structured event payload
• Trigger GitHub workflow_dispatch for one known workflow on a branch
• Fetch the latest prompt in GitHub Actions and open a pull request automatically

How It Works​

Create an automation in your project settings and subscribe it to deployment events. When a prompt revision is committed to an environment, Agenta sends an HTTP POST to your target.

For generic webhooks, Agenta sends the event payload plus delivery headers such as X-Agenta-Event-Type, X-Agenta-Delivery-Id, and Idempotency-Key. In signature mode, Agenta also signs the raw request body with HMAC-SHA256.

For GitHub automations, Agenta calls the GitHub API directly. repository_dispatch sends a richer JSON payload that includes event metadata and references. workflow_dispatch sends a smaller set of string inputs for a specific workflow file and branch.

Typical Use Cases​

• Sync the latest deployed prompt into a repository
• Open a pull request for prompt changes after each deployment
• Trigger validation or approval workflows in GitHub Actions
• Notify internal tools when a production prompt changes
• Mirror deployment metadata into another system for auditing

Getting Started​

Start with these guides:

• Webhooks
• GitHub
• Fetch Prompts via SDK/API

The Problem​

Building agentic applications that interact with external services means writing integration code: OAuth flows, API wrappers, error handling. You do all of that before you can even test whether your prompt works with the tool.

The Solution​

You can now connect external tools to your prompts and call them from the playground. Browse a catalog of 150+ integrations, authenticate with OAuth or an API key, and attach tool actions to your prompt config. When the LLM generates a tool call, you execute it with one click and send the result back to the chat.

What You Can Do​

• Use Google Sheets or Notion as data sources. Connect a spreadsheet or Notion database and let the LLM query it. Build RAG applications directly from the playground without writing any code.
• Send emails and messages. Attach Gmail or Slack and your prompt can draft and send emails, post to channels, or create threads.
• Automate developer workflows. Connect GitHub to create issues, Jira to update tickets, or any of the 150+ available integrations.
• Manage connections in Settings. A dedicated Tools page shows all your connected integrations, their status, and lets you add or revoke connections.

How It Works​

1. Go to Settings > Tools and connect an integration (e.g., Gmail). You authenticate via OAuth or paste an API key.
2. In the Playground, open the tools panel in your prompt config. Your connected integrations appear there.
3. Select the actions you want available to the LLM (e.g., "Send Email", "List Emails").
4. Run your prompt. When the LLM generates a tool call, click "Execute" to run it and return the result to the conversation.

Getting Started​

Tool integrations are available on Agenta Cloud. Connect your first integration in Settings > Tools.

The Problem​

Prompt engineering is iterative. You write a prompt, test it, notice it's too vague or missing constraints, and rewrite it. Each cycle requires you to figure out how to improve it, not just what to improve. That's the hard part.

The Solution​

You can now refine prompts with AI directly in the playground. Click the wand icon on any prompt section, describe what you want to change in plain English, and get back a refined version with an explanation of what changed and why.

How It Works​

1. Open a prompt in the playground and click the wand icon in the prompt section header.
2. A two-panel modal opens. On the left, type what you want to improve (e.g., "add output format instructions" or "make it more concise"). On the right, see the refined prompt.
3. Each refinement builds on the previous result, so you can iterate. Ask for one change, review it, then ask for another.
4. Toggle the Diff view to see exactly what changed compared to the original.
5. Edit the refined prompt directly if you want to adjust anything before applying.
6. Click Use refined prompt to apply the changes to your playground session.

You can also use the built-in quick action "Optimize the prompt using best practices" for a one-click improvement.

What You Can Do​

• Describe improvements in plain English: "Add a constraint that the output must be valid JSON" or "Make the tone more professional."
• Iterate: Each refinement uses the latest version, so you can make incremental changes across multiple rounds.
• Review diffs: Toggle the diff view to see a side-by-side comparison of the original and refined prompts.
• Edit before applying: The refined prompt is fully editable. Adjust anything before committing.
• Quick optimize: Use the built-in "Optimize the prompt using best practices" shortcut for an instant improvement.

Getting Started​

Open any prompt in the playground. Look for the wand icon in the prompt section header. If you don't see it, the feature may not be enabled for your organization yet.

Overview​

Agenta has new enterprise features. Multi-organization support, SSO, domain verification, and a US region.

Multi-Organization Support​

You can now create separate organizations for different teams or clients. Each organization has its own billing, projects, and roles. One account can belong to multiple organizations, and you switch between them without signing out.

Inside each organization, you create workspaces and projects. Workspaces group related projects together. Projects scope your prompts, evaluations, and traces.

Roles are per-organization. You can be an owner in one org and a viewer in another. Available roles include owner, workspace admin, editor, viewer, evaluator, and deployment manager.

SSO​

Connect your identity provider to Agenta. We support any OIDC-compliant provider: Okta, Azure AD, Auth0, OneLogin, Google Workspace, and others.

You configure SSO per-organization through the API. Add your provider's issuer URL, client ID, and client secret. Agenta discovers the rest from the OIDC discovery endpoint. You can test the connection before enabling it.

If your security policy requires it, you can enforce SSO-only for an organization and disable password login entirely.

Domain Verification​

Verify your company domain via a DNS TXT record. Once verified, anyone who signs up with a matching email address joins your organization automatically. No need to send invitations one by one.

Domain verification also works with SSO enforcement. Users with a verified domain email are automatically routed to your SSO provider during login.

US Region​

Agenta Cloud now has a US-based region. If your data needs to stay in the United States, you can run your projects there. The US region has the same features as the EU region.

Availability​

SSO, and domain verification are available on Business and Enterprise plans. The US region is available on all plans.

Getting Started​

• Organizations documentation
• SSO documentation

The Problem​

When you're building agents or managing multiple use cases, prompts multiply fast. Finding the right one turns into scrolling through a flat list.

The Solution​

You can now create folders and subfolders to organize your prompts. It works like any file system you've used before.

Create a folder. Drag prompts into it. Create subfolders. Move things around as your structure evolves.

What You Can Do​

• Create folders and nest them as deep as you need
• Move prompts and folders via drag-and-drop or the actions menu
• Search across all folders from the root level
• Share folder URLs with teammates (folder location persists in the URL)

Getting Started​

Go to the Prompts page. Click "Create new" and select "New folder". Name it, and start organizing.

Overview​

Getting started with a new platform can be overwhelming. You now have an onboarding widget that guides you through Agenta's key features step by step. The widget appears in the sidebar and tracks your progress as you explore the platform.

Each walkthrough highlights the relevant UI elements and explains what they do. You learn by doing, not by reading documentation.

Key Capabilities​

• Progress Tracking: The widget shows which features you've explored and what's left to discover
• Interactive Walkthroughs: Step-by-step tours that highlight UI elements as you go
• Contextual Guidance: Each walkthrough focuses on a specific workflow, from creating prompts to running evaluations
• Dismissible: Once you're comfortable, dismiss the widget and it stays out of your way

Available Walkthroughs​

The onboarding widget includes guided tours for:

• Playground Basics: Create and test prompts, compare variants, and iterate on your configurations
• Running Evaluations: Set up test sets, configure evaluators, and analyze results
• Observability: Trace your LLM calls, add annotations, and debug production issues
• Deployment: Deploy prompts to environments and fetch configurations in your code

How It Works​

The onboarding widget appears automatically for new users. Each walkthrough takes you through the actual UI, highlighting buttons, menus, and features as you go. Complete a walkthrough to mark it done, or skip ahead if you already know the basics.

You can reopen the widget anytime from the sidebar to revisit walkthroughs or check your progress.

Getting Started​

The widget appears automatically when you sign up or create a new project. If you've dismissed it and want it back, look for the help icon in the sidebar.

For detailed documentation on each feature, check out:

• Playground Quick Start
• Evaluation Quick Start
• Observability Quick Start

Overview​

You can now click directly from any trace to the application, variant, version, or environment that generated it. This makes debugging much faster. Instead of manually searching for which configuration produced a specific output, you can jump there in one click.

This is especially useful when you're investigating production issues. You see a problematic trace, click through to the exact prompt version, and start iterating immediately.

Key Capabilities​

• Clickable Application Links: Jump from a trace to the application that generated it
• Variant Navigation: Go directly to the specific variant and version used
• Environment Context: See and navigate to the environment (production, staging, development) where the trace originated
• Drawer Integration: Links appear in both the trace table and the detailed drawer view

How to Add References to Your Traces​

For the navigation links to appear, you need to store references in your traces. Here's how to do it with the Python SDK and OpenTelemetry.

Using the Python SDK​

import agenta as ag

ag.init()

# Store references to link traces to your configuration
ag.tracing.store_refs({
    "application.slug": "my-chatbot",
    "variant.slug": "gpt-4-optimized",
    "variant.version": "3",
    "environment.slug": "production",
})

# Your LLM calls are now linked to this configuration
response = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Hello!"}]
)

Using OpenTelemetry​

If you're using OpenTelemetry for instrumentation, add references as span attributes:

import { trace } from '@opentelemetry/api';

const tracer = trace.getTracer('my-app');
const span = tracer.startSpan('chat-interaction');

// Add references to link the trace
span.setAttribute('ag.refs.application.slug', 'my-chatbot');
span.setAttribute('ag.refs.variant.slug', 'gpt-4-optimized');
span.setAttribute('ag.refs.variant.version', '3');
span.setAttribute('ag.refs.environment.slug', 'production');

// Your code here
span.end();

Available Reference Keys​

You can use any combination of these reference keys:

Use Cases​

Debug Production Issues​

When a user reports a problem, find the trace and click through to see exactly which prompt version was used. No more guessing or searching through commit history.

Compare Configurations​

Filter traces by variant, then click through to compare how different configurations perform. Jump between variants to understand what changed.

Track Deployments​

See which environment generated each trace. Verify that production is running the expected version by clicking through from any trace.

Getting Started​

Learn more about storing references in our documentation:

• Reference Prompt Versions (Python SDK)
• Semantic Conventions (OpenTelemetry)
• Observability Overview

Overview​

When you compare evaluation results from last week to today, how do you know the test data didn't change? You don't. Until now.

Test set versioning tracks every change to your test sets. Each edit, upload, or programmatic update creates a new version. Evaluations link to specific versions, so you can trust your comparisons.

We also rebuilt the test set UI from scratch. It handles hundreds of thousands of rows without slowing down. Editing is faster, especially for chat messages and complex JSON data.

Test Set Versioning​

Every change to a test set creates a new version. You can see the version history, compare versions, and revert to previous versions.

What gets versioned:

• Adding, editing, or deleting test cases
• Uploading new data (CSV, JSON)
• Programmatic updates via SDK or API
• Column changes

Evaluation linking: When you run an evaluation, it links to the specific test set version used. This means:

• You can compare evaluations knowing they used the same test data
• If someone updates the test set, your historical evaluations still reference the original version
• You can filter evaluations by test set version

Programmatic versioning: Upload test sets via the SDK or API. The system detects changes and creates new versions automatically.

import agenta as ag

# Upload a test set - creates a new version if content changed
testset = ag.testsets.upload(
    name="my-test-set",
    data=test_cases,  # Your test case data
)

# The testset object includes version information
print(f"Version: {testset.version}")

New Test Set UI​

The test set view is completely rebuilt. It uses virtualized rendering, so it stays fast with large datasets.

What's new:

• Scale: Handle 100,000+ rows without performance issues
• JSON support: View and edit complex JSON directly. Toggle between raw JSON and formatted views
• String or JSON columns: Choose how each column stores data. Use JSON for structured data like chat messages

Chat message editing: Test cases with chat messages (like [{"role": "user", "content": "..."}]) now have a dedicated editor. Add, remove, or reorder messages. Edit content with proper formatting.

Upload options:

• Upload CSV or JSON files
• Create test sets in the UI
• Create programmatically via SDK
• Add spans from observability to test sets

Traceability​

Everything connects. When you view a trace in observability:

• See which test case it came from
• See which test set version
• Filter traces by test case or test set

When you view an evaluation:

• See the exact test set version used
• Compare only evaluations that used the same version
• Navigate to the test set to see the data

Getting Started​

Test set versioning is automatic. Any change creates a new version.

To use versioned test sets in evaluations:

1. Create or upload a test set
2. Make your edits (each save creates a version)
3. Run an evaluation (it links to the current version)
4. Later, compare evaluations knowing they used the same test data

For programmatic access, check the test sets documentation.

Overview​

This release brings three quality-of-life improvements to the Playground that make testing and iterating on prompts faster and more convenient.

What's New​

Provider Cost Display​

You can now see the cost per million tokens directly in the provider selection dropdown. This helps you make informed decisions about which model to use based on both capability and cost. No more switching to external pricing pages to compare costs.

Run Evaluations from the Playground​

You can now trigger evaluations directly from the Playground without navigating to the evaluation menu. When you're testing a prompt and want to run a full evaluation, click the evaluate button to start an evaluation run with your current configuration. This keeps you in flow when iterating on prompts.

Collapsible Test Cases​

Test cases in the Playground can now be collapsed. This is especially useful when working with large test sets or test cases with long inputs and outputs. Collapse completed test cases to focus on what you're working on. You can still see a preview of each test case to maintain context while navigating through your data.

Getting Started​

These features are available now in the Playground. Start using them today:

• Open the Playground and check the provider dropdown to see costs
• Click the evaluate button to run evaluations directly
• Use the collapse controls on test cases to manage your view

Overview​

Chat sessions bring conversation-level observability to Agenta. You can now group related traces from multi-turn conversations together, making it easy to analyze complete user interactions rather than individual requests.

This feature is essential for debugging chatbots, AI assistants, and any application with multi-turn conversations. You get visibility into the entire conversation flow, including costs, latency, and intermediate steps.

Key Capabilities​

• Automatic Grouping: All traces with the same ag.session.id attribute are automatically grouped together
• Session Analytics: Track total cost, latency, and token usage per conversation
• Session Browser: Dedicated UI showing all sessions with first input, last output, and key metrics
• Session Drawer: Detailed view of all traces within a session with parent-child relationships
• Real-time Monitoring: Auto-refresh mode for monitoring active conversations

How to Use Sessions​

Using the Python SDK​

Add session tracking to your application with one line of code:

import agenta as ag

# Initialize Agenta
ag.init()

# Store the session ID for all subsequent traces
ag.tracing.store_session(session_id="conversation_123")

# Your LLM calls are automatically tracked with this session
response = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Hello!"}]
)

Using the Chat Run Endpoint​

You can also instrument sessions when calling Agenta-managed prompts via the /chat/run endpoint:

import agenta as ag

# Initialize the Agenta client
agenta = ag.Agenta(api_key="your_api_key")

# Call the chat endpoint with session tracking
response = agenta.run(
    base_id="your_base_id",
    environment="production",
    inputs={
        "chat_history": [
            {"role": "user", "content": "What is the weather like?"}
        ]
    },
    # Add session metadata to group related conversations
    metadata={
        "ag.session.id": "user_456_conv_789"
    }
)

# Follow-up in the same session
follow_up = agenta.run(
    base_id="your_base_id",
    environment="production",
    inputs={
        "chat_history": [
            {"role": "user", "content": "What is the weather like?"},
            {"role": "assistant", "content": response["message"]},
            {"role": "user", "content": "What about tomorrow?"}
        ]
    },
    metadata={
        "ag.session.id": "user_456_conv_789"  # Same session ID
    }
)

Using OpenTelemetry​

If you're using OpenTelemetry for instrumentation:

import { trace } from '@opentelemetry/api';

const tracer = trace.getTracer('my-app');
const span = tracer.startSpan('chat-interaction');

// Add session ID as a span attribute
span.setAttribute('ag.session.id', 'conversation_123');

// Your code here
span.end();

The UI automatically detects session IDs and groups traces together. You can use any format for session IDs: UUIDs, composite IDs like user_123_session_456, or custom formats.

Use Cases​

Debug Chatbots​

See the complete conversation flow when users report issues. Instead of viewing isolated requests, you can analyze the entire conversation context and understand why a particular response was generated.

Monitor Multi-turn Agents​

Track how your agent handles follow-up questions and maintains context across turns. See which turns are expensive, identify where latency spikes occur, and understand conversation patterns.

Analyze Conversation Costs​

Understand which conversations are expensive and why. Session-level cost tracking helps you identify optimization opportunities and set appropriate pricing for your application.

Optimize Performance​

Identify latency issues across entire conversations, not just single requests. See which conversational patterns lead to performance problems and optimize accordingly.

Getting Started​

Learn more in our documentation:

• Track Chat Sessions (Python SDK)
• Session Tracking (OpenTelemetry)
• Observability Overview

What's Next​

We're continuing to enhance session tracking with upcoming features like session-level annotations, session comparisons, and automated session analysis.

The JSON Multi-Field Match evaluator lets you validate multiple fields in JSON outputs simultaneously. This makes it ideal for entity extraction tasks where you need to check if your model correctly extracted name, email, address, and other structured fields.

What is JSON Multi-Field Match?​

This evaluator compares specific fields between your model's JSON output and the expected JSON values from your test set. Unlike the old JSON Field Match evaluator (which only checked one field), this evaluator handles any number of fields at once.

For each field you configure, the evaluator produces a separate score (either 1 for a match or 0 for no match). It also calculates an aggregate score showing the percentage of fields that matched correctly.

Key Features​

Multiple Field Comparison​

Configure as many fields as you need to validate. The evaluator checks each field independently and reports results for all of them.

If you're extracting user information, you might configure fields like name, email, phone, and address.city. Each field gets its own score, so you can see exactly which extractions succeeded and which failed.

Three Path Format Options​

The evaluator supports three different ways to specify field paths:

Dot notation (recommended for most cases):

• Simple fields: name, email
• Nested fields: user.address.city
• Array indices: items.0.name

JSON Path (standard JSON Path syntax):

• Simple fields: $.name, $.email
• Nested fields: $.user.address.city
• Array indices: $.items[0].name

JSON Pointer (RFC 6901):

• Simple fields: /name, /email
• Nested fields: /user/address/city
• Array indices: /items/0/name

All three formats work the same way. Use whichever matches your existing tooling or personal preference.

Nested Field and Array Support​

Access deeply nested fields and array elements without restrictions. The evaluator handles any level of nesting.

Per-Field Scoring​

See individual scores for each configured field in the evaluation results. This granular view helps you identify which specific extractions are working well and which need improvement.

Aggregate Score​

The aggregate score shows the percentage of matching fields. If you configure five fields and three match, the aggregate score is 0.6 (or 60%).

Example​

Suppose you're building an entity extraction model that pulls contact information from text. Your ground truth looks like this:

{
  "name": "John Doe",
  "email": "john@example.com",
  "phone": "555-1234",
  "address": {
    "city": "New York",
    "zip": "10001"
  }
}

Your model produces this output:

{
  "name": "John Doe",
  "email": "jane@example.com",
  "phone": "555-1234",
  "address": {
    "city": "New York",
    "zip": "10002"
  }
}

You configure these fields: ["name", "email", "phone", "address.city", "address.zip"]

The evaluator returns:

You can see immediately that the model got the email and zip code wrong but correctly extracted the name, phone, and city.

Auto-Detection in the UI​

When you configure the evaluator in the web interface, Agenta automatically detects available fields from your test set data. Click to add or remove fields using a tag-based interface. This makes setup fast and reduces configuration errors.

Migration from JSON Field Match​

The old JSON Field Match evaluator only supported checking a single field. If you're using it, consider migrating to JSON Multi-Field Match to gain:

• Support for multiple fields in one evaluator
• Per-field scoring for detailed analysis
• Aggregate scoring for overall performance tracking
• Nested field and array support

Existing JSON Field Match configurations continue to work. We recommend migrating to JSON Multi-Field Match for new evaluations.

Next Steps​

Learn more about configuring and using the JSON Multi-Field Match evaluator in the Classification and Entity Extraction Evaluators documentation.

The Playground now supports PDF attachments for chat applications. You can include PDF documents in your prompts to build applications that analyze documents, answer questions about content, or extract information from files.

What is PDF Support?​

PDF support lets you attach PDF documents to chat messages when testing prompts in the Playground. The feature works with vision-capable models from OpenAI, Gemini, and Claude. These models can read and understand PDF content to answer questions or perform analysis.

This is useful when you're building applications that need to work with documents. Examples include invoice processing, contract analysis, document Q&A, or content extraction.

Supported Providers​

PDF support works with vision-capable models that handle document inputs.

How to Attach PDFs​

To attach a PDF to a chat message, click "Add attachment" in the message input. You'll see three options:

Upload a File​

Select a PDF from your computer. The file is converted to base64 and sent with your prompt.

Provide a URL​

Paste the URL to a publicly accessible PDF. The model fetches the PDF from the URL.

Use a File ID​

If you've uploaded a file through a provider's API (like the Gemini Files API), you can use the file ID instead. The model retrieves the file from the provider's storage.

Using PDFs in Evaluations​

PDF attachments work in both automatic and human evaluations. You can include PDFs in your test sets and run evaluations across multiple documents.

PDFs in Observability and Tracing​

When you trace requests that include PDFs, you can see the PDF attachment information in the trace data.

Example Use Cases​

Invoice Processing​

Create a prompt that extracts key information from invoices:

Extract the following information from this invoice:
- Invoice number
- Date
- Total amount
- Vendor name
- Line items

Return the information as structured JSON.

Attach sample invoices as PDFs. Test the prompt with different invoice formats to ensure reliable extraction across vendors.

Contract Analysis​

Build a prompt that analyzes legal contracts:

Review the attached contract and identify:
- Key obligations for each party
- Important dates and deadlines
- Termination clauses
- Liability limitations

Provide a summary in plain language.

Attach contract PDFs and verify that the model identifies critical terms consistently.

Document Q&A​

Create an assistant that answers questions about documents:

You are a document assistant. Answer the user's question based on the
attached PDF. Be specific and cite page numbers when possible.

Question: {{question}}

Attach various document types (reports, manuals, research papers) and test question-answering accuracy across different content.

Next Steps​

Learn more about using the Playground to develop and test prompts with PDF attachments.

AI coding agents like Cursor, Claude Code, VS Code Copilot, and Windsurf can now access Agenta documentation directly through the Agenta MCP server.

The MCP server implements the Model Context Protocol, allowing AI assistants to search and retrieve Agenta documentation on demand. Instead of manually searching docs, your AI agent can answer questions about Agenta features, APIs, and code examples.

Read the full setup guide →

You can now create projects within an organization. This feature helps you organize your work when you're building multiple AI products or managing different teams working on separate initiatives.

What Are Projects?​

Projects provide a way to isolate and organize your AI work within an organization. Each project maintains its own scope for:

• Prompts: All prompt templates and variants stay within the project
• Traces: Observability data is scoped to the project that generated it
• Evaluations: Test sets, evaluators, and evaluation results belong to specific projects

This scoping prevents clutter and makes it easy to focus on one product at a time.

Creating and Managing Projects​

You can create a new project directly from the sidebar in the Agenta interface. Once created, you can switch between projects using the sidebar navigation.

Each team member can work in different projects simultaneously. The interface remembers your last active project, making it easy to pick up where you left off.

When to Use Projects​

Projects work well when you need to:

• Build multiple AI products for different use cases
• Separate development work for different teams or departments
• Keep client work isolated from internal tools

Next Steps​

If you're managing complex AI initiatives across multiple products, projects give you the structure to keep everything organized. You can create your first project from the sidebar and start organizing your prompts and evaluations.

For questions about projects or organizational structure, check the FAQ or reach out through our support channels.

The Playground now supports provider built-in tools. You can use web search, code execution, file search, and other native provider tools directly when developing prompts.

What Are Provider Built-in Tools?​

Provider built-in tools are capabilities that LLM providers offer natively. Unlike custom tools that you define with JSON schemas, these tools are managed by the provider. When the model needs them, the provider handles execution and returns results automatically.

Common built-in tools include:

• Web search: Fetch current information from the internet
• Code execution: Run Python or JavaScript code
• File search: Search through uploaded documents
• Bash scripting: Execute shell commands (Anthropic)

Supported Providers and Tools​

Different providers offer different built-in tools:

OpenAI​

• Web Search: Access current information from the web
• File Search: Search through files you upload to OpenAI

Anthropic​

• Web Search: Retrieve information from the internet
• Bash Scripting: Execute bash commands in a sandboxed environment

Gemini​

• Web Search: Search the web for current information
• Code Execution: Run Python code to perform calculations and data analysis

How to Use Built-in Tools​

Adding Tools in the Playground​

1. Open your prompt in the Playground
2. Click the "Add Tool" button in the configuration panel
3. Choose the tools you want to enable for your prompt
4. Test your prompt; the model will automatically use tools when needed

The tools are saved with your prompt configuration. When you commit changes, the tool configuration is stored with the variant.

Invoking with Tools via LLM Gateway​

When you invoke prompts through Agenta as an LLM gateway, the tools are automatically included in the request. The provider handles tool execution during the call.

Your application receives the final response after all tool calls complete. You don't need to handle tool execution yourself.

Tool Definitions in the Registry​

Tool definitions follow the LiteLLM format. You can view the exact tool schemas in the Prompt Registry. This helps you understand what parameters each tool accepts and how the provider will use it.

Example Use Cases​

Research Assistant with Web Search​

Create a prompt that answers questions using current information:

You are a research assistant. Answer the user's question with accurate,
current information. Use web search when you need recent data.

Question: {{question}}

Enable web search in the tool configuration. When users ask about current events or recent data, the model automatically searches the web for information.

Data Analysis with Code Execution​

Build a data analysis prompt that performs calculations:

Analyze the following data and provide insights:

{{data}}

Calculate statistics and create visualizations as needed.

Enable code execution for Gemini. The model can run Python code to calculate statistics, process data, and generate visualizations.

Document Q&A with File Search​

Create a prompt that answers questions about uploaded documents:

Answer the user's question based on the uploaded documentation.
Be specific and cite relevant sections.

Question: {{question}}

Enable file search for OpenAI. The model searches through your uploaded files to find relevant information.

Next Steps​

Learn more about using the Playground to develop and test prompts with provider built-in tools.

You can now configure reasoning effort for models that support this parameter, such as OpenAI's o1 series and Google's Gemini 2.5 Pro.

Reasoning effort controls how much computational thinking the model applies before generating a response. This is particularly useful for complex reasoning tasks where you want to balance response quality with latency and cost.

The reasoning effort parameter is part of your prompt template configuration. When you fetch prompts via the SDK or invoke them through Agenta as an LLM gateway, the reasoning effort setting is included in the configuration and applied to your requests automatically.

This gives you fine-grained control over model behavior directly from the playground, making it easier to optimize for your specific use case.