Experiment on LLM prompts and AI-agent configurations

Use Kameleoon feature flags to run controlled experiments on the prompts, models, and configurations behind an LLM application or AI agent, and measure impact with AI-quality, operational, and business metrics.

LLM-powered applications and AI agents are non-deterministic. Changing a prompt, model, retrieval strategy, tool configuration, or agent workflow can affect response quality, operating cost, latency, and user behavior in different, hard-to-predict ways. Evaluation frameworks such as RAGAS score whether an individual response meets a quality bar, but that score doesn’t tell you whether the change helps users accomplish what they came to your product to do.

You can use Kameleoon Feature Experimentation capabilities to customize, test, and roll out the configuration behind a generative AI application or AI agent. A feature variable holds a single piece of that configuration (a prompt, a model parameter, a retrieval strategy, a tool definition), so your team can manage it outside your application code. Each variation represents a candidate configuration, which lets you iterate, experiment, and release changes more safely, without redeploying. For example, an AI customer-support agent might expose its system prompt, model, reasoning effort, and retrieval settings as four separate variables, so you can test a new combination of all four at once instead of waiting on a deploy for each one.

Split traffic between variations and compare their impact using several types of metrics:

• AI-quality metrics, such as correctness, groundedness, relevance, context quality, safety, or compliance
• Agent-performance metrics, such as successful tool use or task completion
• Operational metrics, such as latency, token consumption, errors, and cost
• User and business metrics, such as satisfaction, escalation, conversion, retention, and revenue

Because the configuration lives in a feature flag instead of your source code, you can add, edit, or roll back a variation directly from the Kameleoon platform at any time.

Evaluation and experimentation solve different problems

Evaluation determines whether an individual model or agent output meets a defined quality standard. Observability tools let you inspect the prompts, responses, traces, retrieval steps, and tool calls behind that output. Experimentation determines whether a change to the underlying configuration causes a measurable improvement for users or the business: a question neither evaluation nor observability answers.

For example, an LLM judge might score a customer-support agent’s rewritten system prompt as more grounded than its current one. A Kameleoon experiment answers the questions your team is accountable for: whether that same configuration resolves more tickets without escalating to a human, and what it costs in latency and token spend to get there.

Kameleoon doesn’t replace your LLM observability or evaluation stack. Feed evaluator scores from RAGAS, an LLM judge, or a human review process into Kameleoon as a custom goal, and track those scores alongside the behavioral and business goals your experiment already measures. Your model-level quality signals then connect to a statistically reliable measurement of real user impact.

For most AI experiments, combine several metric types rather than relying on one:

• Set a user or business outcome as the primary goal.
• Track AI-quality metrics as secondary goals or guardrails.
• Monitor latency, cost, errors, and safety as operational guardrails.
• Validate automated judges against a sample of human-reviewed examples before you trust their scores at scale.

How it works

An experiment on an LLM application or AI agent moves through five stages in Kameleoon:

1. A feature flag holds the configuration. Each part of the configuration, such as a prompt or a model name, becomes a feature variable on the flag.
2. Each variation sets its own values. A variation is one complete candidate configuration, with a value for every variable.
3. The SDK assigns each visitor a variation. When a visitor reaches your application, your code requests the flag and receives the assigned variation’s values, then uses them to call the LLM or configure the agent.
4. Goals record what happened. Your application tracks a conversion against each goal attached to the flag, including a guardrail conversion that fires only when a response breaches an acceptable quality or latency threshold.
5. The results page compares the variations. After you collect enough traffic, compare the variations across every attached goal to decide which configuration to roll out.

Prerequisites

• A Kameleoon account with a project set up for Feature Experimentation.
• Your account’s client ID and client secret. To find these values, see API credentials.
• A server-side application where you can install a Kameleoon SDK, for example a Python application.

Set up your AI-agent experiment in Kameleoon

The following steps build one concrete experiment: does a rewritten system prompt, combined with a stronger model and a higher reasoning effort, help a customer-support AI agent resolve more tickets on its own, and does it keep latency within an acceptable range while doing so? Configure the feature flag, variations, and tracking goals in the Kameleoon platform before you touch your application code.

Create the feature flag

Create a feature flag to hold your agent’s configuration and control the rollout of your experiment.

1. In the Kameleoon app, click Features > Flags & Experiments > New feature flag.
2. Enter a name, for example AI support agent config, and select the project for the flag.
3. In the Description field, note what the flag controls, for example “Controls the system prompt, model, reasoning effort, and retrieval settings for the support chatbot,” so other members of your team understand its purpose.
4. Click Validate.
5. Kameleoon generates a feature key from the flag’s name. Note the generated key, or edit it to ai_support_agent. Your application code identifies the flag by this key, not by its name, so the two must match.

New flags start in the OFF state. You turn the flag on in the Rollout Planner after you finish configuring it.

For more detail, see Create a feature flag.

Store the agent configuration in feature variables

Add one feature variable for each part of the agent’s configuration you want to test, so you can change any of them from the Kameleoon platform without editing your application code. This example tests four variables:

Variable key	Type	Example default value
system_prompt	String	You are the support assistant for Kameleoon's help center. Answer only using the article excerpts provided below, and say you'll escalate to a human agent if the answer isn't in them.
model	String	claude-sonnet-5
reasoning_effort	String	low
retrieval_top_k	Number	3

1. On the flag’s page, in the left sidebar, click Set Up > Variables > Add Variable.
2. Set the variable’s Type to match the table, either String or Number.
3. Enter the Variable Key from the table, for example system_prompt.
4. Set the Default Value to the value your application uses in production today. Every variation you create later starts pre-filled with these values, so accurate defaults save you work and give you a known-good configuration to fall back on.
5. Click Save.
6. Repeat these steps for each remaining variable in the table.

Kameleoon also offers an Enum type, where you enter the allowed values as a comma-separated list and then pick from a drop-down when you define variations. Consider it for a variable that only accepts a fixed set of values, since a drop-down prevents a typo from reaching your LLM provider. In this example, you could define model as an Enum with the list claude-sonnet-5,claude-opus-5, and reasoning_effort as an Enum with the list low,medium,high.

For more detail, see Define feature variables.

Create your variations

A delivery rule can only serve Off or a variation you created yourself, so an A/B comparison needs a variation for your current configuration as well as one for each new configuration you want to test. This example creates two variations: Baseline, which mirrors what your application already serves in production, and Grounded, high reasoning, a challenger that combines a rewritten, more explicit prompt with a stronger model and more reasoning effort.

Don’t use Off as the comparison arm of an experiment, even if your application already has a hardcoded fallback for when it can’t read the flag. Off represents your application with the flag turned off, and it carries none of the flag’s feature variables, so code that reads system_prompt or model from an Off assignment gets nothing back. A hardcoded fallback doesn’t solve that problem either: it lives in your source code, not in Kameleoon, so promoting a winning configuration later still means a deploy, and nothing keeps it in sync if the flag’s variables change. Create an explicit Baseline variation instead, so your current configuration stays visible and editable alongside the challenger you’re testing against it.

Variation	system_prompt	model	reasoning_effort	retrieval_top_k
Baseline	Same as the default value	claude-sonnet-5	low	3
Grounded, high reasoning	You are the support assistant for Kameleoon's help center. Using only the article excerpts provided below, answer the visitor's question and cite the specific article title for each claim you make. If the excerpts don't fully answer the question, say so explicitly and escalate to a human agent instead of guessing.	claude-opus-5	high	5

1. In the left sidebar, click Set Up > Variations > Add variation.
2. Enter a Name, for example Baseline, and a matching Variation Key, for example baseline. Each variable is pre-filled with its default value, so leave all four unchanged.
3. Click Save.
4. Repeat these steps for a second variation named Grounded, high reasoning (key grounded_high_reasoning), editing each of the four variables to match the value in the table for this variation.
5. Click Save.

For more detail, see Define feature variations.

Attach goals for business impact, AI quality, and latency

Attach several goals to the feature flag so you can compare variations across the outcomes your team is accountable for, not just response quality. Because Kameleoon is a unified platform, you can attach any goal that already exists in your organization, or create a goal specific to your LLM-powered feature. This example attaches three goals, one from each of the metric categories that matter for an AI agent:

Goal	Metric category	Type	What it answers
Ticket resolved without escalation	User and business	Custom goal	Did the agent actually solve the visitor’s problem?
Response groundedness score	AI quality	Custom goal	Did more responses fall below an acceptable groundedness threshold?
Response latency	Operational	Custom goal	Did more responses take longer than an acceptable threshold?

Create all three as Custom goals that your backend triggers, since your application fires them rather than the visitor’s browser. When you create each goal, select Custom goal as the Type, then choose the option for a back-end event via SDK.

1. On the flag’s page, in the Set Up menu, click Goals > Add goal.
2. Select an existing goal, or click Create a new goal to define one. Add Ticket resolved without escalation first, since Kameleoon automatically sets the first goal you attach as the Primary goal.
3. Click Save.
4. Repeat these steps for Response groundedness score and Response latency, which Kameleoon attaches as Secondary goals.

If a goal ends up with the wrong designation, click the three dots next to it to reassign which goal is primary.

Response groundedness score and Response latency don’t carry a numeric value. Your application decides whether a given response breached an acceptable threshold, either too slow or too ungrounded, and triggers the goal’s conversion only when it did. Kameleoon then reports each goal’s conversion rate per variation, telling you what fraction of responses breached that threshold.

For more detail on goal types, including how to trigger a custom goal from your backend, see Create a goal.

Roll out the experiment

Add an Experiment rule that splits traffic between your two variations, then turn the flag on to start collecting data. The Add a rule menu groups rules by purpose. Feature testing contains the Experiment rule, which splits traffic and measures a statistically significant comparison between variations, while Feature delivery contains Progressive delivery and Targeted delivery, which release a single variation gradually or to a specific segment without comparing arms. An A/B test needs the Experiment rule.

1. In the Rollout Planner, select the environment you want to target, for example Production.
2. Click Add a rule, then under Feature testing, select Experiment.
3. Under Variations to serve, set Baseline as the Control and add Grounded, high reasoning as a Treatment. Kameleoon measures every treatment’s results against the control, so the control must be the configuration you already run in production.
4. Set the traffic distribution across the two variations, for example 50% each.
5. Set the rule’s targeting to include the visitors you want to test, for example all visitors who open a support conversation.
6. In the Then, for everyone else in production, serve drop-down, select Baseline. Visitors who fall outside the rule’s targeting then receive your current, validated configuration, and your application still gets a full set of variables for them.
7. Turn the flag’s ON/OFF toggle to ON.
8. Click Save.

For more detail, see Create feature experiments.

After you save the rule, Kameleoon starts assigning visitors to a configuration and serving the corresponding variables. To change a value or add a variation later, edit it directly in the Kameleoon platform. You don’t need to redeploy your application to make these changes.

Retrieve the configuration in your application

Install the Kameleoon Python SDK, then retrieve the visitor’s assigned configuration and track a conversion for each goal as the visitor’s ticket progresses. The same pattern applies to any Kameleoon server-side SDK, including Node.js, Java, and Go. Because Kameleoon only supplies the configuration values, the same pattern also works with any agent framework, such as the OpenAI Agents SDK, the Claude Agent SDK, or LangChain. Most agent code runs on Python or TypeScript, so pick whichever matches your application.

1. Install the SDK as a dependency:

   pip install kameleoon-client-python
   

2. Initialize the client with your site code and credentials. Set environment to the same Rollout Planner environment that holds your Experiment rule, otherwise the SDK evaluates a different environment’s rules:

   from kameleoon import KameleoonClientConfig, KameleoonClientFactory
   
   SITE_CODE = "a8st4f59bj"
   FEATURE_KEY = "ai_support_agent"
   TASK_GOAL_ID = 12345      # "Ticket resolved without escalation"
   QUALITY_GOAL_ID = 12346   # "Response groundedness score"
   LATENCY_GOAL_ID = 12347   # "Response latency"
   
   configuration = KameleoonClientConfig(
       client_id="your-client-id",
       client_secret="your-client-secret",
       environment="production",
   )
   kameleoon_client = KameleoonClientFactory.create(SITE_CODE, configuration)
   
   if not await kameleoon_client.wait_init_async():
       # The SDK failed to initialize. Fall back to your production configuration.
       ...
   

3. Retrieve the assigned configuration before you call your LLM, and track a conversion for each goal as the visitor’s ticket progresses:

   from anthropic import AsyncAnthropic
   
   judge_client = AsyncAnthropic()
   
   # Mirrors the flag's default values, for use when the SDK can't reach Kameleoon.
   FALLBACK_CONFIG = {
       "system_prompt": "You are the support assistant for Kameleoon's help center. ...",
       "model": "claude-sonnet-5",
       "reasoning_effort": "low",
       "retrieval_top_k": 3,
   }
   
   # A response breaches the latency guardrail above this value.
   LATENCY_THRESHOLD_MS = 5000
   # A response breaches the groundedness guardrail below this value.
   QUALITY_THRESHOLD = 0.7
   
   
   def get_agent_config_for_visitor(visitor_code: str) -> dict:
       try:
           variation = kameleoon_client.get_variation(visitor_code, FEATURE_KEY)
       # These exception classes ship with the Kameleoon SDK.
       except (FeatureNotFound, FeatureEnvironmentDisabled, VisitorCodeInvalid):
           # The visitor isn't in the experiment, so serve the known-good configuration.
           return FALLBACK_CONFIG
   
       variables = variation.variables
       if not variables:
           return FALLBACK_CONFIG
   
       return {
           "system_prompt": variables["system_prompt"].value,
           "model": variables["model"].value,
           "reasoning_effort": variables["reasoning_effort"].value,
           "retrieval_top_k": int(variables["retrieval_top_k"].value),
       }
   
   
   def track_ticket_resolved(visitor_code: str) -> None:
       kameleoon_client.track_conversion(visitor_code, TASK_GOAL_ID)
   
   
   # LLM-as-a-judge: extracts the factual claims in `response` and scores
   # the fraction that `retrieved_context` supports.
   async def score_response_groundedness(retrieved_context: str, response: str) -> float:
       judge_prompt = (
           "List the factual claims in RESPONSE. For each claim, check "
           "whether CONTEXT supports it. Return only a number from 0 to 1: "
           "the fraction of RESPONSE claims that CONTEXT supports.\n\n"
           f"CONTEXT:\n{retrieved_context}\n\nRESPONSE:\n{response}"
       )
       judge_reply = await judge_client.messages.create(
           model="claude-opus-5",
           max_tokens=10,
           messages=[{"role": "user", "content": judge_prompt}],
       )
       return float(judge_reply.content[0].text)
   
   
   def track_quality_score(visitor_code: str, score: float) -> None:
       if score < QUALITY_THRESHOLD:
           kameleoon_client.track_conversion(visitor_code, QUALITY_GOAL_ID)
   
   
   def track_response_latency(visitor_code: str, latency_ms: float) -> None:
       if latency_ms > LATENCY_THRESHOLD_MS:
           kameleoon_client.track_conversion(visitor_code, LATENCY_GOAL_ID)
   

   Call get_agent_config_for_visitor() with the visitor’s visitor_code before you send a request to your LLM, and use the returned values to build the request, the system prompt, model, reasoning effort, and number of retrieved documents. Call track_ticket_resolved() when the agent resolves the visitor’s issue without escalating to a human. After the agent responds, call score_response_groundedness() with the documents it retrieved and the response it generated, then pass the returned score to track_quality_score(). Call track_response_latency() with the response time in milliseconds after each LLM call. Both track_quality_score() and track_response_latency() track a conversion only when the value breaches its threshold, so a response that stays within both guardrails triggers neither goal.

   score_response_groundedness() is a minimal LLM-as-a-judge example: it asks a model to compare the response’s claims against the retrieved context and return the fraction it supports. RAGAS’s Factual Correctness metric scores the same underlying idea, and you can substitute it, or another evaluation framework your team already uses, for a hand-rolled judge prompt.

   Kameleoon then reports each goal’s conversion rate per variation, telling you what fraction of responses breached the quality or latency guardrail, rather than tracking the raw score or millisecond value itself.

Always handle the case where a visitor falls outside the experiment. An LLM call built from a missing prompt or model fails at request time, so return a complete fallback configuration rather than letting a lookup raise or return None.

Use get_visitor_code() to assign a unique ID to each visitor, and set_legal_consent() if your application requires visitor consent before tracking data. For the full client initialization and configuration reference, see the Python SDK developer guide.

Monitor and iterate

Open the feature flag’s results page to compare Baseline and Grounded, high reasoning across all three attached goals. Kameleoon tracks exposures and conversions automatically whenever your application calls get_variation() and track_conversion(), so you don’t need any additional instrumentation.

Read the three goals together rather than in isolation. The challenger in this example runs a larger model at a higher reasoning effort and retrieves more documents, so it costs more per conversation and is more likely to breach the latency guardrail. A win on Ticket resolved without escalation justifies that trade only if Response latency and Response groundedness score don’t convert more often for the challenger than for Baseline. If the primary goal moves but a guardrail’s conversion rate rises past what you’re willing to accept, keep serving Baseline and refine the challenger.

You don’t need to watch the results page yourself to catch a challenger that underperforms. Add a rollback condition to the experiment rule, for example turning it off when Response groundedness score breaches a threshold you define, and Kameleoon automatically disables the rule and returns all visitors to Baseline once the condition triggers.

See Automatically rollback a feature.

When a challenger wins, promote it: update each variable’s Default Value to the winning configuration so it becomes the new known-good baseline, then retire the experiment rule or reuse the variation for your next hypothesis.

For more detail, see Analyze a feature flag’s overall results.

Next steps

• Read the Python SDK reference for advanced options such as custom data, cross-device experimentation, and targeting conditions.
• Attach precise segmentation criteria to target the experiment at a specific audience, for example only tickets tagged with a certain product area.
• Add a token-cost goal alongside latency, tracking tokens consumed per conversation as a numeric custom goal, so you can price the difference between a Sonnet and an Opus configuration directly. See Create a goal.
• Add a context-relevance goal to check whether raising retrieval_top_k actually improves which documents the agent retrieves, since a grounded answer can still come from the wrong documents. See Create a goal.
• Add a direct user-feedback goal, such as a thumbs-up or thumbs-down control after each response, to capture visitor satisfaction alongside the behavioral signals this example already tracks. See Create a goal.
• Add more variables to test other parts of the agent’s configuration, such as temperature, tool definitions, or a fallback model for retries. See Define feature variables.
• Validate your Response groundedness score threshold by comparing a sample of automated scores against a human review before you trust it at scale. See Create goals for feature flags.