> ## Documentation Index
> Fetch the complete documentation index at: https://koreai-content-gov.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# By AI Agent Metric

The By AI Agent metric enables supervisors to use AI agents to evaluate conversations across multiple dimensions. Quality AI supports two versions of this metric:

* **By AI Agent V1**, which uses the existing Agent Platform V1 integration.
* **By AI Agent V2 (Artemis)**, which uses Agent Platform V2 (Artemis) with project-based authentication.

Both versions use a parent metric with multiple sub-metrics, each with its own question, weight, and logic. All sub-metrics are evaluated in a single API call, with detailed justifications provided for each aspect.

Supervisors can also pass `requestMeta`-based metadata for the Execute API request, including `customConversationId` and configured custom fields.

This guide explains how to configure and use the By AI Agent metric in Quality AI, including version selection, agentic application configuration, sub-metrics, custom field propagation, response format, and evaluation results.

## When to Use This Metric

Use this metric type for evaluation scenarios that require:

| Scenario                          | Description                                                                                                   |
| --------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| **Multi-dimensional Assessments** | Evaluate several facets (sub-metrics) under one parent metric.                                                |
| **Autonomous AI Analysis**        | Leverage AI agents to interpret, reason, and assess interactions using contextual understanding.              |
| **Weighted Evaluations**          | Assign different weights to sub-metrics to prioritize specific aspects.                                       |
| **Efficient Execution**           | Reduce redundant API calls by evaluating multiple sub-metrics within one agentic request.                     |
| **Seamless Configuration**        | Select agentic applications from the same workspace without entering endpoint URLs.                           |
| **Context-aware Evaluations**     | Pass custom metadata (for example, customer ID, ticket ID) to enable external data lookups during evaluation. |

## Prerequisites

Before creating a By AI Agent metric, confirm:

* You have access to both Quality AI and Agent Platform.
* The same workspace is available across both platforms.
* You have permissions to view and deploy agentic applications.
* The By AI Agent Metric feature is enabled for your workspace account.
* You have configured at least one agentic app on the Agent Platform with the required response structure.
* Custom fields exist in the Quality AI custom field registry if you want to pass request metadata.

**For By AI Agent V2 (Artemis):** You must have access to Agent Platform V2 (Artemis) in the same workspace and have a deployed Artemis project with its **Project Key**, **Project ID**, **Deployment ID**, and **Channel ID** available.

For information about setting up and deploying an Artemis project and obtaining these connection details, see [Quality AI integration with Artemis](/agent-platform/integrations/quality-ai#configure-a-by-ai-agent-metric).

## Choose a Version: V1 or V2

When you add a By AI Agent metric in Quality AI, select the version you want to use.

* **By AI Agent V1**: Uses the existing Agent Platform V1 integration.
* **By AI Agent V2 (Artemis)**: Uses Agent Platform V2 (Artemis) with project-based authentication.

| Capability                    | By AI Agent V1                                             | By AI Agent V2 (Artemis)                                                                    |
| ----------------------------- | ---------------------------------------------------------- | ------------------------------------------------------------------------------------------- |
| **Connection**                | Select an agentic app and environment from your workspace. | Connect to an Artemis project using Project Key, Project ID, Deployment ID, and Channel ID. |
| **Authentication**            | Uses the current Agent Platform V1 configuration settings. | Project-key-based token, created and refreshed automatically by the system.                 |
| **Sub-metrics and weightage** | Supported.                                                 | Supported (same as V1).                                                                     |
| **Custom field propagation**  | Supported.                                                 | Supported (same as V1).                                                                     |
| **Audit logs**                | Standard evaluation results.                               | Adds a per-evaluation request payload, response, and debug trace.                           |

***

## Configure By AI Agent Metric

### Step 1: Navigate to Metric Configuration

1. Navigate to **Quality AI** > **Configure** > **Evaluation Forms** > **Evaluation Metrics**.
2. Select **+ New Evaluation Metric**.
3. From the **Evaluation Metrics Measurement Type** dropdown, select **By AI Agent**.
4. Select the version, **By AI Agent V1** or **By AI Agent V2 (Artemis)**.

### Step 2: Create the Parent Metric

1. Enter a descriptive **Name** (for example, Compliance Disclosure).
2. Select the **Language** for the AI Agent's evaluation.
3. The **Question** field is defined later under sub-metrics.

### Step 3: Connect to the Agent Platform

Provide the connection details for the version you selected.

#### By AI Agent V1

1. In the **Agent App** dropdown, choose from available apps in your workspace.
2. Select the **Environment** (for example, Draft, Version 1, Version 2).

#### By AI Agent V2 (Artemis)

Enter the details of the Artemis project to evaluate against:

| Field             | Description                                                                                                     |
| ----------------- | --------------------------------------------------------------------------------------------------------------- |
| **Project Key**   | Identifies the Artemis project. The system uses it to create and periodically refresh the authentication token. |
| **Project ID**    | Used with the authentication token to call the agent API and determine the specific agent.                      |
| **Deployment ID** | Identifies the specific agent deployment used for evaluation.                                                   |
| **Channel ID**    | Identifies the channel that attributes the source of the request.                                               |

<Note> By AI Agent V2 uses project-key-based authentication. The system creates the authentication token from the Project Key, refreshes it periodically, and sets the required origin header on the run API automatically, no manual token management is needed.</Note>

### Step 4: Test Connection and Fetch Sub-Metrics

1. Select **Test Connection**.
2. The system sends a test call to the selected app and retrieves available sub-metrics for configuration.
3. On success, the panel confirms the connection and shows how many sub-metrics the agent application includes.

<Note> Test Connection behaves the same for V1 and V2. If the agentic app response doesn't match the required contract, Test Connection fails and blocks configuration.</Note>

***

### Step 5: Configure Sub-Metrics

Upon successful connection, the system displays all sub-metrics returned by the agentic app with their reference names.

Select **Edit** next to the **Weightage** field to open the sub-metric configuration panel, where you can define the following:

| Field                   | Description                                                                             |
| ----------------------- | --------------------------------------------------------------------------------------- |
| **Display Name**        | Label for the sub-metric.                                                               |
| **Question**            | Evaluation question for this sub-metric.                                                |
| **Evaluation Criteria** | Assign a weightage to each outcome (**Yes** / **No**).                                  |
| **Fatal Error**         | If enabled, failing this sub-metric marks the entire interaction as a critical failure. |

<Note> Sub-metric weightages must add up to 100%, they split the score within the parent metric's total.</Note>

### Step 6: Configure Custom Field Propagation

Optionally configure custom fields to send to the Agent Platform as metadata for conversation dependent evaluation. These fields populate the `requestMeta` object of the [Execute API request](/agent-platform/v1/apis/agentic-apps/execute#sample-request).

For Agent AI and Express sources, `customConversationId` is automatically included in `requestMeta`.

1. Select a conversation-level **Custom Field**.
2. Define **Header Name** as the key in `requestMeta`.
3. Add multiple mappings using **+ Add Custom Field**.

When all details are configured, select **Create** to save the metric for AI Agent evaluation.

***

## Configure the Agent Platform response

Quality AI expects the Agent Platform application to return the response in the JSON format below to process the sub-metric results.

1. Navigate to your AI Agent configuration in the Agent Platform.
2. Locate the Description field.
3. Follow the [Response format](/ai-for-service/quality-ai/configure/evaluation-criteria/metrics-measurement-types/by-ai-agent#response-format-for-sub-metrics) for Sub-metrics specification.

***

## Example Use Case: UDAP Compliance

For financial services compliance, a single parent metric can evaluate multiple aspects in one API call:

| Sub-Metric             | Weight | What It Evaluates                         |
| ---------------------- | ------ | ----------------------------------------- |
| Fee Disclosure         | 25%    | All applicable fees are clearly explained |
| Interest Rate Accuracy | 30%    | Interest rate information is accurate     |
| Benefit Explanation    | 20%    | Benefits are clearly described            |
| Exclusion Details      | 15%    | All exclusions are clearly listed         |
| Terms Clarity          | 10%    | Overall clarity of terms                  |

Each sub-metric is evaluated independently with a single API call, providing detailed justifications for each aspect.

***

## Evaluation Flow

The system sends a single evaluation request that includes:

* Conversation data (transcripts and sub-metrics).
* `requestMeta` (`customConversationId` and configured custom fields).

The agent evaluates all sub-metrics and returns structured results. The system maps results and displays adherence with reasoning.

***

## Request Metadata in Execute API

The system sends metadata in the `requestMeta` object of the Execute API.

The `requestMeta` object includes:

* **Contents**: The `customConversationId` (automatically included for Agent AI and Express sources) and custom fields configured for the metric, represented as key-value pairs.
* **Custom Field Mapping Rules**: The system derives keys from Header Names and sources values from conversation-level custom fields. It supports configuration of multiple custom fields per metric.

<Note> `customConversationId` is used in the `requestMeta` request object, while `conversationId` is returned as part of the evaluation response.</Note>

**Example**:

```json theme={null}
{
  "requestMeta": {
    "customConversationId": "AWS_Mono_22April0xxxx",
    "phone_number": "862684xxxx"
  }
}
```

<Note> This metadata is used only during evaluation execution and isn't stored in the results. </Note>

***

## Response Format for Sub-metrics

The Agent Platform application must return the following JSON structure for Quality AI to process the sub-metric results.

```json theme={null}
{
  "botId": "string",
  "accountId": "string",
  "conversationId": "string",
  "agentEvaluation": [
    {
      "PARENTMETRIC_ID_VALUE": {
        "subMetrics": [
          {
            "subMetricId": "string",
            "subMetricName": "string",
            "justification": "string",
            "messageIds": ["array"],
            "timestamps": ["array"],
            "source": "agent | customer",
            "isQualified": "YES | NO | NA",
            "failureReason": "string"
          }
        ]
      }
    }
  ]
}
```

### Sample Response

```json theme={null}
{
  "botId": "bot_001",
  "accountId": "account_001",
  "conversationId": "conv_001",
  "agentEvaluation": [
    {
      "eval_001": {
        "subMetrics": [
          {
            "subMetricId": "sm_001",
            "subMetricName": "Loan Inquiry Identification",
            "justification": "Agent correctly identified the customer's loan-related query.",
            "messageIds": ["msg_001"],
            "timestamps": ["2025-10-17T10:00:00Z"],
            "source": "agent",
            "isQualified": "YES",
            "failureReason": ""
          },
          {
            "subMetricId": "sm_002",
            "subMetricName": "Loan Eligibility Explanation",
            "justification": "Agent provided loan eligibility information.",
            "messageIds": ["msg_002", "msg_004"],
            "timestamps": ["2025-10-17T10:00:10Z", "2025-10-17T10:00:35Z"],
            "source": "agent",
            "isQualified": "YES",
            "failureReason": ""
          }
        ]
      }
    }
  ]
}
```

<Note> The Agent Platform contract strictly defines the response format, and no one can modify it. Quality AI only consumes and maps the response. </Note>

## Audit Logs (By AI Agent V2)

**For By AI Agent V2 (Artemis)** metrics, the Audit Screen records an audit log for each evaluation so you can review and debug agent calls. The log lists one entry per sub-metric evaluation:

| Column              | Description                                                              |
| ------------------- | ------------------------------------------------------------------------ |
| **Timestamp**       | Date and time of the evaluation call.                                    |
| **Request Payload** | The payload sent to the agent API.                                       |
| **Response**        | The response returned by the agent.                                      |
| **Sub-metric**      | The sub-metric evaluated in the call.                                    |
| **Status**          | Outcome of the call, **Success** or **Failed**.                          |
| **View Payload**    | Opens the full request and response, including the trace, for debugging. |

## Managing Evaluation Metrics

### Edit and Delete Evaluation Metrics

Steps to edit and delete existing Evaluation Metrics:

1. Select an **AI Agent** metric.

2. Select **Edit** to update the required metric details and fields.

<Note> When you update a By AI Agent V2 connection field (Project Key, Project ID, Deployment ID, or Channel ID), the system records a change log entry, for example: "Project ID was updated from \{old value} to \{new value} for metric \{metric name} of type By AI Agent."</Note>

### Delete Evaluation Metrics

Before deleting a metric:

* Remove it from all associated evaluation forms (for example, Chat Form – COMMON, New Points Based).
* Reassign any linked attributes (for example, Agent AI Metric Attribute-1) to a different metric.

The system allows deletion only after you resolve all dependencies and save the changes.

***