> ## Documentation Index
> Fetch the complete documentation index at: https://koreai-content-gov.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.
# Calibration Sessions
Calibration Sessions help QA teams improve scoring consistency by having multiple participants independently evaluate the same customer conversations using the same evaluation form.
A Calibration Manager creates the session, selects conversations and the evaluation form, and assigns participants. Participants score conversations independently in **Calibration Mode**, with peer scores hidden to prevent influence.
After submissions, the manager uses **Debrief View** to compare evaluations, resolve scoring differences, and set **Reference Scores**. Closing the session generates a shareable **Calibration Session Report** with agreement results, analyst accuracy, and resolution notes.
The experience supports both session management and participant scoring. Calibration Managers can also participate when assigned, while users with **QA functionalities access+** can manage their assigned work through **Assigned to Me** and review completed sessions through **Closed**.
## Key Capabilities
Calibration Sessions provide the following capabilities.
| Capability | Description |
| -------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Session Configuration** | Create a session with selected conversations, an evaluation form or scorecard, participants, and session settings. |
| **Assign Participants** | Assign multiple participants to the same conversations so they can evaluate them independently. |
| **AutoQA Visibility** | Configure when AutoQA recommendations are visible during calibration. |
| **Calibration Mode** | Provide a dedicated experience where participants evaluate conversations independently while peer scores remain hidden. |
| **Submission Confirmation** | Allow participants to review and confirm their evaluations before submitting them. |
| **Monitor Evaluation Progress** | Track participant submission and conversation-level scoring progress. |
| **Debrief View** | Compare participant scores side by side and review metric-level scoring differences after all participants submit their evaluations. |
| **Scoring Analysis** | Review metric-level disagreements, participant notes, and AutoQA AI Justification when available. |
| **Reference Score** | Establish the final expected score for a metric after reviewing participant evaluations and debrief discussions. |
| **Agreement and Accuracy** | Measure participant agreement and accuracy using the established Reference Score as the benchmark. |
| **Calibration Session Report** | Review and export calibration outcomes, agreement results, participant accuracy, conversation-level results, and Resolution Notes. |
| **Resolution Notes** | Document decisions and outcomes from the debrief and Reference Score resolution process. |
| **Session History** | Enable Calibration Managers to review closed sessions within their management scope and participants to review sessions in which they participated. |
| **Support Participant Coaching** | Use calibration results to identify scoring inconsistencies and coaching opportunities. |
## Prerequisites
Before creating a Calibration Session, confirm that you have the following:
| Requirement | Description |
| -------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Quality AI Permissions** | Require Calibration Manager permission for session management, including creating and managing sessions, accessing management views, debriefing, establishing Reference Scores, resolving scoring differences, and closing sessions. QA functionalities access+ provides access to the participant-facing Calibration experience. |
| **Evaluation Form** | Use a published Human Agent or AI Agent evaluation form that applies to the selected calibration conversations. |
| **Conversations** | Select conversations that match the evaluation form and configured filters. |
| **Participants** | Assign at least two participants to enable scoring comparison. |
| **Calibration Scenarios** | Select conversations that represent the scoring scenarios the team needs to calibrate. |
| **AutoQA Evaluation** | Require available AutoQA evaluation results when the session uses AutoQA recommendations. |
## Access Calibration Sessions
Navigate to **Quality AI** > **AutoQA** > **Calibration Sessions**.
Users see the Calibration experience based on their permissions and session participation:
* **Calibration Managers** can access the management experience.
* Users with **QA functionalities access+** can access the participant experience through **Assigned to Me** and **Closed**, even without **Calibration Manager** permission.
* A **Calibration Manager** who is also assigned as a participant retains management access and can additionally use **Assigned to Me** for personal participant assignments.
Users without **QA functionalities access+** or **Calibration Manager** permission can't access Calibration Sessions.
## Navigation and Tab Visibility
The **Calibration** navigation item is available to users with **QA functionalities access+**, even without Calibration Manager permission.
The following table defines tab and control visibility:
| Tab / Control | Calibration Manager | Manager + Participant | QA functionalities access+ without Calibration Manager |
| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| **Assigned to Me** | Visible when the manager participates in an Active session. | Visible for the manager's participant assignments. | Always visible. Shows Active sessions assigned to the user and an empty state when none exist. |
| **All** | Visible. Displays sessions within the manager's management scope across Draft, Active, and Closed statuses. | Visible. | Not shown. |
| **Active** | Always visible. Displays all Active sessions within the manager's management scope. | Visible. | Not shown. |
| **Closed** | Always visible. Displays all Closed sessions within the manager's management scope, whether or not the manager participated. | Visible. Shows Closed sessions within the manager's management scope. | Always visible. Shows only Closed sessions where the user is a listed participant. Displays an empty state when none exist. |
| **Create New Session** | Visible when the user has session-creation permission. | Visible when the user has session-creation permission. | Not shown. |
| **Reference Score** | Available according to management permissions. | Available according to management permissions. | Not available. |
| **Resolution Notes** | Available according to management permissions. | Available according to management permissions. | Not available. |
| **Close Session** | Available according to management permissions. | Available according to management permissions. | Not available. |
| **Debrief View** | Available according to debrief rules. | Available according to debrief rules. | Not available. |
| **Score Now** | Available for the manager's participant assignments. | Available for the manager's participant assignments. | Available for personal assignments. |
| **View Report** | Available according to management or report permissions. | Available according to management or report permissions. | Available for Closed sessions in which the user participated. |
Users without Calibration Manager permission have a participant-only experience consisting of **Assigned to Me** and **Closed**. They can't access **All**, **Active**, management-scoped Closed content, or management actions.
## Roles and Participation
Permissions determine which parts of Calibration a user can access. Session participation determines whether the user receives participant-specific scoring access.
| User Type | Available Experience |
| --------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Calibration Manager — not a participant** | **All**, **Active**, and management-scoped **Closed** views. Can create, monitor, debrief, resolve, and close sessions according to available permissions. |
| **Calibration Manager — also a participant** | All management capabilities plus **Assigned to Me**, personal scoring progress, and **Score Now** for participant assignments. |
| **QA functionalities access+ without Calibration Manager permission** | **Assigned to Me** for Active sessions in which the user participates and **Closed** for sessions in which the user participated. |
| **QA functionalities access+ with no current assignment** | **Assigned to Me** remains visible and displays the appropriate empty state. |
## Calibration Sessions Landing Page
The **Calibration Sessions** landing page provides access to calibration sessions based on the user's permissions and participation.
* When no calibration sessions exist, **All** displays: `No Calibration Sessions — Create a session to start calibrating auditor scoring.`
* Use **Search** to find a session by name.
* Select **+ Create New Session** to create a session when available.
* The page includes four tabs: **All**, **Active**, **Closed**, and **Assigned to Me**.
### All
The **All** tab displays calibration sessions within the user's available scope.
| Column | Description |
| ----------------- | -------------------------------------------------------------------- |
| **Name** | Displays the calibration session name. |
| **Conversations** | Displays the number of conversations included in the session. |
| **Auditors** | Displays assigned participants as an avatar stack. |
| **Created** | Displays the session creation date. |
| **Status** | Displays the session status: Draft, Active, or Closed. |
| **Agreement** | Displays the session agreement rate when available. |
| **Actions** | Provides actions based on the session status and user's permissions. |
#### Available Actions
* **View:** Opens session details or reports. Active and Closed sessions open in read-only or monitoring mode.
* **Edit:** Available for Draft sessions to continue configuration.
* **Draft Actions:** **Cancel**, **Save as Draft**, and **Next** are available during Draft configuration.
### Active
The **Active** tab is available to Calibration Managers and displays Active sessions within their management scope.
Opening an Active session enables the manager to monitor participant progress, review scoring activity, access **Debrief View**, resolve scoring differences, and close the session.
The **Active** tab lists calibration sessions that are in progress.
| Column | Description |
| ----------------- | ---------------------------------------------------------------------------------- |
| **Name** | Displays the calibration session name. |
| **Conversations** | Displays the number of conversations included in the session. |
| **Auditors** | Displays assigned participants as an avatar stack. |
| **Created** | Displays the session creation date. |
| **Status** | Displays the current session status: Active. |
| **Agreement** | Displays the current agreement rate when available. |
| **Actions** | Provides actions available for the Active session based on the user's permissions. |
#### Available Actions
* **View:** Opens the Active session in read-only or monitoring mode to review current calibration results and progress.
* **Session Management Actions:** Provides additional actions based on the user's permissions.
#### Active Session View
When a user opens an Active session, the page displays:
* **Session Details:** Session name, conversation count, auditor count, creation date, conducted-by user, and current status.
* **Submission Progress:** Overall submitted evaluations and each auditor's scoring progress.
* **Conversation-Level Summary:** Conversation details, including:
* **Handled By:** Agent associated with the conversation.
* **Flow or Queue:** Flow or queue associated with the conversation.
* **Conversation ID:** Unique conversation identifier.
* **Date:** Conversation date.
* **Duration:** Conversation duration.
* **AutoQA Score:** Automated quality-assurance score.
#### Session Views
| Status | View |
| ---------- | ----------------------------------------------------------------------------------------------------------------------- |
| **Draft** | Displays General, Select Conversations, and Assign Auditors, along with conversation filters and configuration actions. |
| **Active** | Displays Agreement Breakdown, Metric Calibration Gaps, Conversation-Level Summary, and Consensus Alignment Summary. |
| **Closed** | Displays the final Calibration Session Report and completed calibration results. |
### Closed
The **Closed** tab provides access to completed calibration sessions based on the user's permissions.
| User Type | Closed Tab Scope |
| ------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| **Calibration Manager** | Closed sessions within the manager's management scope, regardless of participation. |
| **Calibration Manager + Participant** | Closed sessions within the manager's management scope. Participation doesn't restrict the management view. |
| **Non-Manager Participant** | Only Closed sessions in which the user participated. |
#### Participant Closed View
For participant-facing users, each session displays:
| Field | Description |
| ------------------ | ------------------------------------------------------------- |
| **Session Name** | Displays the calibration session name. |
| **Conducted By** | Displays the Calibration Manager who created the session. |
| **Date Closed** | Displays the date the manager closed the session. |
| **Conversations** | Displays the number of conversations included in the session. |
| **Auditors** | Displays assigned participants as an avatar stack. |
| **Agreement Rate** | Displays the final agreement rate. |
Select **View Report** to open the read-only Calibration Session Report.
When no applicable sessions exist, the tab displays: `No closed calibration sessions yet.`
## Assigned to Me
The **Assigned to Me** tab provides the participant entry point for calibration scoring. It's available to Calibration Managers assigned as participants and to users with **QA functionalities access+**.
For users without Calibration Manager permission, the tab remains visible even when no sessions are assigned.
**Assigned to Me** displays only **Active** calibration sessions in which the logged-in user participates. It excludes Draft sessions, Closed sessions, and sessions the user manages without participating.
### Session Details
Each session displays:
| Field | Description |
| -------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| **Session Name** | Displays the calibration session name. |
| **Created By** | Displays the Calibration Manager who created the session. |
| **Scoring Progress** | Displays the participant's progress as `X of Y conversations scored.` |
| **Personal Status** | Displays **Not Started** (grey), **In Progress** (blue), or **Submitted** (green) based on the participant's own scoring status. |
### Available Actions
* **View:** Opens the participant's assigned conversations.
* **Score Now:** Opens the first unscored conversation in **Calibration Mode**.
* **Submitted:** When the participant submits all assigned evaluations, **Score Now** changes to an inactive Submitted state.
**Assigned to Me** doesn't provide management actions such as **Debrief View**, **Reference Score**, **Resolution Notes**, or **Close Session**.
### Calibration View
The participant's view displays: Handled By, Flows/Queues, Conversation ID, Duration, AutoQA Score, Status, and Score Now.
### Score Now
Each conversation card displays **Score Now** when the participant has conversations remaining to score. Selecting **Score Now** opens the corresponding conversation in **Calibration Mode**.
After the participant submits all assigned evaluations, the action changes to an inactive **Submitted** state.
**Calibration Mode** keeps participant scores hidden from other participants during scoring.
When the user has no Active sessions assigned, display: `No calibration sessions are assigned to you yet`.
## Session Status
| Status | Description |
| ---------- | ------------------------------------------------------------------------ |
| **Draft** | The manager has started configuring the session but hasn't activated it. |
| **Active** | The session is in progress and assigned evaluations remain incomplete. |
| **Closed** | The manager has completed the calibration review and closed the session. |
## Calibration Session Workflows
Calibration Sessions help QA teams identify and resolve scoring differences by having multiple participants independently evaluate the same conversations.
The experience supports two complementary workflows:
### Management Workflow
The Calibration Manager creates and configures the session, selects the conversations and evaluation form, and assigns participants. After participants submit their evaluations, the manager uses **Debrief View** to compare scores, review disagreements, establish **Reference Scores**, add **Resolution Notes**, and close the session.
### Participant Workflow
Participants with **QA functionalities access+** access the Calibration experience through **Assigned to Me** and **Closed**. **Assigned to Me** provides access to their assigned Active sessions, where they score conversations in **Calibration Mode**, track personal progress, and submit evaluations.
Calibration Mode keeps peer participant scores hidden during scoring. After the manager closes the session, participants can access **Closed** and select **View Report** to review the read-only **Calibration Session Report**.
A Calibration Manager assigned as a participant can perform both workflows.
## Create a Calibration Session
Select **+ Create New Session** to start a calibration exercise.
The wizard has three steps: **General**, **Select Conversations**, and **Assign Auditors**.
A Calibration Manager creates a session by configuring the session, selecting conversations, and assigning QA Analysts. After the manager activates the session, assigned analysts receive the calibration work in their audit queue.
### Step 1: General
Define the session configuration.
1. Enter a **Name** for the calibration exercise.
2. Enter a **Description** with instructions or context for participating analysts.
3. Turn on **Analyst Justification** > **Mandatory** when participants must add a comment before submitting a metric score.
4. Select an **AutoQA Visibility** option.
5. Select the **Evaluation Form** participants use to score the conversations.
6. Select **Next**, or **Save as Draft** to continue later.
### AutoQA Visibility
The Calibration Manager can control when analysts see AutoQA recommendations during calibration.
The available options are:
| Option | Description |
| -------------------------------------- | ---------------------------------------------------------------------------------------------- |
| **Hidden until all analysts submit** | AutoQA recommendations remain hidden until every assigned analyst submits the session. |
| **Visible throughout** | AutoQA recommendations remain visible during the complete evaluation. |
| **Visible after each analyst submits** | AutoQA recommendations become visible only after an individual analyst submits the evaluation. |
### Step 2: Select Conversations
Select the conversations for participants to evaluate.
1. Select a **Conversation Type** to define which conversations are available to add.
* **Audited**: Displays audited conversations that are already audited.
* **Unaudited**: Displays conversations that aren't audited.
2. Select a conversation source tab:
* **All Conversations:** Conversations that match the selected filters.
* **Marked for Calibration:** Conversations that analysts or supervisors previously flagged using **Mark for Calibration** on the audit screen. See [Mark a Conversation for Calibration](#mark-a-conversation-for-calibration).
3. Use the **Search** field to find specific conversations.
4. Select **Filter** to narrow the list of available conversations.
5. Configure the required filters and select **Apply**.
6. Select individual conversations or **Select all**.
7. Select **Add Interactions** to add it to the **Selected Conversations** panel.
8. Select **View Conversation** to preview a conversation, or remove a conversation from the selection. Select at least one conversation to proceed.
9. Select **Next**.
Each selected conversation displays: **Conversation ID**, **Flows or Queues**, **Duration**, **Sentiment Score**, and **AutoQA Score**.
**Audit Screen Navigation**: **Mark for Calibration** is available from the **Ellipsis (⋮)** menu on the Audit screen. After the action is successfully completed, the option becomes disabled.
#### Conversation Filters
The filter panel helps narrow conversations before adding them to the session.
| Filter | Description |
| ------------------- | --------------------------------------------------------- |
| **Date Range** | Filters conversations using evaluation date. |
| **Evaluation Form** | Displays conversations evaluated with the selected form. |
| **Queues** | Filters conversations by queue or experience flow. |
| **Agents** | Filters conversations by evaluated agents. |
| **Duration** | Filters conversations using minimum and maximum duration. |
| **AutoQA Score** | Filters conversations using an AutoQA score range. |
| **Sentiment Score** | Filters conversations using a sentiment score range. |
Select **Apply** to apply the filter.
#### Mark a Conversation for Calibration
Supervisors and auditors can flag a conversation for calibration directly from the standard audit screen, without opening the Calibration workflow first.
1. Open a conversation on the audit screen.
2. Select the more options **Ellipsis** (⋮) in the header.
3. Select **Mark for Calibration**.
Flagged conversations appear under the **Marked for Calibration** tab in [Step 2: Select Conversations](#step-2-select-conversations) of session creation, so a Calibration Manager can add them to a session later.
### Step 3: Assign Auditors
Assign the QA Analysts who can evaluate every selected conversation independently.
1. Use **Search Auditors to Add** to find participants.
2. Select one or more participants.
3. Review the selected participants listed in the **Auditors** section.
4. Select **Create** and **Notify Participants** to activate the session.
5. Alternatively, select **Save as Draft** to continue configuring the session later.
6. Select **Save** to save the current configuration.
Each participant receives the same set of selected conversations and evaluates them independently. A Calibration Session requires at least two participants.
Selecting **Create and Notify Participants** changes the session status to **Active** and sends an email notification to the assigned participants.
## Monitor an Active Session
Opening an active session from the landing page displays the manager's monitoring view.
The session header displays the **Created Date**, **Auditors** avatar stack, **Conducted by**, and **Status**.
### Submission Progress
Submission Progress displays the overall participant submission status and individual participant progress. Each participant card displays the participant's name, number of **Conversations Scored**, and a progress bar.
A participant card reaches 100% and turns green when all assigned conversations are submitted.
### Conversation-Level Summary
The **Conversation-Level Summary** lists the conversations included in the session.
| Column | Description |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| **Handled By** | Identifies the agent who handled the conversation. |
| **Flows/Queues** | Displays the Experience Flow for conversations handled by an AI Agent and the queue for conversations handled by a Human Agent. |
| **Conversation ID** | Identifies the conversation. |
| **Date** | Displays the conversation date. |
| **Duration** | Displays the conversation duration. |
| **AutoQA Score** | Displays the AutoQA score when available. |
If the logged-in manager is a participant, Score Now is available for the manager's pending participant assignments. For more details, see [Manager as Participant](#manager-as-participant).
The **Flows or Queues** column displays the **Experience Flow** for conversations handled by an **AI Agent**, and the queue for conversations handled by a **Human Agent**.
## Calibration Evaluation
**Calibration Mode** is the scoring experience for QA Analysts participating in a Calibration Session. It uses the selected evaluation form and keeps peer scores hidden so analysts can independently evaluate each conversation.
A **Calibration Mode** indicator identifies that the analyst is completing a calibration evaluation rather than a production audit.
During calibration:
* Peer scores remain hidden.
* AutoQA visibility follows the session's configured **AutoQA Visibility** setting.
* Analysts evaluate each metric independently.
* Add comments when **Analyst Justification** has the **Mandatory** option enabled.
* **Audit Progress** displays evaluation completion status.
* Analysts can't modify an evaluation after submission.
* Calibration Session conversations aren't routed to the agent for review, acknowledgement, or dispute.
### Complete a Calibration Evaluation
1. Open an assigned Calibration Session.
2. Select a pending conversation.
3. Review the conversation transcript.
4. Evaluate each metric using the selected evaluation form.
5. Add notes to provide additional scoring context.
6. Review the completed evaluation.
7. Confirm that the evaluation is final.
8. Select **Submit Score**.
After submission, the evaluation becomes final. The analyst can't modify it unless a separate correction workflow is available. Submitted calibration scores don't affect agent dashboards, leaderboard rankings, or scorecard scores.
### Calibration Mode Options
Participants can:
* Review the **Agent** and **Audit Review** details.
* View **Audit Progress**.
* **Bookmark** the conversation.
* **Cancel** scoring.
* **Submit** Score.
Scores submitted in **Calibration Mode** apply only to the calibration session.
### Manager as Participant
When a Calibration Manager participates in a session, the manager can use **Assigned to Me** and **Score Now** to complete their participant evaluation while retaining management capabilities.
The manager's participant status operates independently of other participants. After the manager submits their evaluation, [Open Debrief](#debrief-view) becomes available according to the applicable debrief rules. The manager can access **Open Debrief** without waiting for other participant assignments to change.
#### Available Actions
* **View:** Opens the read-only Calibration Session Report.
* **Export CSV:** Exports available calibration results.
#### Session Report
The Closed session report includes: **Agreement Breakdown**, **Metric Calibration Gaps**, **Conversation-Level Summary**, **Consensus Alignment Summary**\*, **Conversation-level comparison details**, and **Final calibration results**.
For more information, see [Calibration Session Report](#calibration-session-report).
#### Audit Allocations as a Secondary Access Path
Existing Calibration access in **Audit Allocations** remains available to eligible users as a secondary path to calibration work. **Assigned to Me** remains the primary entry point for participant scoring.
#### Existing Calibration Access
Existing calibration sessions, scoring records, reports, and agreement calculations remain unchanged. A Calibration Manager who does not participate continues to use **All**, **Active**, and **Closed** management views.
#### Capability Access Summary
| Capability | Calibration Manager | Manager + Participant | QA functionalities access+ only |
| ---------------------- | --------------------------------------------- | ------------------------------------- | ---------------------------------------------- |
| **Assigned to Me** | Only when participating | Yes, for personal assignments | Yes |
| **All** | Yes | Yes | No |
| **Active** | Yes | Yes | No |
| **Closed** | Management scope | Management scope | Participated sessions only |
| **Create New Session** | When creation permission is available | When creation permission is available | No |
| **Score Now** | Only for personal assignments | Yes, for personal assignments | Yes |
| **Debrief View** | Yes, subject to debrief rules | Yes, subject to debrief rules | No |
| **Reference Score** | Yes | Yes | No |
| **Resolution Notes** | Yes | Yes | No |
| **Close Session** | Yes | Yes | No |
| **View Report** | According to management or report permissions | According to permissions | Closed sessions in which the user participated |
## Debrief View
Debrief View is a management function that allows the Calibration Manager to compare participant evaluations and resolve scoring differences.
Debrief View is available only to Calibration Managers according to the applicable debrief rules. It is not available to non-manager participants.
For a Calibration Manager who is also a session participant, Open Debrief becomes available after the manager submits their own participant evaluation, according to the applicable debrief rules. The manager can then review participant evaluations, compare scoring differences, and establish the [Reference Score](#reference-score).
Debrief View is available only to Calibration Managers and isn’t available to non-manager participants.
The view opens from a conversation in the session and provides a metric-by-metric comparison of participant evaluations. Depending on the available data, the comparison can include: Participant Scores, Participant Notes, AutoQA information (when available), Reference Score, Resolution Notes, and Accuracy, when Reference Scores are available.
The [Calibration Session Report](#calibration-session-report) uses the same comparison grid structure when the manager selects **View** for a conversation in the **Conversation-Level Summary**.
### Debrief View Components
| Component | Description |
| --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Conversation** | Identifies the conversation under review. Each conversation appears as a tab with **Handled By**, **Flow / Queue**, **Agreement Rate**, and **Most Disputed Metric**. |
| **Metric** | Displays the evaluation metric under comparison. |
| **Participant Scores** | Displays each participant's response for the metric in a separate column with a checkmark, cross, or N/A indicator. |
| **Agreement Rate** | Displays the agreement result for the metric. |
| **AutoQA Verdict** | Displays the AutoQA result when available. |
| **Participant Notes** | Displays notes entered by participants. |
| **AutoQA AI Justification** | Displays the AutoQA explanation for the metric when available. |
| **Accuracy** | Displays accuracy based on the established Reference Score when available. |
| **Reference Score** | Records the manager's final score for the metric. |
| **Resolution Notes** | Records the manager's explanation for the final resolution. |
The **Participant Summary** displays participant scoring and alignment information when available.
For example, for the **Resolution Effectiveness** metric, the comparison grid displays each participant's **Adhered**, **Not Adhered**, or **N/A** response, the **AutoQA Verdict** when available, and the metric's **Agreement Rate**.
At the bottom of **Debrief View**, the **Participant Summary** displays **Metrics Scored**, **Aligned with Consensus (%)**, and **Accuracy** for each participant.
### Review a Disagreement
1. Open **Debrief View** after all required participant evaluations are submitted.
2. Select a conversation.
3. Review participant scores for each metric.
4. Identify metrics where the scores differ.
5. Review participant notes and **AutoQA AI Justification**, when available.
6. Determine the appropriate Reference Score.
7. Add **Resolution Notes** explaining the decision.
8. Resolve all required disagreements before proceeding.
9. Review the final calibration results.
### Reference Score
The Reference Score is the Calibration Manager's final determination for a metric after reviewing participant evaluations and supporting evidence. It represents the expected scoring outcome for the calibration exercise and is used to calculate individual participant accuracy.
## Close a Calibration Session
The Calibration Manager closes the session after completing the required calibration review.
Before closing the session, the manager should:
1. Review all analyst scores.
2. Resolve the required scoring differences.
3. Set **Reference Scores**.
4. Add **Resolution Notes**.
5. Review the calibration results.
6. Close the session.
After the session Is closed:
* The session status changes to Closed.
* Final calibration results become available.
* Participants can access the read-only **Calibration Session Report** from the **Closed** session.
* Calibration Managers can access the report for sessions within their management scope.
## Calibration Session Report
Closing a session generates a read-only **Calibration Session Report**, which is available to the Calibration Manager and all participating QA Analysts. The report summarizes calibration outcomes, analyst alignment, scoring consistency, and areas that require additional calibration.
The report header displays the **session name**, **date closed**, **conducted by**, **participants**, and **conversation count**.
### Report Sections
| Section | Description |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Agreement Breakdown** | Displays the distribution of **Full Agreement**, **Partial Agreement**, and **No Agreement** results, along with the **Overall Agreement Rate** for the session. |
| **Metric Calibration Gaps** | Lists evaluation metrics ranked by agreement rate, from lowest to highest, and identifies metrics that may require additional calibration. |
| **Conversation-Level Summary** | Displays each conversation included in the session, along with its agreement results and the most disputed metric. Select the **View** icon to expand a conversation and review detailed metric-level comparisons. |
| **Consensus Alignment Summary** | Displays per-analyst results for Parameters Evaluated, Consensus Alignment, Inter-Auditor Agreement, and Accuracy. An AutoQA benchmark row appears below the analyst rows and compares AutoQA results with human consensus without including them in the consensus calculation. |
### Conversation-Level Comparison
Select the view icon on a **Conversation-Level Summary** row to expand the row inline and reveal the metric-by-analyst comparison grid directly beneath it, without leaving the report.
This expanded grid uses the same structure as [Debrief View](#debrief-view).
#### Accuracy
In the reviewed report data, **Accuracy** measures how well an analyst's scores match the final **Reference Scores** established by the Calibration Manager during Debrief.
For example, if an analyst's scores match the Reference Scores for **8 out of 10 metrics**, their accuracy is **80%**.
If the Calibration Manager hasn't established Reference Scores, the report can't calculate Accuracy.
### Export the Report
Select [Export CSV](https://raw.githubusercontent.com/Koredotcom/docs-v2/refs/heads/main/ai-for-service/assets/agentai-interaction-calibration.csv) to download the calibration results as a CSV file.
### Exported CSV Fields
The exported CSV contains calibration results from the report.
| **Field** | **Description** |
| ------------------------ | ----------------------------------------------------------------------- |
| **Session Details** | Calibration session name and relevant session information. |
| **Conversation** | Conversations included in the calibration session. |
| **Analyst Scores** | Scores submitted by participating QA Analysts. |
| **Reference Scores** | Final scores established by the Calibration Manager during Debrief. |
| **Agreement Rate** | Agreement result for the calibration session. |
| **Per-Analyst Accuracy** | Accuracy of each analyst compared with the established reference score. |
| **Resolution Notes** | Manager's explanation for resolved scoring differences. |
| **Calibration Outcome** | Final results of the calibration exercise. |
Participants can use the report to compare their evaluations with the final Reference Scores and review Resolution Notes for resolved scoring differences. When AutoQA results are available, the report also compares them with human consensus to show how closely AutoQA aligns with human evaluators.
## Calibration Results and Formulas
The system calculates calibration results after participant evaluations are available and Reference Scores are established where required.
### Variables
Use distinct variables to avoid ambiguity.
| Variable | Definition |
| -------- | --------------------------------------------- |
| **N** | Number of human participants in the session. |
| **C** | Number of conversations in the session. |
| **M** | Number of metrics evaluated per conversation. |
| **A** | Number of Adhered responses for a metric. |
| **B** | Number of Not Adhered responses for a metric. |
| **D** | Number of N/A responses for a metric. |
For each metric:
`A + B + D = N`
### Metric Agreement Rate
A metric reaches consensus when more than half of the participants select the same verdict.
If no verdict receives more than `N / 2` responses, classify the metric as **No Consensus**.
For metrics with consensus:
$$
\text{count} > N / 2
$$
Otherwise, the system labels the metric **No Consensus** and excludes it from further calculations.
$$
\text{Metric Agreement Rate} = \frac{\max(A, B, C)}{N}
$$
The system computes the Metric Agreement Rate only when the metric reaches consensus.
### Session Agreement Rate
The Session Agreement Rate averages the agreement rates for all metrics that reach consensus. The system excludes metrics labeled No Consensus.
sum of all metric agreement rates / number of consensus metrics
$$
\text{Session Agreement Rate} = \frac{\text{sum of all metric agreement rates}}{\text{number of consensus metrics}}
$$
### Agreement Distribution
The system groups metrics into the following agreement categories:
| Category | Condition |
| ---------------------- | ----------------------------------------------------- |
| **Full Agreement** | Metric Agreement Rate = **100%** |
| **Majority Agreement** | Metric Agreement Rate is between **50%** and **100%** |
| **No Consensus** | No verdict receives more than **N / 2** responses. |
### Session N/A Rate
The Session N/A Rate measures the proportion of metric responses that analysts marked N/A.
$$
\text{Session N/A Rate} = \frac{\text{total N/A verdicts across all metrics}}{C \times M \times N}
$$
### Auditor Agreement Rate (Aligned with Consensus)
For each analyst, the proportion of their verdicts that matched the consensus verdict, excluding **No Consensus** metrics from both the numerator and denominator.
$$
\text{Auditor Agreement Rate} = \frac{\text{number of matching verdicts}}{\text{number of consensus metrics}}
$$
### Auditor N/A Rate
The **Auditor N/A Rate** measures the proportion of metrics for which an analyst selected **N/A**.
$$
\text{Auditor N/A Rate} = \frac{\text{metrics where the analyst responded N/A}}{C \times M}
$$
The Participant Summary flags an analyst when their N/A rate exceeds the session average by more than 15 percentage points.
### Auditor Inter-Auditor Agreement (Cohen's Kappa)
Measures whether an analyst's alignment with the group consensus reflects genuine agreement rather than shared scoring bias, using the three verdict categories (`Adhered`, `Not Adhered`, and `N/A`).
Where:
`P_o` = the analyst's Auditor Agreement Rate.
`P_chance` = the probability the analyst and the consensus would match by chance, using each side's verdict distribution.
$$
\text{Kappa} = \frac{\text{P\_o} - \text{P\_chance}}{1 - \text{P\_chance}}
$$
The report displays Kappa only when the analyst scored 10 or more consensus metrics, and shows a dash otherwise.
### Kappa Interpretation
| Kappa Range | Interpretation |
| ------------- | -------------- |
| **Below 0.4** | Poor |
| **0.4–0.6** | Moderate |
| **0.6–0.8** | Substantial |
| **Above 0.8** | Near-perfect |
### Per-Conversation Agreement Rate
The **Conversation-Level Summary** calculates the agreement rate for each conversation using the same method as the **Session Agreement Rate**, scoped to that conversation.
### Per-Analyst Accuracy
A recommended calculation for individual analyst accuracy is:
$$
\text{Analyst Accuracy (\%)} = \frac{\text{Number of metrics matching the Reference Score}}{\text{Total metrics evaluated by the analyst}} \times 100
$$
For example, if an analyst's scores match the Reference Score for 42 of 50 metrics:
$$
\text{Analyst Accuracy} = \frac{42}{50} \times 100 = 84\%
$$
This result helps the Calibration Manager identify analysts who consistently score differently from the established team standard.
### Most Contested Metric
The Most Contested Metric is the metric with the lowest agreement rate, shown per conversation and at the session level.
When two metrics share the lowest rate, the report displays the one that appears first in the evaluation form.
### Metric Calibration Gaps — Thresholds
The **Metric Calibration Gaps** panel sorts metrics ascending by agreement rate, lowest first, and applies these thresholds across the Calibration module.
| Label | Agreement Rate | Color |
| ------------------------ | -------------- | ----- |
| **Calibrated** | 80% or above | Green |
| **Needs Attention** | 65–79% | Amber |
| **Calibration Required** | Below 65% | Red |
### AutoQA in Calibration Calculations
AutoQA appears in the **Consensus Alignment Summary** as a reference row. It isn't included in consensus or agreement calculations based on human analyst verdicts.
* **Consensus calculation**: Human analyst verdicts determine consensus. The system excludes the AutoQA verdict from **N** and the **A**, **B**, and **C** counts.
* **AutoQA Consensus Alignment**: Uses the Auditor Agreement Rate formula and treats AutoQA as an additional participant. The system compares its verdict with the human-only consensus.
* **AutoQA Cohen's Kappa:** Computed using the Auditor Kappa formula, treating AutoQA as an additional analyst, with the consensus distribution drawn from human analysts only.
The report displays the AutoQA row below the human analyst rows with the following note:
`AutoQA evaluated against human consensus; not included in consensus calculation`.
## Calibration Session Lifecycle
A Calibration Session follows a structured workflow from setup to reporting.
| Stage | Calibration Manager | Participant |
| ------------ | ------------------------------------------------------------------------------------- | -------------------------------------------------- |
| **Draft** | Creates and configures the session. | Can't access the session for scoring. |
| **Activate** | Assigns participants and activates the session. | Receives the assignment and email notification. |
| **Score** | Monitors progress and, when assigned as a participant, scores personal conversations. | Scores assigned conversations in Calibration Mode. |
| **Submit** | Waits for all required submissions. | Completes and submits the evaluation. |
| **Debrief** | Compares scores and reviews disagreements. | — |
| **Resolve** | Sets Reference Scores and Resolution Notes. | — |
| **Results** | Reviews agreement and participant accuracy. | — |
| **Close** | Closes the session and generates the report. | — |
| **Report** | Reviews the final report. | Reviews the final report after closure. |
When a Calibration Manager participates in a session, the manager follows the same **Score** and **Submit** stages as other participants while continuing management responsibilities. See [Manager as Participant](#manager-as-participant).
Each stage helps the Calibration Manager configure the session, collect independent participant evaluations, compare scoring differences, establish Reference Scores, and review final calibration results.