Creating a scorecard
A scorecard is a set of criteria used to evaluate a conversation. It tells reviewers, and the AI, what to look for and how to score what they observe. The same scorecard framework applies to both real calls and roleplay sessions, so practice scores and real-call scores are directly comparable.
This article covers building a scorecard manually, objective by objective. If you would rather have Outdoo AI draft one for you, see Generating a scorecard with AI.
What a scorecard contains
- Scorecard details: a title, a final objective (what the scorecard is for), the AI scoring mode, what the scorecard applies to, the scorecard model, and optional reference resources.
- Objectives: the individual behaviors being evaluated, for example "Confirmed budget" or "Asked about timeline". Each objective has its own scoring criteria, split into Must do (points required for a passing score) and Good to do (points that go above and beyond), and can be tagged with skills. On Caliber 2 scorecards, each objective also has a Format and a When to mark N/A note.
- Overall score: how the objectives combine into one score, either with equal weights or with a weight per objective.
Keep scorecards focused: 5 to 10 objectives is usually enough. More than that and reviewers start skimming.
The Scorecards page, where every scorecard you create appears alongside its publish status.
Create a scorecard manually
- Go to Manage > Scorecards and click Add Scorecard. The Create a new scorecard window opens.
- Click Start from scratch to begin with an empty scorecard, or click Select on a template that is close to your use case. A template pre-fills a set of objectives that you can edit or delete. To have Outdoo AI draft the scorecard instead, use Build with AI (see Generating a scorecard with AI).
- On Scorecard Details, enter the Scorecard Title and Final Objective, and pick an AI Scoring Mode (Standard, Strict, or Very Strict). What each mode does depends on the scorecard model. See "AI Scoring Mode by scorecard model" below.
- Under Apply to, choose where the scorecard is used: Roleplay Agent (the default), Team, Team Member, Category, or Deal Stage. Choose Roleplay Agent to use the scorecard only for roleplay sessions.
- Under Scorecard model, choose Caliber 1 or Caliber 2. New scorecards start on Caliber 2, which is marked Recommended. See "Choose a scorecard model" below.
- Optionally, set Recommendation Context (which data the AI draws on for coaching recommendations) and attach a reference document under Resources. See Scorecards with attached resources.
- In the left panel, under Objectives, select an objective to edit it, or click + to add one. For each objective, write what the team member should demonstrate, fill in Must do and Good to do, and add skills if relevant. On Caliber 2, also set Format and When to mark N/A. See "Set up each objective" below. Use Prev and Next to move between objectives.
- Select Overall Score in the left panel and choose a Score Methodology: Equal Distribution (every objective counts the same) or Weighted Distribution (set a weight from 1 to 10 for each objective).
- Click Publish to make the scorecard available, or Save Draft to finish it later.
Templates are fully editable. Change the objectives, criteria, or weights at any point, including after publishing.
To use this scorecard in practice sessions, select it on a roleplay agent. See Creating a scorecard for the roleplay agent.
Choose a scorecard model
The Scorecard model setting on Scorecard Details controls how the scorecard is evaluated.
Pick Caliber 1 or Caliber 2 in Scorecard model, at the bottom of Scorecard Details.
Setting up a Caliber 2 objective.
Model | How range scores work | When a question is N/A |
|---|---|---|
Caliber 1 | The AI picks each range score. It can return any number, capped at 5. | The AI marks a question N/A when it decides the question does not apply. |
Caliber 2 (Recommended) | Scores are calculated from the Must do and Good to do checklist, in half-point steps from 0 to 5. | A question is N/A when it has no checklist items to evaluate. You can also describe when it should be marked N/A. |
New scorecards, including ones generated with AI, use Caliber 2. Scorecards created before the scorecard model setting existed open as Caliber 1. You can switch an existing scorecard at any time.
Switching to Caliber 1. Caliber 1 does not use a custom format or a when-to-mark-N/A note. If any objective has a Yes/No format or a When to mark N/A note, the Switch to Caliber 1? window lists how many objectives will return to the Range format and how many will have their N/A note cleared. Objective text, scoring criteria, and skills stay as they are. Click Switch to Caliber 1 to confirm.
The Switch to Caliber 1? window tells you what will be reset before you confirm.
Switching to Caliber 2. If you try to edit Format or When to mark N/A on a Caliber 1 scorecard, the Switch to Caliber 2 window opens. Click Switch to change the scorecard model. You stay on the same objective.
AI Scoring Mode by scorecard model
The three modes have the same names on both models, but they work differently.
Mode | Caliber 1 | Caliber 2 |
|---|---|---|
Standard | The AI picks a score with some tolerance and gives partial credit when coaching points are partly followed. | Must do items count for 70% of the score and Good to do items for 30%. A partial mark counts as half. With no Good to do items, the score comes only from Must do items. |
Strict | More demanding than Standard. Coaching points need to be clearly followed for a high score, with some flexibility in phrasing. | More demanding than Standard. Missed Must do items pull the score down further. Good to do items still add credit once Must do items are largely met. |
Very Strict | A high score only when coaching points are followed exactly. Best for compliance-heavy teams or high-stakes calls. | Every Must do item has to be met before the score can go above 4. All Must do items met but Good to do items missed scores 4. The score is 5 only when every checklist item is met. |
On Caliber 2, each Must do and Good to do item is marked met, partial, or not met, and scores land on half-point steps from 0 to 5.
Set up each objective
- Scoring Criteria (Must do and Good to do): on Caliber 2 scorecards, each is a list of separate points. Type a point and press Enter, or click Add point, to start the next one. Click Remove point to delete one. Write one observable behavior per point, because each point is marked on its own. On Caliber 1 scorecards, Must do and Good to do are free-text boxes.
- Format (Caliber 2 only): Range (0 to 5) scores the objective from 0 to 5. Yes (5)/ No (0) scores Yes as 5 and No as 0.
- When to mark N/A (Caliber 2 only): describe when the objective should be marked N/A and left out of the score, for example "Mark N/A if the customer never raises an objection."
- Skills: tag up to 10 skills per objective.
On Caliber 2, each Must do and Good to do item is its own point.
Choose a Format and describe when the objective should be marked N/A.
Tips
- Use clear, observable language. "Team member asked at least two open-ended discovery questions" is easier to score consistently than "Team member showed curiosity."
- Align each objective with a real behavior you want team members to change or reinforce.
- Review the scorecard after the first few uses. Criteria that score the same way on every call are either too easy or too hard and should be adjusted.
Hope this article was helpful. For any queries, please write to support@outdoo.ai
Related articles
- Generating a scorecard with AI: let AI draft the scorecard from a prompt
- Create a scorecard using AI (with resources): ground an AI-generated scorecard in your own document
- Scorecards with attached resources: attach a reference document to an existing scorecard
- Creating a scorecard for the roleplay agent: use a scorecard to grade roleplay sessions