Project outputs
Key Performance Indicators
Tangible, measurable outputs that reflect what a project delivers. KPIs track the immediate results of an AI initiative: tools implemented, actions taken, services improved.
Track what you build and what changes because of it.
A rubric decides whether an initiative should proceed. Success metrics decide whether it worked. The Compass uses three kinds: KPIs for what a project delivers, OKRs for what the institution gains, and efficacy measures for whether the AI itself works and caused the gain.
Outputs, outcomes, and efficacy
Mission outcomes
Objectives & Key Results
Mission-aligned outcomes: what changes for students, faculty, or the institution. OKRs connect AI implementation to strategic goals like student success, equity, research excellence, or operational transformation.
AI performance
Efficacy
Evidence that the AI does what it claims, accurately and fairly across groups, and that it, rather than other factors, produced the change. Efficacy is what makes KPIs and OKRs trustworthy.
- Problem or opportunityWhy act at all
- ContributionHow the AI helps
- OutputsMeasured by KPIs
- EfficacyAccuracy, fairness, attribution
- OutcomesMeasured by OKRs
- MissionStudent success, equity, research, operations
Exit criteria & re-review triggers
Decide in advance what would make you pause or stop, and what would send an approved initiative back for re-scoring. Retiring a tool should be a planned outcome, not an afterthought.
Set at Plan
Exit criteria
- An efficacy measure stays below its floor after a set period (e.g., precision under 70% after six weeks).
- Performance gaps across groups exceed the agreed tolerance.
- A privacy, security, or safety incident occurs.
- KPIs or adoption miss target by a set margin at the mid-pilot check.
- Costs exceed budget, or ongoing funding is not secured.
Any time after approval
Re-review triggers
- The vendor changes the model, terms, or data practices.
- New data types are added, or the AI's role in decisions grows.
- Use expands to a new population, unit, or scale.
- A contract comes up for renewal.
- A significant complaint, incident, or audit finding.
- Scheduled: at least annually for High and Critical initiatives.
Record both in the scorer's Plan step. A re-review starts again at Score with the current evidence.
Six planning questions
Every initiative answers the same six questions (three about the mission, three about the project) plus an efficacy check that ties them together. Draft them at intake; finalize them after selection.
| Mission outcomes: what the institution gains | Project outputs: what the project delivers |
|---|---|
| What problem or opportunity? | How will the project contribute? |
| What will be the outcome(s)? | What will the project produce or enable? |
| How will you know the outcomes have been realized? → OKRs | How will you know the outputs have been achieved? → KPIs |
Efficacy check. How will you know the AI is working as intended? Name how you'll measure accuracy and error rates, whether performance holds across groups, and how you'll attribute outcomes to the AI (for example, a comparison group or a staged rollout) rather than to other changes made at the same time.
Teaching & Learning
Worked example: AI-driven early warning system for at-risk students
An AI system that analyzes student data (e.g., grades, attendance) to identify at-risk students and recommend interventions, aiming to improve retention and academic success.
1 · Score
| Dimension | Score | Reasoning |
|---|
Total 17: High Strong alignment and clear operational potential, with notable risks around data governance, financial sustainability, and stakeholder trust. Recommended only with clear ethical oversight, a well-defined budget and support infrastructure, transparent communication with stakeholders, and ongoing monitoring.
2 · Plan
Mission outcomes
- Problem
- Students at risk of failing courses often go unnoticed, leading to higher dropout rates.
- Outcome
- Improved student retention and academic success, supporting equitable education.
- Objective
- Enhance student retention through early AI interventions.
- Key result
- Increase course pass rates for at-risk students by 15% within one academic year.
- Key result
- Reduce dropout rates by 10% by the end of the academic year.
Project outputs
- Contribution
- Analyzes student data (e.g., grades, attendance) to predict risk and recommend interventions.
- Deliverable
- An AI early warning system integrated into the learning management system.
- KPI
- Identifies 90% of at-risk students within the first four weeks of the semester.
- KPI
- Delivers intervention recommendations to 95% of identified students within 48 hours.
- Efficacy
- At least 80% of flagged students are confirmed at risk by advisors (precision).
- Efficacy
- Flag accuracy stays within 5 points across student demographic groups.
- Efficacy
- Retention gain is measured against a matched comparison group, not the prior year alone.
3 · Track & reflect
After the pilot, compare actuals to targets and record what the institution learned. Illustrative results:
| Measure | Target | Actual | Notes |
|---|---|---|---|
| KPI | Identifies 90% of at-risk students within four weeks | 85% | Close, but advisor workload slowed response. |
| OKR | Reduce dropout rates by 10% | 7% | Improved, but needs a longer pilot. |
| Efficacy | 80% of flagged students confirmed at risk | 68% | False positives confused advisors; threshold needs tuning. |
What worked
The tool accurately flagged at-risk students; advisors liked the interface.
What didn't
Efficacy fell short: 68% precision meant false positives that confused advisors; student trust was low due to lack of communication.
Next step: iterate
Add opt-in consent and retrain advisors on interpreting flags; pilot again in the fall.
Try it with this example
Open the scorer pre-filled with the early warning system, then adjust the scores and see how the band changes.
Plan your metricsTemplate (Word)One-page guide (PDF)
Representative targets. The percentages and thresholds on this page are examples, not benchmarks. Set targets from your own baselines and goals.