Skip to content
  • There are no suggestions because the search field is empty.

AI Evaluation at Breakout Learning

 
1

How does the AI work?

Our AI evaluation system analyzes the textual content of student discussions to assess their contributions based on instructor-defined rubrics. Here's the process:

What it evaluates:

"Discussion Quality" Rubrics: The AI reads through discussion transcripts and scores students on specific criteria—like how well they apply concepts, articulate a reasoned position with evidence, or engage with their peers' ideas.

Pass/Fail Rubrics: Binary evaluation that renders a "pass" or "fail" based on pre-defined criteria. For example, in a negotiation role play, a pass/fail rubric might be "was the student truthful to the facts outlined in their role sheet?"

How it scores:

If using Discussion Quality Rubrics, the AI generates a score 0-4. Pass/fail rubrics generate a "pass" or "fail" score. Scores from both rubrics are accompanied by a detailed justification. Importantly, students see not just their score, but also:

  • The AI's reasoning for that score
  • Key arguments the AI identified in their contributions
  • Direct citations from the transcript supporting those arguments
  • Cited transcript excerpts

Instructors and students can review this information.

Context matters: We inject assignment-specific information submitted by instructors including: learning objectives, key concepts, and assignment descriptions, so the AI focuses only on what's relevant to that particular discussion, not everything students said. Before any assignment goes live, instructors review these learning objectives with our instructional designers to ensure they align with what's being taught in class. This alignment process ensures the AI evaluates students on criteria that actually matter for the course.

In addition to rubric-based scoring, the AI system also synthesizes patterns across groups. When multiple groups complete the same assignment, the system categorizes topics by themes and aggregates findings at the assignment level, answering questions like:

  • "Are my students making connections between the discussion and what I've been teaching them in class?"
  • "What was the most divisive topic in the discussion?"
  • "Were there any debates that would serve as a good jumping off point for an in-class lecture?"
  • "What are my students struggling to understand?"
  • "Can you suggest an in-class activity that will help my students better understand what they're struggling with?"

Participation and discourse patterns are analyzed across all groups working on the same assignment. This includes:

  • Talk-time balance
  • Emerging conceptual gaps
  • Recurring misconceptions
Part of a bigger picture: The AI evaluation is just one component of a student's grade. By default, it accounts for 30% of the final score, with the rest coming from non-AI components including pre-discussion multiple choice quiz performance (40%), attendance - did the student join the discussion room? (15%), and completion - did the student stay in the discussion room from beginning to end? (15%).

Instructors have full control to adjust these weightings or even exclude the AI evaluation entirely.
2

How do I know I can trust it?

We've built multiple layers of reliability and transparency into our system:

Consensus-based scoring: We don't rely on a single AI evaluation. Instead, we run the AI multiple times and only accept scores that appear consistently across at least 7 evaluations. If we can't reach consensus after up to 27 attempts, the system flags the assignment for human review by our team.

Full transparency: Students can see exactly how the AI arrived at their score—the reasoning, their identified arguments, and the specific parts of the transcript that support the evaluation. This transparency helps students learn from the feedback and allows them to identify any potential errors.

Student data is not used to train AI models: Breakout Learning does not use student information or session transcripts to train AI models, including when that data has been de-identified. Student discussion data is used to provide the evaluation, feedback, and insights associated with the learning experience, not to train AI.

Human oversight: We actively monitor AI results through an internal dashboard. When issues arise, our team investigates. We're also launching a grading adjudicator system that will allow students to contest grades, with streamlined workflows for instructors to review and resolve concerns.

Instructor control: Instructors design rubrics in close collaboration with our instructional designers to ensure criteria align with their learning objectives. They can adjust grade weightings, exclude AI evaluation entirely, and review the same detailed justifications their students see. The first run of any assignment often leads to refinements, but then becomes reliably reusable.

Built for substance, not shortcuts: The AI works best when discussions involve articulating positions, applying concepts, and engaging with ideas. We design experiences that invite substantive discussion rather than simple fact recall, and we help ensure evaluation criteria match the discussion format.

Continuous improvement: We're currently testing multiple AI models against exemplar discussions to ensure we're using the best technology available. Our approach evolves as the field advances.

The bottom line: AI is a powerful tool for evaluating discussions at scale, but it's not infallible. That's why we've made it transparent, contestable, flexible, and just one part of a holistic assessment approach. Instructors maintain full agency over how—and whether—to use it.
3

What about video recordings and privacy?

Our system transcribes discussions but does not store video recordings. Here's what that means:

Text transcription, not video storage: The AI analyzes text transcripts of what students say during discussions. We do not save or store video files of students' faces or the video feed itself. Once the discussion is transcribed, students' video and audio streams are not retained.

Transcription occurs for any participants while they are actively in the session and continues until the session is completed. If all participants leave, transcription pauses and resumes when someone returns. In a solo session, transcription similarly pauses if the participant leaves and resumes when they return.

Personal information in discussions: The system filters what instructors see. Instructors don't receive full transcripts—they only see snippets that are relevant to the assignment and learning objectives in the student session results. If instructors want to explore additional context or quotes, they can query for that information through the insights tool. Full transcripts are only accessible to the internal Breakout Learning team for quality monitoring and troubleshooting purposes.