Fair Grades for Team Projects: An AI System That Reads the Evidence

Status: 🟡 In Progress — Topic Proposal Stage (July 2026)

This is a living case study. I'll update it as each part of the system gets built, tested, and evaluated.

The Problem

In team-based software projects, everyone on the team usually gets the same grade — no matter who actually did the work.

Picture a five-person team. One student writes 80% of the code, works steadily every sprint, and reviews teammates' pull requests properly. Another commits a few files the night before the deadline and doesn't really understand the code they submitted. Both get the same mark.

Instructors know this happens. But manually reading through hundreds of commits, cross-checking Scrum boards, and judging code quality — for every student, every sprint, across ten or more teams — isn't something a human can realistically do at scale. So effort goes unmeasured, and the grading system quietly rewards free-riding.

My part of our four-person final year project tackles this directly: can we fairly and automatically measure what one student actually contributed to a team codebase — using nothing but their GitHub activity and Scrum board data?

My Role

Our team's overall project is a multi-agent AI framework that supports software engineering students end-to-end — requirements planning, adaptive tutoring, project health monitoring, and grading. Four of us each own one sub-system. Mine is the last piece in that pipeline: turning raw developer activity into a fair, explainable individual score.

The Research Question

Can individual student contribution in a team software project be fairly and accurately evaluated by measuring effort, consistency, quality, collaboration, task complexity, behavioral pattern, and code ownership — extracted automatically from GitHub and Scrum board data?

I'm not claiming to build a perfect grader. The honest claim is narrower and more defensible: the system should produce scores that are explainable, consistent, and useful enough that instructors can grade faster and fairer than they could by hand.

Where the Data Comes From

The system needs no new tools or surveys — everything it needs is already produced by any team running GitHub + Agile Scrum:

Between these two sources, you can reconstruct what a student did, when, how well, and how they engaged with the team — without asking them to fill in a single form.

The Seven Factors

Instead of just counting commits, I broke down what an experienced instructor actually looks for when judging a student manually, and turned each of those into a measurable factor:

None of these are invented — each is grounded in existing research (mining-software-repositories work on commit activity, Zimmerman's self-regulated learning theory for consistency, LLM-as-Judge methodology for quality, GitHub social-coding research for collaboration, and educational data mining for behavioral classification).

A design rule I set early on: use an LLM only where genuine judgment is needed, not where a formula will do. Counting commits is a formula. Deciding whether a code diff actually solves the assigned task requires reading comprehension — that needs an LLM. Of the eleven components in the pipeline, six use an LLM and five are pure computation, and each choice is justified by that test.

Consistency, for example, is pure math — no LLM needed. I split the sprint into equal time windows, count commits per window, and compute the coefficient of variation (standard deviation ÷ mean), then invert it: C = 1 − (σ/μ). Even spread across the sprint → score near 1. Everything crammed into the last two days → score near 0. Dividing by the mean (rather than using raw standard deviation) matters because it makes the score fair between a student who commits twice a day and one who commits forty times a day — it measures relative unevenness, not raw activity.

Behavioral Pattern is where it gets interesting: instead of scoring in isolation, the system classifies each student into a contributor profile — consistent deep contributor, last-minute rusher, reviewer/coordinator, burst worker, or free rider — based on combinations of the other scores, and applies that as a multiplier (0.8×–1.2×) on the final result. This is what catches the student who contributes almost entirely through code review rather than raw commits, and correctly credits that instead of penalizing it.

The Part I'm Most Proud Of: The Code Ownership Quiz

Passive git analysis has a blind spot: git blame shows who committed code, not who understands it. A student can commit code copied from a teammate, Stack Overflow, or an AI tool, and the history looks identical to genuine work.

To close that gap, the Code Ownership Agent generates targeted questions directly from a student's own committed code — for example, "You set a 15-minute expiry on this JWT refresh token — why that duration?" — and evaluates their answer against the actual code. It's the difference between measuring "who touched this file" and "who could explain this file." As far as I've found, no existing automated assessment tool combines passive repository analysis with this kind of active knowledge check.

System Architecture

The pipeline runs in four layers, plus a fifth I'm adding for real-time prediction:

1. Data Collection — GitHub and Scrum board collectors, triggered by webhook on every push.

2. Factor Computation — six agents/calculators produce the seven factor scores (five LLM-based, one pure algorithm).

3. Aggregation — the Behavioral Pattern Classifier and Score Aggregator combine everything into one weighted score: S = BP_modifier × (w₁E + w₂C + w₃Q + w₄Co + w₅TC + w₇KO).

4. Output — an Explanation Agent writes a plain-language justification, a radar chart visualizes the seven-axis profile, and an instructor report ties it together (with override capability — the instructor always has the final word).

5. Prediction (new) — runs during an active sprint on partial data, using a logistic regression classifier trained on completed sprint history, to flag students as On Track, At Risk, or Critical before the sprint ends, while there's still time for an instructor to step in.

The weights aren't guessed — I'm deriving them two ways: Analytic Hierarchy Process (instructors pairwise-comparing factor importance) and empirical regression against real instructor-assigned grades. Where the two methods agree, that's strong evidence the weighting is right; where they disagree, that disagreement becomes a finding worth discussing in its own right.

What's Genuinely New Here

Progress Log

(I'll keep adding to this table as development moves through data collection, agent implementation, evaluation, and the final dissertation.)

Check back for updates as each layer comes online.