The University of Pittsburgh has developed an AI-powered course review tool that can assess online learning materials against 50 quality criteria in under three minutes, helping instructional designers improve consistency while retaining human oversight.
The University of Pittsburgh has developed an Artificial Intelligence tool designed to reduce the time required to review online courses from several hours to less than three minutes.
Created by the university’s Cloud Innovation Center (CIC), powered by Amazon Web Services (AWS), the Canvas OLC Insights tool uses AI to evaluate online and hybrid courses against established quality standards.
The technology was developed by CIC student developer Avi Kottakota following a request from Rae Mancilla, Executive Director of University Digital Education at the University of Pittsburgh Center for Excellence in Digital Education (Pitt EDGE).
Before launching, online courses at Pitt EDGE undergo a quality review based on the Online Learning Consortium’s Quality Scorecard, which assesses 50 objectives covering course design, content, structure and student engagement.
Previously, reviewing a course manually took between two and four hours. With hundreds of courses undergoing revision approximately every 12 to 18 months, the process placed significant demands on instructional designers.
The university also identified inconsistencies in how reviewers interpreted certain assessment criteria.
An earlier trial of a ChatGPT-based review assistant encountered limitations involving course size, data governance and integration with existing systems.
The new tool was developed to address these challenges while maintaining the role of human reviewers in evaluating course quality.
Rae Mancilla, Executive Director of University Digital Education at Pitt EDGE, said: “Working with the Pitt CIC was a great experience. Our instructional designers know this scorecard inside and out, but there aren’t enough hours in the day to run every course through it by hand, so we knew we needed to leverage AI. This tool will make our team more efficient and give them more time to focus on the judgment calls that need human intervention.”
Combining AI assessment with human oversight
Canvas OLC Insights retrieves course materials through the Canvas application programming interface (API), including syllabuses, modules, assignments, discussions, quizzes, rubrics and supporting documents.
It then uses Claude Sonnet 4.6 through AWS Bedrock to evaluate the material against the scorecard’s 50 objectives.
A second model, Claude Haiku 4.5, generates recommendations for improvement once a review has been completed.
The system examines both structural information, such as assignment counts and rubric coverage, and course content requiring more detailed interpretation.
For each objective, it provides a proposed score alongside supporting evidence and an explanation of its assessment.
Human reviewers can confirm or override these recommendations through a dashboard before finalising the review.
The university found that AI scoring alone produced agreement with expert reviewers at approximately chance level. However, introducing human confirmation and adjustment increased agreement to a level comparable with expert review.
The system also identifies criteria that cannot be assessed reliably from static course materials, such as whether an instructor responds to students promptly throughout a semester.
These objectives are flagged for human assessment rather than being assigned an unsupported score.
Completed scorecards can be exported as PDFs, while the platform can generate improvement plans and track course quality across successive review cycles.
Potential for wider adoption
The university tested the technology using a synthetic course containing deliberately constructed evidence to validate the assessment process.
Although initially developed for Pitt EDGE, the system has been designed so that other institutions belonging to the Online Learning Consortium could adapt it to their own course review processes.
The underlying code allows institutions to incorporate their interpretations of the scorecard’s objectives without rebuilding the wider system.
The university has also made the project’s code, architecture and documentation available through GitHub.
Avi Kottakota, Student Developer at the University of Pittsburgh Cloud Innovation Center, said: “It’s really rewarding to build a tool that helps make someone’s job easier. I learned a lot on this project, and it’s nice to be able to support staff so they can focus on what really matters, which is improving course quality. That way, other students at Pitt, like me, get the best possible learning experience.”
The project demonstrates how universities can use AI to accelerate administrative and academic quality assurance processes while preserving human responsibility for decisions requiring professional judgement.
