Beginner level
No prior experience required
A/B Testing for Learning Design
Flexible schedule
1 week, 6 to 8 hours per week
Instructor feedback
Get guidance on your work
Verified certificate
Share on LinkedIn
*Proof of full-time student enrollment required. Acceptable forms of ID include a letter from your university’s registrar office or an unofficial transcript. Email your documents to learnlab-help@lists.andrew.cmu.edu.
What you will learn
- Frame meaningful A/B tests around learning design decisions.
- Choose outcome measures and comparison conditions that support valid interpretation.
- Analyze A/B test results and identify practical limits of the findings.
- Use experimental evidence to guide iteration in courses and learning products.
Course description
When teams have multiple plausible design options, intuition alone is not enough to choose among them. A/B testing provides a practical way to compare alternatives and learn which design choices produce better outcomes in real contexts.
In this course, you will learn how to design and interpret A/B tests for learning products and courses. You will examine how to formulate testable comparisons, select meaningful measures, and use experimental evidence to inform iteration and product decisions.
Syllabus
Module 1: In Vivo Experimentation and A/B Testing
- Identify the characteristics of a good experiment.
- Explain why optimizing near-term performance, such as through better UX design, does not guarantee optimal learning.
- Explain why experimentation is important and illustrate with cases where intuition and theory are inadequate.
- Design a learning experiment, or A/B test, to evaluate whether one e-learning design is better than a close alternative.
- Explain ethical considerations for running A/B testing studies in education.
Meet the instructor
Dr. Ken Koedinger
Ken Koedinger is the Hillman University Professor of Computer Science at Carnegie Mellon University, with appointments in Human-Computer Interaction and Psychology. He holds an M.S. in Computer Science and a Ph.D. in Cognitive Psychology and has experience teaching in an urban high school. He has developed data-sharing and analytics infrastructures that support innovations in learning, including DataShop and LearnSphere, and has used them to improve learning as illustrated in his hundreds of publications. He directs LearnLab and co-founded Carnegie Learning in 1998, the first AI in Education company to bring intelligent tutoring technology into widespread use in schools. His PLUS project provides hybrid human-AI tutoring to middle school math students in schools around the country. He is a fellow of the Cognitive Science Society, the Association for Psychological Science, and the Association for Computing Machinery.
Where to go next
Learning Analytics Foundations: Predicting Student Success
Analyze educational data, build regression and classification models, and predict student outcomes to support better learning decisions.
Learning Curves and Data Mining for Course Improvement
Use learning curves and educational data mining to identify learning patterns and improve course design.
Predictive Modeling and Knowledge Tracing for Adaptive Learning
Build predictive models and use knowledge tracing to model learner progress and support adaptive learning systems.