Test Your Assumptions Early
Many of the interventions that are designed to make a difference for students fail to realize the results they are designed for. Usually, this is because some assumption we have made does not hold. In the Innovation Box process, I train participants to identify and test their assumptions. The act of identifying and testing the assumptions that an intervention or new idea is built upon reduces risk and uncertainty and increases the potential that the idea will have an impact.
I train Innovation Box participants to take the following steps: (1) frame your problem, (2) research how others have tried to solve your problem, (3) identify and develop a solution, and (4) test the assumptions built into your solution. First, framing your problem carefully ensures that there is something worth solving and you have a way to tell if your idea has made a difference. Second, I always encourage participants to study how others have tried to solve problems similar to theirs. Looking at what others have done provides insight into how the nature of the problem and may provide inspiration. Third, participants select a solution that they think will solve the problem (or a part of the problem). Fourth, participants identify the assumptions that their solution is built upon. Identifying the assumptions allows the participants to determine the riskiest parts of their idea and develop tests to see whether their assumption van be validated. Below is an example of assumption testing.
After years of seeing students in 9th grade fail algebra, Bob decided that something needed to change if more kids were going to pass algebra and graduate from high school. The school district where Bob worked had put in place planning teams and was dedicated to exploring “best practice”, but didn’t have a good track record of implementing the plans or investing in big projects. Bob figured the best way to make change was to start small and validate a simple idea instead of developing the perfect plan. Bob spent some time looking at the past student data and studied some prior reports on why students fail algebra and concluded that weak fraction knowledge was holding students back and contributing to their failure. Bob proposed that his school run a parallel math class for students that entered algebra with weak fraction knowledge. Below is the series of assumptions and proposed tests along the way.
Problem: Students that enter algebra with weak knowledge of fractions struggle to complete the class with a passing grade.
Solution: Run a parallel intervention course to boast fraction knowledge while students are enrolled in algebra.
Assumption #1 - An assessment can be developed that screens students for fraction knowledge (current district placement exam does not provide adequate information).
Tested by -
1. Colleague review of proposed 10-15 question screening assessment.
2. Field test screening assessment in a variety of math classes to determine whether it is sensitive enough to select students for additional support.
Validated if -
Colleagues provide feedback on how to improve and support the deployment.
The assessment screens our students for additional support in level 2 algebra course (students that scored in the middle of the placement exam).
Assumption #2 - Students will agree to participate in a supplementary math class.
Tested by - Ask students to volunteer to participate in the supplementary math course.
Validated if - At least five students sign-up.
Assumption #3 - An intervention curriculum with a visual models and hands-on activities could be used teach students missing fraction knowledge.
Tested by -
1. Design a curriculum using visual models and hands-on activities and share with colleagues.
2. Test the curriculum with students to see if students gain knowledge over a short intervention period (10 days).
Validated if -
Majority of colleagues that review the proposed curriculum support it.
Students that participate in the full ten days of intervention show improvement in fraction knowledge.
Assumption #4 - A semester long parallel course for students enrolled in algebra that score below a cut point for fraction knowledge will ensure that students that are at-risk for failure will pass algebra and enroll in higher level math classes.
Test by - Run a semester-long parallel course for students that are at-risk (based on fraction screening and placement assessment) to teach fraction knowledge using the experimental curriculum.
Validated by - Compare fraction (and other basic math) knowledge pre- and post- and compare algebra, geometry, and algebra II completion rates for those students in the intervention and those not (all with weak fraction knowledge).
85% pass rate for algebra.
75% enrollment rate for algebra II within 2 years.
Bob worked to identify how riskiest assumptions, which included that they could even identify students that needed support, that students would agree to attend an extra math class, and that teaching fractions a different way would yield results. Bob tried to find low-cost and pragmatic ways to test these assumptions rapidly to build support for the solution he was proposing and to learn what worked and what didn’t.
It is this kind of process that more schools should be engaging in. Empower and trust your teachers to develop solutions to vexing problems and teach them a process for identifying and testing their riskiest early in the process. Leaders should be incentivizing and empowering teachers, but demanding that we move rapidly from belief that something will work to data that shows it works.