ÌÇÐÄvlog¹ÙÍø

Despite rising school enrollment, global learning levels are stagnant. In 2024, we find ourselves over the halfway mark towards the 2030 Sustainable Development Goal (SDG) number 4 of . Yet, the most recent Sustainable Development Goals from July 2023 predicts that, unless something changes, 300 million students globally will lack basic numeracy and literacy skills by 2030. COVID-19 school closures were a further setback to learning over the last four years. The World Bank estimates that between 2019 and 2022, the global share of children in Low- and Middle-Income Countries (LMICs) in learning poverty rose from 57% to 70%. Given these sobering statistics, the urgency to address the learning crisis is even more pronounced.

One of the most promising solutions to this crisis is targeted instruction (TI), where content is tailored to children’s actual learning levels rather than their age or grade. Driven by innovation and evidence from the Teaching at the Right Level (TARL) program in India, TI has expanded to countries around the world, and studies suggest . In Pakistan, the Learning and Educational Achievement in Pakistan Schools () team have been implementing and evaluating a localized, tech-enabled version of TI, the (TIP). The latest iteration of the TIP program, funded by the , was conducted in the Islamabad Capital Territory (ICT) from 2022 to 2023, and included just over 91,000 students in classes 1-5 and about 2,200 teachers from 560 public primary schools

TIP implementation in schools entails four steps: diagnosing student learning gaps; optionally sorting students into remedial groups (within the classroom) based on their learning level; targeting instruction for 40 days; and tracking student and overall remedial group performance through various types of testing - including two key tests at the beginning and end of the study (referred to as baseline and endline). All teachers and head teachers were trained on program administration and usage of ready-to-use tools and content in math. During the 40 days, regular classes of math for classes 2-4 were substituted with TIP classes. While classes 1 and 5 were not directly included in the evaluation of the TIP program, we did test children in those classes as well to check for any indirect impact the program may have on them.

TIP-ICT was implemented through a randomized control trial (RCT) to allow the research team to causally measure the impact of the program. Teachers were randomized into one of four groups:

• Control Group – teachers in this group did not receive any intervention from our team.
• Workbook Only Group – in this group, teachers received student workbooks (with answer keys) that included learning materials from grades 2 through 4.
• Information Group – teachers in this group were provided access to workbooks alongside test score information about how each student was doing in various math student learning objectives (SLOs). They were also granted access to the TIP Tech Tool - a low-cost, smartphone app that teachers used to run the program. The tool allows teachers to monitor learning progress and gaps, access lesson plans and teaching materials, sort students into peer learning groups, watch training videos, and other similar tasks. The idea behind this treatment group was to empower teachers by giving them tools, but not be too prescriptive on how they use them.
• Guidance Group – in this final group, teachers were provided with everything in the information group (including the TIP Tech Tool) as well as specific instructions on which learning activities from the workbook to assign to each child. In this group, we also sorted students into 3 distinct peer learning groups on behalf of the teacher (visible on the tech tool). This was a more structured treatment than information, and this group relied more heavily on the TIP Tech Tool to direct and target their teaching activities.

While full results are forthcoming, there are exciting preliminary insights from the program. These findings tentatively shed light on the specificities of where edtech may play the most meaningful role in improving both the teaching experience and student learning outcomes. This analysis focuses on the impact of TIP in the classroom, particularly on teachers and student learning outcomes, although TIP in ICT included a parental component as well.

Finding #1: Learning improved for all groups

Overall, the preliminary results of the TIP program show great improvements in student learning outcomes. Compared to the control group, students taught by teachers in the treatment groups (collectively) scored 0.16 standard deviations (SDs) higher than the control group at the final testing stage. This translates to about 2 months of extra learning over one academic year.

This result holds both when we account for students’ baseline learning levels (which have a strong relationship to how they perform at endline) and when we separate out the three different treatment groups. When looking at the test results more closely, we see that learning improved on a range of SLOs, including both some basic ones like addition, and some more complex ones like fractions.

Finding #2: The three treatment groups appear to have very similar results

At this stage, all three treatment groups seem to have very similar results - particularly information and workbook only. While the guidance group seems to have marginally higher results (about 0.17 SD compared to 0.14 SD for information and 0.15 SD for workbook only), we cannot statistically say these effects are different from the other treatment impacts. All three round to roughly 2 additional months of learning, with guidance slightly above 2 months and the others slightly below.

The team is currently working to understand the mechanisms behind these similar results and whether there are variations among the results we haven’t spotted yet (there are still many layers of test results, surveys, and implementation data to work through). Specifically, we are interested in understanding the fine details of implementation, how teachers used the materials provided to them, and whether splitting the results by certain demographics (e.g. by student gender or teacher characteristics) reveals something new. Finding 3 below touches on this.

From qualitative fieldwork, we know that both parents and teachers highly praised the workbooks. They mentioned that the workbooks have less text, simple explanations (compared to the complexity of standard, single-grade curriculum textbooks) and more space to write than standard textbooks, which increased student engagement. The collection of materials from different grade levels also allowed students to quickly pick up concepts and practice them. Parents, especially less educated ones, often used the workbooks as a tool to track student performance, which they did not feel comfortable doing using the standard textbook. In fact, there is such strong demand for these workbooks that the team is in discussions with the relevant Ministry about distributing workbooks for optional use for the coming academic year.

This finding suggests that technology needs to have higher value-added than simply providing learning materials. At least in settings like Pakistan, paper still dominates for simple tools like that, especially for parents. These preliminary findings seem to suggest that in order benefit from technology, edtech needs to focus on tools (such as guidance, AI-based student sorting, or real-time learning advice) that cannot easily be replicated on paper. This also suggests that edtech may act as a complement to paper materials such as these workbooks, which can benefit students differently when paired with technology (see finding 3).

Finding #3: The students who are most behind - especially the oldest - benefitted the most

The above results are strongest for learners who were most behind. If we divide the sample in two, test scores for students in the bottom half of the distribution gained about 2.7 months of additional learning (0.21 SDs), whereas students in the top half gained 1.3 months (0.1 SDs). Similarly, learning improved most for class 4 and least for class 2 (compared to the control classes). This is likely due to a similar mechanism, where students in class 4 had fallen most behind over time. Overall, the program appears to reduce the (large existing) variation in learning levels within a classroom, which makes teaching significantly easier. While this result is somewhat expected, it is encouraging that programs such as this do indeed benefit those furthest behind.

If we break these results down by treatment group, there is an interesting trend. The workbook only group appears to benefit the students in the bottom third of the class most, and those at the top least, when compared to control. In the case of top performers (who -  it is important to note - are still behind, just less so), workbook only appears to have almost no benefit compared to control. However, both the information and guidance groups seem to have more consistently positive impacts for all students - although the overall trend remains that lower performers learn more.

While we are still exploring the mechanisms as to why this is, these findings suggest that different students (and classrooms) may benefit from a varied combination of physical and edtech tools, depending on their existing learning levels. While physical materials may help those furthest behind (often by 2 grade levels or more) catch up, pairing that with edtech is more successful at helping those slightly less behind catch up, and potentially those at grade level advance even further. One hypothesis as to why this is the case may be that students who are that far behind don’t need elaborate targeting or nuanced AI models - their needs are more obvious to the teacher. Whereas students who are only slightly (e.g. 1 grade level) behind may be harder to spot and therefore teachers benefit more from technology helping them do so. It is also possible that there is some underlying teacher characteristic explaining this trend (see finding 4) that we have not uncovered yet.

Finding #4: Different types of teachers appear to respond differently to the treatment options

Lastly, teacher skills and beliefs appear to influence how impactful treatments were in their classrooms. For example, for teachers who believe they have to work very hard to increase test scores (e.g. it would take a lot of time on their part, or they would need a raise to do so), the information treatment did not appear to be as effective at improving test scores as for other teachers. Similarly, teachers who thought that student performance was not within their control (e.g. a student’s home life was more influential), guidance was particularly effective. Both guidance and information were particularly effective for teachers who viewed their jobs as a stepping stone to something else, or who expected to be in a higher paying job in the near-future.

These results are likely explained by the amount of effort teachers put into their classes. Both information and guidance are designed to make teachers’ work easier and more impactful, but between the two, information required more teacher effort to be meaningful whereas guidance reduced the burden of sorting students and deciding which SLOs to focus on with who.

Keep in mind that students under all these groups of teachers - those that have a high cost of effort, low sense of control, and/or other career aspirations - tend to perform worse than students of other teachers. This suggests that targeting tools supported by edtech may help counteract some negative effects of certain teacher characteristics that may be hard or impossible to regularly measure. This finding in particular may be key for future targeting of different iterations of similar programs, especially considering that guidance is a much heavier lift and more costly than information.

Conclusion

Where do we go from here? The team is working on diving deeper into these results - this is just a glimpse into what we have uncovered so far. Our focus is first understanding the mechanisms behind some of these preliminary findings and documenting the multitude of lessons we learnt from implementing such a complex program. As the new initiative highlights, it is increasingly important to publish findings relevant for program implementation - otherwise exciting results such as these may not be replicable let alone scalable.

First, later this year, look for a working paper on the development of our tools, in particular the workbooks, which were a crucial underpinning to this program. That paper will have lessons relevant to the edtech realm by understanding why the workbooks were so useful to students, teachers, and parents, and therefore how edtech may replicate or complement their success. Second, also later this year, we will work on a parallel publication exploring the take-up of the TIP Tech Tool. Which teachers used it, which didn’t, and why. That paper should have lessons for future iterations of similar programs and comment on the implementation conditions necessary for edtech take-up and scaling.

Lastly, we will continue carefully examining student learning outcomes, and how both the teachers’ side of TIP and the parental side (which we have not addressed here at all yet!) changed student learning. This should include another round of data collection to understand the long-term impacts of the program.