Share on facebook
Share on twitter
Share on linkedin
Share on email
Best Practices Report: How Data Analytics Informs District Decision-making

Data Review & Analysis

An Evaluation of the Potential of CatchOn Data to Inform District Decision-making

Analysis of findings from participating Digital Promise schools

Prepared by Project Tomorrow

Project Tomorrow®, a national education nonprofit organization, was contracted by CatchOn to do an external analysis and evaluation of the data collected through this special pilot with seven (7) school districts that are part of Digital Promise’s League of Innovative Schools. The primary objective of the analysis work was to understand the potential and power of data to support key district needs for better information about how various digital products are being used by students and teachers.

The methodology for the analysis included reviewing feedback collected from leaders in the seven study districts through an online survey, and a secondary data analysis of the usage and engagement data provided through the CatchOn dashboard. Where appropriate, individual school district-level data is compared to the aggregated Digital Promise pilot group and some district side-by-side analysis was conducted around similar product or product category usage. Project Tomorrow leveraged its extensive knowledge about district technology practices from our annual Speak Up Research Project™ to provide additional context for the analytical processes and reporting of key findings. This report documents the overall perspective of the district leaders and provides insights with specific data examples of how CatchOn’s data dashboard can support K-12 education decision-making.

To support data privacy for the participating pilot districts, individual districts’ identifying names have been removed from this report. References are made to Districts A, B, C, D, E, F, and G. 

“CatchOn provides the visibility necessary to consider a variety of questions related to instruction and learning. These include the implementation fidelity of educational technology tools, degree of student engagement vs. enticement, and the full range of EdTech tools in use throughout the district.”

– Technology Officer, School District C

“CatchOn provides the visibility necessary to consider a variety of questions related to instruction and learning. These include the implementation fidelity of educational technology tools, degree of student engagement vs. enticement, and the full range of EdTech tools in use throughout the district.”

Technology Officer, School District C

Key findings about the valuation of the CatchOn data utilized within the pilot program

Based upon the latest Speak Up Research Project findings from the 2020-21 school year, 90% of district administrators say their district has successfully implemented a one-to-one device program for their students where students can use their device in school and at home. This represents a significant increase since the 2018-19 school year when only 41% of administrators reported that they provided their students with devices to use both at school and to take home for extended learning. Alongside the increase in student access to technology for learning, teachers report a 20% increase in their integration of digital content within everyday instruction in the 2020-21 school year compared to the previous year. 

Education leaders highly value the impact that effective technology usage can have on student learning. For example, 85% of district administrators strongly agree that the “effective use of technology within learning is important for students to develop the skills and knowledge they need for their future success.” But the key is the effective usage of technology. To measure the efficacy of the use of technology, 65% of administrators look to students’ engagement with learning content as a meaningful indicator. When implementing digital solutions, education leaders are also highly cognizant of the need to address inequity in learning experiences, close the achievement gap between student groups, and understand the return or value of their investment in these technology solutions. Within that context, one of the most important questions that district leaders are asking today is the following: 

How well do the digital products and resources we have licensed and implemented within our classrooms help us achieve our educational mission in the most cost-effective way?

Survey feedback from the Digital Promise pilot districts indicates that the CatchOn data can help district leaders answer this important question. Most notably, the CatchOn data provides leaders with visibility into information that that is highly actionable for them. The key findings from our analysis of the leader survey feedback include the following: 

100% of the leaders from the Digital Promise pilot school districts report that reviewing their CatchOn data helps them identify gaps in student engagement that can indicate inequity.

100% of the leaders from the Digital Promise pilot school districts say that their CatchOn data is valuable for informing their ROI analysis on technology investments.

100% of the leaders from the Digital Promise pilot school districts believe that their CatchOn data is valuable for supporting their district’s online learning initiatives

This focus on the impact of the CatchOn data supporting a district’s online learning initiatives is especially timely because many districts are implementing new virtual academies or schools. Additionally, many districts are interested in leveraging the experiences of their teachers over the past 18 months to expand on the effective use of technology within learning. 

In the survey, the pilot’s district leaders identified the following strategic and operational benefits (Table 1) of using CatchOn to support online learning initiatives:

Table 1: Strategic and Operational Benefits of Using CatchOn to Support Online Learning Initiatives
Strategic Benefits
Operational Benefits

CatchOn provides visibility into usage to inform future decisions.

CatchOn supports our messaging efforts to our community about product choices.

CatchOn provides visibility into usage to evaluate past decisions especially pertaining to ROI considerations.

CatchOn informs our utilization analytics with access to regular and timely data.

CatchOn provides a meaningful way for us to focus on decisions on digital products to align with our district mission.

CatchOn helps us ensure that we are effectively monitoring product usage and that we follow all requirements.

Table 1: Strategic and Operational Benefits of Using CatchOn to Support Online Learning Initiatives

“The CatchOn tool has helped us focus the conversation on more strategic alignment with software implementation. IT has begun conversations with the Instruction department to review measurement of the effectiveness of specific applications.”

– Senior Technology Staff, School District F

“The CatchOn tool has helped us focus the conversation on more strategic alignment with software implementation. IT has begun conversations with the Instruction dept to review measurement of the effectiveness of specific applications.”

Senior Technology Staff, School District F

CatchOn Pilot Project Data Results

Leveraging CatchOn data to support district goals – 5 principal use cases and examples of pilot data usage

To truly understand the potential of data to support district decision-making, Project Tomorrow did a deep dive into the CatchOn data dashboards to explore how a typical school district could use the data to support district goals. It was important for our analysis that we use the same tools as a district leader would use, seeing the same types of data and reports. Our analysis identified five principal data use cases that uniquely align with and address the Digital Promise priority challenges. For purposes of our analysis, all data used in the examples represents Digital Promise pilot school districts’ product usage in the time period February 1, 2021, through May 28, 2021. The five principal use cases are as follows:

From a data analysis standpoint, the key findings in this report are considered highly generalizable due to the diversity of school districts that participated in the Digital Promise and CatchOn pilot. As would be expected, the goals of the districts in terms of their digital learning initiatives and their status using data to inform product and resources decisions varied widely. The common denominator, however, was an appreciation for the value of using data such as provided on the CatchOn database to support decision-making. 

Appendix Table 2: Contextual information about the school districts that participated in the pilot.
School District Identifier
Community Type
# of Schools
Total Student Population
% of Students in Free or Reduced Lunch Program

A

Rural

44

30,602

38%

B

Rural

3

1,080

64%

C

Urban

57

42,020

44%

D

Suburban

10

5,216

38%

E

Urban

89

48,710

40%

F

Suburban

31

26,737

38%

G

Suburban

48

40,879

36%

Appendix Table 2: Contextual information about the school districts that participated in the pilot.
1
1
Technology Access Use Case:
Using the CatchOn data to support a Return on Investment (ROI) analysis

Technology Access Use Case: Using the CatchOn data to support a Return on Investment (ROI) analysis

In this use case, a school district can utilize the data provided by CatchOn to make more informed decisions as to whether a financial investment in a particular product license is worthwhile. Quite often, this type of analysis is done when a product is up for renewal or relicensing. An ROI analysis is used to evaluate the cost of the product license relative to the impact it is having on the educational delivery system or process.

As educators well know, it is difficult to tie the use of a particular product directly to some student outcomes, such as student achievement results, due to the variables in the learning process that cannot be controlled. However, there are several metrics that a district leader can use within the CatchOn dashboard to support an ROI type of analysis including evaluating the number of student engagements with the digital products and also calculating a cost per engagement or user. Additionally, a district may want to evaluate the ROI of using a paid product vs. a similar free product.

Finally, the CatchOn data is also useful to help a district leader understand if they have too many (or too few) licenses for a particular product based upon the level of usage or engagement with that product. These are all important considerations when evaluating the ROI for technology usage.

Example: District A’s CatchOn data analysis – examining comparative costs per engagement or user

Like other districts within the pilot, District A is paying closer attention today to not only how students and teachers are using digital content and products for learning, but which products are being used most often. That type of information is increasingly important in District A as it evolves into what the Chief Technology Officer calls “an enterprise K-12 school district” with more centralized control over the purchasing of learning apps and solutions. To accomplish this, the district leadership needs greater visibility into the products being used. In this example, we demonstrate how that visibility can also support an ROI analysis by examining the cost per engagement and cost per user based upon the CatchOn data.

We first evaluated the number of engagements per user and device for the top five trending applications in Math and English Language Arts during the data evaluation period of February through May 2021.Those top trending apps are identified in a special report generated by the CatchOn dashboard. Several district leaders in the Digital Promise pilot specifically called out the value of the trending report in their comments. As depicted in Chart 1, the top trending apps for this period included Learning and Assessment Program 1, Learning and Assessment Program 2,  Supplemental Curriculum 1, and Supplemental Curriculum 2*. During the study period, the Learning and Assessment Program 1 product averaged 51.18 engagements per user and a slightly lesser number of engagements per device (45.19).

*Product names have been changed to represent product type.

Chart 1: District A - # of engagements per user and device for 5 trending products

As noted earlier, district administrators and school board members are increasingly interested in ROI type analysis relative to technology investments. The key question is always, “Are we getting our money’s worth with our purchase/license of this product?” The CatchOn dashboard and resulting data is especially effective in helping with those types of discussions. 

A district has the opportunity to include in their dashboard the cost of the product licenses. That information was available to us for the analysis for the four licensed products that were top trending products for District A: Learning and Assessment Program 1, Learning and Assessment Program 2, Supplemental Curriculum 1, and Supplemental Curriculum 2. 

Using that information from District A, we were able to create a new metric for the district from the CatchOn data – the cost per engagement. Chart 2 represents that new metric for the four licensed products within District A.

Chart 2: District A – cost per engagement for 4 trending licensed products

Per this data analysis, an engagement with the Learning and Assessment Program 1 product is the most cost effective at $.33 per engagement. An engagement with the Learning and Assessment Program 2 product is costing the district $4.01 per interaction by a student or teacher. As noted in Chart A, the Learning and Assessment Program 1 product has the highest number of engagements on average. The cost per engagement is directly related to the number of engagements or interactions with the product. Thus, the lower cost per engagement make sense in this example.

This type of data analysis can be helpful to a school district in many ways. For example, this cost per engagement metric may indicate to a district leadership team that they need to provide more support to their teachers to ensure greater usage of a product. It may also stimulate an internal evaluation as to why a particular product is not being used more frequently. That evaluation may lead to a greater understanding of the fit of the product with the curriculum or teachers’ goals for their class. 

Additionally, this type of analysis could be used by a district to articulate to their community the value of using a particular product if the ROI for that product is deemed positive. This type of messaging could resonate with leaders looking to demonstrate cost effectiveness and good stewardship of district funds.

2
2
Opportunity Gaps Use Case:
Using the CatchOn data to support a Value of Investment (VOI) analysis

Opportunity Gaps Use Case: Using the CatchOn data to support a Value of Investment (VOI) analysis

It is understood that not all school districts are interested in a financial investment analysis with their licensed products. Rather, some districts prefer to examine the efficacy of their product usage with a lens on the value provided by that product to instruction or in meeting very specific district goals. A value of investment analysis (VOI) is applicable in this environment.

The CatchOn data can support this type of analysis as well. Using the CatchOn dashboard, district leadership can examine the engagement levels of their students and teachers across products and schools to understand the alignment of that product usage with their district goals. The ability to disaggregate the data by schools or specific student populations can support that type of value-based analysis on the investment as well. 

Example: District E’s CatchOn analysis – examining product engagement through an equity lens

In District E a strong focus on providing equitable learning experiences for all students precedes the pandemic and the national response to examine inequity in our society after the George Floyd murder. This laser attention on equity is pervasive throughout the district culture and influences decisions on technology products and access by students and teachers. District E leaders examine the value of their investments (the VOI) in digital products through an equity lens, asking themselves, “Are our decisions about products providing equitable learning experiences for all students?” 

With that lens in mind, the district administrators decided last year to migrate from the Learning Management System Program 1 to Learning Management System Program 2 for middle and high school students and Learning Management System 3 for elementary students. The goal was to provide a more appropriate, responsive, and accessible platform for all students and their families within District E. The universality of the usage of the products provides an optimum environment for understanding the equity of learning environments in District E. And the CatchOn data can help the district leaders understand if their value-based decisions regarding these products are achieving the desired goal. 

We evaluated the product usage statistics for Learning Management System Program 2 at eight high schools in spring 2021 and for Learning Management System Program 3 at eight elementary schools during that same time period. Chart 3 documents the number of engagements per user for Learning Management Program 2 comparatively across the eight high schools. As depicted, High School 7 had on average 271.65 engagements with Learning Management System Program 2 at their school from February 1 through March 28. Comparatively, High School 3 had only 133.67 engagements per user during the same time. This type of visibility into the usage of a core foundational product such as their learning management system could indicate that the learning experiences being provided to students at High School 3 may not be equitable with the experience students are having at High School 7.

Chart 3: District E - # of engagements per user with Learning Management System Program 2 at 8 high schools

Chart 4 illustrates the results of a similar analysis. District E has 39 elementary schools and the 8 schools used in this analysis were selected randomly. As with the high school usage analysis, the data points to differences in the number of engagements per user. For example, the usage levels appear to be similar in Elementary School 2 (221.00 engagements per user) and Elementary School 8 (224.27 engagements per user). But the number of engagements per user at Elementary School 4 are less than half as much as Schools 2 and 8. 

There may be many reasons to explain the differences both within the high school data and the elementary school data including variances in the implementation strategies of both products, lack of teacher training on effective usage, and possibly a delay in teacher buy-in in this first year of usage. Regardless, the data provides visibility to assess the extent in which the district can leverage access to certain digital products as a way to ensure equity of learning in their schools.  

Chart 4: District E - Number of engagements per user with Learning Management System Program 3 at 8 elementary schools

Equity in education is about more than providing every student with a Chromebook and playlist of learning content. District leaders striving to ensure equity for their students must also consciously produce equitable learning experiences for all students, create environments that enable every student to be successful, and support individual student agency and efficacy in learning. 

Good intentions abound but measuring success with these goals can be challenging. The CatchOn data as illustrated in this example from District E can provide an innovative way for district leaders to assess if the learning environments they are creating for students (especially through access to digital products, content, and tools) are being equitably implemented and leveraged to support learning. 

3
3
Student Engagement Use Case:
Using the CatchOn data to evaluate engagement

Student Engagement Use Case: Using the CatchOn data to evaluate engagement

Per the recent Speak Up findings, district administrators highly value student engagement with digital products as a meaningful metric for evaluating the efficacy of those tools within the learning process. Traditionally, principals and teachers evaluate student engagement through highly subjective means: facial gestures, attentive eyes, and nodding heads. 

School district access to data such as provided by the CatchOn dashboard provides a much more objective approach to evaluating engagement with products. This is especially important for leaders who want to make the best decisions around product usage, both in terms of the financial impact of those decisions and also to ensure that the best apps and solutions are available for student and teacher usage in the classroom. 

Using the CatchOn data, district leaders have access to information about the number of engagements with a product per user and/or device for an entire school year, a semester, a month, or whatever time period is most appropriate. The dashboard can disaggregate the results by grade level or by trending app. Used in combination with other reports available through the dashboard, the leaders can make more informed decisions that impact the educational opportunities of their students.   

Example: District B’s CatchOn data analysis – examining comparative engagement levels

According to the Assistant Superintendent in School District B, there are several digital products that always catch his attention because they represent higher cost licenses. Prior to the access to CatchOn in this study, the district would pull usage reports from the various platforms, but they mostly relied upon the anecdotal feedback from teachers and administrators at the various schools when it came time to make repurchasing or relicensing decisions.

It is the Assistant Superintendent’s hope that this year the process can involve a more sophisticated analysis that can create a stronger validation for their decisions. 

The focus in this analysis was the level of engagements (interactions) by user and by device (Chart 5). The five products evaluated using the CatchOn data were four currently licensed products (Learning and Assessment Program 1, Supplemental Curriculum, Learning, and Assessment Program  3, and Learning and Assessment Program 2) and one free product (Quiz App).  

Chart 5: District B - Number of engagements per user and device for 5 different digital products

During the study period, the Learning and Assessment Program 3 product averaged 109.57 engagements per user and a similar number of engagements per device (108.08). This data is in line with the expectations from District B’s leadership who predicted high usage of the  Learning and Assessment Program 3 product within instruction based upon teacher feedback. The value is that, rather than a hunch or a guess, the district leadership now has quantitative data to validate their assumptions. 

As districts become more sophisticated with their use of data, they need resources that can help support that usage. The CatchOn dashboard can provide a valued service to districts across the entire spectrum of data sophistication supporting those on the beginning stages of the journey such as District B or others that are already more fluent with their internal data analysis.    

4
4
Teaching With Technology Use Case:
Using the CatchOn data to understand the efficacy of technology usage

Teaching With Technology Use Case: ​Using the CatchOn data to understand the efficacy of technology usage

Analyzing the efficacy of digital content, tools, or resources can be very challenging. It is difficult to isolate the variables that can impact product efficacy in an education setting. And the evaluation of product impact cannot happen in a vacuum but must be within a real-world classroom setting. The bottom line is that it is important for school- and district-level leadership to know what products work in their classrooms and why. 

The CatchOn data provides a new way of understanding the impact of technology usage within learning. By examining the usage metrics around a particular product in conjunction with appreciating the environment for that usage, district leaders can approximate a more comprehensive picture about the efficacy of the product. Additionally, that type of efficacy evaluation can often lead to a deeper understanding of the value of certain products (productivity tools, content or curriculum, teacher support resources) in different settings. 

Example: District D’s use of the CatchOn data to connect the value of teacher training with product efficacy

To support the effective use of a new product (Science Simulation App) by their science teachers, District D leadership paid for add-on professional development sessions for their teachers. Through that additional investment, the leadership team hoped to speed up the adoption process by teachers which can be lengthy for some new products. 

The need for a quick adoption was necessary since Science Simulation App was slated to support students’ virtual learning environments. The CatchOn data provided a way for the administration to both see how well the teachers were adapting to the new product as measured by the usage as well as validate their decision to invest in  additional professional learning.   

In this example, we disaggregated the engagement data by month (February, March, April, and May 2021) to provide visibility to the district in terms of any variances in usage (Table 2). And then we examined the user per engagement metric as illustrated in Chart 6. 

Per that analysis, the level of engagements over the four-month period was relatively consistent with a slight dip in April for spring break. This again validates that the impact of the professional learning investment was meaningful as the level of engagement with the product is highly consistent. This also speaks to the efficacy of the product usage as teachers continue to use it on a consistent basis.

Table 3: District D – Use of the Science Application App as measured by users, engagements, and devices in spring 2021
Month
Total users
Total engagements
Total devices

February 2021

1650

19159

1821

March 2021

1944

23707

2103

April 2021

1918

20824

2100

May 2021

1659

19452

1804

Table 3: District D – Use of the Science Application App as measured by users, engagements, and devices in spring 2021​
Chart 6: District D - Number of engagements per user for the Science Application App in spring 2021

Since the dashboard also allows for school-level data disaggregation, we were able to see that for the most part the same schools had the highest levels of engagements during this time period, and the same schools had the lowest levels of engagement. This insight can provide the administrator with knowledge about where they need to provide additional support or additional professional development to support teacher usage of Science Simulation App with their students. 

The CatchOn data provides a new level of visibility for administrators interested in evaluating and understanding the efficacy of technology products within a learning setting. District leaders know that teachers are the best evaluators of product efficacy. Per the latest Speak Up results, 68% of district administrators said they are using teacher feedback as an effective metric for evaluating efficacy of product usage. For districts that are interested in quantifying that feedback with tangible data, using the CatchOn data to see the consistency of usage by students and teachers across time and schools is an effective method for evaluating the value of your digital products. 

5
5
Personalized Learning Use Case:
Using the CatchOn data to analyze how technology is supporting personalized learning ​

Personalized Learning Use Case: Using the CatchOn data to analyze how technology is supporting personalized learning

Teachers are increasingly enthusiastic about the use of online and digital games to support student learning. Per 2019-20 school year results, 44% of teachers say they are incorporating games within classroom instruction at least weekly. The teachers see the use of games such as Quiz Game 1, Quiz Game 2, and Quiz Game 3 as vehicles for driving student engagement (83%) and also helping them differentiate the learning process to meet individual student needs (70%).

It follows that administrators will be interested in how the effective use of games within the classroom can support more personalized learning experiences for students. By evaluating student engagement with games, both in school and at home on school-owned devices, administrators can better understand the potential of games to address individual student needs.  The CatchOn data provides district leaders with a high level of visibility and access to engagement data by product category (example: adaptive software) or specific product (Quiz Game 1). 

Example: District A, C, and F’s use of the CatchOn data to evaluate engagement in game products

To understand the viability of using the CatchOn data to inform how certain products are supporting personalized learning, we identified three commonly used game apps (Quiz Game 1, Quiz Game 2, and Quiz Game 3) and investigated their usage across three districts within the Study Group (District A – a rural district, District C – an urban district, and District F – a suburban district).

Our driving question was to see if the level of engagements differed by product or by district and then compare those engagement statistics with a sampling of other CatchOn districts. While our access provided us with the ability to see the usage across districts, an education leader could do the same in their own district comparing school usage metrics. Chart 7 illustrates the comparative analysis. 

Chart 7: District A, C, and F – engagement level across the districts for three game products

From the analysis, we can see across the three districts high levels of engagement with the three identified game products. As to be expected, the mix of products differs by district in some cases, but the overall engagement levels are not differentiated by community type (rural, suburban, or urban). Whereas Quiz Game 1 is the predominant game product used in District C, District F is slightly more likely to use Quiz Game 2.  

Compared to the CatchOn sampling of other districts (an artifice established just for comparison with the Study Group), the three districts from the Digital Promise League of Innovative Schools pilot demonstrate greater usage of game technology within their classrooms. This is not surprising as Digital Promise and the League have long advocated for more personalized learning and the effective use of innovative solutions such as games within learning. 

The CatchOn dashboard provides ways for administrators also to view data on product usage when the school-owned devices are at home with students. That data is helpful for making the connection between product usage and personalized learning. For example, District A was using Remediation Game fairly consistently during the school day in spring 2021 with an average of 30 engagements per user and 98,832 total engagements.

With CatchOn, the leadership in District A can also see the number of engagements with Remediation Game outside of school. During the same period of time, there were 15,166 engagements outside of school with 1,633 unique users using  Remediation Game to supplement their learning and self-remediate where needed with additional game play time. This could indicate to the administrators the value of their Remediation Game subscription in supporting at-home student game play as well as in school. Being able to support students’ learning at home is another metric for connecting the dots between product usage and personalized learning. 

Page Contents