Tag: MLTI

Evaluating the Success of Your Ed Tech Program

  • This article originally appeared in THE Journal. Republished with permission.
  • By Jeff Mao 
  • 07/12/17

I’ve been a part of the education technology crowd for a long time now. My first teaching job was with Brewster Academy, a small boarding school in New Hampshire. In the fall of 1993, Brewster launched its first 1-to-1 laptop program with its ninth-grade students. We had no internet. The laptop batteries lasted fewer than two hours if you were lucky, and our software tools were mostly limited to a productivity suite.

Despite the limitations of the technology back then, I learned a lot from my time at Brewster Academy. One of the things that we did well, that I still recommend to schools today, is to be targeted and intentional about how you use the technology. At Brewster, we measured success one skill at a time and one student at a time.

A decade later, I joined the team at the Maine Department of Education that led the Maine Learning Technology Initiative (MLTI) — a much larger initiative. Whereas Brewster’s total student body was fewer than 350 students, MLTI served about 35,000 students in more than 230 schools. 

How did we measure success? At the state-level, we looked at metrics different from the ones at Brewster. We looked at data points that we believed were indicators of positive movement forward, such as the percentage of teachers who engaged kids with the technology in either instruction or a learning activity, who used the technology to conduct research for lesson plans, or who used the technology to aid with assessing student progress. Looking at this data, we saw positive indicators of success.

However, the question remained: Were student performance and learning improving? This question is hard to answer from the state-level and what I’ve learned over the years is to return to what I found at Brewster. Schools need to be targeted and intentional with their use of technology, as well as mindful of student growth at the individual skill level.

Technology’s Impact in One Classroom

In 2009, MLTI published a research brief about a single science teacher in a small coastal town. Kevin Crafts taught eighth-grade science at the Bristol Consolidated School (where he continues today). Working with our researchers from the University of Southern Maine (USM), we examined student learning on a tricky science concept: the axis angle of the Earth and its impact. Crafts had two sections of eighth-grade science, and he taught the concept to the classes. He used the MLTI laptops with one group to create animated podcasts while the other group completed a more traditional poster assignment to demonstrate what they’d learned. 

All students completed pre- and post-assessments, as well as a retention assessment about a month later to compare growth and learning between the groups over time. The results showed that the group that used the technology to create animated podcasts outperformed their peers and a month later they showed very good retention of the concepts as compared to the other group.

Reading through the research brief, you find other interesting details the researchers captured through their observations. They also conducted interviews that uncovered more insight into why the podcast assignment led to better student learning. I know that almost all schools and teachers will not be able to go into this much depth and, in particular, may only be able to conclude anecdotal and intuitive understanding of the “why.” But with some preparation and planning, teachers can measure whether one method or another works better. Kevin Crafts no longer assigns the poster project and that’s really what matters. He’s found a better way.

How to Begin Your Ed Tech Evaluation

While it may seem daunting to a classroom teacher to replicate the work that Crafts and the USM researchers performed, there is a tool that could help you. The United States Department of Education commissioned the creation of an online tool — the Ed Tech RCE Coach — that not only helps you identify what you’re really trying to measure but also will help you crunch the data and analyze it.

Using the Ed Tech RCE Coach requires some work and planning and I don’t believe it’s necessary for each and every lesson that you teach. However, if you’re considering adopting a new technology tool for your district, school, or classroom or a significant change in how you teach a skill or concept, taking the extra time to leverage the tool can help you better gauge if you’re moving in the right direction.

Regardless of whether or not you use the tool, these three simple steps embedded in it can be terrific as guideposts for your teaching:

  1. Identify your desired outcome; 
  2. Identify what success looks like; and
  3. Identify the tools and pedagogical approach that you’re using. 

By following these simple steps, even without instituting a pre- and post-assessment or control and intervention groups, you can begin to better understand success. With these in place, you can make more direct comparisons between group performance and growth. Lacking these, you can compare this year’s students to last year’s students. It isn’t perfect, but it’s a start.

Tough Tech Choices: Laptops, Tablets, Chromebooks, BYOD?

Practical Advice to Help You Find Your School’s Solution

  • This article originally appeared in THE Journal. Republished with permission.
  • By Jeff Mao 
  • 05/24/16

I began my education career right out of college in the fall of 1992. My first teaching job was at Brewster Academy, a small boarding school in central New Hampshire. That year, the school began planning efforts that led to one of the nation’s first 1-to-1 learning programs. Little did I know that, only a decade later, I’d be working in a small gray office cubicle in the Maine Department of Education, helping lead the nation’s first statewide 1-to-1 learning program: the Maine Learning Technology Initiative (MLTI). 

One of my responsibilities as Maine’s coordinator of education technology was buying the technology and associated services for the MLTI. To say that I’ve been responsible for buying a lot of computers (and tablets) is an understatement. It’s from this perspective that I share some thoughts about the selection and purchasing of students’ primary learning devices.

Choosing a Purchasing Model: BYOD vs. District-Purchased Devices
There are generally two purchasing models: The district pays, or families pay (bring your own device/BYOD). In some instances, I’ve seen a cost-sharing model. 

BYOD has its supporters and opponents. On page 72 of the 2016 National Education Technology Plan (NETP), the U.S. Department of Education shares some issues for districts to consider when looking at BYOD, including three key concerns: 

  1. Economic disparity
  2. Instructional burden, and 
  3. Privacy and security. 

Many BYOD programs have attempted to mitigate economic disparity by budgeting district funds to buy devices for families that qualify for free and reduced lunch. However, I argue that doing so can jeopardize student privacy and security. By using district funds to buy devices for these families, the district makes visible which students have means and which do not. Districts should seriously consider whether this practice violates student privacy.

Another option, a district-purchased device for all students, also has its challenges, but I believe those are outweighed by the benefits. Economically, a BYOD program means less spending by the district because spending has been shifted to the family. However, the district-purchased option can be less expensive overall to the community since districts qualify for educational discounts. Depending on the size of the district, competitive bidding and subsequent negotiations can be a significant cost saver to the community as well. 

A district-purchased program also tends to lead to a standardized device. I learned from the mission statement of my second school, the Allendale Columbia School in New York, that “structure should liberate, not confine.” While some in your community may want the personalization that BYOD affords, the device you select should be a proverbial Swiss Army Knife. Personalization shouldn’t focus on variations in design, color, brand or operating system so much as the capacity to shape how an individual interacts, creates and consumes media with the device. 

Defining the Education Goals of Your 1-to-1 Program
This leads to one of the most critical considerations: What do you envision students and teachers doing with the devices? If your conversations have been dominated by gigabytes, screen size, operating systems and cost, then you should ask yourself: “Am I choosing a device based on my education goals? Or are my goals being defined by my device?” If you believe that structure should liberate, not confine, then seek a solution that solves for your educational goals, not the other way around.

Use the SAMR model of technology integration developed by Dr. Ruben Puentedura to help guide discussions about how the technology will be used — how it will change how teachers teach and students learn. You can use Common Sense Education’s quick video introduction to SAMRalong with videos and blog posts written by Dr. Puentedura; also, the MLTI iTunes U page hosts an excellent podcast series by Dr. Puentedura himself. An additional benefit to using this or another model is it will provide a common vocabulary for your school community to discuss technology use. Broad terms such as “integrating technology” are better defined, and, as a result, discussions can be more meaningful and useful.

With a model like SAMR in hand, you can define your educational vision and goals. This will help you better determine your device needs. As you go along, note who is involved. Including students and teachers is important. Remember: Your students outnumber the teachers by a large margin, and they’ll be the most common user of the device. Never assume what they want or what will be easy or hard for them, and don’t project your perspective on student-facing tools. Otherwise, you’re allowing what I like to call Adult Paradigm Paralysis Syndrome (APPS) to drive your thinking. Go ask a student. 

Additional Questions to Drive Your Planning
Finally, let’s get to the more nitty-gritty aspects. What about tablets versus laptops? Or a Chromebook versus a traditional computer? These are some questions to consider.

What age are the students? I mentioned the benefit of a standardized device, but in a real-world context, that may mean one standard for early elementary students, which differs from that for upper elementary, middle or high school students. Your students’ developmental needs are an important factor. 

Which devices do the teachers need? Whichever device you select for students you should also distribute to the teachers. That may mean teachers have two devices. Teachers’ needs are different from students’; institutional realities may mean compatibility and productivity needs that aren’t met by the student device. However, if the teacher doesn’t have a student device, how will she be able to fully understand the student experience? How can the teacher test that the resources she creates or curates are fully compatible on the student device?

What type of software is needed? The software you use is as critical as the device itself. Productivity software is still highly leveraged by students and teachers. Don’t ignore it, but also don’t fixate on it. Beyond the productivity suite are tools that facilitate communication, collaboration and creativity. Additionally, consider visualization, modeling, simulation and gaming. You may decide you don’t need all these tools, but it’s better to at least consider them than discover later that you need them.

How much storage space will we need? Increasingly, schools are leveraging rich digital content. Much of this content is accessible over the Internet but is often stored on the device itself. For storage of content and space for creativity, don’t skimp on storage capacity! Even when content is accessible online, remember that not all students will have ready access to high-quality broadband outside of school.

What can we afford? Delay looking at cost for as long as you can. Seek the solution that suits your needs, and then try to find a way to pay for it. If you quickly rule out options purely on cost, you’ll likely settle for a solution that doesn’t really meet your needs. I always prefer to justify the cost of a solution than to try to justify why students or teachers can’t do what you agree to be educationally sound and desirable. 

Though the cost of computing devices has fallen in recent years, you will still find that inexpensive devices can come with hidden or indirect costs. Those costs aren’t reason to avoid inexpensive devices, but you want to make sure you have accounted for the total cost of ownership (TCO) as best you can. What you may save in cost of the device may be spent on bandwidth. Or, more expensive devices may have a longer lifetime than less expensive alternatives.

For more on this topic, watch the webinar “Laptops, Tablets and Chromebooks? BYOD? Practical Advice to Help You Find Your School’s Solution“