|GRADING THE DIGITAL SCHOOL
September 3, 2011
In Classroom of Future, Stagnant Scores
By MATT RICHTEL
CHANDLER, Ariz. — Amy Furman, a seventh-grade English teacher here, roams among 31 students sitting at their desks or in clumps on the floor. They’re studying Shakespeare’s “As You Like It” — but not in any traditional way.
In this technology-centric classroom, students are bent over laptops, some blogging or building Facebook pages from the perspective of Shakespeare’s characters. One student compiles a song list from the Internet, picking a tune by the rapper Kanye West to express the emotions of Shakespeare’s lovelorn Silvius.
The class, and the Kyrene School District as a whole, offer what some see as a utopian vision of education’s future. Classrooms are decked out with laptops, big interactive screens and software that drills students on every basic subject. Under a ballot initiative approved in 2005, the district has invested roughly $33 million in such technologies.
The digital push here aims to go far beyond gadgets to transform the very nature of the classroom, turning the teacher into a guide instead of a lecturer, wandering among students who learn at their own pace on Internet-connected devices.
“This is such a dynamic class,” Ms. Furman says of her 21st-century classroom. “I really hope it works.”
Hope and enthusiasm are soaring here. But not test scores.
Since 2005, scores in reading and math have stagnated in Kyrene, even as statewide scores have risen.
To be sure, test scores can go up or down for many reasons. But to many education experts, something is not adding up — here and across the country. In a nutshell: schools are spending billions on technology, even as they cut budgets and lay off teachers, with little proof that this approach is improving basic learning.
This conundrum calls into question one of the most significant contemporary educational movements. Advocates for giving schools a major technological upgrade — which include powerful educators, Silicon Valley titans and White House appointees — say digital devices let students learn at their own pace, teach skills needed in a modern economy and hold the attention of a generation weaned on gadgets.
Some backers of this idea say standardized tests, the most widely used measure of student performance, don’t capture the breadth of skills that computers can help develop. But they also concede that for now there is no better way to gauge the educational value of expensive technology investments.
“The data is pretty weak. It’s very difficult when we’re pressed to come up with convincing data,” said Tom Vander Ark, the former executive director for education at the Bill and Melinda Gates Foundation and an investor in educational technology companies. When it comes to showing results, he said, “We better put up or shut up.”
And yet, in virtually the same breath, he said change of a historic magnitude is inevitably coming to classrooms this decade: “It’s one of the three or four biggest things happening in the world today.”
Critics counter that, absent clear proof, schools are being motivated by a blind faith in technology and an overemphasis on digital skills — like using PowerPoint and multimedia tools — at the expense of math, reading and writing fundamentals. They say the technology advocates have it backward when they press to upgrade first and ask questions later.
The spending push comes as schools face tough financial choices. In Kyrene, for example, even as technology spending has grown, the rest of the district’s budget has shrunk, leading to bigger classes and fewer periods of music, art and physical education.
At the same time, the district’s use of technology has earned it widespread praise. It is upheld as a model of success by the National School Boards Association, which in 2008 organized a visit by 100 educators from 17 states who came to see how the district was innovating.
And the district has banked its future and reputation on technology. Kyrene, which serves 18,000 kindergarten to eighth-grade students, mostly from the cities of Tempe, Phoenix and Chandler, uses its computer-centric classes as a way to attract children from around the region, shoring up enrollment as its local student population shrinks. More students mean more state dollars.
The issue of tech investment will reach a critical point in November. The district plans to go back to local voters for approval of $46.3 million more in taxes over seven years to allow it to keep investing in technology. That represents around 3.5 percent of the district’s annual spending, five times what it spends on textbooks.
The district leaders’ position is that technology has inspired students and helped them grow, but that there is no good way to quantify those achievements — putting them in a tough spot with voters deciding whether to bankroll this approach again.
“My gut is telling me we’ve had growth,” said David K. Schauer, the superintendent here. “But we have to have some measure that is valid, and we don’t have that.”
It gives him pause.
“We’ve jumped on bandwagons for different eras without knowing fully what we’re doing. This might just be the new bandwagon,” he said. “I hope not.”
A Dearth of Proof
The pressure to push technology into the classroom without proof of its value has deep roots.
In 1997, a science and technology committee assembled by President Clinton issued an urgent call about the need to equip schools with technology.
If such spending was not increased by billions of dollars, American competitiveness could suffer, according to the committee, whose members included educators like Charles M. Vest, then president of the Massachusetts Institute of Technology, and business executives like John A. Young, the former chief executive of Hewlett-Packard.
To support its conclusion, the committee’s report cited the successes of individual schools that embraced computers and saw test scores rise or dropout rates fall. But while acknowledging that the research on technology’s impact was inadequate, the committee urged schools to adopt it anyhow.
The report’s final sentence read: “The panel does not, however, recommend that the deployment of technology within America’s schools be deferred pending the completion of such research.”
Since then, the ambitions of those who champion educational technology have grown — from merely equipping schools with computers and instructional software, to putting technology at the center of the classroom and building the teaching around it.
Kyrene had the same sense of urgency as President Clinton’s committee when, in November 2005, it asked voters for an initial $46.3 million for laptops, classroom projectors, networking gear and other technology for teachers and administrators.
Before that, the district had given 300 elementary school teachers five laptops each. Students and teachers used them with great enthusiasm, said Mark Share, the district’s 64-year-old director of technology, a white-bearded former teacher from the Bronx with an iPhone clipped to his belt.
“If we know something works, why wait?” Mr. Share told The Arizona Republic the month before the vote. The district’s pitch was based not on the idea that test scores would rise, but that technology represented the future.
The measure, which faced no organized opposition, passed overwhelmingly. It means that property owners in the dry, sprawling flatlands here, who live in apartment complexes, cookie-cutter suburban homes and salmon-hued mini-mansions, pay on average $75 more a year in taxes, depending on the assessed value of their homes, according to the district.
But the proof sought by President Clinton’s committee remains elusive even today, though researchers have been seeking answers.
Many studies have found that technology has helped individual classrooms, schools or districts. For instance, researchers found that writing scores improved for eighth-graders in Maine after they were all issued laptops in 2002. The same researchers, from the University of Southern Maine, found that math performance picked up among seventh- and eighth-graders after teachers in the state were trained in using the laptops to teach.
A question plaguing many education researchers is how to draw broader inferences from such case studies, which can have serious limitations. For instance, in the Maine math study, it is hard to separate the effect of the laptops from the effect of the teacher training.
Educators would like to see major trials years in length that clearly demonstrate technology’s effect. But such trials are extraordinarily difficult to conduct when classes and schools can be so different, and technology is changing so quickly.
And often the smaller studies produce conflicting results. Some classroom studies show that math scores rise among students using instructional software, while others show that scores actually fall. The high-level analyses that sum up these various studies, not surprisingly, give researchers pause about whether big investments in technology make sense.
One broad analysis of laptop programs like the one in Maine, for example, found that such programs are not a major factor in student performance.
“Rather than being a cure-all or silver bullet, one-to-one laptop programs may simply amplify what’s already occurring — for better or worse,” wrote Bryan Goodwin, spokesman for Mid-continent Research for Education and Learning, a nonpartisan group that did the study, in an essay. Good teachers, he said, can make good use of computers, while bad teachers won’t, and they and their students could wind up becoming distracted by the technology.
A review by the Education Department in 2009 of research on online courses — which more than one million K-12 students are taking — found that few rigorous studies had been done and that policy makers “lack scientific evidence” of their effectiveness.. A division of the Education Department that rates classroom curriculums has found that much educational software is not an improvement over textbooks.
Larry Cuban, an education professor emeritus at Stanford University, said the research did not justify big investments by districts.
“There is insufficient evidence to spend that kind of money. Period, period, period,” he said. “There is no body of evidence that shows a trend line.”
Some advocates for technology disagree.
Karen Cator, director of the office of educational technology in the United States Department of Education, said standardized test scores were an inadequate measure of the value of technology in schools. Ms. Cator, a former executive at Apple Computer, said that better measurement tools were needed but, in the meantime, schools knew what students needed.
“In places where we’ve had a large implementing of technology and scores are flat, I see that as great,” she said. “Test scores are the same, but look at all the other things students are doing: learning to use the Internet to research, learning to organize their work, learning to use professional writing tools, learning to collaborate with others.”
For its part, Kyrene has become a model to many by training teachers to use technology and getting their ideas on what inspires them. As Mr. Share says in the signature file at the bottom of every e-mail he sends: “It’s not the stuff that counts — it’s what you do with it that matters.”
So people here are not sure what to make of the stagnant test scores. Many of the district’s schools, particularly those in more affluent areas, already had relatively high scores, making it a challenge to push them significantly higher. A jump in students qualifying for free or reduced-price lunches was largely a result of the recession, not a shift in the population the district serves, said Nancy Dundenhoefer, its community relations manager.
Mr. Share, whose heavy influence on more than $7 million a year in technology spending has made him a power broker, said he did not think demographic changes were a good explanation.
“You could argue that test scores would be lower without the technology, but that’s a copout,” he said, adding that the district should be able to deliver some measure of what he considers its obvious success with technology. “It’s a conundrum.”
Results aside, it’s easy to see why technology is such an easy sell here, given the enthusiasm surrounding it in some classrooms.
Engaging With Paper
“I start with pens and pencils,” says Ms. Furman, 41, who is short and bubbly and devours young-adult novels to stay in touch with students. Her husband teaches eighth grade in the district, and their son and daughter are both students.
At the beginning of the school year, Ms. Furman tries to inspire her students at Aprende Middle School to write, a task she says becomes increasingly difficult when students reach the patently insecure middle-school years.
In one class in 2009 she had them draw a heart on a piece of paper. Inside the heart, she asked them to write the names of things and people dear to them. One girl started to cry, then another, as the class shared their stories.
It was something Ms. Furman doubted would have happened if the students had been using computers. “There is a connection between the physical hand on the paper and the words on the page,” she said. “It’s intimate.”
But, she said, computers play an important role in helping students get their ideas down more easily, edit their work so they can see instant improvement, and share it with the class. She uses a document camera to display a student’s paper at the front of the room for others to dissect.
Ms. Furman said the creative and editing tools, by inspiring students to make quick improvements to their writing, pay dividends in the form of higher-quality work. Last year, 14 of her students were chosen as finalists in a statewide essay contest that asked them how literature had affected their lives. “I was running down the hall, weeping, saying, ‘Get these students together. We need to tell them they’ve won!’ ”
Other teachers say the technology is the only way to make this generation learn.
“They’re inundated with 24/7 media, so they expect it,” said Sharon Smith, 44, a gregarious seventh-grade social studies teacher whose classroom is down the hall from Ms. Furman’s.
Minutes earlier, Ms. Smith had taught a Civil War lesson in a way unimaginable even 10 years ago. With the lights off, a screen at the front of the room posed a question: “Jefferson Davis was Commander of the Union Army: True or False?”
The 30 students in the classroom held wireless clickers into which they punched their answers. Seconds later, a pie chart appeared on the screen: 23 percent answered “True,” 70 percent “False,” and 6 percent didn’t know.
The students hooted and hollered, reacting to the instant poll. Ms. Smith then drew the students into a conversation about the answers.
The enthusiasm underscores a key argument for investing in classroom technology: student engagement.
That idea is central to the National Education Technology Plan released by the White House last year, which calls for the “revolutionary transformation” of schools. The plan endorses bringing “state-of-the art technology into learning to enable, motivate and inspire all students.”
But the research, what little there is of it, does not establish a clear link between computer-inspired engagement and learning, said Randy Yerrick, associate dean of educational technology at the University of Buffalo.
For him, the best educational uses of computers are those that have no good digital equivalent. As examples, he suggests using digital sensors in a science class to help students observe chemical or physical changes, or using multimedia tools to reach disabled children.
But he says engagement is a “fluffy term” that can slide past critical analysis. And Professor Cuban at Stanford argues that keeping children engaged requires an environment of constant novelty, which cannot be sustained.
“There is very little valid and reliable research that shows the engagement causes or leads to higher academic achievement,” he said.
Instruct or Distract?
There are times in Kyrene when the technology seems to allow students to disengage from learning: They are left at computers to perform a task but wind up playing around, suggesting, as some researchers have found, that computers can distract and not instruct.
The 23 kindergartners in Christy Asta’s class at Kyrene de las Brisas are broken into small groups, a common approach in Kyrene. A handful stand at desks, others sit at computers, typing up reports.
Xavier Diaz, 6, sits quietly, chair pulled close to his Dell laptop, playing “Alien Addition.” In this math arcade game, Xavier controls a pod at the bottom of the screen that shoots at spaceships falling from the sky. Inside each ship is a pair of numbers. Xavier’s goal is to shoot only the spaceship with numbers that are the sum of the number inside his pod.
But Xavier is just shooting every target in sight. Over and over. Periodically, the game gives him a message: “Try again.” He tries again.
“Even if he doesn’t get it right, it’s getting him to think quicker,” says the teacher, Ms. Asta. She leans down next to him: “Six plus one is seven. Click here.” She helps him shoot the right target. “See, you shot him.”
Perhaps surprisingly given the way young people tend to gravitate toward gadgets, students here seem divided about whether they prefer learning on computers or through more traditional methods.
In a different class, Konray Yuan and Marisa Guisto, both 7, take turns touching letters on the interactive board on the wall. They are playing a spelling game, working together to spell the word “cool.” Each finds one of the letters in a jumbled grid, touching them in the proper order.
Marisa says there isn’t a difference between learning this way and learning on paper. Konray prefers paper, he says, because you get extra credit for good penmanship.
But others, particularly older students, say they enjoy using the technology tools. One of Ms. Furman’s students, Julia Schroder, loved building a blog to write about Shakespeare’s “As You Like It.”
In another class, she and several classmates used a video camera to film a skit about Woodrow Wilson’s 14-point speech during World War I — an approach she preferred to speaking directly to the class.
“I’d be pretty bummed if I had to do a live thing,” she said. “It’s nerve-racking.”
Teachers vs. Tech
Even as students are getting more access to computers here, they are getting less access to teachers.
Reflecting budget cuts, class sizes have crept up in Kyrene, as they have in many places. For example, seventh-grade classes like Ms. Furman’s that had 29 to 31 students grew to more like 31 to 33.
“You can’t continue to be effective if you keep adding one student, then one student, then one student,” Ms. Furman said. “I’m surprised parents aren’t going into the classrooms saying ‘Whoa.’ ”
Advocates of high-tech classrooms say computers are not intended to replace teachers. But they do see a fundamental change in the teacher’s role. Their often-cited mantra is that teachers should go from being “a sage on the stage to a guide on the side.”
And they say that, technology issues aside, class sizes can in fact afford to grow without hurting student performance.
Professor Cuban at Stanford said research showed that student performance did not improve significantly until classes fell under roughly 15 students, and did not get much worse unless they rose above 30.
At the same time, he says bigger classes can frustrate teachers, making it hard to attract and retain talented ones.
In Kyrene, growing class sizes reflect spending cuts; the district’s maintenance and operation budget fell to $95 million this year from $106 million in 2008. The district cannot use the money designated for technology to pay for other things. And the teachers, who make roughly $33,000 to $57,000 a year, have not had a raise since 2008.
Many teachers have second jobs, some in restaurants and retail, said Erin Kirchoff, president of the Kyrene Education Association, the teacher’s association. Teachers talk of being exhausted from teaching all day, then selling shoes at the mall.
Ms. Furman works during the summer at the Kyrene district offices. But that job is being eliminated in 2014, and she is worried about the income loss.
“Without it, we don’t go on vacation,” she said.
Money for other things in the district is short as well. Many teachers say they regularly bring in their own supplies, like construction paper.
“We have Smart Boards in every classroom but not enough money to buy copy paper, pencils and hand sanitizer,” said Nicole Cates, a co-president of the Parent Teacher Organization at Kyrene de la Colina, an elementary school. “You don’t go buy a new outfit when you don’t have enough dinner to eat.”
But she loves the fact that her two children, a fourth-grader and first-grader, are learning technology, including PowerPoint and educational games.
To some who favor high-tech classrooms, the resource squeeze presents an opportunity. Their thinking is that struggling schools will look for more efficient ways to get the job done, creating an impetus to rethink education entirely.
“Let’s hope the fiscal crisis doesn’t get better too soon. It’ll slow down reform,” said Tom Watkins, the former superintendent for the Michigan schools, and now a consultant to businesses in the education sector.
Clearly, the push for technology is to the benefit of one group: technology companies.
It is 4:30 a.m. on a Tuesday. Mr. Share, the director of technology at Kyrene and often an early riser, awakens to the hard sell. Awaiting him at his home computer are six pitches from technology companies.
It’s just another day for the man with the checkbook.
“I get one pitch an hour,” he said. He finds most of them useless and sometimes galling: “They’re mostly car salesmen. I think they believe in the product they’re selling, but they don’t have a leg to stand on as to why the product is good or bad.”
Mr. Share bases his buying decisions on two main factors: what his teachers tell him they need, and his experience. For instance, he said he resisted getting the interactive whiteboards sold as Smart Boards until, one day in 2008, he saw a teacher trying to mimic the product with a jury-rigged projector setup.
“It was an ‘Aha!’ moment,” he said, leading him to buy Smart Boards, made by a company called Smart Technologies.
He can make that kind of decision because he has money — and the vendors know it. Technology companies track which districts get federal funding and which have passed tax assessments for technology, like Kyrene.
This is big business. Sales of computer software to schools for classroom use were $1.89 billion in 2010. Spending on hardware is more difficult to measure, researchers say, but some put the figure at five times that amount.
The vendors relish their relationship with Kyrene.
“I joke I should have an office here, I’m here so often,” said Will Dunham, a salesman for CCS Presentation Systems, a leading reseller of Smart Boards in Arizona.
Last summer, the district paid $500,000 to CCS to replace ceiling-hung projectors in 400 classrooms. The alternative was to spend $100,000 to replace their aging bulbs, which Mr. Share said were growing dimmer, causing teachers to sometimes have to turn down the lights to see a crisp image.
Mr. Dunham said the purchase made sense because new was better. “I could take a used car down to the mechanic and get it all fixed up and still have a used car.”
But Ms. Kirchoff, the president of the teachers’ association, is furious. “My projector works just fine,” she said. “Give me Kleenex, Kleenex, Kleenex!”
Last November, Kyrene went back to voters to ask them to pay for another seven years of technology spending in the district. The previous measure from 2005 will not expire for two years. But the district wanted to get ahead of the issue, and leave wiggle room just in case the new measure didn’t pass.
It didn’t. It lost by 96 votes out of nearly 50,000 cast. Mr. Share and others here said they attributed the failure to poor wording on the ballot that made it look like a new tax increase, rather than the continuation of one.
They say they will not make the same wording mistake this time. And they say the burden on taxpayers is modest.
“It’s so much bang for the buck,” said Jeremy Calles, Kyrene’s interim chief financial officer. For a small investment, he said, “we get state-of-the-art technology.”
Regardless, some taxpayers have already decided that they will not vote yes.
“When you look at the big picture, it’s hard to say ‘yes, spend more on technology’ when class sizes increase,” said Kameron Bybee, 34, who has two children in district schools. “The district has made up its mind to go forward with the technologically advanced path. Come hell or high water.”
Other parents feel conflicted. Eduarda Schroder, 48, whose daughter Julia was in Ms. Furman’s English class, worked on the political action committee last November to push through an extension of the technology tax. Computers, she says, can make learning more appealing. But she’s also concerned that test scores haven’t gone up.
She says she is starting to ask a basic question. “Do we really need technology to learn?” she said. “It’s a very valid time to ask the question, right before this goes on the ballot.”
October 8, 2011
Inflating the Software Report Card
By TRIP GABRIEL and MATT RICHTEL
The Web site of Carnegie Learning, a company started by scientists at Carnegie Mellon University that sells classroom software, trumpets this promise: “Revolutionary Math Curricula. Revolutionary Results.”
The pitch has sounded seductive to thousands of schools across the country for more than a decade. But a review by the United States Department of Education last year would suggest a much less alluring come-on: Undistinguished math curricula. Unproven results.
The federal review of Carnegie Learning’s flagship software, Cognitive Tutor, said the program had “no discernible effects” on the standardized test scores of high school students. A separate 2009 federal look at 10 major software products for teaching algebra as well as elementary and middle school math and reading found that nine of them, including Cognitive Tutor, “did not have statistically significant effects on test scores.”
Amid a classroom-based software boom estimated at $2.2 billion a year, debate continues to rage over the effectiveness of technology on learning and how best to measure it. But it is hard to tell that from technology companies’ promotional materials.
Many companies ignore well-regarded independent studies that test their products’ effectiveness. Carnegie’s Web site, for example, makes no mention of the 2010 review, by the Education Department’s What Works Clearinghouse, which analyzed 24 studies of Cognitive Tutor’s effectiveness but found that only four of those met high research standards. Some firms misrepresent research by cherry-picking results and promote surveys or limited case studies that lack the scientific rigor required by the clearinghouse and other authorities.
“The advertising from the companies is tremendous oversell compared to what they can actually demonstrate,” said Grover J. Whitehurst, a former director of the Institute of Education Sciences, the federal agency that includes What Works.
School officials, confronted with a morass of complicated and sometimes conflicting research, often buy products based on personal impressions, marketing hype or faith in technology for its own sake.
“They want the shiny new one,” said Peter Cohen, chief executive of Pearson School, a leading publisher of classroom texts and software. “They always want the latest, when other things have been proven the longest and demonstrated to get results.”
Carnegie, one of the most respected of the educational software firms, is hardly alone in overpromising or misleading. The Web site of Houghton Mifflin Harcourt says that “based on scientific research, Destination Reading is a powerful early literacy and adolescent literacy program,” but it fails to mention that it was one of the products the Department of Education found in 2009 not to have statistically significant effects on test scores.
Similarly, Pearson’s Web site cites several studies of its own to support its claim that Waterford Early Learning improves literacy, without acknowledging the same 2009 study’s conclusion that it had little impact.
And Intel, in a Web document urging schools to buy computers for every student, acknowledges that “there are no longitudinal, randomized trials linking eLearning to positive learning outcomes.” Yet it nonetheless argues that research shows that technology can lead to more engaged and economically successful students, happier teachers and more involved parents.
“To compare this public relations analysis to a carefully constructed research study is laughable,” said Alex Molnar, professor of education at the National Education Policy Center at the University of Colorado. “They are selling their wares.”
Carnegie officials say 600,000 students in 44 states use its products, many taking teacher-led classes three times a week with Carnegie-provided workbooks and spending the other two class periods in computer labs using Cognitive Tutor. The full curriculum can cost nearly three times as much as a typical textbook over six years.
Officials declined to release annual revenue figures, but Carnegie Learning was acquired in August for $75 million by the parent of the for-profit University of Phoenix. Carnegie Mellon University, which had retained ownership of the Cognitive Tutor software and licensed it to Carnegie Learning, earned an additional $21.5 million from the sale.
Steve Ritter, a founder and the chief scientist of Carnegie Learning, said there were flaws in the What Works Clearinghouse evaluations of Cognitive Tutor and disputed the Education Department’s judgment of what makes a worthy study.
“What you want to focus on is more of the why,” he said, “and less of a horse race to find out what works and doesn’t.”
A Carnegie spokeswoman, Mary Murrin, said in a statement that the company used “the data from all studies with varying outcomes to continuously improve our programs.”
Karen Billings, a vice president of the Software and Information Industry Association — a trade group representing many education companies — said the problem was not that companies overpromise, but that schools often do not properly deploy the products or train teachers to use them. Ms. Billings’s group helped design the field trials, in 132 schools, for the landmark 2009 government study of 10 software products, which was ordered by Congress and cost $15 million.
Then came the deflating results. The industry “became very hostile,” recalled Mr. Whitehurst, now director of education policy at the Brookings Institution. “It seems to me,” he added, “ ‘hypocrisy’ is the right word for loving something until the results are not what you expect.”
The Hard Sell
Shelly Allen, the math coordinator for public schools in Augusta, Ga., has seen a lot of curriculum salespeople pass through. She is wary of their sweet words and hard sell.
In June, when representatives from Carnegie Learning visited, Dr. Allen warned: “I just want everybody to know I grew up here. I graduated from here. My children go to school here. When you guys get back where you live, our kids have to still be able to reach goals we set.”
Augusta is famous for its magnolia-shaded National Golf Club, host to the Masters Tournament, but its public schools are typical of struggling urban districts. Three-quarters of the 32,000 students in the district, Richmond County, are black, and 72 percent are poor enough to qualify for the federal lunch program. The mean SAT math score last year was 443, below Georgia’s mean of 490 and the nation’s 516.
Six years ago, the district adopted Cognitive Tutor for about 3,000 students at risk of failing, paying $101,500 annually to use it. As students work through problems, the computer analyzes their weaknesses and serves up new items until they grasp the skill and are allowed to move on. To a student, the promotional materials say, it feels “as if the software is getting to know her and supporting her like a tutor.”
So when the screen says: “You are saving to buy a bicycle. You have $10, and each day you are able to save $2,” the student must convert the word problem into an algebraic expression. If he is stumped, he can click on the “Hint” button.
“Define a variable for the time from now,” the software advises. Still stumped? Click “Next Hint.”
“Use x to represent the time from now.” Aha. The student types “2x+10.”
The software likes this and moves on to highlight a series of questions in green, beginning with, “How many more days must you save to buy a bike that costs $60?” Using his 2x+10 formula, the student enters “25.”
After solving several questions of this sort and plotting them on a graph, the student would click “Skillometer” to see how he had fared. A series of forest-green bars would show that he did well labeling axes for his graph, but not so well writing the initial formula.
Moving on, Cognitive Tutor would bump him down to an easier problem: “A skier noticed that he can complete a run in about 30 minutes (half an hour).” The expression relating ski runs to time would be 2x, with x representing hours.
“Immediate feedback,” Carnegie Learning explains on its Web site, “enables the student to self-correct and leads to more effective learning.”
Augusta officials liked the program enough that when concerns arose last winter that many 11th graders were not on track to pass a new state graduation test, the district asked to expand the software’s use to all 9,400 of its high school students. The company agreed to provide access for no additional charge — temporarily.
“As a company, it makes sense to give you the opportunity to prove it works for all students,” Anita Sprayberry, a regional sales manager, told school leaders. That way, she said, “We can talk about a bigger sale.”
Going forward, Ms. Sprayberry said, the cost would be about $34,000 for each of the district’s 11 high schools.
In a recent interview, Dr. Allen said she was familiar with the What Works Clearinghouse, but not its 2010 finding that Cognitive Tutor did not raise test scores more than textbooks.
Though the clearinghouse is intended to help school leaders choose proven curriculum, a 2010 Government Accountability Office survey of district officials found that 58 percent of them had never heard of What Works, never mind consulted its reviews.
“Decisions are made on marketing, on politics, on personal preference,” said Robert A. Slavin, director of the Center for Research and Reform in Education at Johns Hopkins University. “An intelligent, caring principal who’d never buy a car without looking at Consumer Reports, when they plunk down serious money to buy a curriculum, they don’t even look at the evidence.”
Founded in 1998 by cognitive and computer scientists along with math teachers, Carnegie Learning is proud of its academic heritage, and many education researchers consider it a model of rigor and transparency.
One founder, John R. Anderson, received the 2011 Benjamin Franklin Medal in Computer and Cognitive Science for work on how humans perceive, learn and reason. The company’s Web site promises that its curriculums “provide the research-based foundation for proven results,” citing “success stories” from around the country.
At Dundalk Middle School in Baltimore County, Md., for example, Carnegie Learning says that Cognitive Tutor led to an increase in the passing rate on a state assessment, to 86 percent in 2004 from 49 percent in 2002. What it does not say is that the rate remained at 85 percent last year, even though Dundalk dropped Cognitive Tutor in 2007 because of difficulties arranging lab time.
That is why many academics dismiss case studies: it is too easy for slices of data to be taken out of context, or for correlation to be confused with causation.
Instead, the gold standard of education research is a field trial in which similar groups of students are randomly assigned to classes where one uses the curriculum and the other does not.
The Carnegie Web site lists five such trials and says they all show positive results for Cognitive Tutor.
Three of these studies, however, were rejected by the What Works Clearinghouse for flaws in their design; in a fourth, the clearinghouse identified a problem with part of the study — the part that purported to show benefits. One of the rejected studies had found that users of Cognitive Tutor in 10 Miami high schools scored better on Florida state exams than a control group, but the clearinghouse found that the students being compared were not equivalent.
“The entire ‘effect’ of Cognitive Tutor possibly can be traced to other factors,” said Mark Dynarski, a former director of the clearinghouse, “and the way in which the research was carried out does not allow one to know if this is the case.”
Dr. Ritter, Carnegie’s chief scientist, noted that the clearinghouse’s 2010 review was limited to high schools and that a year earlier it found that Cognitive Tutor had “potentially positive effects” in middle school.
The middle school finding rested on one study, out of 14 reviewed. That study is featured prominently on the Carnegie Web site, which omits mention of two others that the Education Department judged to be well designed but showed no benefits.
Dr. Ritter said he had excluded those studies, in Hawaii and Virginia, because the students had not used Cognitive Tutor precisely as the company intended. The researcher who did the Hawaii study, Denis Newman, said it reflected how Cognitive Tutor was used in the real world.
Dr. Newman is also the author of research guidelines for the Software and Information Industry Association, where Dr. Ritter sits on the education research working group. One of those guidelines states, “An expectation in the scientific community is that research findings are made available regardless of the result.”
Karen Cator, a former Apple executive who directs the Office of Educational Technology at the Department of Education, said the clearinghouse reports on software should be “taken with a grain of salt” because they rely on standardized test scores. Those tests, Ms. Cator said, cannot gauge some skills that technology teaches, like collaboration, multimedia and research.
Ms. Cator’s office is developing a new framework to measure the educational value of technology, but she advised schools and districts not to wait to invest in software like Cognitive Tutor.
“They know what their students need to know and what they need to be able to do,” she said.
In Augusta, Dr. Allen, the math coordinator, said her district did not have the means to study the effectiveness of Cognitive Tutor formally. But she and her staff saw that low-achieving students who used it were able to join mainstream classes. And teachers appreciated the way the software transmits assessments in real time to Carnegie Learning, then kicks back a report indicating the strengths and weaknesses of each student.
Teachers “just didn’t know, skill by skill, the same type of data they are getting now,” Dr. Allen said.
On the other hand, when the new state math test was given in March, 27 percent of the district’s 11th graders did not pass, which Dr. Allen described as “something that makes us not real excited.”
At the June meeting with Carnegie Learning’s sales team, Dr. Allen said Cognitive Tutor could be worthwhile if the district, which has recently cut $7 million from its budget and furloughed employees for nine days, could scrape together the financing. “Our negotiations are intense because we don’t have any money,” she said to laughter around the table.
In Georgia, where the state negotiates prices with publishers, an annual license for Cognitive Tutor software is $32 per student, and the workbook, which must be replaced annually, is $24 — for a total of $336 over six years, a typical lifespan of a math textbook that costs about $120.
Ultimately, Dr. Allen’s district did not have the money, so she focused on getting the most out of her staff. “Giving them the right tools and resources certainly helps,” she said, “but our teachers are the ones making that difference.”
Gregory W. Capelli, co-chief executive of the Apollo Group, which runs the 400,000-student University of Phoenix and bought Carnegie Learning this summer, said his company first ran its own pilot project with the software and also examined independent research.
But Mr. Capelli, like others, relied at least in part on personal experience.
“My daughter, who’s in eighth grade, used this product,” he said.
“She would do very well” in some lessons “and not in others,” Mr. Capelli said. “What I liked about it is that once she got it, it would allow her to go on to the next part of the tree.”
This article has been revised to reflect the following correction:
Correction: October 13, 2011
An article on Saturday about technology companies that produce software for use in classrooms misspelled the given name of a researcher whose study showed that the use of Cognitive Tutor, one such program, produced no benefits. He is Denis Newman, not Dennis.
October 22, 2011
A Silicon Valley School That Doesn’t Compute
By MATT RICHTEL
LOS ALTOS, Calif. — The chief technology officer of eBay sends his children to a nine-classroom school here. So do employees of Silicon Valley giants like Google, Apple, Yahoo and Hewlett-Packard.
But the school’s chief teaching tools are anything but high-tech: pens and paper, knitting needles and, occasionally, mud. Not a computer to be found. No screens at all. They are not allowed in the classroom, and the school even frowns on their use at home.
Schools nationwide have rushed to supply their classrooms with computers, and many policy makers say it is foolish to do otherwise. But the contrarian point of view can be found at the epicenter of the tech economy, where some parents and educators have a message: computers and schools don’t mix.
This is the Waldorf School of the Peninsula, one of around 160 Waldorf schools in the country that subscribe to a teaching philosophy focused on physical activity and learning through creative, hands-on tasks. Those who endorse this approach say computers inhibit creative thinking, movement, human interaction and attention spans.
The Waldorf method is nearly a century old, but its foothold here among the digerati puts into sharp relief an intensifying debate about the role of computers in education.
“I fundamentally reject the notion you need technology aids in grammar school,” said Alan Eagle, 50, whose daughter, Andie, is one of the 196 children at the Waldorf elementary school; his son William, 13, is at the nearby middle school. “The idea that an app on an iPad can better teach my kids to read or do arithmetic, that’s ridiculous.”
Mr. Eagle knows a bit about technology. He holds a computer science degree from Dartmouth and works in executive communications at Google, where he has written speeches for the chairman, Eric E. Schmidt. He uses an iPad and a smartphone. But he says his daughter, a fifth grader, “doesn’t know how to use Google,” and his son is just learning. (Starting in eighth grade, the school endorses the limited use of gadgets.)
Three-quarters of the students here have parents with a strong high-tech connection. Mr. Eagle, like other parents, sees no contradiction. Technology, he says, has its time and place: “If I worked at Miramax and made good, artsy, rated R movies, I wouldn’t want my kids to see them until they were 17.”
While other schools in the region brag about their wired classrooms, the Waldorf school embraces a simple, retro look — blackboards with colorful chalk, bookshelves with encyclopedias, wooden desks filled with workbooks and No. 2 pencils.
On a recent Tuesday, Andie Eagle and her fifth-grade classmates refreshed their knitting skills, crisscrossing wooden needles around balls of yarn, making fabric swatches. It’s an activity the school says helps develop problem-solving, patterning, math skills and coordination. The long-term goal: make socks.
Down the hall, a teacher drilled third-graders on multiplication by asking them to pretend to turn their bodies into lightning bolts. She asked them a math problem — four times five — and, in unison, they shouted “20” and zapped their fingers at the number on the blackboard. A roomful of human calculators.
In second grade, students standing in a circle learned language skills by repeating verses after the teacher, while simultaneously playing catch with bean bags. It’s an exercise aimed at synchronizing body and brain. Here, as in other classes, the day can start with a recitation or verse about God that reflects a nondenominational emphasis on the divine.
Andie’s teacher, Cathy Waheed, who is a former computer engineer, tries to make learning both irresistible and highly tactile. Last year she taught fractions by having the children cut up food — apples, quesadillas, cake — into quarters, halves and sixteenths.
“For three weeks, we ate our way through fractions,” she said. “When I made enough fractional pieces of cake to feed everyone, do you think I had their attention?”
Some education experts say that the push to equip classrooms with computers is unwarranted because studies do not clearly show that this leads to better test scores or other measurable gains.
Is learning through cake fractions and knitting any better? The Waldorf advocates make it tough to compare, partly because as private schools they administer no standardized tests in elementary grades. And they would be the first to admit that their early-grade students may not score well on such tests because, they say, they don’t drill them on a standardized math and reading curriculum.
When asked for evidence of the schools’ effectiveness, the Association of Waldorf Schools of North America points to research by an affiliated group showing that 94 percent of students graduating from Waldorf high schools in the United States between 1994 and 2004 attended college, with many heading to prestigious institutions like Oberlin, Berkeley and Vassar.
Of course, that figure may not be surprising, given that these are students from families that value education highly enough to seek out a selective private school, and usually have the means to pay for it. And it is difficult to separate the effects of the low-tech instructional methods from other factors. For example, parents of students at the Los Altos school say it attracts great teachers who go through extensive training in the Waldorf approach, creating a strong sense of mission that can be lacking in other schools.
Absent clear evidence, the debate comes down to subjectivity, parental choice and a difference of opinion over a single world: engagement. Advocates for equipping schools with technology say computers can hold students’ attention and, in fact, that young people who have been weaned on electronic devices will not tune in without them.
Ann Flynn, director of education technology for the National School Boards Association, which represents school boards nationwide, said computers were essential. “If schools have access to the tools and can afford them, but are not using the tools, they are cheating our children,” Ms. Flynn said.
Paul Thomas, a former teacher and an associate professor of education at Furman University, who has written 12 books about public educational methods, disagreed, saying that “a spare approach to technology in the classroom will always benefit learning.”
“Teaching is a human experience,” he said. “Technology is a distraction when we need literacy, numeracy and critical thinking.”
And Waldorf parents argue that real engagement comes from great teachers with interesting lesson plans.
“Engagement is about human contact, the contact with the teacher, the contact with their peers,” said Pierre Laurent, 50, who works at a high-tech start-up and formerly worked at Intel and Microsoft. He has three children in Waldorf schools, which so impressed the family that his wife, Monica, joined one as a teacher in 2006.
And where advocates for stocking classrooms with technology say children need computer time to compete in the modern world, Waldorf parents counter: what’s the rush, given how easy it is to pick up those skills?
“It’s supereasy. It’s like learning to use toothpaste,” Mr. Eagle said. “At Google and all these places, we make technology as brain-dead easy to use as possible. There’s no reason why kids can’t figure it out when they get older.”
There are also plenty of high-tech parents at a Waldorf school in San Francisco and just north of it at the Greenwood School in Mill Valley, which doesn’t have Waldorf accreditation but is inspired by its principles.
California has some 40 Waldorf schools, giving it a disproportionate share — perhaps because the movement is growing roots here, said Lucy Wurtz, who, along with her husband, Brad, helped found the Waldorf high school in Los Altos in 2007. Mr. Wurtz is chief executive of Power Assure, which helps computer data centers reduce their energy load.
The Waldorf experience does not come cheap: annual tuition at the Silicon Valley schools is $17,750 for kindergarten through eighth grade and $24,400 for high school, though Ms. Wurtz said financial assistance was available. She says the typical Waldorf parent, who has a range of elite private and public schools to choose from, tends to be liberal and highly educated, with strong views about education; they also have a knowledge that when they are ready to teach their children about technology they have ample access and expertise at home.
The students, meanwhile, say they don’t pine for technology, nor have they gone completely cold turkey. Andie Eagle and her fifth-grade classmates say they occasionally watch movies. One girl, whose father works as an Apple engineer, says he sometimes asks her to test games he is debugging. One boy plays with flight-simulator programs on weekends.
The students say they can become frustrated when their parents and relatives get so wrapped up in phones and other devices. Aurad Kamkar, 11, said he recently went to visit cousins and found himself sitting around with five of them playing with their gadgets, not paying attention to him or each other. He started waving his arms at them: “I said: ‘Hello guys, I’m here.’ ”
Finn Heilig, 10, whose father works at Google, says he liked learning with pen and paper — rather than on a computer — because he could monitor his progress over the years.
“You can look back and see how sloppy your handwriting was in first grade. You can’t do that with computers ’cause all the letters are the same,” Finn said. “Besides, if you learn to write on paper, you can still write if water spills on the computer or the power goes out.”