By now, many Florida residents are fairly acquainted with the controversial teacher scoring formula known as the value-added model or VAM.The lengthy mathematical equation determines teacher impact on student growth and up to 50 percent of a teacher’s overall score by measuring a student’s expected test score against his or her actual results. And it does so by plugging in several variables apart from instruction that can potentially influence a student’s performance.
Florida uses 10 such variables in its model: a student’s disability status; his or her English language learner status; gifted status; mobility; number of subject-relevant courses; attendance; class size; age difference; prior achievement scores; and similarities of prior test scores.
But it’s the variable missing from that list that is causing more questions in a measurement system already steeped in controversy. Florida’s model does not account for student poverty status.
“It was explicitly disallowed by the legislature and it didn’t proceed any further,” said Sam Foerster, Deputy Chancellor for School Improvement and Student Achievement for the Florida Department of Education.
In 2011, Foerster—then, the associate superintendent of support services in Putnam County—chaired a committee of 27 teachers, parents and administrators tasked by the state department with designing Florida’s value-added model.
The state department of education contracted with D.C.-based American Institutes for Research to develop the model—the same nonprofit that it recently selected to develop the new statewide assessment on which the student growth measurement is based.
Florida legislation from the same year, prohibited setting different student expectations on the basis of gender, race, or socio-economic status.
“That was pretty clearly laid out, of the many, many, many things that could be considered as covariates,” Foerster said.
But in a state where more than half of students live in poverty, some fear it’s an omission that could be placing teachers in high-poverty schools at a disadvantage.
Suzanne Colvin, associate director of teaching and learning at the University of Florida, has co-authored a study examining teacher evaluation models across the country.
The research and similar studies show links between a student’s socio-economic status and performance. The economic gap was particularly apparent in the area of reading, Colvin said.
“That seemed to be one of the main contributing factors to this differentiation between reading performance, so it certainly was a variable that I was a little curious, quite frankly, as to why it was not accounted for in the Florida model,” she said.
Could leaving poverty out of the equation lead to biased results?
It’s possible, says Eric Isenberg, senior researcher at D.C.-based policy research firm Mathematica.
“To really know, you’d actually have to run the model two different ways,” he said. “One with the free lunch variable; one without the free lunch variable.”
Isenberg helped develop the IMPACT value-added model used in D.C. Public Schools. The model does adjust for poverty.
“In terms of why they wanted to account for a measure of student poverty status, it’s because they’re concerned if we don’t fully account for what the kids bring to the table, that they’ll be getting biased information about which teachers are relatively more effective and which teachers are less effective,” he said.
Isenberg has been working on a series of government reports examining teacher effectiveness in schools with disadvantaged students, the first of which was was released November 2013.
“We find that even accounting for everything else...past levels of baseline achievement of students, accounting for special education status, English language learner status and their attendance in the previous year…two students who are otherwise equivalent in terms of those characteristics but one is eligible for a free lunch and the other is not eligible, you see a difference in their predicted performance of about one point on the state test,” he said.
To put that in perspective, special education status which Florida does factor into its value-added model, accounts for about a one to four point variation in predicted performance, Isenberg says. And a one-point difference in a student’s predicted test score can translate into standard deviation of about one-fifth to one-half in a teacher’s value-added score, he says. That small deviation can mean a potentially significant difference in where a teacher ranks in effectiveness.
“It means that between those two teachers, one is going to score at the 50th percentile and the other is going to score somewhere between the 58th percentile and the 70th percentile,” he said.
It’s cause for concern said Florida Education Association President Andy Ford.
“If we’re going to be using the VAM as a way to judge teachers performance and their pay, we need to make sure that all things that impact a student’s life (are) taken into account, and poverty is definitely one of them,” he said.
The state’s poorest district is Gadsden County, near Tallahassee, where about 84 percent of students are eligible for free or reduced lunch.
About 67 percent of teachers in the district scored lower than average, according to state department data. Conversely, in St. John’s County where about 22 percent of students qualify for free or reduced lunch, just 35 percent of teachers rated below average.
In Duval County, where about 49 percent of children are considered poor, more than half of teachers ranked lower than expected.
Duval Schools Superintendent Nikolai Vitti said while he does believe there is a relationship between poverty and student achievement, he also believes that including it as a variable in VAM could lead to lowered expectations.
“Instead, I think you account for the reality and the negative impact that poverty has on student achievement by looking at growth over time,” he said. “The accountability system in Florida only looks at growth...from a year to year comparison.”
There are other high-poverty districts in the state where most teachers score above expectations. For example, in Miami-Dade County, 73 percent of children qualify for free or reduced lunch, and more than half of teachers scored above average in student growth.
Still, researchers say it would take a side-by-side analysis of the same students to understand if poverty has any real effect on the student growth measurement.
Sam Foerster, from the state Department of Education, says that an IMPACT analysis has already been conducted by American Institutes for Research.
“What they found in that IMPACT analysis is, looking at the scores, had successfully been de-coupled, that there was virtually no correlation between those two things and that gave us a fair degree of confidence,” he said.
Members of AIR were not available to comment in time for publication.
FEA President Andy Ford said the teachers group plans to conduct an independent analysis of their own to determine any potential impact.
“We are waiting for some analysis on whether teachers who work with students who live in poverty are handicapped the way we believe it is...We’re going to pursue every angle,” he said.
You can follow Rhema Thompson on Twitter @RhemaThompson.
