The Department of Education’s Misleading Academic Score System
In an apparent effort to put the best face possible on failing schools in the state, the Pennsylvania Department of Education (DoEd) has concocted a complex scoring scheme that is intended to blunt the impact of the reality of the poor performance at many of the state’s public schools. This is done by giving dubious credit for some measures and irrationally weighting some factors, all of which are aimed at making the academic performance at failing schools look better than it actually is.
A comparison of two schools in Allegheny County will illustrate the absurdity of the DoEd’s rating scheme. For the latest school year, 2013-2014, Lincoln Elementary in Mt. Lebanon received a DoEd academic score of 85.2. In the same year Kelly Elementary in Wilkinsburg achieved a score of 62.9. One would assume from these scores that even though Lincoln has a better score, Kelly is only 22 points (25 percent) below Lincoln. Not good but not terrible.
Here is the problem. The results on the PSSA (state assessment) tests reveal a vastly different picture from that portrayed by the DoEd’s Academic Scores. At Lincoln, 93.3 percent of students tested proficient or advanced in math: at Kelly 38.9 percent were proficient or higher. 72.5 percent of Lincoln students scored at the advanced level on the math exam, only 9.4 percent scored at the advanced level at Kelly. The reading results were similar; 92.8 percent either proficient or advanced with 61.2 percent at the advanced level at Lincoln. At Kelly, only 29.4 percent were either proficient or advanced with only 3.9 percent advanced. Finally, on what is called a crucial indicator for elementary education and a predictor of future accomplishment of students, 95.8 percent of Lincoln third graders were proficient or advanced in reading; only 34.7 percent of third graders at Kelly attained proficiency or better.
So how does this enormous difference in test results for the two schools get converted into the mere 22 point Academic Score difference reported by DoEd?
There are three very large elements in this distorted picture. First, actual test results on four PSSA tests, including science and writing plus the score for third grade reading, account for a maximum of only 40 points in the final total—7.5 points for the four PSSA tests for all grades together and 10 points for third grade reading. Kelly earned 17.3 points of these but only 8.6 of those from math and reading (out of a possible 25 points in those two testing categories), the foundational framework for other learning. Meanwhile, Lincoln earned 37.7 points, 23.6 of those from the 25 possible in math and reading. Note that earned points are calculated by multiplying the possible points per test by the percentage of students scoring proficient and advanced on the PSSA exams.
Second, 40 possible points toward the final score are based on the percentage of students “meeting annual academic growth expectations.” This seems to be a meaningless measure in terms of actual progress. Nonetheless, meeting expectations on each of the four PSSA tests is assigned a weight of 10 points so if 100 percent of students meet expectations on each test, the school gets 40 points.
Lincoln students received 29.3 points in the “meeting growth expectations” category, while Kelly students received 31.3 points. Interestingly, the lowest point count any school can get in this category is 20 because the percentage of students meeting academic growth expectations on each test is set by arbitrary rule so that it cannot fall below 50 percent.
A big part of the distortion in final score totals arises from the fact that actual test results, i.e., the percentage of students scoring proficient or above, are weighted at 7.5 points for each PSSA test while “meeting academic growth expectations” are weighted at 10 percent for each of the four tests. Thus, a school where 100 percent of students score at the proficient level on math will get 7.5 points toward the final Academic Score. If 30 percent are at the proficient level in math, the school will earn 2.25 points toward the Academic Score. This is repeated for the other three PSSA tests. A school could theoretically receive 30 points total from the results on the four tests. The third grade reading weight is set at 10 percent, so a school could get anywhere between zero and 40 points for actual achievement on PSSA exams.
At the same time, the “meeting growth expectations” component assigns weights of 10 percent for the four PSSA tests. However, there is an arbitrary minimum guarantee of 50 percent “meeting growth expectations” whether that level is actually reached or not. Therefore, all schools are guaranteed 20 points toward their Academic Score through this component even if not a single student showed any academic improvement. A school will earn 20 to 40 points in this category depending on how many students above 50 percent meet expectations.
In short, the category of “meeting growth expectations” is given higher weights per test than actual performance on the tests. Further complicating the issue is just how meaningful is the concept of “meeting growth expectations”? Say a student in third grade falls well short of proficiency in math and in fourth grade the same student is still not proficient. So how do we know whether there has been any academic growth? If year after year the percentage of students scoring proficient in math is around 35 percent in each grade, would it not be reasonable to assume that, for the most part, it must be the same group of students scoring proficient as they move through higher grades? Giving points for “meeting expectations” for students who never reach grade level competence is useless and completely misleading.
So how do we get 70 or 80 percent “meeting growth expectations” at Kelly Elementary? Or said another way, if children cannot do third grade math, why would one expect they would be equipped to do fourth grade math?
And even more astounding, how can the number of students at Lincoln “meeting growth expectations” in math be only 53 percent when 93.3 percent of the students score at the proficient or advanced level? Indeed, it is even more puzzling considering that 72.3 percent of the students score at the advanced level. This suggests that the procedure for measuring “meeting growth expectations” is irrational, to be kind.
The third distortion in the state’s academic rating scheme: A possible 10 points are awarded based on promotion rate (5) and attendance rate (5). With Kelly’s promotion rate at 97.37 percent, the school earned 4.9 points and with attendance at 92 percent, worth 4.6 points, for a total of 9.5 toward the final score. Lincoln received 9.8 points for the two measures.
With promotion rates in most schools at or near 100 percent, and attendance in elementary schools almost always 90 percent or better, all schools are essentially being given 9.5 to 10 points even if students learn almost nothing.
And how can promotion rates be close to 100 percent at Kelly when 65 to 70 percent of students are scoring below proficient in math and reading?
Adding to this 9.5 to 10 points the guaranteed 20 points from the “meeting expectations” measure gives schools a total of almost 30 points for merely existing. On the other hand, the scheme does give a modest possible four points for excellence—one point for each PSSA test times the percentage scoring advanced on each test. Lincoln earned 2.5 additional points on this measure. Kelly earned zero. One would have thought that having 70 percent of students scoring advanced in math would be worth more than 0.7 points toward the Academic Score. Sadly, that is not happening.
Without question there is a lot wrong with this rating scheme. It is misleading in the extreme because of its arbitrariness and its underweighting of actual, measured strong academic achievement. But probably most deleterious is the false sense of academic achievement it gives to schools that score a 70 when 30 points are basically freebies and makes them appear to be not too far behind schools with truly excellent academic performance and receive a score of 85 on the rating scheme.
And it also takes away from schools that are sterling performers when the “meeting expectations” measure is determined in a way that punishes high achieving schools.
In sum, the Department of Education Academic Scoring system for Pennsylvania schools is a travesty. One has to wonder how much it has cost in man hours, employee wages and consultant fees to set it up and to run it.