A Policy Brief from January (Volume 15, Number 3) laid out the case that the School Performance Profiles (SPP) produced by the Pennsylvania Department of Education are misleading at best and useless at worst. The recently released 2015 SPP scores for high schools, generated by the same procedures and factor weighting as the SPP scores from school year 2013-2014, remind in starkest terms of just how far from giving a true picture of school academic achievement the SPPs are.
To illustrate the absurdity of the SPP as an indicator of academic performance, this Brief examines two Allegheny County High Schools at opposite ends of the quality of education spectrum. First, Wilkinsburg is used because there is no other school with as much room for academic improvement. Indeed, things are so bad the 7th through 12th grade students are scheduled to go to Westinghouse Academy next year—not a school known for academic achievement. Second, North Allegheny High is used as an example of what 11th graders at a good school are able to achieve.
How did the schools fare on the 2015 SPP? Wilkinsburg High posted an SPP score of 40.7. North Allegheny High received a score of 89.9. Looking at these scores one might conclude that North Allegheny High is performing at a level that is just over twice as good as Wilkinsburg High. But as it turns out the gap between the academic performances is actually a vast chasm.
To begin, it is important to note the horrendous shortcomings in the SPP measurement system. The fundamental problem is that comparison of actual levels of academic achievement has been rendered virtually impossible because of the weights assigned to the factors used to evaluate schools. Consider that scores on the math, English and Language Arts (ELA) and science tests, and Career and Technical Education program taken by 11th graders along with SAT/ACT College readiness account for 40 of the 100 points scoring system at North Allegheny and only 35 of the 95 points at Wilkinsburg which does not have not have Career and Technical Education. Meanwhile, indicators of meeting annual academic growth (with whatever validity that can be measured) are worth 40 points and indicators of closing academic gaps another ten, and other indicators add ten more points to get to 100 and 95 possible final score points for each school. There are up to seven bonus or extra credit points that can be awarded for the percentages of students scoring at the advanced level on the tests.
A major problem is that on the “academic growth” measure, the school is guaranteed 20 points because each of the three tests (math, ELA, and science) is arbitrarily assigned a minimum share of 50 percent for the number of students meeting annual growth expectations regardless of the actual percentage who do. Thus, every school is guaranteed at least 20 points out of a possible high of 40 points in this category towards its final profile score. And it gets worse. A total of ten points are allotted for four “other academic” factors at 2.5 points each: attendance rate, graduation rate, PSAT participation and advanced placement programs. Even a poorly performing school should get six or seven points out of the ten with practically no effort or achievement. Indeed, as remarkable as it sounds a school with 80 percent attendance will get two points in that category despite the fact that an 80 percent attendance rate means the average student misses school 36 days a year and is almost certainly not learning very much. Attendance rates that low should be viewed as a negative and certainly should not count as a positive.
Then too, a graduation rate of 70 percent will get the school 1.8 points even though a graduation rate that low should be a negative for any performance profile. Then there is the PSAT participation. Since everyone (according to the reporting) takes the preparatory SAT that is another 2.5 points toward the SPP score no matter how poorly the students do on the tests.
In total, a school basically starts out with 27 points if it just opens its doors and hands out diplomas. And with a little actual annual “growth” in language arts, it takes very little effort to boost the profile points to over 30.
Wilkinsburg manages to get 38.7 points on the SPP weighting scheme, which is then normalized by adjusting for the 95 possible points to give a total score of 40.7. Bear in mind that of the 38.7 points, 24.8 were in the “annual growth” component, 5.9 were in the attendance and graduation component, four came in the closing the gap section, and a mere four points for actual academic achievement. There were zero students proficient or higher in math; zero percent proficient or better in science and 26 percent in ELA. None reached the advanced level on any of the three tests. Because ELA has a multiplier weight of 15, it was able to account for the only four points earned for academic achievement. Math, science and college readiness are weighted at 7.5, 7.5 and 5.0 percent respectively. But with zero scores on these factors they contributed nothing to the academic achievement point count.
In sum, only four of the 38.7 earned points (and of 95 total available points) can be viewed as reflecting actual achievement levels. And even that is questionable since a zero percent scoring proficient on a test does not tell how far behind academically those scoring below basic (far below grade level) have fallen. Sadly, an average of 40 percent of Wilkinsburg 11th graders scored below basic in math and ELA.
Most of the remainder of the school’s profile points comes from artificial gifts including attendance (82 percent) and graduation (53 percent) that should, in a meaningful performance measurement, be viewed as a negative. All things considered, Wilkinsburg’s SPP at 40.7 is of no value whatsoever.
Meanwhile, North Allegheny High has one of the best SPP scores in the County and ranks second only to Upper St. Clair in the number of students scoring at the advanced levels on the three tests. 11th graders at the school did very well with 92.5 percent proficient or better in math, 96 percent proficient or better in ELA and 100 percent SAT college readiness. And even more impressive, 44 percent achieved the advanced level in math, 28.6 percent scored advanced in ELA and 46.5 percent were advanced in science. And remarkably, only one percent of North Allegheny test takers scored below basic. The school’s achievements rank among the best performance numbers in the state.
The problem? Only 40 of the 100 points in the basic SPP ranking process are available for academic achievement and only seven bonus points are available for all the superb achievement of the students scoring at the advanced level. Getting all seven bonus points would require 100 percent of students to be advanced. Thus, most of the very best schools will acquire only four or fewer points as extra credits.
As a result of the seemingly jerrybuilt SPP methodology, a school with an academic record so bad it is hardly believable with only four points earned for actual achievement can score 40.7 on the SPP while a school with almost all students proficient in math and reading and large percentages scoring advanced receives a score of only 89.9. Clearly, the SPP is providing no real service to the schools or the taxpayers. It arbitrarily rewards factors that do not measure positive achievement and under values strong academic achievement.
Only education bureaucrats and apologists for poor public school performance can believe this system is worth saving. A performance profile should measure performance, not ancillary factors.
The SPP program should be thoroughly revamped to place far more weight on actual achievement with say 90 points instead of just 40 and only 10 points for “academic growth” (if it can be measured meaningfully) instead of 50. There should be no points for attendance rate, graduation rate, or taking the PSAT. Those are essentially free gifts, unless minimum levels for receiving points are set. Perhaps 95 percent attendance or higher could get two points and anything below would get none.
Similarly, a graduation rate of 90 percent or higher could get two points with below 90 receiving no points. Merely taking the PSAT should receive no points. If students score above a certain minimum level, it might be a consideration. But if the PSAT simply verifies what the Keystone tests already reveal as being grossly inadequate progress, why should there be any credit for merely showing up to take the exam?
Finally, some thought should be given to including a formula that would reduce a school’s profile point count if the percentage of students scoring below basic on tests rises above some level, say 20 percent. Those scoring at that level are far below grade level and that should be viewed as a negative in the rating system.
Well intentioned the SPP in its current form might be, but useful it is not. If the Department of Education is unwilling to overhaul the SPP methodology, they should be instructed by the Governor and Legislature to drop it altogether. The PSSAs and Keystone test results speak for themselves. One can only imagine how much the design, implementation and yearly updating of this program has cost taxpayers.