Corona Daily 249: The Algorithmic Kingdom

United Kingdom has some of the world’s oldest universities. To apply to the old and the new Universities, British teenagers must score well at the A-level exams. In March, with the onset of the pandemic, schools were shut and exams cancelled.

Schools were asked to offer the predicted grades based on the student’s ability and past performance. But as we know; teachers are human. Some schools grade students leniently, some strictly. The purpose of most external exams, conducted on a state or a national level, is to create a level playing field for all students.

England has an exam watchdog called Ofqual (the office of qualifications and examinations regulations). At the end of March, Gavin Williamson, the education secretary asked Ofqual to ensure qualification standards are maintained, and grade distribution patterns are similar to those in the previous years.

*****

Ofqual was one of the few super-busy organisations during the pandemic. Over the next few months, it started developing a series of algorithms. To find that formula which will standardize the grades given by public and private schools, by indulgent and harsh-marking teachers. In June, schools began submitting teacher-assessed grades which were fed into the algorithm.

In July, Sir Jon Coles, former Director-General of the Department of Education, warned that the algorithm would be 75% accurate at best. A bit like the effectiveness of vaccines. Williamson had a video conference with Sir Coles, but decided to press ahead with the algorithm. External experts issued warnings that the Ofqual formula was as volatile as the stock market, and the outcomes were bound to be erratic.

*****

In mid-July, MPs on the education committee asked Ofqual to publish the algorithm for transparency and scrutiny. The exam watchdog refused, saying it would allow schools to calculate the awarded grades. In the last two weeks of July, Ofqual ran summer symposiums, but refused to divulge the algorithm. It admitted testing 12 different approaches, and finalizing one.

*****

On 13 August, the A-level results were published. They caused a mayhem greater than covid or Brexit.

40% of the students were downgraded, 36% by one grade, 3.5% by two grades and others by three grades. As a result, most of them would not be able to join the university of their choice. The proportion of private (elite) school students receiving A and A* was more than double that of students at the comprehensive (public) schools.

For your benefit, I give the algorithm here.

P_kj = (1-r_j)C_kj + r_j(C_kj + q_kj - p_kj)  

The ugly-looking formula considers factors like the history of the school, grades and distribution in the previous years, teacher-student ratio and availability of historical data. If a school has had low average performance in the past, and suddenly there is a bright student from that school, the formula makes sure that he/she is downgraded to be in line with the school’s past.

Students came out on the streets in thousands and protested vocally, with emphasis on the F verb, against the algorithm, against Boris Johnson and Williamson. Many felt their life and future career were ruined.

Boris Johnson, having nothing to do with education for a long time, said the exam results were “robust” and “dependable”. Ofqual said any statistical model, no matter how good, would produce anomalies.

Labour and LibDems supported students starting a legal action, with legal costs paid by the State.

Williamson imagined more than 200,000 students launching court cases against the government in pandemic times. He acknowledged the unfairness of the algorithm, pointed to the danger that a high-performing child in a low-performing school will not get deserving grades. (Something known for months). He apologized, England’s government made a U-turn, scrapped the algorithm and allowed the students to proceed with the grades the teachers had given them.

More than 400,000 students had their places confirmed in the universities. Human beings showed they were superior to algorithms.

Ravi