Bill Gates once said “I have been struck again and again by how important measurement is to improving the human condition.”
On April 7th, Mark Woolhouse Professor of Infectious Disease Epidemiology at the University of Edinburgh said ‘As things stand we could be out by a factor or 10 or even 100‘ [1].
Testing only those admitted to hospitals with symptoms provides a small, biased and unreliable sample which in normal circumstances would not be considered of any use at all. As a result, we have no idea about how many people have actually been infected or recovered or what its true mortality rate is.
Estimates of the number of cases currently range from the government’s official 191,000 (to 5th May) to a possible high of more than 23m – based on 800 cases for each of the 28.7k confirmed deaths. The difference between 191k and 23m is a factor of 120 i.e. close to Mark Woolhouse’s inaccuracy factor of 100.
The range between these two numbers is important because it includes two very different scenarios. Either Coronavirus is much less contagious than we thought, with a high mortality rate or it is far more infections than we thought, but with a much lower mortality rate.
In the first scenario there is a case for lockdown, because it looks like it’s keeping the number of cases under control. But in the second scenario it looks like 50% of the population have had the virus and the lockdown was too late. If we are at 50% we are closer to herd immunity levels which reduce the need for a strict lockdown.
If we can get to some accurate measurement we will be able to:
- Know how many people have actually had the disease in the past which provides us with a framework to calculate true infection, recovery and mortality rates, all of which are currently wrong.
- Understand more about the pattern of infection and by using reproduction rate (r rate) in scenario planning we would be able to establish a clearer picture about how this pandemic might evolve in the coming weeks.
- Estimate whether we are coming close to herd immunity – right now, this remains the only practical weapon against this disease until we have a vaccine – but we have no idea about how close we are to this stage. Understanding levels of herd immunity will be one of the keys to ending the lockdown.
- Understand the geographical distribution of the disease which may in turn help us identify whether different regions and communities are affected in different ways, thereby allowing us to target resources more accurately.
What should the government have done / do next time?
Accurate sampling was the big miss. A huge miss in fact. The government picked a sample that by its own definition is not representative. The sample contains glaring errors so bad as to render it effectively useless. But we could have got the sampling right. We could have gathered vital data through sampling the population for both reported symptoms and actual test results by:
- Building a nationally representative sample for symptom for self-reporting without testing. This disease has a number of known symptoms that can be reported online. Even if these don’t amount to a diagnosis they will help understand how the virus might be spreading. Mobile devices make collecting this data easy as has been shown by the COVID-19 project run by King’s College London.
- Creating a nationally representative sample for weekly testing. A UK nationally representative sample only needs to be about 1k individuals. If tests cost £100 each this this would cost around £100k per week.
- Running anonymous secondary Coronavirus tests on hospital blood samples collected for other reasons.
How do these data help end lockdown?
These simple forms of sample measurement would allow us to understand total numbers much more accurately, we would know how many people have had the infection and how many have survived. From this we could establish true recovery and mortality rates. Through continuous weekly sampling would be able to establish patterns of infection and evidence of any second or subsequent waves. Armed with this information we would be able to stop guessing about the end of lockdown and put in place policies based on accurate evidence.
Without accurate case measurement it is extremely difficult, if not impossible, to build an effective strategy to deal with the current situation and make plans for ending the lockdown. Without accurate measurement we are left guessing and we are building policy on guesswork.
Update 11th May 2020
Sir David Spiegelhalter, one of the worlds leading statisticians and professor of public understanding at the University of Cambridge has accused the government of an extraordinary failure to prioritise randomised testing – the type testing highlighted in this post [2].
1. Daily Telegraph Online ‘Why the Coronavirus tests you’ve never heard of hold the key to exit from Lockdown’ 7 April 2020.
2. The Times ‘Risk expert lambasts number 10 for failing to prioritise randomised testing’ 11 May 2020