In these COVID-19 times, there is much talk about ‘testing’. As we get more and more pressure to reopen our economy, it is clear we have nowhere near the testing capacity or testing certainty that we really need in order to make rational decisions about who can work and where. How did we get to this point?

It may come as a surprise to hear that testing (whether software, biomedical, manufacturing…) is a complex subject involving both science and art. For starters there are many different kinds of tests even in a single domain of endeavor. (With COVID-19 for instance, there are tests for current COVID-19 infection, past exposure to COVID-19, relatedness of different COVID-19 samples, etc.) Each of these requires different (sometimes radically different) technology.

So how is a test designed? And how do we know whether, when, and why a test works (or not)? Designing a test is basically the same methodology CTOs and Chief Architects use every day for any process:

1. Identify the relevant technologies
2. Set forth a plan for how to connect the technologies
3. Check/verify that each technology works in the intended setting (POC)
4. String together the different technologies
5. Verify that the entire process/test works (POC)
6. Verify that you have the supply chain or needed throughput for all the steps (have legal begin drafting the necessary contracts with appropriate exigencies/penalties)
7. Do a complete error analysis for the full process/test
8. Develop more specific ways to identify/troubleshoot potential or actual reliability problems (test the process/test)

Sometimes you are lucky and the process/test you want is similar to another existing process/test. Sometimes you have to start from zero. This is where the art comes into play. In any case the importance of domain knowledge about the process/test cannot be emphasized strongly enough.

What the heck is step 8 all about? If you have already verified that the whole process/test works (step 5) and you can scale it up (step 6), why do you need any more testing or troubleshooting? Well, how do you even know that your process/test works correctly all the time or in other hands? You do this by running controls and intermediate tests of the process/test. This is precisely what is substantially missing in the COVID-19 situation. Just because a technology works once or works in your hands does not mean it will always work in the field. Maybe a reagant is not consistent, or maybe the servers you use are not always available. Maybe when someone else sets things up they use a different source of reagant or maybe they use servers that have different available ports. Maybe your process/test requires some kind of preprocessing or sample preparation and not everyone performs those steps the same way. Finally ALL processes/tests ALWAYS have both false positives and false negatives–blame it on the Uncertainty Principle or the Second Law of Thermodynamics. Anyway, a lot can/will go wrong, and you need a system which can identify the problem(s). MORE TESTING!!!

The starting point for properly analyzing any process/test is a thorough error analysis (step 7). Without going into huge detail, you identify the variables (possible errors) in each process/test step, measure or estimate those errors, and use the Chain Rule to propagate the effect of the errors through to the final result. The Chain Rule propagation directly provides a way to identify potential systematic (determinate) errors. Usually the Chain Rule propagation is also generalized to produce an RMS error propagation that is appropriate for random errors in all the process/test variables. In essence you are determining how sensitive your process/test is to different amounts and types of variation. This sensitivity analysis should help determine the overall sensitivity of the process/test and it should also identify those things most likely to go wrong with the process/test. These are the starting points for ‘testing the process/test’.

If there are many different kinds of tests in any given domain, then it follows that there are different kinds of ‘tests of a process/test’. Most of these are important. There are tests of data or sample preprocessing, tests of instrumentation or servers, throughput tests, consistency tests, and end-to-end tests.

The end-to-end test of a process/test is the most crucial because it verifies that the entire original process/test truly works reliably from start to finish. You do this by inputting data or samples which have known expected results and you verify that the process/test produces those expected results. You do this regularly (repeatedly) while the process/test is in use. If your process/test is supposed to produce results which vary in a defined numerical way for different inputs, then it is important for your end-to-end test to incorporate inputs which will produce results spanning the expected range. This is necessary because processes/tests often saturate, and this is the only way to correctly detect changes in saturation.

Consistency tests can also be quite important. There are different types of consistency, but a common situation is that you expect a certain range of outcomes when running your process/test on a typical population of inputs/samples. You want to know any time your process/test produces results that deviate from your expected outcomes. This may not mean there is anything wrong with your process/test itself if the end-to-end tests come out correct. But you likely still need to know if the population on which you are running your process/test is inconsistent.

I am glossing over quite a few things, but you get the point. A lot of testing needs to be done (continually) on any process/test. Without this you never really know what your process/test results mean.

In the case of COVID-19, the end-to-end test needs to start with several amounts (and none) of actual virus (perhaps carefully inactivated by alteration at the genomic level) mixed in mucus/sputum. The sample tubes should be handled by healthcare workers the same way as actual samples before they are sent on to the actual testing facility. The testing facility needs to also incorporate several samples with different known amounts (and none) of isolated or synthesized COVID-19 RNA. Positive and negative controls need to be included in each instrumentation series or multi-well plate. Multiple analyses of the same sample likely need to be done for both sensitivity and fidelity. Finally there are standard tests of the PCR instrumentation and any associated reagants (especially enzymes) that need to be carried out periodically. Only a few of these different tests seem to be done routinely, and there is NO end-to-end testing. Tests of machinery, reagants, etc. appear to vary significantly from site to site because our COVID-19 testing (United States) is so decentralized.

Many of the decisions not to run certain tests of the process/test are perhaps understandable when process/test materials are in short supply. Know however that failures of any real world process/test almost invariably occur because someone makes the decision that tests of the process/test are too costly or time consuming. We make these compromises at our peril!!!