Coronavirus Tests Are Being Fast-Tracked by the FDA, but It’s Unclear How Accurate They Are.

10 Apr

by Lydia DePillis and Caroline Chen

ProPublica is a nonprofit newsroom that investigates abuses of power. Sign up to receive our biggest stories as soon as they’re published.

Kendra Boroff believes she contracted the coronavirus on her 71st birthday, Feb. 20, when her family went out for a celebratory dinner, perhaps from their waiter, who was coughing into his elbow. Four days later, she developed a fever and a raging sore throat.

“You feel like you’re suffocating,” recalled Boroff, a real estate agent in Maineville, Ohio. “You cough and breathe with the top fourth or maybe less of your chest, because everything else is in a vise.”

Over the course of the next three weeks, as Boroff started getting chills and nausea, a series of doctors would suggest that it could be the common cold, bronchitis or pneumonia. She tested negative for the flu, and her chest X-rays showed signs of lung damage, including white patches called “ground-glass opacities” that are common in COVID-19 cases. By March 7, she was pretty sure it was COVID-19, but she couldn’t get a test until she arrived at the emergency room at the University of Cincinnati Health Center on March 19. She was so sick that she was admitted to an intensive care unit.

Nearly a week later, as Boroff’s condition was stabilizing, the test results came back: negative.

Boroff was flummoxed, but her physician was clear that she had the virus, no matter what her test said.

“‘This is my diagnosis,’” she recalled him saying. “‘There is no other explanation.’”

Tests turning up negative even when all signs point to COVID-19 has been a common experience in American hospitals over the past month, public health experts have told ProPublica. It’s unclear what proportion of these negative results are inaccurate — known as “false negatives” — and whether that’s due to some external factor, like bad sample collection, or because of an issue inherent in the tests’ design.

Neither the major test manufacturers, the U.S. Food and Drug Administration or the U.S. Centers for Disease Control and Prevention would say how common false negatives are. While the FDA requires test-makers to report any known instances of false negatives as a condition of granting them provisional approval, known as emergency use authorizations, no such reports are visible in a database the agency maintains for that purpose.

Without much data on how COVID-19 tests are performing in the real world, concerns are mounting that a lack of accurate testing will make it more difficult for America to relax social distancing, as the ability to track and trace new infections will be critical for any strategy to reopen the country.

Those warnings have reached Capitol Hill, where Texas Democratic Rep. Lloyd Doggett had heard from a doctor in his district about the accuracy of the tests. On Thursday, he and Rep. Rosa DeLauro of Connecticut sent a letter to the FDA demanding more data about the prevalence of false negatives in both the diagnostic tests currently in widespread use, as well as inaccuracies in the coming wave of rapid blood tests that detect immunity once the infection has passed.

“I’m very concerned about it,” Doggett told ProPublica. Too many false results, he worries, could lead to a new surge of infections when people go back to work or are allowed to gather in bars, sports arenas and restaurants. “They have to monitor this very closely to ensure that we’re not creating false expectations, and in the process ending up with an epidemic that is even worse than the one we have now.”

Lowering the Bar in an Emergency

In the early days of the pandemic, the FDA, which regulates diagnostic tests, was criticized for not moving quickly enough to make testing widely available. For much of February, the only available test was the CDC’s, which initially had flaws when it was sent out to public health labs. Only on Feb. 29 did the FDA announce a new policy that made it easier for private labs and academic medical centers to make tests available as well.

Since then, the ongoing need for even more testing capacity across the United States has pushed the agency to loosen its typical requirements for manufacturers to prove that their tests are accurate before allowing them onto the market.

Normally, to get FDA approval, diagnostic makers need to run trials to gather evidence on their tests’ performance, a process that can take months or even years. The agency is currently skipping a lot of those steps by issuing emergency use authorizations.

Manufacturers are now required to run their COVID-19 tests on a minimum of 30 positive samples and 30 negative samples. They must demonstrate to the agency that the test has at least a 95% sensitivity, meaning it must correctly identify at least 95% of the positive samples as having the coronavirus, and 100% specificity, meaning that it must accurately identify all the negative samples as not having the coronavirus.

But the manufacturers are demonstrating their diagnostics’ performance with what’s known as “contrived samples,” which are not taken from actual patients. A contrived sample is made by taking coronavirus RNA made in a lab and putting it into a medium that mimics nasal mucus.

“This is supposed to represent a swab specimen, but it’s not a positive sample from a real patient, and that does make a real difference,” said Benjamin Pinsky, medical director of the Clinical Virology Laboratory for Stanford Health Care.

It’s not clear if the concentrations of virus on the simulated samples are representative of the full range of material taken from patients’ bodies in the real world. Pinksy says that it’s reasonable for the FDA to allow the use of contrived samples, because it makes it much faster for a manufacturer to run validation studies, and the need for speed has been pressing.

“But then we need to have studies to compare these assays and see how they perform with real-world samples, and whether some are more or less sensitive and whether some are more or less specific,” Pinsky said. “We don’t know the answer to these questions at this point.”

To compensate for the lower standard up front, experts say the FDA should track data on accuracy to make sure the tests are performing as expected, but this is easier said than done.

“In diagnostic tests in particular, it’s very difficult to know if something is failing,” said Alberto Gutierrez, former director of the FDA’s Office of In Vitro Diagnostics and Radiological Health. “When are you getting more erroneous results than you should? It’s not always easy to figure out.”

Swiss manufacturer Roche, whose test was authorized by the FDA on March 12, told ProPublica it couldn’t give specific numbers about its test’s actual rate of false negatives and false positives, though it said studies have demonstrated its test could detect very low levels of the coronavirus.

“Clinical studies, which take months to run and would be part of a regular (nonemergency) test approval process, are needed to give us an exact percentage of false negatives and false positives,” Roche spokesman Mike Weist wrote in an email. “We will continue to work with the FDA on ongoing studies post-EUA that will allow us to potentially say more in the future.”

Abbott, which makes a rapid COVID-19 test, also said that “performance characteristics, including accuracy data, will continue to be collected in the field.”

Abbott and the testing firms LabCorp and Quest Diagnostics all told ProPublica that tests should be used by physicians along with other information to form a diagnosis.

Even Good Tests Can Give Inaccurate Results

Clinicians and researchers said that a number of factors could cause inaccurate results on COVID-19 tests, and many of them have nothing to do with the test’s design.

For starters, the timing of when a patient receives the test matters. “If you’re far out from the initial exposure, the more days you are after onset, viral load goes down,” explained Stanford’s Pinsky. Viral load refers to the amount of virus that is being emitted from an infected person’s cells, and if that drops too low, even a person who still has an active infection may test negative.

Another issue is where the virus is in a person’s body. As the disease progresses, scientists think the virus tends to move down into a patient’s lungs, so the window of time when a nose swab will return a positive result may be limited.

“One of the issues with this stupid virus is that if it’s down in your lungs, and we’re putting a swab up your nose, that’s not the best way to measure what’s in your lungs,” said Alex Greninger, assistant director of the clinical virology lab at the University of Washington Medical Center.

While it is possible to stick a scope down a patient’s airway to collect a sample from the bottom of the lungs, this is a much more complex procedure that requires sedating the patient. Technicians can ask a patient to cough up phlegm, known as sputum, but doing so substantially raises the risk of infecting health care workers. Even with a nasopharyngeal sample collected with a nose swab, one needs to collect it properly, which involves sticking the swab quite far up a patient’s nose.

A doctor administers a coronavirus swab test at a drive-through testing center at Lehman College in the Bronx.
(John Moore/Getty Images)

Daniel Brook, a freelance journalist and historian in New Orleans, says he thinks his test result may have been a false negative because he was incorrectly swabbed.

During Mardi Gras, he hung out with a friend who was visiting from Manhattan. A few days later, as he started to get night sweats and chills, Brook’s friend texted to say that he had tested positive for COVID-19. Brook has asthma, so when he started to have trouble breathing, he went to an urgent care center, which said it didn’t have enough tests to give him one.

Four days later, as Brook found himself even more winded going up stairs, he and his girlfriend, who also had symptoms, received a letter from an emergency room doctor that would get them a test at a drive-through center. They first were tested for the flu and then finally for COVID-19.

“This flu test was way the hell in there. It was almost like you ate too much hot pepper,” Brook said. “And then we had this COVID test, and it was barely in the nose at all, which may be one of the issues.” Nine days later, they received their results: Both were negative.

Brook was confused. He had been trying to tell all of the people he had been in contact with, like his barber, that they might have been exposed, and he shared the good news with many of them. But his doctor told him that clinically, he had all the symptoms of COVID-19, and that his diagnosis would not change based on his test result.

Even if the sample is taken correctly, mishandling of the swab can also invalidate the result. RNA is similar to DNA but due to chemical differences is a much more fragile material and degrades more readily. This coronavirus is an RNA virus, essentially a string of RNA encased in a membrane “envelope.”

Abbott, one of the test makers, said that it recommends that samples be kept for no more than eight hours at about 60 to 85 degrees Fahrenheit, or refrigerated for 72 hours. “People should make sure it is tested in a timely fashion,” Abbott said in its statement to ProPublica.

None of this bodes well for the numerous labs that have reported backlogs of tens of thousands of samples that are waiting to be tested.

A technician at an academic laboratory, who asked for anonymity because he is not authorized to speak on behalf of his university, described seeing basic mishandling of samples that is probably ruining dozens of patients’ test results.

“I don’t know why, but with COVID, we’ve just been awash with problems,” he told ProPublica. “Even simple things like caps not screwed on tightly — we’ll get a bag of samples, and two or three of them will be leaking, so you have this media completely soaking the inside of the bag. If one of those leaking samples is positive, you’ll have droplets all over the bag.”

Those samples, the technician said, often can’t be processed at all. His experience isn’t unique: In Alabama in late March, hundreds of samples were ruined in transit to a lab in Montgomery.

The Dangers of Inaccuracy

In the absence of data, physicians and public health officials are left to guess how many false negatives may be occuring — which could have serious consequences both for individuals and for combating the spread of the disease.

“You want to be right every time, because you miss somebody, and tell them that they’re negative, then you’re infecting people,” said Gutierrez, the former FDA official. “Let’s say you consider Amazon essential, and at the warehouse they’re testing people, even if they miss one or five people out of 100, that can be problematic.”

In addition, false negatives can make it more difficult to track spread of the virus, since those patients are not reported as confirmed cases and people who die without a positive test result won’t be counted in COVID-19 mortality statistics.

False positives also present problems. If you mistakenly think a patient has COVID-19, “then you have the potential to clog up the health care system and waste personal protective equipment and the time and effort of health care workers who think they are caring for individuals with COVID-19,” Stanford’s Pinsky said. “In addition, you’re producing a lot of anxiety for the patient.”

Pinsky says he hopes that real-world data will be gathered on the tests’ performance, especially as more and more come on the market: “If physicians have this information, they could move on to a different, better performing test and use that instead.”

Dr. Yukari Manabe, associate director of Global Health Research and Innovation at Johns Hopkins Medicine, estimates that 10% to 25% of test results are false negatives. That’s not based on any data, she cautions, since hard evidence isn’t available. But she has been noticing many patients in the Hopkins system being tested more than once, when the first result doesn’t match their clinical symptoms.

Like others, Manabe acknowledges that the FDA has needed to greenlight tests quickly in order to get them out into the public. But she laments that companies weren’t encouraged to develop diagnostics earlier, which might have allowed the agency to keep the bar for approval higher, and also churn out more tests sooner.

“If people had seen the writing on the wall back in December, someone should’ve paid these companies what they needed to develop these tests on platforms that could’ve been rapidly ramped up to millions of tests,” Manabe said.

Instead, a test shortage caused doctors to limit tests to only the sickest patients, at a time when the virus had probably moved out of the back of the nasal cavity and into their lungs. A larger supply would have allowed for testing more people as soon as they started showing symptoms. That would have resulted in a lower rate of false negatives, Manabe said, since nose swabs are more likely to detect the virus soon after it’s been contracted.

The Next Wave of Tests May Be Even Less Accurate

The questions swirling around the accuracy of the COVID-19 diagnostic tests are likely to persist as the next set of tests — antibody blood tests — start hitting the market. Already, the FDA has authorized the first of these tests, which search for molecules in a patient’s blood that can indicate if the immune system did battle with the coronavirus. Unlike the swab-based tests, which look for the viral RNA that indicate active infection, antibody tests are used to seek evidence of a past encounter with the virus.

Antibody tests are already seen as a critical tool in lifting lock-down measures, because they could potentially be used to figure out who has immunity to the coronavirus. In this case, false positives would be the greater concern, because it could be dangerous to tell someone that they have antibodies and are safe to go back to work when that is a false signal.

There are issues that need to be figured out before rushing to rely on these tests, Stanford’s Pinsky warned. What level of antibodies are needed to mean that someone is protected? And if you are protected, how long are you protected? The answers to these basic questions are still unknown, he said. This week, the World Health Organization put out guidance recommending against using antibody tests for clinical decision making.

The FDA, meanwhile, is lowering the bar even further. On March 16, it issued new guidance allowing manufacturers to distribute tests even before receiving emergency use authorization, for a “reasonable period of time” — about 15 days — after a diagnostic maker had validated the test internally and while preparing its request to the agency for an EUA.

Local governments are desperate enough for tests that they’ll buy them without assurances of accuracy at all. Chicago recently ordered 11,000 antibody tests made in South Korea that had not been reviewed by the FDA but are legal to distribute as long as they include several disclaimers including a recommendation that any negative result be confirmed with a diagnostic test.

“There’s no time really to put the effort into saying, ’Where’s the problem here?’” said Catherine Troisi, an epidemiologist at the University of Texas Health Science Center. “I’m not saying the test is bad. But what good is a test if you don’t know it’s giving you reliable results? We just don’t know.”


Coronavirus Tests Are Being Fast-Tracked by the FDA, but It’s Unclear How Accurate They Are.

10 Apr

by Lydia DePillis and Caroline Chen

ProPublica is a nonprofit newsroom that investigates abuses of power. Sign up to receive our biggest stories as soon as they’re published.

Kendra Boroff believes she contracted the coronavirus on her 71st birthday, Feb. 20, when her family went out for a celebratory dinner, perhaps from their waiter, who was coughing into his elbow. Four days later, she developed a fever and a raging sore throat.

“You feel like you’re suffocating,” recalled Boroff, a real estate agent in Maineville, Ohio. “You cough and breathe with the top fourth or maybe less of your chest, because everything else is in a vise.”

Over the course of the next three weeks, as Boroff started getting chills and nausea, a series of doctors would suggest that it could be the common cold, bronchitis or pneumonia. She tested negative for the flu, and her chest X-rays showed signs of lung damage, including white patches called “ground-glass opacities” that are common in COVID-19 cases. By March 7, she was pretty sure it was COVID-19, but she couldn’t get a test until she arrived at the emergency room at the University of Cincinnati Health Center on March 19. She was so sick that she was admitted to an intensive care unit.

Nearly a week later, as Boroff’s condition was stabilizing, the test results came back: negative.

Boroff was flummoxed, but her physician was clear that she had the virus, no matter what her test said.

“‘This is my diagnosis,’” she recalled him saying. “‘There is no other explanation.’”

Tests turning up negative even when all signs point to COVID-19 has been a common experience in American hospitals over the past month, public health experts have told ProPublica. It’s unclear what proportion of these negative results are inaccurate — known as “false negatives” — and whether that’s due to some external factor, like bad sample collection, or because of an issue inherent in the tests’ design.

Neither the major test manufacturers, the U.S. Food and Drug Administration or the U.S. Centers for Disease Control and Prevention would say how common false negatives are. While the FDA requires test-makers to report any known instances of false negatives as a condition of granting them provisional approval, known as emergency use authorizations, no such reports are visible in a database the agency maintains for that purpose.

Without much data on how COVID-19 tests are performing in the real world, concerns are mounting that a lack of accurate testing will make it more difficult for America to relax social distancing, as the ability to track and trace new infections will be critical for any strategy to reopen the country.

Those warnings have reached Capitol Hill, where Texas Democratic Rep. Lloyd Doggett had heard from a doctor in his district about the accuracy of the tests. On Thursday, he and Rep. Rosa DeLauro of Connecticut sent a letter to the FDA demanding more data about the prevalence of false negatives in both the diagnostic tests currently in widespread use, as well as inaccuracies in the coming wave of rapid blood tests that detect immunity once the infection has passed.

“I’m very concerned about it,” Doggett told ProPublica. Too many false results, he worries, could lead to a new surge of infections when people go back to work or are allowed to gather in bars, sports arenas and restaurants. “They have to monitor this very closely to ensure that we’re not creating false expectations, and in the process ending up with an epidemic that is even worse than the one we have now.”

Lowering the Bar in an Emergency

In the early days of the pandemic, the FDA, which regulates diagnostic tests, was criticized for not moving quickly enough to make testing widely available. For much of February, the only available test was the CDC’s, which initially had flaws when it was sent out to public health labs. Only on Feb. 29 did the FDA announce a new policy that made it easier for private labs and academic medical centers to make tests available as well.

Since then, the ongoing need for even more testing capacity across the United States has pushed the agency to loosen its typical requirements for manufacturers to prove that their tests are accurate before allowing them onto the market.

Normally, to get FDA approval, diagnostic makers need to run trials to gather evidence on their tests’ performance, a process that can take months or even years. The agency is currently skipping a lot of those steps by issuing emergency use authorizations.

Manufacturers are now required to run their COVID-19 tests on a minimum of 30 positive samples and 30 negative samples. They must demonstrate to the agency that the test has at least a 95% sensitivity, meaning it must correctly identify at least 95% of the positive samples as having the coronavirus, and 100% specificity, meaning that it must accurately identify all the negative samples as not having the coronavirus.

But the manufacturers are demonstrating their diagnostics’ performance with what’s known as “contrived samples,” which are not taken from actual patients. A contrived sample is made by taking coronavirus RNA made in a lab and putting it into a medium that mimics nasal mucus.

“This is supposed to represent a swab specimen, but it’s not a positive sample from a real patient, and that does make a real difference,” said Benjamin Pinsky, medical director of the Clinical Virology Laboratory for Stanford Health Care.

It’s not clear if the concentrations of virus on the simulated samples are representative of the full range of material taken from patients’ bodies in the real world. Pinksy says that it’s reasonable for the FDA to allow the use of contrived samples, because it makes it much faster for a manufacturer to run validation studies, and the need for speed has been pressing.

“But then we need to have studies to compare these assays and see how they perform with real-world samples, and whether some are more or less sensitive and whether some are more or less specific,” Pinsky said. “We don’t know the answer to these questions at this point.”

To compensate for the lower standard up front, experts say the FDA should track data on accuracy to make sure the tests are performing as expected, but this is easier said than done.

“In diagnostic tests in particular, it’s very difficult to know if something is failing,” said Alberto Gutierrez, former director of the FDA’s Office of In Vitro Diagnostics and Radiological Health. “When are you getting more erroneous results than you should? It’s not always easy to figure out.”

Swiss manufacturer Roche, whose test was authorized by the FDA on March 12, told ProPublica it couldn’t give specific numbers about its test’s actual rate of false negatives and false positives, though it said studies have demonstrated its test could detect very low levels of the coronavirus.

“Clinical studies, which take months to run and would be part of a regular (nonemergency) test approval process, are needed to give us an exact percentage of false negatives and false positives,” Roche spokesman Mike Weist wrote in an email. “We will continue to work with the FDA on ongoing studies post-EUA that will allow us to potentially say more in the future.”

Abbott, which makes a rapid COVID-19 test, also said that “performance characteristics, including accuracy data, will continue to be collected in the field.”

Abbott and the testing firms LabCorp and Quest Diagnostics all told ProPublica that tests should be used by physicians along with other information to form a diagnosis.

Even Good Tests Can Give Inaccurate Results

Clinicians and researchers said that a number of factors could cause inaccurate results on COVID-19 tests, and many of them have nothing to do with the test’s design.

For starters, the timing of when a patient receives the test matters. “If you’re far out from the initial exposure, the more days you are after onset, viral load goes down,” explained Stanford’s Pinsky. Viral load refers to the amount of virus that is being emitted from an infected person’s cells, and if that drops too low, even a person who still has an active infection may test negative.

Another issue is where the virus is in a person’s body. As the disease progresses, scientists think the virus tends to move down into a patient’s lungs, so the window of time when a nose swab will return a positive result may be limited.

“One of the issues with this stupid virus is that if it’s down in your lungs, and we’re putting a swab up your nose, that’s not the best way to measure what’s in your lungs,” said Alex Greninger, assistant director of the clinical virology lab at the University of Washington Medical Center.

While it is possible to stick a scope down a patient’s airway to collect a sample from the bottom of the lungs, this is a much more complex procedure that requires sedating the patient. Technicians can ask a patient to cough up phlegm, known as sputum, but doing so substantially raises the risk of infecting health care workers. Even with a nasopharyngeal sample collected with a nose swab, one needs to collect it properly, which involves sticking the swab quite far up a patient’s nose.

A doctor administers a coronavirus swab test at a drive-through testing center at Lehman College in the Bronx.
(John Moore/Getty Images)

Daniel Brook, a freelance journalist and historian in New Orleans, says he thinks his test result may have been a false negative because he was incorrectly swabbed.

During Mardi Gras, he hung out with a friend who was visiting from Manhattan. A few days later, as he started to get night sweats and chills, Brook’s friend texted to say that he had tested positive for COVID-19. Brook has asthma, so when he started to have trouble breathing, he went to an urgent care center, which said it didn’t have enough tests to give him one.

Four days later, as Brook found himself even more winded going up stairs, he and his girlfriend, who also had symptoms, received a letter from an emergency room doctor that would get them a test at a drive-through center. They first were tested for the flu and then finally for COVID-19.

“This flu test was way the hell in there. It was almost like you ate too much hot pepper,” Brook said. “And then we had this COVID test, and it was barely in the nose at all, which may be one of the issues.” Nine days later, they received their results: Both were negative.

Brook was confused. He had been trying to tell all of the people he had been in contact with, like his barber, that they might have been exposed, and he shared the good news with many of them. But his doctor told him that clinically, he had all the symptoms of COVID-19, and that his diagnosis would not change based on his test result.

Even if the sample is taken correctly, mishandling of the swab can also invalidate the result. RNA is similar to DNA but due to chemical differences is a much more fragile material and degrades more readily. This coronavirus is an RNA virus, essentially a string of RNA encased in a membrane “envelope.”

Abbott, one of the test makers, said that it recommends that samples be kept for no more than eight hours at about 60 to 85 degrees Fahrenheit, or refrigerated for 72 hours. “People should make sure it is tested in a timely fashion,” Abbott said in its statement to ProPublica.

None of this bodes well for the numerous labs that have reported backlogs of tens of thousands of samples that are waiting to be tested.

A technician at an academic laboratory, who asked for anonymity because he is not authorized to speak on behalf of his university, described seeing basic mishandling of samples that is probably ruining dozens of patients’ test results.

“I don’t know why, but with COVID, we’ve just been awash with problems,” he told ProPublica. “Even simple things like caps not screwed on tightly — we’ll get a bag of samples, and two or three of them will be leaking, so you have this media completely soaking the inside of the bag. If one of those leaking samples is positive, you’ll have droplets all over the bag.”

Those samples, the technician said, often can’t be processed at all. His experience isn’t unique: In Alabama in late March, hundreds of samples were ruined in transit to a lab in Montgomery.

The Dangers of Inaccuracy

In the absence of data, physicians and public health officials are left to guess how many false negatives may be occuring — which could have serious consequences both for individuals and for combating the spread of the disease.

“You want to be right every time, because you miss somebody, and tell them that they’re negative, then you’re infecting people,” said Gutierrez, the former FDA official. “Let’s say you consider Amazon essential, and at the warehouse they’re testing people, even if they miss one or five people out of 100, that can be problematic.”

In addition, false negatives can make it more difficult to track spread of the virus, since those patients are not reported as confirmed cases and people who die without a positive test result won’t be counted in COVID-19 mortality statistics.

False positives also present problems. If you mistakenly think a patient has COVID-19, “then you have the potential to clog up the health care system and waste personal protective equipment and the time and effort of health care workers who think they are caring for individuals with COVID-19,” Stanford’s Pinsky said. “In addition, you’re producing a lot of anxiety for the patient.”

Pinsky says he hopes that real-world data will be gathered on the tests’ performance, especially as more and more come on the market: “If physicians have this information, they could move on to a different, better performing test and use that instead.”

Dr. Yukari Manabe, associate director of Global Health Research and Innovation at Johns Hopkins Medicine, estimates that 10% to 25% of test results are false negatives. That’s not based on any data, she cautions, since hard evidence isn’t available. But she has been noticing many patients in the Hopkins system being tested more than once, when the first result doesn’t match their clinical symptoms.

Like others, Manabe acknowledges that the FDA has needed to greenlight tests quickly in order to get them out into the public. But she laments that companies weren’t encouraged to develop diagnostics earlier, which might have allowed the agency to keep the bar for approval higher, and also churn out more tests sooner.

“If people had seen the writing on the wall back in December, someone should’ve paid these companies what they needed to develop these tests on platforms that could’ve been rapidly ramped up to millions of tests,” Manabe said.

Instead, a test shortage caused doctors to limit tests to only the sickest patients, at a time when the virus had probably moved out of the back of the nasal cavity and into their lungs. A larger supply would have allowed for testing more people as soon as they started showing symptoms. That would have resulted in a lower rate of false negatives, Manabe said, since nose swabs are more likely to detect the virus soon after it’s been contracted.

The Next Wave of Tests May Be Even Less Accurate

The questions swirling around the accuracy of the COVID-19 diagnostic tests are likely to persist as the next set of tests — antibody blood tests — start hitting the market. Already, the FDA has authorized the first of these tests, which search for molecules in a patient’s blood that can indicate if the immune system did battle with the coronavirus. Unlike the swab-based tests, which look for the viral RNA that indicate active infection, antibody tests are used to seek evidence of a past encounter with the virus.

Antibody tests are already seen as a critical tool in lifting lock-down measures, because they could potentially be used to figure out who has immunity to the coronavirus. In this case, false positives would be the greater concern, because it could be dangerous to tell someone that they have antibodies and are safe to go back to work when that is a false signal.

There are issues that need to be figured out before rushing to rely on these tests, Stanford’s Pinsky warned. What level of antibodies are needed to mean that someone is protected? And if you are protected, how long are you protected? The answers to these basic questions are still unknown, he said. This week, the World Health Organization put out guidance recommending against using antibody tests for clinical decision making.

The FDA, meanwhile, is lowering the bar even further. On March 16, it issued new guidance allowing manufacturers to distribute tests even before receiving emergency use authorization, for a “reasonable period of time” — about 15 days — after a diagnostic maker had validated the test internally and while preparing its request to the agency for an EUA.

Local governments are desperate enough for tests that they’ll buy them without assurances of accuracy at all. Chicago recently ordered 11,000 antibody tests made in South Korea that had not been reviewed by the FDA but are legal to distribute as long as they include several disclaimers including a recommendation that any negative result be confirmed with a diagnostic test.

“There’s no time really to put the effort into saying, ’Where’s the problem here?’” said Catherine Troisi, an epidemiologist at the University of Texas Health Science Center. “I’m not saying the test is bad. But what good is a test if you don’t know it’s giving you reliable results? We just don’t know.”


He Spent $500,000 to Buy Coronavirus Tests. Health Officials Say They’re Unreliable.

10 Apr

by Jeremy Schwartz

ProPublica is a nonprofit newsroom that investigates abuses of power. Sign up to receive our biggest stories as soon as they’re published.

This article is co-published with The Texas Tribune, as part of an investigative partnership

When thousands of rapid COVID-19 tests were delivered to the South Texas city of Laredo late last month, it looked as if a visiting dignitary had arrived. With lights flashing and sirens blaring, Webb County sheriff’s deputies escorted a red tractor-trailer carrying the tests to a local emergency room, whose owner had purchased them from a Chinese manufacturer.

Longtime Laredo Congressman Henry Cuellar, who helped facilitate the arrival of the tests, smiled broadly as he carried boxes of them inside the clinic. Believing the tests would detect an active infection, Laredo leaders hustled to set up a drive-through testing site to welcome anxious residents the following morning.

But the promise of the 20,000 tests would soon become a bitter example of what can go wrong when local governments and private medical firms try to buy supplies on the open market from unknown manufacturers, as policies from the U.S. Food and Drug Administration shift and anxiety increases over a lack of test kits from official sources.

As they tried to validate the tests to ensure their accuracy, city health workers in Laredo quickly determined that they were unreliable and unusable. And even if they had passed the city’s testing, it’s unclear how helpful they would have been for the city at that early point in its battle against the coronavirus. They were antibody tests, which seek evidence that someone’s immune response has encountered the virus, not diagnostic tests that detect active infection.

Within a week, investigators from the U.S. Department of Homeland Security would seize the 20,000 tests, as Laredo police and federal authorities tried to determine the validity of the tests’ FDA status.

“We’re very disappointed, because we thought we had secured a supply,” Laredo Mayor Pete Saenz said. “It set us back, but we can’t give up. Plan B is acquiring (tests) wherever we can find them.”

Laredo’s attempt to secure the testing kits highlights what is happening not only in Texas but across the country, as a stubborn shortage of COVID-19 tests has led private sector and government leaders to take risks that they normally would not.

But as local governments and private laboratories across the nation similarly seek out stores of test kits, critics say the FDA isn’t keeping up with a growing number of international manufacturers peddling kits in online emporiums, sometimes claiming FDA authorizations that don’t exist.

“This is one of those things where, can you imagine … a city government trying to figure this out? This shouldn’t be the process,” said Scott Becker, chief executive officer with the Association of Public Health Laboratories. “The concern is that we’re starting to see tests that have not been vetted, where we don’t know their quality, and they are already coming into the country.”

Cuellar refused to answer questions this week from ProPublica and The Texas Tribune about his role in the procurement of the tests or their FDA status, issuing a statement thanking federal authorities for seizing the tests.

Today, Laredo, a city of 260,000 on the banks of the Rio Grande, faces a much more dire coronavirus situation than it did a week ago when the tests arrived. As the kits sat unused, six Laredoans died of COVID-19 over seven days, giving the city one of the highest per capita death rates in the state.

“We haven’t tested enough,” said Dr. Hector Gonzalez, Laredo’s health director. “The number of residents tested is still too small to predict any model or scenario with any confidence.”

A Rush to Buy COVID-19 Tests. Confusion Ensues.

The test saga in Laredo began last month with a chance encounter in the waiting room of Cuellar, a Democrat serving his eighth term representing the area in Congress. Robert Castañeda, who last year built what he says is Laredo’s most modern free-standing emergency room, Clear Choice ER, had just left a meeting with Cuellar to discuss the lack of protective equipment and testing in the border city.

In the lobby, as he was leaving, Castañeda ran into another group waiting to talk to the congressman about telemedicine. According to Castañeda, a member of that group, whom he declined to identify, said he knew how to score a huge quantity of 15-minute COVID-19 tests from a Chinese company called Anhui DeepBlue Medical Technology.

The only catch: Castañeda would have to buy 20,000 units, the minimum amount the company’s American distributor would sell.

“That was one of the causes of concern,” Castañeda told ProPublica and The Texas Tribune. “It was about a half million dollar buy.”

In Laredo, city leaders were desperate for additional tests and agreed to buy 2,500 of the kits (Clear Choice would donate another 2,500). Like their neighboring counties along the Texas border, they were scrambling to find enough tests for a population with high levels of preexisting health conditions that doctors warned would leave residents more susceptible to complications from the coronavirus. As a state, Texas ranked low in per capita testing units, leaving many local governments hunting for tests anywhere they could find them.

“We felt like we were all competing with each other for whatever was out there,” Saenz said.

And so with the congressman facilitating the deal, Castañeda paid $500,000 up front and bought the tests. Cuellar would later write on Facebook that he worked with the FDA and U.S. Customs and Border Protection to get the kits “approved and sent down to Laredo.”

Despite Warning Signs, the Tests Are Bought

But even before the purchase, there were warning signs.

First, Anhui DeepBlue, a 10-year-old firm specializing in diagnostic devices, only had so-called antibody tests, not the diagnostic tests that are used to detect an active infection. Antibody tests are meant to detect molecules in the blood that can indicate an immunity after an infection has passed, and the FDA has warned against using them to diagnose active infection.

It also wasn’t clear to Castañeda if the Chinese tests had received authorization from the FDA. The company’s website indicated the tests were registered with the U.S. government. Castañeda said he contacted the FDA and a representative told him the tests weren’t currently approved, but that the agency was authorizing new manufacturers and relaxing regulations at a rapid rate.

Over the past two months, the FDA has been loosening regulations in hopes of spurring more supply. In February, it allowed manufacturers to apply for emergency authorizations and, in a March 16 policy update, distribute test kits even without the authorization if they could show proof of the test’s validation.

On Monday, the FDA reported that some firms “are falsely claiming that their (antibody) tests are FDA approved or authorized, or falsely claiming that they can diagnose COVID-19.” The agency vowed to take action against firms that are falsely marketing tests that aren’t accurate or reliable.

On Thursday night, the FDA provided ProPublica and The Texas Tribune with an updated list of companies that had received an emergency authorization or were listed among the companies operating under the FDA’s March 16 policy. Anhui DeepBlue was not on those lists.

Also late Thursday, Anhui DeepBlue sent ProPublica and The Texas Tribune documentation that the FDA had received its submission for an emergency authorization, the same documentation it previously sent Castañeda, and said it continues to submit materials for FDA review. The company also noted that its antibody tests should not be used as the sole basis to diagnose patients in the early stage of infection, but rather be used as a supplemental test, a warning statement in line with FDA guidance.

Castañeda said trying to decipher Anhui DeepBlue’s claims about FDA registration, certification and, ultimately, authorization was vexing, but in the end he agreed to buy the antibody tests, figuring they were better than nothing:

“It was something that would allow us to do an initial screening and calm the fears of people who wanted to get tested but couldn’t because of the shortage. It was not the be-all. It was just an additional tool.”

Unreliable Tests. Law Enforcement Steps In.

The day after the tests arrived, city health workers began expressing alarm over what they were seeing as they tried to validate their accuracy, and they left the drive-thru site shuttered.

“Until we finish our quality controls, we can’t provide them,” Gonzalez, the health director, said during the city’s daily web-based press briefing.

The following day, the fifth in a row on which city leaders would announce a COVID-19 death, Gonzalez was even more pessimistic. “We had been told by the manufacturer of the test and the literature that it was 93% or 97% effective, which is great,” he said. “But doing our controls, we didn’t like the results that we were getting because they weren’t anywhere near” those numbers.

Anhui DeepBlue told ProPublica and The Texas Tribune that Laredo health officials should have reached out to them before declaring the kits unusable and said it is seeking a detailed report of the city’s validation test.

Gonzalez said the health department tested individuals known to be positive or negative through previous swab testing, and that the Anhui DeepBlue tests only had about 20% accuracy, “too low of a rate to use.”

The health director added that he believed the kits could have been used to screen for active infection, since they could detect a type of antibody called IgM, which is one of the immune system’s first responders during an infection . According to the FDA, when igM antibodies are found in the blood, they “can indicate” an active or recent infection.

“It would have allowed us to test large numbers of persons and give us an indicator of the level of possible infection,” he said.

On April 2, Laredo police began asking questions about the FDA documentation the company’s distributor had presented, according to a press release announcing the city would not use the tests.

The same day, Homeland Security Investigations, an arm of U.S. Immigration and Customs Enforcement, seized the kits and launched an investigation of its own.

Then, in yet another example of the confusing rules around international test manufacturers, Castañeda said that Homeland Security on Wednesday agreed to return his tests. ICE spokeswoman Nina Pruneda declined to explain why.

What happened in Laredo isn’t the first time the company has run into controversy. Last month, federal agents seized 100 of the company’s COVID-19 tests, which were on their way to a Portland, Oregon, man for home use. It’s illegal to send testing kits to a private individual and not a health care provider, according to FDA rules. But in an example of how murky FDA regulations can be, an FDA consumer safety officer told The Oregonian that he couldn’t say at that time whether the company was authorized to distribute its tests to clinical labs.

Laredo health administrators are not the first to complain about unreliable testing kits from China. European nations have complained for weeks about receiving substandard tests from that country.

Disputed purchases include 1.2 million faulty antibody tests in Slovakia, a batch of 300,000 quick tests in the Czech Republic that local authorities said were about one-third defective, and tests sent to Turkey that officials said were just 35% accurate. Spain returned about 9,000 faulty rapid tests that it purchased from an unauthorized Chinese firm last month.

$500,000 Worth of Tests, in Limbo

When Homeland Security releases the tests, Castañeda said he is not sure how they will be used, and there aren’t firm guidelines on how antibody tests should be deployed. Some have argued the tests should be used to clear first responders and other front-line workers to return to their jobs; others believe they will be part of a massive testing effort nationally to identify those with immunity.

“We’ll try to use them wisely to make them available to the community as soon as possible,” he said.

Saenz, the Laredo mayor, said he is hopeful that Texas Gov. Greg Abbott, who has come under criticism for the state’s slow pace of test procurement, will follow through on a promise to ensure a reliable flow of authorized testing kits to cities and counties. On Wednesday, Abbott announced that Walgreens intends to begin offering drive-thru testing, potentially serving thousands of Texans per day.

“We need a secure, proven supply chain where we don’t have to question the validity of the product,” Saenz said. “We know there are people trying to take advantage of a lot of entities.”

Caroline Chen contributed reporting.


He Spent $500,000 to Buy Coronavirus Tests. Health Officials Say They’re Unreliable.

10 Apr

by Jeremy Schwartz

ProPublica is a nonprofit newsroom that investigates abuses of power. Sign up to receive our biggest stories as soon as they’re published.

This article is co-published with The Texas Tribune, as part of an investigative partnership

When thousands of rapid COVID-19 tests were delivered to the South Texas city of Laredo late last month, it looked as if a visiting dignitary had arrived. With lights flashing and sirens blaring, Webb County sheriff’s deputies escorted a red tractor-trailer carrying the tests to a local emergency room, whose owner had purchased them from a Chinese manufacturer.

Longtime Laredo Congressman Henry Cuellar, who helped facilitate the arrival of the tests, smiled broadly as he carried boxes of them inside the clinic. Believing the tests would detect an active infection, Laredo leaders hustled to set up a drive-through testing site to welcome anxious residents the following morning.

But the promise of the 20,000 tests would soon become a bitter example of what can go wrong when local governments and private medical firms try to buy supplies on the open market from unknown manufacturers, as policies from the U.S. Food and Drug Administration shift and anxiety increases over a lack of test kits from official sources.

As they tried to validate the tests to ensure their accuracy, city health workers in Laredo quickly determined that they were unreliable and unusable. And even if they had passed the city’s testing, it’s unclear how helpful they would have been for the city at that early point in its battle against the coronavirus. They were antibody tests, which seek evidence that someone’s immune response has encountered the virus, not diagnostic tests that detect active infection.

Within a week, investigators from the U.S. Department of Homeland Security would seize the 20,000 tests, as Laredo police and federal authorities tried to determine the validity of the tests’ FDA status.

“We’re very disappointed, because we thought we had secured a supply,” Laredo Mayor Pete Saenz said. “It set us back, but we can’t give up. Plan B is acquiring (tests) wherever we can find them.”

Laredo’s attempt to secure the testing kits highlights what is happening not only in Texas but across the country, as a stubborn shortage of COVID-19 tests has led private sector and government leaders to take risks that they normally would not.

But as local governments and private laboratories across the nation similarly seek out stores of test kits, critics say the FDA isn’t keeping up with a growing number of international manufacturers peddling kits in online emporiums, sometimes claiming FDA authorizations that don’t exist.

“This is one of those things where, can you imagine … a city government trying to figure this out? This shouldn’t be the process,” said Scott Becker, chief executive officer with the Association of Public Health Laboratories. “The concern is that we’re starting to see tests that have not been vetted, where we don’t know their quality, and they are already coming into the country.”

Cuellar refused to answer questions this week from ProPublica and The Texas Tribune about his role in the procurement of the tests or their FDA status, issuing a statement thanking federal authorities for seizing the tests.

Today, Laredo, a city of 260,000 on the banks of the Rio Grande, faces a much more dire coronavirus situation than it did a week ago when the tests arrived. As the kits sat unused, six Laredoans died of COVID-19 over seven days, giving the city one of the highest per capita death rates in the state.

“We haven’t tested enough,” said Dr. Hector Gonzalez, Laredo’s health director. “The number of residents tested is still too small to predict any model or scenario with any confidence.”

A Rush to Buy COVID-19 Tests. Confusion Ensues.

The test saga in Laredo began last month with a chance encounter in the waiting room of Cuellar, a Democrat serving his eighth term representing the area in Congress. Robert Castañeda, who last year built what he says is Laredo’s most modern free-standing emergency room, Clear Choice ER, had just left a meeting with Cuellar to discuss the lack of protective equipment and testing in the border city.

In the lobby, as he was leaving, Castañeda ran into another group waiting to talk to the congressman about telemedicine. According to Castañeda, a member of that group, whom he declined to identify, said he knew how to score a huge quantity of 15-minute COVID-19 tests from a Chinese company called Anhui DeepBlue Medical Technology.

The only catch: Castañeda would have to buy 20,000 units, the minimum amount the company’s American distributor would sell.

“That was one of the causes of concern,” Castañeda told ProPublica and The Texas Tribune. “It was about a half million dollar buy.”

In Laredo, city leaders were desperate for additional tests and agreed to buy 2,500 of the kits (Clear Choice would donate another 2,500). Like their neighboring counties along the Texas border, they were scrambling to find enough tests for a population with high levels of preexisting health conditions that doctors warned would leave residents more susceptible to complications from the coronavirus. As a state, Texas ranked low in per capita testing units, leaving many local governments hunting for tests anywhere they could find them.

“We felt like we were all competing with each other for whatever was out there,” Saenz said.

And so with the congressman facilitating the deal, Castañeda paid $500,000 up front and bought the tests. Cuellar would later write on Facebook that he worked with the FDA and U.S. Customs and Border Protection to get the kits “approved and sent down to Laredo.”

Despite Warning Signs, the Tests Are Bought

But even before the purchase, there were warning signs.

First, Anhui DeepBlue, a 10-year-old firm specializing in diagnostic devices, only had so-called antibody tests, not the diagnostic tests that are used to detect an active infection. Antibody tests are meant to detect molecules in the blood that can indicate an immunity after an infection has passed, and the FDA has warned against using them to diagnose active infection.

It also wasn’t clear to Castañeda if the Chinese tests had received authorization from the FDA. The company’s website indicated the tests were registered with the U.S. government. Castañeda said he contacted the FDA and a representative told him the tests weren’t currently approved, but that the agency was authorizing new manufacturers and relaxing regulations at a rapid rate.

Over the past two months, the FDA has been loosening regulations in hopes of spurring more supply. In February, it allowed manufacturers to apply for emergency authorizations and, in a March 16 policy update, distribute test kits even without the authorization if they could show proof of the test’s validation.

On Monday, the FDA reported that some firms “are falsely claiming that their (antibody) tests are FDA approved or authorized, or falsely claiming that they can diagnose COVID-19.” The agency vowed to take action against firms that are falsely marketing tests that aren’t accurate or reliable.

On Thursday night, the FDA provided ProPublica and The Texas Tribune with an updated list of companies that had received an emergency authorization or were listed among the companies operating under the FDA’s March 16 policy. Anhui DeepBlue was not on those lists.

Also late Thursday, Anhui DeepBlue sent ProPublica and The Texas Tribune documentation that the FDA had received its submission for an emergency authorization, the same documentation it previously sent Castañeda, and said it continues to submit materials for FDA review. The company also noted that its antibody tests should not be used as the sole basis to diagnose patients in the early stage of infection, but rather be used as a supplemental test, a warning statement in line with FDA guidance.

Castañeda said trying to decipher Anhui DeepBlue’s claims about FDA registration, certification and, ultimately, authorization was vexing, but in the end he agreed to buy the antibody tests, figuring they were better than nothing:

“It was something that would allow us to do an initial screening and calm the fears of people who wanted to get tested but couldn’t because of the shortage. It was not the be-all. It was just an additional tool.”

Unreliable Tests. Law Enforcement Steps In.

The day after the tests arrived, city health workers began expressing alarm over what they were seeing as they tried to validate their accuracy, and they left the drive-thru site shuttered.

“Until we finish our quality controls, we can’t provide them,” Gonzalez, the health director, said during the city’s daily web-based press briefing.

The following day, the fifth in a row on which city leaders would announce a COVID-19 death, Gonzalez was even more pessimistic. “We had been told by the manufacturer of the test and the literature that it was 93% or 97% effective, which is great,” he said. “But doing our controls, we didn’t like the results that we were getting because they weren’t anywhere near” those numbers.

Anhui DeepBlue told ProPublica and The Texas Tribune that Laredo health officials should have reached out to them before declaring the kits unusable and said it is seeking a detailed report of the city’s validation test.

Gonzalez said the health department tested individuals known to be positive or negative through previous swab testing, and that the Anhui DeepBlue tests only had about 20% accuracy, “too low of a rate to use.”

The health director added that he believed the kits could have been used to screen for active infection, since they could detect a type of antibody called IgM, which is one of the immune system’s first responders during an infection . According to the FDA, when igM antibodies are found in the blood, they “can indicate” an active or recent infection.

“It would have allowed us to test large numbers of persons and give us an indicator of the level of possible infection,” he said.

On April 2, Laredo police began asking questions about the FDA documentation the company’s distributor had presented, according to a press release announcing the city would not use the tests.

The same day, Homeland Security Investigations, an arm of U.S. Immigration and Customs Enforcement, seized the kits and launched an investigation of its own.

Then, in yet another example of the confusing rules around international test manufacturers, Castañeda said that Homeland Security on Wednesday agreed to return his tests. ICE spokeswoman Nina Pruneda declined to explain why.

What happened in Laredo isn’t the first time the company has run into controversy. Last month, federal agents seized 100 of the company’s COVID-19 tests, which were on their way to a Portland, Oregon, man for home use. It’s illegal to send testing kits to a private individual and not a health care provider, according to FDA rules. But in an example of how murky FDA regulations can be, an FDA consumer safety officer told The Oregonian that he couldn’t say at that time whether the company was authorized to distribute its tests to clinical labs.

Laredo health administrators are not the first to complain about unreliable testing kits from China. European nations have complained for weeks about receiving substandard tests from that country.

Disputed purchases include 1.2 million faulty antibody tests in Slovakia, a batch of 300,000 quick tests in the Czech Republic that local authorities said were about one-third defective, and tests sent to Turkey that officials said were just 35% accurate. Spain returned about 9,000 faulty rapid tests that it purchased from an unauthorized Chinese firm last month.

$500,000 Worth of Tests, in Limbo

When Homeland Security releases the tests, Castañeda said he is not sure how they will be used, and there aren’t firm guidelines on how antibody tests should be deployed. Some have argued the tests should be used to clear first responders and other front-line workers to return to their jobs; others believe they will be part of a massive testing effort nationally to identify those with immunity.

“We’ll try to use them wisely to make them available to the community as soon as possible,” he said.

Saenz, the Laredo mayor, said he is hopeful that Texas Gov. Greg Abbott, who has come under criticism for the state’s slow pace of test procurement, will follow through on a promise to ensure a reliable flow of authorized testing kits to cities and counties. On Wednesday, Abbott announced that Walgreens intends to begin offering drive-thru testing, potentially serving thousands of Texans per day.

“We need a secure, proven supply chain where we don’t have to question the validity of the product,” Saenz said. “We know there are people trying to take advantage of a lot of entities.”

Caroline Chen contributed reporting.