Developing a diagnostic test with high sensitivity and specificity is hard. One must (1) choose a good set of observable variables (e.g. each instance is a real-valued vector or a combination of real-value variables and nominal variables), (2) make some reasonable assumptions about the data (e.g. instances are drawn IID from heteroskedastic Gaussians), (3) label each instance as positive or negative for use in estimation (this can be problematic if no method is known to absolutely confirm the presence of the target disease), (4) employ an algorithm for estimating parameters of the prediction model (a machine learning algorithm), and (5) employ statistical validation to evaluate the accuracy of the prediction framework.

Choosing a good set of variables (step 1) is hard and often costly. For example, if one were designing a diagnostic test for brain tumors, patient-reported headache severity on a scale from 1 to 10 is one observable variable that could hypothetically be used. From a set of 100 people, if we knew which ones had brain tumors and which ones did not, we should expect even the best estimated/learned predictors to perform poorly. There just isn’t enough information in the data.

Even when we have a set of variables with a reasonable amount of information and a good set of assumptions, we cannot expect to predict with infallibility. In the real world, false positives and false negatives are inevitable. If we assume for the moment that the data are drawn from Gaussians, false positives and false negatives are guaranteed to occur given enough trials because a Gaussian distribution has infinite support, and thus two or more Gaussians must overlap.

By making reasonably valid assumptions, we can develop a prediction framework in which mathematically provable statistical guarantees can be made. This is difficult. As a simple example, if the data are assumed to be drawn from Gaussians with common covariance, we can predict optimally with a linear discriminant. However, if this assumption doesn’t agree with reality (i.e. the data are drawn from Gaussians of different widths), a linear discriminant will perform sub-optimally (i.e. lower risk can be achieved with better assumptions). The better the assumptions, the better the optimality guarantees that can be made.

In statistics and machine learning we can quantify the cost of false positives and false negatives and the benefit of a true positive with a loss function. The quality of a predictor function is simply the expected loss wrt to the distribution, which is called the risk. The lowest risk achievable given vectors drawn IID from a joint distribution is the Bayes Risk. In the real world, the Bayes Risk is usually always >0, which means mistakes (false positives and false negatives) will always occur.

The goal is to employ a learning/estimation algorithm that generates a predictor function that achieves acceptably low (and hopefully minimum) risk. A second goal, if possible, is to mathematically show that as a sample size gets larger, the risk of the predictors converges toward the Bayes Risk. When this is true, we say the algorithm is consistent. If the learner is consistent regardless of the family of distributions, it is universally consistent. k-NN was the first to be shown universally consistent (Stone, 1975) given the right rate of k and sample size.

Algorithms prone to fitting the model to the data too closely are said to overfit. Even universally consistent algorithms are prone to overfitting and can perform poorly in practice. Some agorithms control overfitting with regularization, which puts a penalty on model complexity. Other algorithms make guarantees about the rate at which an algorithm makes progress towards the Bayes Risk.

To make a reasonable estimate of an algorithm’s accuracy, it must undergo statistical validation. This involves applying a learned model to sequestered data. The estimate of accuracy will be biased if the sequestered data is ever used to estimate a model’s parameters. This a common mistake made when designing a diagnostic test: using the data to find the right model parameters then using the same data to demonstrate the accuracy of the prediction model.

Sometimes a predictor function outputs a real number (sometimes called a signal) indicating the strength of its predictions. By thresholding this signal to make a final positive or negative prediction, we can tradeoff between true positive rate (tested positive given actual positive) and false positive rate (or false negative rate).

A ROC curve plots the detection rate (y-axis) vs. false positive rate (x-axis) for each trade-off threshold, illustrating the true positive/false positive trade-off. A diagnostic test that predicts randomly would generate a line across the plot, which can be used as a baseline. If a predictor does not much better than random then it provides little to no predictive value and shouldn’t be used as a diagnostic test. One unfortunate but common problem is having a detection rate that isn’t acceptably high until an unfavorable false positive rate is chosen. For some conditions, the most accurate diagnostic tests known are prone to either many false positives or false negatives. In these instances, performing a test when it isn’t indicated can be misleading.

The process of developing a good diagnostic test for a disease is a difficult and an error prone process. Even if a good, rigorous methodology is properly followed, there may be inherent limitations in the observable variables chosen, the assumptions made, the data collected, the process for labeling the data instances, the learning algorithm, or the final prediction model. For example, the sample size may be too small or the sample not be representative of the population for the test is targeted. The test might be erroneously applied to subjects not intended for in the test’s design, leading to false positives. The labeling of the data might be contaminated because the method for absolutely confirming disease may be error-prone. Significant money may be spent conducting a study to find a good diagnostic test for a disease and the final result may be a less than desirable ROC curve.

While this company may or may not be reputable, other companies may exploit hypochondriacal consumers who are convinced they have certain diseases by designing tests for them that predict no more informatively than a random coin flip. Placing value on diagnostic tests with performance not much better than random prediction is as bogus as homeopathy. Without proper statistical validation, one can’t make any assumptions about the validity of a test. However, patients in need of answers are prone to swear by these tests anyway. It gives them an explanation for their ailments, serves as a false confirmation of their false self-diagnoses, or a false reassurance they don’t have a disease.

Given the difficulty in designing a single test and demonstrating its validity, what should a consumer do if they test positive for diseases when their risk factors are low? What should a physician do when worried patients come into their office with a print out of positive results? Test batteries such as these, while they may or may not be performed by reputable companies, distract well-intentioned doctors who would ordinarily select diagnostic tests based on evidence but feel obligated to address test results put before them.

Thanks for writing, but I think you missed my main point. Without knowing true positives vs false positives, and without evidence of net clinical outcomes, we cannot know if the screening you offer is beneficial, or even not harmful. Your premise seems to be that more screening is always better – but this is naive.

Given that 1 in 20 blood tests, on average, assuming 2 standard deviations as the norm, are going to be false positives, 250 chances for such is a lot. I understand your company’s claim that you reduce this by stacking tests – but let’s see some evidence for this. In any case – we need some studies looking at net clinical outcomes.

Further, your test is just a snap shot. How often should people get this $3,400 screen done?

I am, of course, not against testing or screening. But there is a rational evidence-based way to go about it. And just doing every test under the sun is not it, especially when you are charging a huge fee out of pocket.

Which stemmed from my pondering why my GP doesn’t have an ultrasound machine. In the UK the reason boils down to he isn’t paid for that, we pay ultrasound specialists and he would refer. This system prevents capital investments the GP might make into things which might work, but wouldn’t save enough of the doctor’s time to justify. So he has a stethoscope and a thermometer, and very little over and above what his grandfather might have used to make his diagnosis on site (blood or other lab tests being the predominant advance over his predecessors). However I’ve no doubt there are many things that would save the patient time, money and suffering, if it were their interests being considered rather than the GPs, and more that could be invented.

As a thyroid patient, my particular bugbear stems from problems getting diagnosed. Surveys suggest the average time to diagnosis of a thyroid problem is still about 5 years from onset of symptoms. The symptoms are well documented in the literature, the at risk groups well understood, the diseases are common (>2% of women at any one point), the blood tests relatively cheap. Yet we still see report after report of these disorders being picked up after referral – psychiatrists being a common one – rather than by the front line doctors, or diagnosis after serious complications set in, or at the start of pregnancy (when a test is done!).

So whilst I agree this kind of screening may be expensive and counter productive. I think there is a good case to argue that we should perhaps do more routine automated tests of patients (at least in the UK). Not necessarily thyroid function screening (the maths says this is not justified in all women), but when I switched GP due to previous doctor over medicating me they did a standard set of lab tests, and I was abnormal on 8 out of 13, due almost entirely to known mild hypothyroid state due to poor prescribing by my previous doctor. These tests are absolutely routine in medicine, I assume they are fully automated after the blood draw. I don’t think it would be that challenging to invent a suitable set of automated tests to be done on patients in the waiting room, including things like weight, blood pressure, pulse, temperature. Ideally non-intrusive tests.

I’d argue the reasons we don’t do these kinds of simple tests routinely on all patients are more cultural than scientific. In the UK we know precisely the proportion of hypertensives not diagnosed as we make doctors report the number of diagnosed hypertensives as part of their contractual arrangements. The contractual arrangements don’t specify (AFAIK) that they should screen for hypertension. As we know the prevalence of hypertension in the population, we can confidently state precisely how many people don’t know they have a blood pressure problem brewing.

I agree. I was commenting in response to superdave’s question.

I would not go as far as to call it pseudoscience. They do, in fact, test many things that medical workups would include. Their cancer panel includes PSA, for example, which is commonly screened for prostate cancer surveillance.

For the most part it is based on science…But I would say some of the stuff they test may be a bit premature, especially if they are claiming it is soundly correlated to a problem. They avoid saying the panel serves as a diagnosis for anything. Their “sex and energy” panel, for example, is a bit iffy. To their credit they do not seem to offer very preliminary/candidate biomarker analysis on their panels like Abeta for Alzheimer’s.

Overall I agree with Dr. Novella’s assessment of the service..It is not AWFUL, but probably unnecessary. Most of the important things are already tested for regularly (cholesterol markers, etc.) This is definitely preying on the paranoid diagnosis-seekers.