By: David Dai

March 3, 2020

Incorporating domain knowledge from subject matter experts is critical for building useful models. As statistician George Box said, “All models are wrong, but some are useful.” How useful our models are depend on how well we apply domain knowledge to extract the signal from the noise.

The Data Analyst Mindset

A feature common to many data analysts (myself included) is that we have an inherent trust in the data and with it, our ability to extract useful signal from the noise. We are naturally skeptical of domain expert opinions; this is understandable, given the many forms of cognitive biases that humans are susceptible to, and is likely what drew us to the practice of data analysis, acting as “speakers for the data”.

“Sure, clinical expertise is helpful, but it must be riddled with subjectivity and circumstance. On the other hand, the data is objective and provides unbiased quantitative evidence.” – My Subconscious Brain

The above thought has probably crept up in many data analysts’ mind. On a surface level, many of the same data analysts would probably agree that this way of thinking is flawed and incomplete. However, I would wager that a much smaller proportion of data analysts actively consider the consequences of “auto-pilot data analysis”. The thought of pursuing the essence of what makes a data analysis good is difficult and rather abstract, making us vulnerable to the very things that we try to stay away from: subjectivity and circumstance. This is a mental hurdle that must be crossed if we are to create analyses with practical utility.

ML on medical datasets vs. ML for health

With the increasing hype of data science and various avenues of exercising modeling prowess (eg. Kaggle), model building has become a sport of sorts. In turn, the current culture of predictive model building encourages analysts to devote a disproportionate amount of effort on improving an arbitrary performance metric rather than improving their understanding of the model’s objective, and how their model translates to practical, real-life utility.

Within medicine and healthcare applications, this approach to data analysis has serious consequences. Medical datasets are expensive to gather and label, and while the implications of creating scalable algorithmic approaches to medical diagnostics may be game-changing, the lack of domain knowledge and understanding of the data can lead to misleading results that lack clinical utility. Sometimes, it may even lead to asking the wrong questions, as was the case below.

 

The influx of big-tech and other prominent players into medicine and the healthcare sector has created excitement, but also stirred some controversy. In a similar vein to the previous example, this following example illustrates that building machine learning systems for health requires more than a well-performing model, but also a thorough understanding of the risks and consequences of such systems in deployment, such as over-diagnosis.

The Google AI mammogram paper is FLAWED.

Want to learn why?

AND why cancer screening is the LAST thing you should pick FIRST to work on with AI?

Read this #TWEETORIAL
This is educational, I promise pic.twitter.com/W63Mr0N53l

— Vinay Prasad January 2, 2020

 

While neither of these examples directly imply immediate clinical utility, these illustrate the fact that without subject matter expertise and critical inspection of the data and our assumptions, we may simply be applying ML on medical datasets, rather than applying ML for the betterment of health.

First, do no harm

As the use of data becomes more pervasive and the technological systems supporting our way of life grow in complexity, the impact of even seemingly simple data analyses can be far-reaching. As Hilary Parker and Roger Peng suggested in their podcast, Not So Standard Deviations, the notion of a “hippocratic oath” for data analysts may not seem so far-fetched in the current technological climate, especially in the field of healthcare where lives are at stake. While no such regulation exists currently, it is important to remember that behind the numbers are people’s lives.