As I wrote in my last essay, real world evidence (RWE) generated from “big” real world data (RWD) is upending the hegemony of traditional randomized controlled trials and the evidence hierarchy. RWE is being used for epidemiological evidence to identify targets for drug development, for safety surveillance of approved medical products, for examining changes in patterns of therapeutic use, and for measuring quality in health care delivery by linking outcomes to treatments. However, as this shift unfolds it is important to recognize the limitations and challenges imposed by this evidence source. Electronic health records (EHRs) and claims data are not collected or organized with the goal of generating evidence, nor have they typically been optimized for such purposes. Additionally, data generated from wearables, smartphone applications, or social media feeds can be unreliable and of uncertain quality. In general, key concerns of RWD include the potential for bias, confounding that cannot necessarily be overcome simply by the large sample sizes and the ubiquity of “big data.”
The move to digitized data collection in EHRs as a source of patient data has expanded the potential of real world evidence generation from these datasets. However, EHR is not optimized for evidence generation but is a tool used to document a patient encounter, justify physician decisions, generate claims for billing, and protect against malpractice claims. For example, practices such as upcoding or even the use of synonymous diagnostic codes that result in higher reimbursements are known sources of skewed distribution of diagnostic codes in EHR datasets. Additionally, although EHRs capture medical data such as diagnoses, laboratory results, and medications, they inconsistently and incompletely record socio-behavioral-environmental determinants of health. At first glance this might not be a relevant oversight but analyses have been estimated that nearly 60% of health outcomes are driven by these measures, whereas, only 10% are a function of medical care. If EHR data is not capturing these drivers, then real world outcome conclusions will obviously be biased and erroneous.
Even for data that is captured in EHRs, the fragmented structure of the US healthcare system makes data predominantly episodic and event based. Thus, EHR captures patient encounters episodically and only following a health event. This not only creates a bias towards the overestimation in prevalence of illness and disability, but pre and post event signals are incompletely captured in the EHR. Complete disease and patient journeys require more continuous tracking of variables. While the industry is looking at smartphones, wearables, and social media feeds to bridge this gap, these data sources can not only be unreliable and inaccurate but when used in isolation can drive patients to inappropriate conclusions. For example, in one study, the Apple Watch’s abnormal pulse detection led to unnecessary healthcare utilization because of a “high false positive rate as a screening tool for undiagnosed cardiovascular disease.” Health outcomes are inherently multidimensional and nuanced, thus for “every problem, there’s a solution that is neat, plausible, and wrong.” (H.L. Mencken)
Throughout history, grand narratives (future essay) and revolutions “have never lightened the burden of tyranny; they have only shifted it to another shoulder” (George Bernard Shaw). In the 20th century, medicine followed science and adopted evidence as the basis of decision making, thereby minimizing the role of physician expertise and intuition in the process. Thereafter, randomized controlled trials were ossified as the gold standard of evidence generation and minimized the role of observational and mechanistic studies in the process. Now, as data has become digitized and ubiquitous, real world evidence is primed to assume the mantle of evidence generation. The British philosopher, Alfred North Whitehead, wrote, “[the] art of progress is to preserve order amid change and to preserve change amid order.” Are pragmatism (next essay), the middle way, and “fit for purpose” evidence, the grand narrative and the revolution we should pursue?