The development of requirements will vary across types of digital health solutions based on functionality (diagnostics, monitoring, care coordination, etc.), which can also be modeled from other industry approaches. It is critical to incorporate the preferences of the clinicians and patients impacted by the digital health solution into the requirement development process. Once requirements are established, the proposed framework that could form the basis for evaluation includes the following domains: technical, clinical, usability, and cost (Fig. 3).
Technical validation is the most traditional type of evaluation in product testing. Does the solution actually perform to its self-proclaimed functionality with accuracy and precision? For example, how accurately and reliably does a wearable measure heart rate compared with a gold standard? Other elements of technical validation could also include security and interoperability assessments. Applications that claim to perform the functions of established medical devices, such as those reflecting biological processes like heart rate, blood pressure, or respiratory rate, should be able to demonstrate equivalence according to rigorous standards set for other non-smartphone based novel devices.26 System failure and redundancy must also be considered—what happens if a digital health system degrades over time as phone camera lenses become scratched or occluded? Is there a need for back-end monitoring of user engagement or daily calibration to ensure appropriate system performance? Do certain processors or sensors fail to meet stringent minimum standards required for reliable system outputs?
A system architecture, the sum of structure, behavior, and components of a technology, has unique considerations in digital health. Developers must consider the privacy and security requirements of handling patient data that may be confidential and even linked to larger electronic health record systems. Robust, enterprise architecture standards exist to guide developers on issues, such as the levels of encryption and user authentication necessary to safeguard patient information. This requirement may vary with the degree of inherent confidentiality of the health condition in focus— ranging from low, when dealing with step-counters or accelerometers—to high if managing a socially stigmatized disease or storing test results.
Clinical validation to demonstrate efficacy is generally well understood and considered vitally necessary in the context of traditional clinical or translational research. However, analogous studies for digital health products is uncommon.27 This gap may be due in part to the lack of clinical subject-matter experts engaged in the digital health product development.28 Regardless of what part of the continuum of care (prevention, detection, and management) the product addresses, a validation study must compare it to relevant clinical gold standards. Particularly for studies aiming to demonstrate the clinical impact of the a product, these may take the form of accepted care quality metrics, such as measures of clinical outcomes (e.g., presence of disease or complications, clinical functional scores) and process (e.g., laboratory values, adherence to treatment guidelines).
A digital health product that aims to prevent diabetes mellitus would be measured by standard clinical quality measures, such as clinically validated disease diagnostic criteria,29 glycemic control, or diabetes-specific complications like stroke and retinal disease.30 This is important not only to ensure that content is based on current state-of-the-art evidence and guidelines but even to assure users that simple digital interventions, such as behavior-modification applications leverage and incorporate decades of proven, efficacious strategies.
An integral validation step is standardized and critical appraisal of any existing evidence (e.g., an approximation of the rigorous GRADE system31) to support any clinical claims made by the product or solution. Existing digital health review resources may provide subjective assessments of the quality of clinical evidence by technical experts, but generally do not provide systematic and objective analysis.
More advanced validation efforts would require external testing within a simulated or actual trial settings to determine if results can be duplicated. Furthermore, such testing would determine how the product performs across the relevant provider systems, clinical settings, and integrated technologies, in which it is likely to be deployed. For example, the data that flows from an app that monitors cardiac function should actually assess the intended cardiac measure or function, assess it in the targeted population (e.g., congestive heart failure), and be accessible to the clinics and hospitals relevant to the target patient population. The data must flow across emergency, intensive care, and ambulatory settings to electronic health records, provider alert systems, and ambulances of these hospital systems. Without meeting such system requirements, deployment of such an app, even if shown to be clinically effective in a research setting, may have no meaning in the real healthcare system for patients.
An added consideration for “prescription-strength” digital health applications may involve post-market surveillance (PMS), as it is required for most drugs and medical devices.32 Often, pre-release clinical studies are not powered to detect rare, but serious, potential device failures which may cause unintended harm to users. Currently, the FDA monitors reports of device and drug adverse events and may require PMS to clarify possible adverse events associated with a device—or verify that efficacy of a device persists in the “real world”, especially if clinical tests used a surrogate outcome for the purposes of expediency.33
Formal usability assessment of traditional healthcare goods and services (e.g., pharmaceuticals, procedures, and treatment plans) is frequently embedded within the regulatory and patient experience pathways that lead to their development. Similarly, usability testing in the setting of medical devices is common (from a clinician/operator perspective) due to regulatory requirements, although generally with a safety focus.
When considering digital technology, there is no assurance or protection that the technology will align with user needs or preferences. Usability is arguably among the most important considerations with patient-oriented mobile- and digital-based solutions. These technologies are frequently literally in the hands of patients and consequently demand a more patient-centered approach to usability. Existing digital health qualitative reviews address only some aspects of usability. All stakeholders would benefit from a standardized approach, unlike the status quo, which is often ad hoc, qualitative, or dependent on the volume of reviews.34
Digital health apps must be easy to use for their intended purpose, require minimal effort to complete tasks, have minimal data entry burden, and allow the user to control preferences when appropriate (e.g., notifications). Since systems can be designed for users with different requirements (e.g., impaired vision, motor deficits, cognitive dysfunction), design considerations must reflect the target user-audience.
The World Wide Web Consortium has summarized a user-centered design process (UCD),35 which takes into account multiple frameworks and mirror some elements of the International Organization for Standardization’s multi-part standard, ISO 9421.36 The key objectives outlined in this standard are applicable to the evaluation of digital health applications: the solution should be useful in helping users achieve their goals, effective (i.e., producing results with minimal user error), learnable (i.e., easy and intuitive to use), and likeable (i.e., enjoyable to use). These considerations, not surprisingly, play an important role in patient engagement—an often neglected, yet essential aspect of digital health; having a unused medical device is tantamount to not having one at all.
At a minimum, a best-practice evaluation framework should be considered,37,38 but these frameworks establish a lowest common denominator, and do not necessarily incorporate the principles of user centered design into the development process. There are multiple efforts underway to codify standards for design principles specifically to digital health solutions, most notably Xcertia and Node.Health. Some criteria are easier to objectively specify than others. For example, criteria such as number of steps required to complete essential tasks, consistency of navigation, visual legibility, and use of recognizable iconography can be objectively developed. More work is needed to formalize subjective aspects of usability, such as utility and user delight and satisfaction. Usability viewed in this way, along with clinical relevance, creates the opportunity to impact patient engagement.
To maximize impact, digital health solutions will likely require clinician input as part of solution development, thereby accounting for UCD on at least two fronts. Just as early electronic health record implementations increased clinician burden by not adequately considering clinician workflow,39 digital health solutions designers need must pay attention to ease of accomplishing the expected tasks.
At face value, cost—defined as the price a consumer must pay to gain access—may be an inadequate differentiator of digital health solutions—many are free or low priced, particularly in the mobile app arena. When integrated in a composite assessment, however, true cost may provide greater discrimination of overall value. Here, cost estimation becomes more complex by incorporating broader considerations, such as costs of the technology lifecycle and those to integrate technology into the clinical workflow.
Furthermore, the long-term cost implications from outcomes improvements are also difficult to calculate but should be taken into consideration, leveraging metrics from pharmaceutical and device industries. While determination and attribution of financial benefit derived from mobile health apps is challenging,40 real value may be derived from increased personal health engagement, improved patient–clinician engagement, or patient and clinician satisfaction. New ways of quantifying and measuring these types of attributes will provide a more comprehensive picture overall cost-benefit.
In the consumer financial industry, the FICO score represents a global score as an amalgamation of credit information to approximate borrower quality and lending risk.41 It contains five subdomains: payment history, current level of indebtedness, types of credit used, length of credit history, and new credit accounts. Correspondingly, by aggregating the individual domain assessments from the Digital Health Scorecard and contextualizing the degree to which a product satisfies end-user requirements, a composite Global Digital Health Score can be created. As a result, consumers and other users would be provided a high-level synthesis of quality and risk for digital health products.
This aggregate score allows gross initial selection of digital solutions. Individual scores allow finer discrimination of particular products. Such scores also allow digital health companies to identify where improvements are needed and inform stakeholders on what gaps will exist when the products are deployed. The scores could become benchmarks and establish thresholds for particular types of digital solutions. These scores would also need to highlight and prioritize how well the product ultimately met the end-user requirements.
The successful Digital Health Scorecard allows better discrimination of digital health products, but just as important, pushes digital health companies to build impactful products that work for real patients, providers, and healthcare systems. Three main types of barriers (conceptual, financial, and operational) could hinder broad execution of this approach.
The framework described is one possible approach to characterize the needs in digital health and may not fully reflect the needs of the current marketplace. Our framework is modeled from other industry approaches and needs we perceive for digital health. Given the multi-stakeholder nature of healthcare and varying stakeholder incentives, the best approach to impactful, useful, and integrated digital health products may differ than the one described here. For example, although our approach aims to maximize clinical impact on patients, a payer may be more interested in efficient resource utilization. A provider system may seek products that boost high-reimbursement activities, such as attracting surgical patients or increasing advanced, high-cost imaging utilization. Thus, our proposal may not be practical for or applicable to every permutation within the current fragmented healthcare landscape, in which different stakeholders have different incentives. We believe our approach, however, would promote a requirements-driven, impact-focused digital health landscape. Given the nascent, shifting nature of digital health, the most realistic initial approach to a Digital Health Scorecard implementation is to create a requirement set broad enough to encapsulate concepts important to all products, but not inclusive of so much detail that the requirements are not realistic or relevant.
The Digital Health Scorecard concept requires a financially viable business model for implementation and sustainability. Sustainability could be realized if companies saw value in, and thus purchased Digital Health Scorecard reports or accreditation ratings. If patients, providers, or payers also saw value in or demanded these reports and ratings, companies may feel or be required to purchase them. It is conceivable that payers and even hospital, provider, and medical specialty associations would fund the creation of such reports and ratings, particularly as they become more financially incentivized to adopt digital health products that improve outcomes.
Organizational and operational challenges
From an organizational perspective, unlike aviation and the construction of airplanes, digital health products have no single owner of the requirements. The Federal Aviation Administration and aviation companies, however, can set specific requirements that guide the development of airplanes. In digital health, neither patients, physicians, hospitals, payers, nor governmental regulatory bodies create or abide by standard requirement sets for digital health products. Also, unlike conventional aircraft design, many digital health products and accessories are so new that the desired or optimal requirements may be unknown. This further complicates which stakeholder should take primary ownership of driving requirements. No centralized body exists to serve as a clearinghouse for digital health application feedback or failure reporting. Like digital health products, however, novel aviation products such as drones, face unstandardized requirements without a clear curator of these requirements. For all new product types, relevant stakeholders should develop broad requirements categories relevant to the product and a corresponding scorecard approach that enables a best-practice product development and robust product assessment.
From an operational perspective, the tremendous volume and growth of products presently precludes rigorous evaluations for all products. Thus, a Digital Health Scorecard would likely fail in the current landscape if applied universally. Initially, a pragmatic approach could be applied to select products, such as those relevant to high burden or high-cost conditions, already popular with patients or providers, or with peer-reviewed studies to demonstrate validation, efficacy, or both. In addition, self-initiated industry evaluation may overcome the practical barrier of creating a fully capable Digital Health Scorecard organization that tests and evaluates products. Companies could be successful if a clear Digital Health Scorecard guided them with product development. The Scorecard itself could present an opportunity for a multi-stakeholder consultation to first generate and then periodically update, as refinements emerge with time and experience.
If independent outside assessment of digital health products is to be realized, scaling up assessment to meet the demand of many new products would require a large organization, certification network, and substantial resources. The latter could be accomplished by identifying or building a network of independent, objective organizations that could complete the different domain assessments, while following universal requirements from the Scorecard. Partners could vary by their role: traditional hardware/software testing firms to perform validation assessments, academic institutions to perform clinical studies, and auditing firms to assess cost.
A way forward
The road to validating digital health will take resources, collaboration, and time. Even if successful, the first iteration of the Digital Health Scorecard will be different than latter versions as the healthcare environment evolves. In particular, a Scorecard approach will be most successful when all stakeholders partake in its construction and all stakeholders’ financial incentives are aligned to outcomes. The first Scorecard version, however, if transparent, rigorous, and pragmatic, would be an important step toward impact-driven digital health products that function in real healthcare settings. We are presently pursuing a small-scale pilot study implementing this approach in granular detail, the results of which will be published upon completion.
We believe there are two non-mutually exclusive initial approaches to the Digital Health Scorecard model. First, governmental regulatory bodies should partner with clinical stakeholders to create a standard set of requirements using the categorical concepts proposed here. The regulatory effort could be driven by one organization, such as the FDA’s Digital Health arm, or a collaboration between varied agencies like the FDA, FTC, the Centers for Medicare & Medicaid Services, and the ISO. This collaboration could produce certifications and ratings, but most importantly, requirements sets that promote the development of digital health products poised to be efficacious for the target patient population and function in a healthcare system with varied stakeholders. We do not expect one regulatory body to have the bandwidth, resources, or expertise to take this on alone.
Second, a provider healthcare or hospital system (or collection of systems) could lead to the development and adoption of a Digital Health Scorecard. Our preference is for a hybrid approach in which leading hospital systems partners with one or more of the aforementioned regulatory bodies, including the FDA, to lead a requirements-driven Scorecard approach. This system would not only lay out the requirements for digital health products that enter it, but only accept products that achieved high marks for each category of assessment. An existing healthcare system with enough patient volume could have ample market clout to influence the development of digital health products. Further, healthcare systems could collaborate to create an even more influential Scorecard model of care that drives requirements and product development across larger geographic regions and patient populations.
Although many large healthcare systems have sufficient market influence to drive this change, most are not nimble enough to do so. For example, many leading healthcare systems remain tied to electronic health record technologies, nearly universally disparaged by end-user practitioners and healthcare leaders.42 Furthermore, although many such systems desire to implement digital health and telemedicine care models, they largely represent a small fraction of clinical care.
The healthcare environment is evolving and converging with the entrance of non-traditional healthcare players, such as the CVS-Aetna partnership43 and the nascent Amazon–JP Morgan Chase–Berkshire Hathaway collaboration.44 Here, healthcare systems unconstrained by typical barriers to innovate are perhaps more realistic and promising champions of requirement-driven digital health.
As digital health companies have become more prolific and the number and diversity of digital health products has multiplied, the need for requirement-driven product realization and systematic validation has become increasingly important. Patients and providers will benefit from and demand the ability to discriminate clinically meaningful solutions. Payers and investors will need to identify high-value opportunities that ultimately guide reimbursement, investment decisions, and impact-focused care. Given the growing pull from non-industry stakeholders to demand impact-focused, interoperable digital health products, industry will also find utility in demonstrating product quality over product claims. We provide a framework to guide the evolution and successful delivery of validated digital health solutions.