A clinical registry is an organized system for the collection, analysis and reporting of clinical data about patient care for the improvement of health outcomes . With the increased use of registries for research, performance evaluation, quality improvement and payment, there is corresponding interest in new registry development and increasing the functionality of established registries. The National Quality Registry Network (NQRN) is a community of organizations interested in clinical registries. NQRN is a program of the PCPI Foundation, a national convener of medical specialty and health care professional societies and associations . To accelerate these trends and guide registry development and use, NQRN has developed a maturational framework for clinical data registries.
Clinical registries have made many contributions to the health care knowledge system [3, 4, 5, 6, 7, 8, 9]. As registry data collection, analysis and reporting capabilities have improved, their use in improving health system effectiveness has expanded (Figure 1). Today’s patchwork system of registries is moving toward a coordinated and transparent system, with improved interoperability envisioned between registries and other health information systems to support better efficiency and outcome evaluations. Additionally, registries have a need to be able to demonstrate the high quality of their data through transparency of performance measures and quality control techniques.
Although the United States leads the world in national health care expenditures, robust data to support quality improvement, accountability and consumer engagement are lacking . As clinically rich data have become more widely available, there is interest in leveraging these data to improve the overall quality and efficiency of the U.S. health care system .
A clinical registry is utilized for many purposes including assessing performance; determining clinical, cost or comparative effectiveness; and measuring or improving the quality, safety or efficiency of care. Specific applications include surveillance, clinical practice guideline development, appropriateness determination, performance evaluation and improvement monitoring, research and public reporting.
All data sources – administrative claims, clinical information systems, electronic health records (EHRs) and registries – have advantages and limitations. Although originally developed for billing and other administrative functions, claims data have subsequently been utilized for a wide variety of purposes including performance evaluation due to their availability, low cost, standardized coding systems such as the International Classification of Diseases (ICD) and Current Procedural Terminology (CPT), and the large volume of available data they represent. However, administrative data often lack clinical granularity and specificity, may be difficult to aggregate and analyze across payers, and have been viewed skeptically by the provider community when used for performance evaluation.
Clinical information systems and EHRs also have advantages and limitations compared with registries. Their data models have been designed to support direct patient care and operations, with a secondary focus on performance evaluation. Roski et al. estimate that, “…less than 15 percent of health data in EHRs might be entered in structured data fields that allow those data to be analyzed using traditional retrieval and analysis methods .” A significant proportion of the data captured in these systems lack standardized data definitions, thus limiting their usefulness for the purposes articulated above.
Registries are designed by clinical experts and contain data that are clinically relevant, structured, precisely-specified, understandable, and acceptable to health care providers. Registries typically include data from real-world populations and multiple payers, distinguishing them from many administrative data sources. For research, registry data may offer an advantage over traditional randomized clinical trials as rigid study population inclusion and exclusion criteria in the latter may limit the generalizability of their results. Finally, registry data are collected using processes that ensure data validity and good inter-rater reliability, with data quality audits ensuring ongoing accuracy.
Registries also have limitations. Currently, most registries focus on short-term outcomes making it difficult to assess care over time. They commonly lack information on readmissions, re-interventions, medication adherence, patient functional status and cumulative costs. This information is becoming increasingly important as responsibility for outcomes and costs extends well beyond a single episode of care.
Given these considerations, one approach may be to combine the positive attributes of each data source, while mitigating their respective weaknesses . To accomplish this, commonly used data elements will need to be standardized to facilitate data linkages . For example, using the registry as the foundation, linkages could be established to EHRs for demographics, to EHRs or clinical information systems for laboratory and diagnostic results, and to administrative claims and other data sources for repeat hospitalizations, procedures, professional services, cost information and long-term survival [15, 16].
There are over 60 national clinical registry programs in the United States, many of which are recognized for the quality of their data and their contributions to improved patient care . While they are typically developed by professional organizations, the value of registries is increasingly recognized by payers. For example, the Medicare Access and Reauthorization Act of 2015 (MACRA) required the Centers for Medicare & Medicaid Services (CMS) to expand the options for registry data submission in the Quality Payment Program (QPP), which consolidated earlier programs such as the Physician Quality Reporting System. Since 2014, CMS has certified over 50 national clinical registries as Qualified Clinical Data Registries (QCDRs). Certification as a QCDR ensures that a registry meets stringent, national requirements. Registries are also being utilized by the U.S. Food and Drug Administration for post-marketing surveillance of medical devices .
There is a growing body of work and increasing interest in registry description, classification and evaluation. In 2012 the NQRN conducted a registry landscape assessment which identified multiple unmet needs of registries, as well as variation in governance and operational processes. The survey was conducted a second time in 2015 . In addition to these efforts, others have offered models and frameworks to help define and classify registries. In 2008 Drolet et al published a classification system for registries that identified five characteristics of a registry, using the term in its broadest sense: mergeable, standardized longitudinal data from multiple users, including outcomes, collected using a rules-based approach . Klaiman et al examined registry use in 2013 and found that “effective registries were successful in 1 or more of 6 key areas: data standardization, transparency, accuracy/completeness of data, participation by providers, financial sustainability, and/or providing feedback to providers .” The Australian Commission on Safety and Quality in Health Care has published a framework for quality registries that describes desired registry attributes across domains including data collection, risk adjustment, data security, data quality, governance, ethics & privacy, reporting and funding . A 2016 report by Crowley et al offered a Public Health Information Technology Maturity Index. Although not specific to registries, the index offered a maturity dimension in addition to domains including scale and scope, quality, human capital, policy, resources and infrastructure .
At the time of publication there was no national standard for evaluating the maturity or performance of registries in general. The QCDR requirements could be considered a baseline at least for reporting performance measure results to evaluating organizations for the purpose of carrying out value-based payment programs. Health Level Seven International (HL7), a standards developing organization, publishes domain analysis models on subjects of interest to the health care data standards and informatics communities. A domain analysis model contains an abstract representation of a specific subject area, oriented to the needs of health IT implementers and users . As part of its Common Clinical Registry Framework project, in 2017 HL7 published a model that “…describes the function, organization, structure and major workflows of a general clinical registry” . As a set these models and frameworks provide useful tools for describing and classifying registries, however none include a maturity model developed specifically for evaluating registry capability through multiple phases of maturity.
In response to the variations in registry development identified in its registry landscape surveys and given the lack of a targeted maturity model, the NQRN developed a clinical registry maturational framework (Framework) to guide the development of registries over time.
Development of the Framework began with the identification of important registry domains such as registry use, scope, data quality and performance measurement. Early, intermediate and mature developmental stages were then described for each domain, and the resulting intersecting cells were populated to demonstrate how a registry might evolve over time. The Framework was further refined with broader stakeholder input from NQRN participants in November 2013, as well as further updates by NQRN staff in 2016 and 2018.
The Framework describes the domains in which a registry might mature over time. The relative importance of each domain is based on the use(s) of the registry. The Framework is a roadmap for achieving the highest level of value from a registry and a guide to identifying the underlying capabilities and infrastructure that will be necessary to achieve that value.
The Framework has been developed according to a set of general principles: (1) it is intended to be a guide to registry development, not a standard; (2) content is organized by topical domain and phase of maturity; (3) content is generally cumulative as maturity increases within a given domain; (4) at any given time, a registry may be at different stages of maturity across the domains; and (5) the desired final stage of maturity in a domain may differ depending on the mission of the registry.
Within the Framework, registry activities are generally cumulative as maturity increases within each domain. However, not all registries will mature along the same trajectory, with differences driven by the needs of its participants and end users. Finally, the Framework reflects the current state of registry development which is expected to evolve over time as medical science, payment models and information technologies advance.
Registries can be described by applying a model comprised of different domains – starting with registry use, followed by additional domains describing underlying capabilities at each phase of maturity (Table 1).
|Registry use||Activities the registry is used for or developed to support|
|Stakeholder relationships||Relationships that registry stewards establish in order to develop and maintain a registry|
|Legal, ethical & oversight||Legal and ethical aspects related to a registry’s governance and operations|
|Participation size & scope||The representativeness of a registry’s patient population|
|Data type||Clinical concepts and metadata captured in a registry|
|Data capture, storage & transmission||The level of automation and standardization for data capture and transmission to and from a registry|
|Data model & quality||Capabilities and methods that influence the ability of a registry to capture high-quality data despite variances in sample size, validity and reliability|
|Measure development & use||The types of performance measures used by a registry, along with its contribution toward measure development efforts|
|Measure type||The kinds of clinical and other concepts measured by a registry|
|Data collection basis||The time domain scope of registry data collection|
|Reporting frequency||The frequency with which the registry provides reports to its participants|
|Public reporting||Registry reporting intended for public audiences|
|Transparency||Details of registry design, activities and operation that are made available to the public|
|Quality improvement||Registry reports, other performance-related feedback to participants including programmatic activities intended to improve the performance of health care operations|
|Research||Capabilities and infrastructure to support the use of registry information for clinical research|
|Human capital needs||Recommended skills and capabilities to have either in staff or outsourced resources|
|Registry networks||The degree to which the registry participates in professional and data-sharing networks|
For a registry to expand its usefulness and overall maturity, increased supporting capabilities are needed. The Framework takes a developmental staged approach to describe what to expect of a registry at different phases of development, along with the capabilities needed. In the early phase of development, a registry typically begins to provide value to participants who use registry data for local quality improvement or performance assessment and reporting. A registry in the intermediate phase has achieved a critical mass of participation to support performance measurement and other activities with increased scope and rigor. Mature registries provide data to support health system decision making at a national or international level as well as for research and health care policy. Registries may achieve different maturity phases across the various domains at any given period of time, but some congruence is anticipated due to the cumulative advancement of underlying capabilities needed to support particular uses in the next higher phase. Examined together, the topical domains and the development structure allow a registry to be assessed holistically. (Figure 2).
A table containing the complete Framework matrix is in Appendix A.
The Framework is intended to guide registry development; it is not a set of standards or requirements. It is hoped that its guidance will facilitate the development of registries in a logical fashion and support the adoption of leading practices and national standards at an accelerated pace.
The Framework has many potential uses. It can guide registry stewards along a consistent path to maturity with a shared understanding of what the different stages of maturation entail. It can be used as a general guide to understand what might be needed for a registry to qualify for use by a private or public payer such as CMS through its QPP. The Framework can help users of registry data e.g., health systems, researchers, evaluators and payers assess a registry’s capabilities. It can also be used by registry stewards to communicate with consumers about how information was collected, analyzed and reported.
The Framework may be a useful assessment tool for registry stewards, those planning a new registry, and users of registry information. The Framework can be used to define the goals of the registry and the developmental pathways to achieve these goals. It is a valuable resource to identify a registry’s strengths, areas where additional focus and resources may be needed, and actions required to expand a registry’s purposes.
With its strong emphasis on descriptiveness combined with flexibility that allows each registry to fulfill its stated purposes, the Framework can provide valuable insight to accelerate the pace and quality of registry development. An existing registry can evaluate its maturity level and gain insight that can help prioritize further development efforts. A new registry can better understand the capabilities that will support its mission. Users of registry data for research, performance improvement and payment can better understand the level of rigor and scope they can expect from the data. There are many other potential uses (Table 2).
|Situation||Goal||How the Framework helps|
|A growing registry is using manual entry data submission that is costly for its participants||Change over to automated data submission from participants’ EHRs||Helps the registry understand the underlying functionality that should be developed in order to support improved interoperability via automated data extraction from electronic sources|
|A registry seeks to measure clinician performance on a national level||Develop national standard measures||Helps the registry understand the expertise and capabilities that support measure development|
|A health plan wants to use registry data for quality evaluation and wants to understand what it can expect from a registry||Evaluate registries for program participation||Provides a context for evaluating the quality, scope and rigor of registry data. Payers can use this information to develop participation requirements.|
|An organization is being asked to submit data that support the execution of value-based payment models||Understand the capabilities needed in a registry that participates in payment programs||Outlines these capabilities and provides a path to achieving them|
Although the Framework is comprehensive and offers guidance to potentially any registry developer, it has limitations. The information it provides, while important and valuable, is necessary but ultimately not sufficient for a sustainable and credible registry to emerge and evolve. The Framework as currently bounded does not address several relevant dimensions such as a) governance and ownership of registries and registry data, b) registry business models and access/use requirements, c) a range of technical issues such as execution of data use agreements to whether data is distributed, consolidated in one place, or some hybrid, and d) approaches to privacy protection. As specialty societies and other registry stewards continue their work in developing and refining sustainable business models, the registry community through the NQRN and other stakeholders has an opportunity to continue the development of this Framework to address these limitations.
It is envisioned that the uses of the Framework will expand in the future. For example, it could serve as an assessment tool for those considering participation in a registry i.e., health systems, practitioners, or health plans, consumers and others interested in using registry data for improvement and related purposes.
Supported by the NQRN and the Framework, as well as additional NQRN resources, registries are expected to continue to mature as an important data source to drive improvement in health care delivery . With strategic investments in data standardization, linkages among registries and other data sources, and alignment of financial incentives, the U.S. health care system will evolve from being “data poor” to “knowledge rich.”
|Legal, ethical & oversight||
|Participation size & scope||
|Data type [27, 28]||
|Data capture, storage & transmission||
|Data model & quality||
|Measure development & use||
|Data collection basis||
|Human capital needs||
Chisara Asomugha, MD, MSPH, CMS
Anthony Asher, MD, American Board of Neurological Surgery
Carmella Bocchino, America’s Health Insurance Plans
Jeremiah R. Brown, PhD, MS, Dartmouth-Hitchcock Medical Center
Dan Campion, MBA, IQVIA
Mythreyi Chatfield, PhD, American College of Radiology
Kate Chenok, MBA, California Joint Replacement Registry
Carol Clothier, American Board of Medical Specialties
Sean Currigan, MPH, American College of Obstetricians and Gynecologists
Richard Dutton, MD, MBA, Anesthesia Quality Institute
Stephen Epstein, MD, American College of Emergency Physicians
Eden Essex, American Society for Gastrointestinal Endoscopy
Jason A. Efstathiou, MD, DPhil, Massachusetts General Hospital
Caryn Etkin, PhD, MPH, American Academy of Orthopaedic Surgeons
Nancy Foster, American Hospital Association
Sheila Heitzig, JD, MNM, American Academy of Allergy, Asthma & Immunology
Kathleen Hewitt, MSN, RN, American College of Cardiology
Stacie Jones, MPH, American College of Emergency Physicians
Danielle Jungst, American College of Chest Physicians
Norman Kahn, MD, Council of Medical Specialty Societies
Jeffrey P. Knezovich, Joint Replacement Registry
Betsy Kohler, North American Association of Central Cancer Registries
Charlotta Levay, PhD, Harvard Medical School
Flora Lum, MD, American Academy of Ophthalmology
Kristen McNiff, MPH, American Society of Clinical Oncology
Thomas Murray, American Gastroenterological Association
Frank Opelka, MD, Louisiana State University Health Sciences Center
Joyce Bruno Reitzner, MBA, MIPH, American College of Chest Physicians
Koryn Rubin, American Medical Association
Dana Gelb Safran, ScD, Blue Cross Blue Shield of Massachusetts
Jill Sage, American College of Surgeons
Lewis Sandy, MD, United Health Group
John Santa, MD, MPH, Consumer Reports
Dale Singer, MHA, Renal Physicians Association
Jean Slutsky, Agency for Healthcare Research and Quality
Katie Sommers, MPH, American Society of Plastic Surgeons
Modena Wilson, MD, American Medical Association
This article represents a team effort in which multiple individuals made contributions in addition to the author. The author gratefully acknowledges their assistance. Appendix B lists the subject matter experts who contributed to the development of the registry maturational framework through their participation in the Registry Maturational Framework Working Meeting, November 20, 2013 in Boston. Gratitude is extended to those who contributed to the validation of the framework, especially Mythreyi Chatfield, Richard Dutton, Jeff Knezovich and Darryl Roberts. Dana Richardson initiated the project and shepherded its development for multiple years. David Shahian and Kathleen Blake provided project oversight in addition to contributing to earlier drafts of the article.
The author has no competing interests to declare.
https://www.thepcpi.org/, accessed 5-22-19.
Ovretveit, J, Nelson, E and James, B. Building a learning health system using clinical registers: a non-technical introduction. Journal of health organization and management. 2016 Oct 10; 30(7): 1105–18. DOI: https://doi.org/10.1108/JHOM-06-2016-0110
Kondziolka, D, Cooper, BT, Lunsford, LD and Silverman, J. Development, implementation, and use of a local and global clinical registry for neurosurgery. Big data. 2015 Jun 1; 3(2): 80–9. DOI: https://doi.org/10.1089/big.2014.0069
1. Reports|Expanding the Science of Patient Input: Building Smarter Patient Registries [Internet]. FasterCures. 2018 [cited 28 June 2018]. Available from: http://www.fastercures.org/reports/view/55.
Lionetti, G, Kimura, Y, Schanberg, LE, Beukelman, T, Wallace, CA, Ilowite, NT, Winsor, J, Fox, K, Natter, M, Sundy, JS and Brodsky, E. Using registries to identify adverse events in rheumatic diseases. Pediatrics. 2013 Oct 1: peds-2013.
Carroll, JD, Shuren, J, Jensen, TS, Hernandez, J, Holmes, D, Marinac-Dabic, D, Edwards, FH, Zuckerman, BD, Wood, LL, Kuntz, RE and Mack, MJ. Transcatheter valve therapy registry is a model for medical device innovation and surveillance. Health Affairs. 2015 Feb 1; 34(2): 328–34. DOI: https://doi.org/10.1377/hlthaff.2014.1010
Chan, PS, Curtis, LH, Ferguson, TB and Jurgens, C. ACC/AHA/STS Statement on the Future of Registries and the Performance Measurement Enterprise. Journal of the American College of Cardiology. 2015; 66(20). DOI: https://doi.org/10.1016/j.jacc.2015.07.010
Papanicolas, I, Woskie, LR and Jha, AK. Health care spending in the United States and other high-income countries. Jama. 2018 Mar 13; 319(10): 1024–39. DOI: https://doi.org/10.1001/jama.2018.1150
Gliklich, RE, Leavy, MB, Karl, J, Campion, DM, Levy, D and Berliner, E. A framework for creating standardized outcome measures for patient registries. Journal of comparative effectiveness research. 2014 Sep; 3(5): 473–80. DOI: https://doi.org/10.2217/cer.14.38
Roski, J, Bo-Linn, GW and Andrews, TA. Creating value in health care through big data: opportunities and policy implications. Health affairs. 2014 Jul 1; 33(7): 1115–22. DOI: https://doi.org/10.1377/hlthaff.2014.0147
Roski, J, Bo-Linn, GW and Andrews, TA. Creating value in health care through big data: opportunities and policy implications. Health affairs. 2014 Jul 1; 33(7): 1115–22. DOI: https://doi.org/10.1377/hlthaff.2014.0147
Pasquali, SK, Jacobs, JP, Shook, GJ, O’brien, SM, Hall, M, Jacobs, ML, Welke, KF, Gaynor, JW, Peterson, ED, Shah, SS and Li, JS. Linking clinical registry data with administrative data using indirect identifiers: implementation and validation in the congenital heart surgery population. American heart journal. 2010 Dec 1; 160(6): 1099–104. DOI: https://doi.org/10.1016/j.ahj.2010.08.010
Shahian, DM, O’brien, SM, Sheng, S, Grover, FL, Mayer, JE, Jacobs, JP, Weiss, JM, DeLong, ER, Peterson, ED, Weintraub, WS and Grau-Sepulveda, MV. Predictors of long-term survival following coronary artery bypass grafting surgery: results from The Society of Thoracic Surgeons Adult Cardiac Surgery Database (The ASCERT Study). Circulation. 2012 Jan 1: CIRCULATIONAHA-111. DOI: https://doi.org/10.1161/CIRCULATIONAHA.112.130989
Reports|Expanding the Science of Patient Input: Building Smarter Patient Registries [Internet]. FasterCures. 2018 [cited 28 June 2018]. Available from: http://www.fastercures.org/reports/view/55.
Blumenthal, S. The Use of Clinical Registries in the United States: A Landscape Survey. eGEMs. 2017; 5(1). DOI: https://doi.org/10.5334/egems.248
Drolet, BC and Johnson, KB. Categorizing the world of registries. Journal of biomedical informatics. 2008 Dec 1; 41(6): 1009–20. DOI: https://doi.org/10.1016/j.jbi.2008.01.009
Klaiman, T, Pracilio, V, Kimberly, L, Cecil, K and Legnini, M. Leveraging effective clinical registries to advance medical care quality and transparency. Population health management. 2014 Apr 1; 17(2): 127–33. DOI: https://doi.org/10.1089/pop.2013.0021
[Internet]. Safetyandquality.gov.au. 2018 [cited 28 June 2018]. Available from: https://www.safetyandquality.gov.au/wp-content/uploads/2014/09/Framework-for-Australian-Clinical-Quality-Registries.pdf.
Crowley, K, Gold, RS, Bandi, S and Agarwal, R. The Public Health Information Technology Maturity Index: An Approach to Evaluating the Adoption and Use of Public Health Information Technology. Frontiers in Public Health Services and Systems Research. 2016; 5(2): 26–33.
5. Domain Analysis Model – HL7Wiki [Internet]. Wiki.hl7.org. 2018 [cited 28 June 2018]. Available from: http://wiki.hl7.org/index.php?title=Domain_Analysis_Model.
HL7 Standards Product Brief – HL7 Domain Analysis Model: Common Clinical Registry Framework, Release 1 [Internet]. Hl7.org. 2018 [cited 28 June 2018]. Available from: http://www.hl7.org/implement/standards/product_brief.cfm?product_id=467.
https://www.thepcpi.org/page/NQRN, accessed May 22, 2019.
[Internet]. Healthit.gov. [cited 28 June 2018]. Available from: https://www.healthit.gov/isa/us-core-data-interoperability-uscdi.