Context: Patient reported outcomes (PROs) are one means of systematically gathering meaningful subjective information for patient care, population health, and patient centered outcomes research. However, optimal data management for effective PRO applications is unclear.
Case description: Delivery systems associated with the Health Care Systems Research Network (HCSRN) have implemented PRO data collection as part of the Medicare annual Health Risk Assessment (HRA). A questionnaire assessed data content, collection, storage, and extractability in HCSRN delivery systems.
Findings: Responses were received from 15 (83.3 percent) of 18 sites. The proportion of Medicare beneficiaries completing an HRA ranged from less than 10 to 42 percent. Most sites collected core HRA elements and 10 collected information on additional domains such as social support. Measures for core domains varied across sites. Data were collected at and prior to visits. Modes included paper, clinician entry, patient portals, and interactive voice response. Data were stored in the electronic health record (EHR) in scanned documents, free text, and discrete fields, and in summary databases.
Major themes: PRO implementation requires effectively collecting, storing, extracting, and applying patient-reported data. Standardizing PRO measures and storing data in extractable formats can facilitate multi-site uses for PRO data, while access to individual PROs in the EHR may be sufficient for use at the point of care.
Conclusion: Collecting comparable PRO data elements, storing data in extractable fields, and collecting data from a higher proportion of eligible respondents represents an optimal approach to support multi-site applications of PRO information.
Creating a patient-centered model of care requires a cultural shift towards asking for, listening to, understanding, and acting on patients’ perspectives. Systematically integrating patient-reported outcomes (PROs) into healthcare delivery can contribute to this change. PROs, defined as “any report of the status of a patient’s health condition, health behavior, or experience with healthcare that comes directly from the patient, without interpretation…by a clinician or anyone else,” have been identified as a critical component of patient-centered care delivery . Initially developed for research, PROs have long been used in clinical trials and epidemiologic studies to quantify constructs such as symptom burden, function, and quality of life, and have become the sine qua non of patient-centered outcomes research (PCOR) [2, 3].
PROs can promote patient-centered care at two levels. At the individual level, PROs can inform treatment decisions and other care processes and guide shared decision making. At the aggregate level, PROs may serve as performance measures to guide quality improvement, health system management, and payment [4, 5, 6, 7]. PROs are also essential outcomes for practice-based patient-centered research .
Most research on PROs has focused on psychometric properties of measures [9, 10, 11]. Additional studies have focused on targeting patients, data collection, processes for reporting results, and developing interventions for positive findings [12, 13, 14]. Based on this evidence and facilitated by improved technology, PROs have been enthusiastically endorsed by patients, clinicians, and policy makers [15, 16]. However, relatively little is known about how and whether PRO use is likely to achieve its intended goals. Unknowns include how to minimize patient and clinician burden, optimize data collection and work flows, prioritize which PROs matter for most delivery of clinical care and for quality measurement/improvement, understand value to clinicians and patients, and apply aggregate data for performance measurement and population health [17, 18]. In short, how to achieve, understand, and optimize PRO implementation.
To systematically use PROs to inform care, it is important to understand how PRO data can be collected and stored, and whether systematic collection is likely to reach the respondent population of interest. If PRO data will be aggregated to assess quality or inform population health, it is additionally important to understand variation in measure selection across delivery settings. Using a case study of implementing the Medicare Health Risk Assessment in multiple care delivery systems, we examine the data management processes underlying systematic PRO collection and use. Figure 1 illustrates elements of systematically collecting and using PRO data.
The Medicare Health Risk Assessment (HRA) is designed to identify patient-reported modifiable risk factors and health needs [19, 20]. HRAs are a component of the Medicare Annual Wellness Visit which was implemented in 2011 as part of the Affordable Care Act. The HRAs are completed at or prior to the visit, and are addressed at the visit or as part of population care management. All Medicare Advantage and Medicare part B beneficiaries are eligible for Medicare Annual Wellness Visits and providers are reimbursed for these visits by the Centers for Medicare and Medicaid Services (CMS). As a result, HRA use is expanding rapidly. Although assessment of certain domains is required, the measures to assess these domains are not specified. HRA implementation provides an opportunity to better understand the spectrum of collecting, storing, and extracting PRO data.
The Health Care Systems Research Network (HCSRN) is comprised of research centers based in large community based integrated delivery systems that provide both clinical care and insurance coverage. Taken together, these health systems care for 16.8 million individuals, with over 75 percent of those individuals remaining continuously enrolled in the systems for at least one year. The systems are based in: Massachusetts, Pennsylvania, Maryland, Georgia, Michigan, Wisconsin, Minnesota, Texas, Colorado, California, Oregon, Washington, and Hawaii. Demographics of each system’s patient population mirrors the underlying geographic area. For example, the population cared for by Kaiser Permanente Mid-Atlantic is 35 percent African American; about 40 percent of the population cared for by Kaiser Permanente Southern California is of Hispanic ethnicity. Most systems are in medium to large urban areas, with the exception of Geisinger, in central Pennsylvania, and Marshfield Clinic, in central Wisconsin. Epic is the predominant electronic health record system used by HCSRN members, with two sites (Marshfield and Henry Ford) using mature in-house systems that have been in place for decades.
Each of these integrated delivery systems provides networked care and collects and stores clinical data in a common data model. Delivery systems associated with the HCSRN serve large populations of Medicare fee for service and Medicare Advantage beneficiaries across a range of service delivery models [21, 22]. HCSRN delivery systems include both early and later adopters of HRA collection. Several store HRA data in extractable fields in electronic health records (EHRs) and aggregate data for population health in addition to using HRA data at the point of care. Health Risk Assessment implementation in these settings provided the opportunity to assess PRO data collection and storage across multiple delivery systems.
Using leaders of HCSRN research departments as points of contact, we sent a questionnaire inquiring about HRA content, data collection, data storage, the proportion of Medicare beneficiaries completing HRAs, and the dates at which HRA data collection became operational to 18 HCSRN delivery systems. Research department representatives collected responses from a range of operational and clinical leaders responsible for HRA implementation and use at each site. Thus responses reflect site-specific information available to these clinical and operational leaders.
The project was reviewed by the Kaiser Permanente Colorado Institutional Review Board and was determined not to be Human Subjects Research.
Of 18 sites contacted, 15 (83.3 percent) sites provided some information on HRA content, collection, and data storage. Not all sites answered all questions, and some sites provided more than one response per question. The proportion of Medicare beneficiaries completing an HRA ranged from less than 10 to 42 percent. Two sites reported data collection since 2011, 6 sites since 2012, and 4 since 2013 or 2014. Data on collection dates were missing for 3 sites and data on proportion completed missing for 7 sites.
Content: All 15 sites reported collecting information on fall risk, mood, and functional status. Fourteen additionally assessed pain and advance directives; and 10 also assessed self-rated health and cognition. For no domain did all sites use the same PRO measure, although all sites used some variation of activities of daily living to assess function. Table 1 lists measures reported for the core HRA domains. Ten sites used the HRA data collection as an opportunity to inquire about other health-related issues. These included hearing and vision, incontinence, oral health, weight loss, sleep, food insecurity, alcohol and tobacco use, social support, physical living arrangements, sexual activity, and personal goals for self-management. (Table 2).
|Domain||Measures (number of sites*)|
|Domain||Number of sites|
|Alcohol and tobacco use||8|
|Seat belt use||7|
|Weight loss or nutrition||6|
|Home conditions (stairs, railings, tub bars, fire alarms||5|
|Self-management goals/primary concern||2|
|Past year medical care needs||2|
Data collection methods: Sites used varying (and often multiple) approaches to engage patients for HRA completion: All sites except one collected HRA data at the point of care, 6 additionally collected data prior to the visit, and one site only collected data prior to the visit. Modes of administration included paper (10 sites), patient portal (5 sites), staff entry into the EHR (10 sites), tablet collection (4 sites), telephone (2 sites), and interactive voice response (4 sites). Five sites used single modes of administration — four with staff entry of data into the EHR and one using paper administration through the mail. The most modalities used by any site was five. Two sites used a combination of paper, interactive voice response, staff entry of information, an interactive patient portal, and tablets at the point of care.
Data storage: The multiple mechanisms for HRA data collection resulted in different approaches to data storage – often within the same site. Four sites reported scanning documents into the EHR, 9 reported storing data in discrete fields in the EHR, and 9 used text fields accessible in the EHR, but not readily extractable into other databases. Nine sites stored data in summary tables or other databases—potentially available for performance measurement, population health, and/or research. Information on HRA data collection and storage is summarized in Table 3.
|Site||Estimated proportion of Medicare beneficiaries completing HRA in 2014||Timing of HRA completion||Mode of administration||Data storage|
|Point of care||Prior to visit||Paper||Patient portal questionnaire||Staff entry in EHR||Tablet||Telephone||IVR||Scanned document||Free text in EHR||Discrete fields in EHR||Summary tables or data warehouse|
This case study of large scale, systematic PRO collection in integrated healthcare delivery systems illustrates a spectrum of approaches to selecting, collecting and storing PRO data. Fully realizing the potential of PROs to enhance patient centered care will require optimizing PRO selection and data management for multiple applications including use at the point of care, population health management, patient-centered outcomes research, and performance measurement within or across sites.
We found substantial variation in measure selection across sites. While CMS prescribes core domains to be included in HRAs, it does not prescribe specific measures, and sites selected different PRO measures to assess these domains. Using varied measures is sufficient for internal use of PRO data either at the point of care or for population health. Responses can prompt referrals for needed care, facilitate shared decision making, or lead to other clinical actions; and can be combined to manage population health or conduct research within a single site. Combining PRO data across sites for broader assessments of population health or research is more likely to be effective if domains are assessed with the same PRO measures. Using the same measures for function, fall risk assessment, cognition, and other domains at multiple sites would permit accurate comparisons of the domains of interest across populations, including assessing changes over time.
Using responses at the point of care requires that the information be available to clinicians for individual decision-making. While some sites reported multidimensional approaches to data collection and storage that resulted in easily visible PRO data, others reported more limited collection and access only to scanned documents. Formatted data in the EHR are usually easiest for clinicians to access and can facilitate use at the point of care. Scanned documents can also be accessed at the point of care; however, scanned documents preclude viewing longitudinal trajectories of PRO data even at the individual level. In comparison, aggregating PRO data for population health, possible performance measurement, and PCOR requires that responses be stored in readily extractable fields in the EHR. Collecting comparable PRO data elements and storing data in extractable fields represent an optimal combination for multi-site applications of PRO information.
Our case study had several limitations. We did not fully assess HRA implementation including how characteristics of the care delivery setting affect data management capacity . As CMS provides financial incentives for the annual wellness visit, our findings reflect PRO implementation in response to external incentives rather than internal needs . We focused on data collection and use and did not examine clinician attitudes towards using these PROs at the point of care or patients’ perceptions of response burden or usefulness. There is currently little evidence on the value and usefulness of PROs to clinicians. While many are supportive, others are wary of management burden and unintended consequences [7, 37]. Information on both patients’ and clinicians’ perceptions of these or any other PRO is essential for making decisions about implementing PRO at scale.
Systematic PRO collection provides a potentially valuable resource for PCOR and longitudinal assessments of population health. For example, two national efforts (Sweden and the United Kingdom) are using pre- and post-operative PRO collection to guide internal quality improvement [1, 38, 39]. Using HRA data on a population level will require data collection from a larger proportion of the population (to ensure respondents are representative) and further study to understand the characteristics of respondents. In our assessment, the site with the highest HRA response rate only reported 42 percent of beneficiaries completing the HRA, and some sites reported less than 10 percent response rates. Having such a low response rate raises concerns that those who respond are not representative of the broader population of older adults. For instance, respondents might be considerably healthier (or sicker) on average than those who did not respond. Relatively little evidence exists on whether the reach and representativeness of PRO collection in general is adequate to support PRO use for performance measurement, PCOR and population health [1, 2, 39, 40, 41, 42]. Target populations most likely to benefit from PRO use include medically and socially complex patients, and individuals for whom repeated measures of function help gauge recovery [43, 44, 45]. The National Quality Forum and others have called for research on best practices in this area [1, 46, 47, 48].
Providing sustainable patient-centered care at scale will require a cultural and process shift to measuring and acting on outcomes that matter to patients such as function and symptom burden [16, 44]. If effectively implemented, systematically-collected and representative PRO data can help clinicians collaborate with patients; help care leaders, policy makers, and the public monitor interventions; and help researchers focus on outcomes that matter to patients. Based on the case of HRA collection within the HCSRN, effective PRO use at scale will require changes including more widespread use of standardized measures, effective and efficient modes of collection, and accessible and readily extractable data that facilitate PRO use on the population level, as well as for providing clinical care to individual patients.
Funder: National Institute on Aging, R24 AG0445050-01: Advancing Geriatrics Infrastructure & Network Growth.
The authors have no competing interests to declare.
Dr. Gill is the recipient of an Academic Leadership Award (K07AG043587) from the National Institute on Aging. Drs. Gill and Allore are supported by the Yale Claude D. Pepper Older Americans Independence Center (P30AG21342). Dr. Gruber-Baldini is supported by the University of Maryland Claude D. Pepper Older Americans Independence Center (P30 AG028747).
Basch, E, Aronson, N, Berg, A, et al. Methodological standards and patient-centeredness in comparative effectiveness research: the PCORI perspective. JAMA-Journal of the American Medical Association. 2012; 307(15): 1636–1640. DOI: https://doi.org/10.1001/jama.2012.466
Schipper, H, Clinch, J, McMurray, A and Levitt, M. Measuring the quality of life of cancer patients: the Functional Living Index-Cancer: development and validation. J Clin Oncol. 1984; 2(5): 472–483. DOI: https://doi.org/10.1200/JCO.19188.8.131.522
Forsberg, HH, Nelson, EC, Reid, R, et al. Using patient-reported outcomes in routine practice: three novel use cases and implications. The Journal of ambulatory care management. 2015; 38(2): 188–195. DOI: https://doi.org/10.1097/JAC.0000000000000052
Segal, C, Holve, E and Sabharwal, R. Collecting and Using Patient-Reported Outcomes (PRO) for Comparative Effectiveness Research (CER) and Patient-Centered Outcomes Research (PCOR): Challenges and Opportunities; 2013.
Van Der Wees, PJ, Nijhuis-Van Der Sanden, MW, Ayanian, JZ, Black, N, Westert, GP and Schneider, EC. Integrating the use of patient-reported outcomes for both clinical practice and performance measurement: views of experts from 3 countries. Milbank Q. 2014; 92(4): 754–775. DOI: https://doi.org/10.1111/1468-0009.12091
Snyder, CF, Jensen, RE, Segal, JB and Wu, AW. Patient-reported outcomes (PROs): putting the patient perspective in patient-centered outcomes research. Med Care. 2013; 51(8 Suppl 3): S73–79. DOI: https://doi.org/10.1097/MLR.0b013e31829b1d84
Tarlov, AR, Ware, JE, Jr., Greenfield, S, Nelson, EC, Perrin, E and Zubkoff, M. The Medical Outcomes Study. An application of methods for monitoring the results of medical care. JAMA. 1989; 262(7): 925–930. DOI: https://doi.org/10.1001/jama.1989.03430070073033
Reeve, BB, Hays, RD, Bjorner, JB, et al. Psychometric evaluation and calibration of health-related quality of life item banks: plans for the Patient-Reported Outcomes Measurement Information System (PROMIS). Medical care. 2007; 45(5): S22–S31. DOI: https://doi.org/10.1097/01.mlr.0000250483.85507.04
Cella, D, Yount, S, Rothrock, N, et al. The Patient-Reported Outcomes Measurement Information System (PROMIS): progress of an NIH Roadmap cooperative group during its first two years. Medical care. 2007; 45(5 Suppl 1): S3. DOI: https://doi.org/10.1097/01.mlr.0000258615.42478.55
Aaronson, N, Elliott, T, Greenhalgh, J, et al. User’s Guide to Implementing Patient-Reported Outcomes Assessment in Clinical Practice, Version 2. International Society for Quality of Life Research; 2015.
Snyder, CF, Aaronson, NK, Choucair, AK, et al. Implementing patient-reported outcomes assessment in clinical practice: a review of the options and considerations. Qual Life Res. 2012; 21(8): 1305–1314. DOI: https://doi.org/10.1007/s11136-011-0054-x
van Vliet, LM, Harding, R, Bausewein, C, Payne, S and Higginson, IJ. How should we manage information needs, family anxiety, depression, and breathlessness for those affected by advanced disease: development of a Clinical Decision Support Tool using a Delphi design. BMC medicine. 2015; 13(1): 263. DOI: https://doi.org/10.1186/s12916-015-0449-6
Patients Like Me. https://www.patientslikeme.com/ Accessed June 24, 2017.
Reuben, DB and Tinetti, ME. Goal-oriented patient care—an alternative health outcomes paradigm. New England Journal of Medicine. 2012; 366(9): 777–779. DOI: https://doi.org/10.1056/NEJMp1113631
Detmar, SB, Muller, MJ, Schornagel, JH, Wever, LD and Aaronson, NK. Health-related quality-of-life assessments and patient-physician communication: a randomized controlled trial. JAMA. 2002; 288(23): 3027–3034. DOI: https://doi.org/10.1001/jama.288.23.3027
Gonçalves Bradley, DC, Gibbons, C, Ricci-Cabello, I, et al. Routine provision of information on patient-reported outcomemeasures to healthcare providers and patients in clinical practice (Protocol). Cochrane Database of Systematic Reviews. John Wiley & Sons, Ltd.; 2015.
Greenlee, RT, Lamerato, LE and Greene, SM. Advancing Population Health: New Models and the Role of Research. An Overview of the 22nd Annual Health Care Systems Research Network Conference. Journal of Patient-Centered Research and Reviews. 2016; 3(3): 155–158. DOI: https://doi.org/10.17294/2330-0698.1284
Health Care Systems Research Network. http://www.hcsrn.org/en/ Accessed June 24, 2017.
Health, R. 36-Item Short Form Survey (SF-36). https://www.rand.org/health/surveys_tools/mos/36-item-short-form.html Accessed June 25, 2017.
WHO World Health Survey. http://www.who.int/healthinfo/survey/en/ Accessed June 25, 2017.
Health, R. 12-Item Short Form Survey (SF-12). https://www.rand.org/health/surveys_tools/mos/12-item-short-form.html Accessed June 25, 2017.
Kroenke, K, Spitzer, RL and Williams, JB. The Patient Health Questionnaire-2: validity of a two-item depression screener. Medical care. 2003; 41(11): 1284–1292. DOI: https://doi.org/10.1097/01.MLR.0000093487.78664.3C
Kroenke, K, Spitzer, RL and Williams, JB. The phq-9. Journal of general internal medicine. 2001; 16(9): 606–613. DOI: https://doi.org/10.1046/j.1525-1497.2001.016009606.x
Kroenke, K, Spitzer, RL, Williams, JB, Monahan, PO and Löwe, B. Anxiety Disorders in Primary Care: Prevalence, Impairment, Comorbidity, and DetectionAnxiety Disorders in Primary Care. Annals of internal medicine. 2007; 146(5): 317–325. DOI: https://doi.org/10.7326/0003-4819-146-5-200703060-00004
Keller, S, Bann, CM, Dodd, SL, Schein, J, Mendoza, TR and Cleeland, CS. Validity of the brief pain inventory for use in documenting the outcomes of patients with noncancer pain. The Clinical journal of pain. 2004; 20(5): 309–318. DOI: https://doi.org/10.1097/00002508-200409000-00005
Borson, S, Scanlan, JM, Chen, P and Ganguli, M. The Mini-Cog as a screen for dementia: validation in a population-based sample. Journal of the American Geriatrics Society. 2003; 51(10): 1451–1454. DOI: https://doi.org/10.1046/j.1532-5415.2003.51465.x
Galvin, J, Roe, C, Powlishta, K, et al. The AD8 A brief informant interview to detect dementia. Neurology. 2005; 65(4): 559–564. DOI: https://doi.org/10.1212/01.wnl.0000172958.95282.2a
Folstein, MF, Folstein, SE and McHugh, PR. Mini-mental state: A practical method for grading the cognitive state of patients for the clinician. J Psychiatric Res. 1975; 12: 189–198. DOI: https://doi.org/10.1016/0022-3956(75)90026-6
Control CfD. Stopping Elderly Accidents, Deaths, and Injuries. https://www.cdc.gov/steadi/index.html Accessed June 25, 2017.
Craig, P, Dieppe, P, Macintyre, S, Michie, S, Nazareth, I and Petticrew, M. Developing and evaluating complex interventions: the new Medical Research Council guidance. International journal of nursing studies. 2013; 50(5): 587–592. DOI: https://doi.org/10.1016/j.ijnurstu.2012.09.010
Chung, S, Lesser, LI, Lauderdale, DS, Johns, NE, Palaniappan, LP and Luft, HS. Medicare annual preventive care visits: use increased among fee-for-service patients, but many do not participate. Health Affairs. 2015; 34(1): 11–20. DOI: https://doi.org/10.1377/hlthaff.2014.0483
Valderas, JM, Kotzeva, A, Espallargues, M, et al. The impact of measuring patient-reported outcomes in clinical practice: a systematic review of the literature. Qual Life Res. 2008; 17(2): 179–193. DOI: https://doi.org/10.1007/s11136-007-9295-0
Health and Social Care Information Centre. PROMs clinical case study: data informs clinical practice. 2015; http://content.digital.nhs.uk/article/6542/PROMs-clinical-case-study-data-informs-clinical-practice Accessed June 25, 2017.
Nilsson, E, Orwelius, L and Kristenson, M. Patient-reported outcomes in the Swedish National Quality Registers. J Intern Med. 2016; 279(2): 141–153. DOI: https://doi.org/10.1111/joim.12409
Bayliss, EA, McQuillan, DB, Ellis, JL, et al. Using electronic health record data to measure care quality for individuals with multiple chronic medical conditions. Journal of the American Geriatrics Society. 2016; 64(9): 1839–1844. DOI: https://doi.org/10.1111/jgs.14248
Marshall, S, Haywood, K and Fitzpatrick, R. Impact of patient-reported outcome measures on routine practice: a structured review. J Eval Clin Pract. 2006; 12(5): 559–568. DOI: https://doi.org/10.1111/j.1365-2753.2006.00650.x
Ahmed, S, Berzon, RA, Revicki, DA, et al. The use of patient-reported outcomes (PRO) within comparative effectiveness research: implications for clinical practice and health care policy. Med Care. 2012; 50(12): 1060–1070. DOI: https://doi.org/10.1097/MLR.0b013e318268aaff
Tinetti, ME, McAvay, GJ, Chang, SS, et al. Contribution of multiple chronic conditions to universal health outcomes. Journal of the American Geriatrics Society. 2011; 59(9): 1686–1691. DOI: https://doi.org/10.1111/j.1532-5415.2011.03573.x
Working Group on Health Outcomes for Older Persons with Multiple Chronic Conditions. Universal health outcome measures for older persons with multiple chronic conditions. Journal of the American Geriatrics Society. 2012; 60(12): 2333. DOI: https://doi.org/10.1111/j.1532-5415.2012.04240.x
Boyce, MB, Browne, JP and Greenhalgh, J. The experiences of professionals with using information from patient-reported outcome measures to improve the quality of healthcare: a systematic review of qualitative research. BMJ Qual Saf. 2013; 23: 508–518. DOI: https://doi.org/10.1136/bmjqs-2013-002524
Valderas, JM, Fitzpatrick, R and Roland, M. Using health status to measure NHS performance: another step into the dark for the health reform in England. BMJ Qual Saf. 2012; 21(4): 352–353. DOI: https://doi.org/10.1136/bmjqs-2011-000184
Wahl, ER and Yazdany, J. Challenges and Opportunities in Using Patient-reported Outcomes in Quality Measurement in Rheumatology. Rheum Dis Clin North Am. 2016; 42(2): 363–375. DOI: https://doi.org/10.1016/j.rdc.2016.01.008