Research design summary for the evaluation of the Health Care Cost Containment Law

Learn more about the Office of the State Auditor's research design to evaluate the impacts of Chapter 224

Each chapter in the Office of the State Auditor's (OSA) report on Chapter 224 answers a research question presented in Section 251, as noted below. Chapter 4, which was subcontracted to Commonwealth Corporation, addresses several questions.

Research Question for Chapter 1: What are the changes to health care costs, including the extent to which savings have reduced out-of-pocket costs to individuals and families, health insurance premium costs, and health care costs borne by the Commonwealth?

Research Question for Chapter 2: What are the changes to access to health care services and quality of care in different regions of the state and for different populations, particularly for children, the elderly, low-income individuals, individuals with disabilities, and other vulnerable populations?

Research Question for Chapter 3: What are the changes to access and quality of care for specific services, particularly primary care and behav­ioral health (which includes substance use disorders and mental health services)?

Research Questions for Chapter 4: How did the industrial, occupa­tional, and geographic structure of health care employment in the Commonwealth change?

1. What is the proper definition of the health care industry (in statistical terms) to measure the size and composition of the state’s health care workforce?
2. What is the impact of structural changes in the health care industry on skill requirements for employment in the state’s health care delivery system as well as impacts on earnings?
3. How did access to employment for racial/ethnic groups, dependence on foreign-born workers for labor supply in some health care occupa­tions, and “benefit cliff effects” on labor supply choices in occupations in which substantial shares of workers participated in non-cash in­come transfer programs change?
4. What is the most likely future growth path for employment in the health care service sector?

Research Question for Chapter 5: What are the changes to public health, including, but not limited to, reducing the prevalence of preventable health conditions, improving employee wellness, and reducing racial/eth­nic disparities in health outcomes?

To respond to these questions, OSA developed a mixed-methods (quan­titative and qualitative), quasi-experimental design for the evaluation. The study explored Chapter 224’s impact on the following:

• Health care costs, access to health care services, and quality of care in different regions of the Commonwealth and for particular populations,
• Access and quality of care for specific services,
• The health care workforce, and
• Public health.

Because the study touched on numerous matters related to health, health systems, population health, and fiscal policy, OSA sought data from many secondary sources, mainly state and federal agencies. OSA conducted unique analyses of datasets from several of these sources, including Massachusetts’ All-Payer Claims Database (APCD), the Massachusetts Department of Public Health, and the Massachusetts Health Reform Survey.

OSA also extensively utilized peer-reviewed and other sources such as the Substance Abuse and Mental Health Services Administration, the National Survey on Drug Use and Health, Centers for Disease Control and Prevention, Healthcare Effectiveness Information and Data, and reports from foundations, including the Kaiser Family Foundation and the Blue Cross Blue Shield of Massachusetts Foundation.

Quantitative Methods

OSA used of a variety of statistical methods for its quantitative research, as follows:

• The logistic regression model to estimate the probability of the dichot­omous outcome variables;
• The method of generalized estimating equations to analyze longitudi­nal data, which accounts for the correlation inherent in using multiple observations for each individual;
• For group comparisons: the Chow test to test whether the coefficients estimated for one group are equal to those for another group;
• For the survey data: complex sampling procedures, including state­ments for stratification, clustering, and sample weights; and
• For mortality data: age-adjusted rates calculated by using the 2010bridged-race population estimates file and the 2015 bridged-race post­censal estimates file, both produced by the National Center for Health Statistics. The rates were then age-adjusted to per-100,000 of the 2000 U.S. Standard Population.

Qualitative Methods

Qualitative study components included two elements:

1. a brief online survey with key stakeholders, published in fall 2015¹ ; and
2. in-depth, semi-structured interviews with key stakeholders, excerpts from which appear as quotations throughout this report.

¹ Reynoso-Vallejo, H., Porche, M., & Stuck-Girard, C. (2015). Chapter 224: stakeholders study. Retrieved April 18, 2017, from mass.gov/auditor/docs/chapter-224/100615-224-stakeholders-survey.pdf

 

Study Limitations

OSA encountered several barriers while attempting to conduct its analyses:

• First, for many health care domains (such as people with disabilities), a paucity of longitudinal data is available to show Massachusetts trends. In addition, some data are available for only group subsets, which do not necessarily reflect trends among the entire group. For example, among people enrolled in MassHealth, many measures used data from the Healthcare Effectiveness Data and Information Set, which reflects only the MassHealth managed care population (approx­imately 60% of MassHealth enrollees).
• Second, although OSA obtained APCD claims data, data from earlier than 2010 were not available, which impacted the accuracy of some measures, including cancer screenings.
• Third, in some cases, available data were insufficient to calculate whether observed trends were statistically significant.
• Another major limitation was OSA’s inability to control for the impact of societal changes and contemporary policy reforms, most importantly Chapter 58 of the Acts of 2006 and the Patient Protection and Affordable Care Act of 2010 (ACA). In addition to these policy changes, other contextual influences, such as an improving economy and societal shifts relating to risk factors (including rates of tobacco use and obesity), contributed to the trends reported here. These limitations in the quality and breadth of the available data prevented OSA from identifying and allocating causal relationships.
• Moreover, many provisions of Chapter 224 had little to no time to take root as of the time of OSA’s analyses, as follows:
• The Health Policy Commission launched certification programs for patient-centered medical homes and accountable-care organizations in 2016 and 2017, respectively.
• The law’s call for transparency among prices of hospital services re­mains aspirational, though the Center for Health Information and Analysis plans to debut a medical pricing website in 2017.
• A mandated price-variation commission was replaced with a special commission on price variation (composed of legislators, governor’s ap­pointees, and representatives from stakeholder groups), which re­ported its findings in March 2017.
• The Pharmaceutical Cost Commission and the Diagnostic Accuracy Task Force proscribed by the law have not convened, and a report on telemedicine due in 2013 has not been issued.

If and when these and other provisions are implemented, it may take sev­eral years for their effects to be observed in longitudinal data. Therefore, OSA’s analysis should be viewed as a provisional and not a final verdict on the impact of Chapter 224.

Finally, it is important to note that OSA finalized the content of this re­port starting in late 2016, so it may not reflect subsequent developments in relevant federal and state policy.