For one week, the American Evaluation Association (AEA) allowed the Massachusetts Office of the State Auditor to submit daily articles related to our health care cost containment law (Chapter 224) evaluation work. The American Evaluation Association is an international professional association of evaluators devoted to the application and exploration of program evaluation, personnel evaluation, technology, and many other forms of evaluation. Each post was originally published on AEA's blog, AEA365. The week of posts detailed important aspects of conducting a a large scale evaluation. Below is a summary of those posts. Please click each title to read the full submission.
Developing an Evaluation plan
Conducting such a large and complex evaluation required the development of a comprehensive evaluation plan capturing all essential pieces of the law. The evaluation design needed to structure all the different topic areas into a logical and concise plan that guides all activities performed by a multidisciplinary team of researchers and data analytic experts. We proposed a longitudinal mixed-methods quasi-experimental design for the evaluation, aiming to determine the law’s impact on health care costs, access to health care services and quality of care, the health care workforce, and the impact on public health. Another important task to address in the evaluation plan is collecting and securing relevant data that will allow the team to perform both quantitative and qualitative analysis.
Using Mixed-methods for Complex Evaluations
In order to capture the broad perspectives of key stakeholders and integrate them with the quantitative analysis, we developed a longitudinal mixed-methods quasi-experimental design for our work in this complex evaluation. This mixed-methods approach is designed to answer Chapter 224 high-level evaluation questions. Activities include:
- open-ended semi-structured interviews through an online survey with key stakeholders.
- In-depth semi-structured interviews as follow-up with selected key stakeholders based on their responses to the online survey.
- face-to-face in-depth semi-structured interviews to explore selected stakeholders interpretation of the quantitative analysis results.
Read our post on using mixed-methods for complex evaluations.
Securing Data
For our Chapter 224 work, we are in various stages of accessing and securing data. These data sets include administrative data (e.g. the All Payer Claims Database (APCD) from the state’s Center for Health Information and Analysis (CHIA) and survey data such as the Behavioral Risk Factor Surveillance Survey (BRFSS) from the MA Department of Public Health. In this post we shared some lessons we have learned.
Using GIS Maps as Proxy for Race/Ethnicity
Using administrative data with a significant number of missing key variables (e.g. race/ethnicity) can be challenging in trying to answer specific evaluation questions. One of our research questions seeks to evaluate the impact of Chapter 224 on racial/ethnic disparities in health outcomes. Both administrative data sets available to us (the Massachusetts Medicaid Program (MassHealth) and the All Payer Claims Database (APCD)) have sparsely populated information about race and/or ethnicity. Since we were not able to apply imputation techniques due to the large number of missing values, we have to use alternative methods. As a proxy for race/ethnicity data, we used US Census Bureau Data and GIS mapping software.
Read our post about using GIS maps as a proxy for race/ethnicity.
Interrupted Time-Series Procedure
A strategy that has been used successfully to investigate the effect of policy changes over time on population outcomes is Interrupted Time-Series (e.g. Ramsey, et al., 2003). Since our project requires us to collect multiple data points previous to the implementation of the Chapter 224 law, as well as several points after, Interrupted Time-Series, for aggregate data, will allow us to determine whether the health care costs containment intervention has an effect significantly greater than a secular trend. By using this method, we will be primarily testing the change in the slope of data trends as a function of Chapter 224.
Using Predictive Modeling in Creating Baseline
In order to implement a comprehensive evaluation on the impact of Chapter 224, our research team is committed to find the most effective statistical procedures to use in order to create a solid baseline that can lead to a sound quantitative analysis of the entire longitudinal project. One of the procedures we found useful is predictive modeling.
Predictive modeling is a process of fitting a statistical model based on existing relationships among variables and making an informed prediction for future behavior(s). In our study, a primary goal is to find the trend or pattern of our variables using a number of data points before the implementation of Chapter 224 (prior to implementation in 2012) as a baseline. We then use the parameter estimates of the baseline to predict the values after 2013. A solid baseline with predicted values will be compared with the actual data once we conclude the longitudinal quantitative analysis.
Read our post about using predictive modeling in creating baseline.
Engaging Stakeholders and Experts as Research Participants and/or as Members of the Advisory Committee
Given the wide scope of the Chapter 224 evaluation project, the research team needed to tap into the healthcare community for its advice, perspectives, and feedback. To this end, we assembled an advisory committee comprised of members from the health insurance industry, health care advocates, academia, labor, and professional organizations to gain diverse input.
In addition, stakeholders will be included as research participants in qualitative interviews complementing the quantitative component of the study. Stakeholders who belong to task forces, councils, commissions, agencies, boards and other groups associated with the enactment of Chapter 224 will be asked to participate in a brief on-line survey to assess initial observations and concerns related to the roll out of Chapter 224 legislation.