The Department of Higher Education’s Administration of its Foster Child Grant Program, Foster Child Tuition Waiver and Fee Assistance Program, and No Interest Loan Program Needs Improvement.

Auditors found a lack of performance measures hinders the department's ability to assess the success of these programs.

Overview

The Department of Higher Education’s (DHE’s) administration of its Foster Child Grant Program, Foster Child Tuition Waiver and Fee Assistance Program, and No Interest Loan (NIL) Program lacked important elements of proper program administration. Specifically, although DHE effectively managed key elements of these programs such as identifying target populations and establishing procedures and standard forms related to program enrollment, it did not establish program performance measures or routinely analyze program data related to program performance, such as the academic progress, graduation rates, dropout rates, and post-college activities of recipients. Without performing these important administrative activities, DHE lacks the ability to assess the impact of these programs or the extent to which they achieve their goals and desired outcomes and to identify the extent of any problems or barriers to success in the programs so they can be addressed in a timely manner.

Authoritative Guidance

DHE’s website states,

The Board [of Higher Education, or BHE], in conjunction with colleges’ Boards of Trustees, holds [DHE] accountable for achieving its goals and establishing a comprehensive system to measure quality by defining educational achievement and success with the use of standards and measurements.

Performance measures define success and indicate progress toward meeting a program’s goals. The federal government and various state governments have recognized the importance of establishing performance goals and measures for assessing the effectiveness of their programs. Examples of this include the following:

• The federal government enacted the Government Performance Results Act in 1993 to ensure that evaluations of its programs’ performance were performed. Federal departments are now required to assess the effectiveness of their programs.
• The Colorado Department of Education has initiated performance measures whereby achievement and educational outcomes of students in foster care, such as four-year high-school completion rates, are reported to its Foster Care Education Program.²
• The state of Washington, in its budget law, requires agencies to measure performance and report measures to its Office of Financial Management. These measures include linking performance measures to results or improvements expected from an investment of resources.
• The state of Georgia has initiated performance measures for state programs. The intent of these metrics is to measure performance and to give agency stakeholders an understanding of how efficiently programs are operated.

We believe these represent best program administration practices that DHE should follow to measure such things as whether program goals are met and to what extent positive outcomes are achieved.

Reasons for Noncompliance

DHE officials told us that although it does establish and assess performance measures for institutions, the legislation that established the Foster Child Grant Program, Foster Child Tuition Waiver and Fee Assistance Program, and NIL Program did not establish metrics or specific data collection requirements that could be used to assess overall program outcomes or the performance of students participating in each of these programs; therefore, it has not established specific policies or procedures for these purposes. These officials added that DHE would have to work with BHE and the state Legislature to determine what the metrics should be. DHE officials further stated that they do not believe they would have sufficient staff or informational resources to effectively perform additional administrative responsibilities.

Recommendations

1. DHE should work with BHE and the Legislature to identify a system for tracking and analyzing data as well as to define key success metrics.
2. DHE should work to identify any program changes that need to be made to enhance program success.

Auditee's Response

The Department welcomes ideas and suggestions on ways that it can continuously improve program administration and effectiveness. The Department also agrees in principle that establishing performance measures to assess the impact of programs is important.

To this point, the Department has, consistent with its legislative mandate, established performance measures that are focused on assessing the performance of public higher education institutions, and the performance of the public higher education system as a whole—rather than the performance of individual financial aid accounts. See M.G.L. c. 15A, § 7A. Current performance measures used by the Department include assessing institutional performance on outcomes related to college/university affordability, student access, student success, fiscal efficiency, employment outcomes as well as other areas. The data that the department collects in this regard is made available to campuses annually with the expectation that institutions use the data, in consultation with the Department, for strategic planning purposes as well as budget and program development. The Commissioner also uses the data to annually evaluate Presidential performance. The Department also generates public accountability and assessment reports that demonstrate our commitment to continuous improvement.

The Department is committed to enhancing its existing Performance Measurement System. To that end, and in collaboration with the campuses, the Department is undertaking a comprehensive review and revision of the Performance Measurement System with a charge from the Board of Higher Education (BHE) to improve the accuracy and relevancy of accountability metrics in the areas of student access, success, and academic progression; affordability; learning outcomes and educational quality; efficiency and stewardship of fiscal resources; employment and post-baccalaureate experiences; and achievement and opportunity gaps. These revisions to the Performance Measurement System are intended to be published in January 2019, and will include a summary of institutional, segmental and system performance outcomes.

As Department staff and the audit team staff discussed early in the audit process, establishing metrics for individual programs that show direct causality—e.g., that funding these particular programs result in particular student outcomes—would be challenging, since success is driven by many factors. For example, while providing students with funding for tuition and fees can increase access to higher education opportunities, other factors such as the availability of academic advising, transportation, child care, and peer mentoring also contribute greatly to student success measures, such as graduation rates.

In addition to the statutory mandate and ongoing efforts to evaluate institutional and segmental performance, the Department would like to point out that it routinely engages in data merges with other state and national agencies for the purpose of assessing and improving student performance in several other ways. For example, through interagency data sharing agreements with the Department of Unemployment Assistance, the Department has been able to track our college and university graduates into the workforce and assess their employment success in the Commonwealth.

Finally, the Department recently completed a comprehensive review of its financial aid programs with the assistance of a consultant. The study was initiated by the Department in February 2016, and completed in the Fall of 2017, with findings presented to the Board of Higher Education (BHE) on March 6, 2018. . . . The study produced a set of recommendations for reforming and consolidating state financial aid programs to improve student success outcomes and to help the Commonwealth move forward on the Department’s three priority objectives which form the underpinnings of its strategic plan for the public higher education system: 1) making college more accessible and affordable for all Massachusetts residents; 2) closing gaps in student opportunity and achievement; and 3) improving college completion rates. The BHE charged the Commissioner with developing policy recommendations, including an implementation plan for the redesign of state financial aid programs aligned with the findings of the report. As he prepares that implementation plan, the Commissioner will give careful consideration to the auditor’s recommendations on performance measurement as it relates to program administration.

Auditor's Reply

The Office of the State Auditor (OSA) does not dispute that DHE has established performance measures that are focused on assessing the performance of public higher education institutions and the performance of the public higher education system as a whole. However, OSA believes that establishing relevant program-specific performance measures and routinely analyzing program data related to program performance (such as the academic progress, graduation rates, dropout rates, and post-college activities of recipients) would provide meaningful information. This information would not only help DHE effect proper administration of the programs in question, but also provide a higher level of transparency to both the Legislature and the public as to what impact, if any, the expenditure of tax dollars on the programs’ activities has on achieving desired outcomes.

Further, we do not dispute that establishing metrics that show direct causality for individual programs would be challenging. However, in OSA’s opinion, proper program administration requires that such metrics be established and that the impact of a program, or the extent to which it achieves its goals and desired outcomes, be assessed. The benefit of assessing a program’s impact is widely accepted by government entities that have been charged with administering programs such as DHE’s that are intended to have a social impact. For example, as noted above, the federal government enacted the Government Performance Results Act in 1993 to ensure that its programs’ performance was evaluated.

Based on its response, DHE is taking measures to address our concerns in this area.

2. The Foster Care Education Program at the Colorado Department of Education was launched in 2012 to support local education agencies and communities in serving students.

Date published:	August 29, 2018