Audit of the Department of Children and Families Objectives, Scope, and Methodology

An overview of the purpose and process of auditing the Department of Children and Families.

Overview

In accordance with Section 12 of Chapter 11 of the Massachusetts General Laws, the Office of the State Auditor (OSA) has conducted a performance audit of certain activities of the Department of Children and Families (DCF) for the period July 1, 2019 through December 31, 2023.

We conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Below is a list of our audit objectives, indicating each question we intended our audit to answer; the conclusion we reached regarding each objective; and, if applicable, where each objective is discussed in the audit findings.

Objective	Conclusion
Did DCF obtain required court approval for antipsychotic medications and document its consent for psychotropic medications prescribed to children in its protective custody as required by Sections 11.14(3)(a), (4)(a), and (6)(a) of Title 110 of the Code of Massachusetts Regulations and DCF Policy 2010-001: Medical Examinations for Children Entering DCF Placement or Custody?	No; see Findings 1 and 2c
Did children in DCF’s protective custody receive follow-up visits and recommended psychosocial services in conjunction with prescriptions for psychotropic medications in accordance with the American Academy of Child and Adolescent Psychiatry’s 2005 “Position Statement on Oversight of Psychotropic Medication Use for Children in State Custody: A Best Principles Guideline” and the American Academy of Child and Adolescent Psychiatry’s 2015 Recommendations about the Use of Psychotropic Medications for Children and Adolescents Involved in Child-Serving Systems?	No; see Findings 3 and 2b
Did DCF maintain medical passports for children in its protective custody who received psychotropic medications according to DCF Policy 85-003: Health Care Services to Children in Placement, DCF Policy 86-011: Ongoing Casework and Documentation, DCF Policy 2010-001, and Section 475 of the Social Security Act?	No; see Finding 2a
Did DCF provide oversight to children in its protective custody who received psychotropic medications that exceeded the US Food and Drug Administration’s (FDA’s) recommended maximum dosages?	No; see Other Matters
Did DCF update its internal control plan to address the COVID-19 pandemic as required by the Office of the Comptroller of the Commonwealth’s “COVID-19 Pandemic Response Internal Controls Guidance,” and did DCF ensure that employees who had access to COVID-19 funds completed cybersecurity awareness training in accordance with the Executive Office of Technology Services and Security’s (EOTSS’s) Information Security Risk Standard IS.010?	No; see Finding 4
Did DCF make outreach efforts to ensure that it reached eligible youths who aged out of DCF care to allocate funds from the John H. Chafee Foster Care Independence Program grant as required by the grant agreement ACYF-CB-PI-21-04?	Yes

To accomplish our audit objectives, we gained an understanding of the aspects of DCF’s internal control environment relevant to our objectives by reviewing its internal control plan and applicable policies and procedures, as well as by interviewing DCF employees and management.

To obtain sufficient, appropriate evidence to address our audit objectives, we performed the procedures described below.

Sampling Strategies for Children in DCF’s Protective Custody

DCF provided us with a list of all 3,899 children in its protective custody who received at least one psychotropic medication during the audit period. We separated this list based on whether the child was enrolled in a fee-for-service (FFS) plan (3,204 children) or managed care organization (MCO) plan (695 children). This separation allowed us to further refine the FFS population based on our detailed access to claim information.

• First, we removed claims for children who were receiving psychotropic medications that exceeded the FDA’s recommended maximum dosages, which totaled 299 children, bringing the population from 3,204 to 2,905 (see the “Maximum Dosages” section of this report).
• We removed the claims for children in the FFS plan population of 2,905 who had fewer than 24 claims for psychotropic medications (meaning they were not prescribed the medication during the length of the audit period). This gave us a population of 1,180 children in DCF’s protective custody with an FFS plan who consistently received at least one psychotropic medication during the audit period.

For Objectives 1, 2, and 3, we selected a random, statistical sample⁷ of 118 out of 1,180 children in DCF’s protective custody enrolled in an FFS plan and who were prescribed at least one psychotropic medication, using a 95% confidence level,⁸ a 50% expected error rate,⁹ and a 15% desired precision range.¹⁰

In addition, we selected a random, nonstatistical sample¹¹ of 50 out of 695 children in DCF’s protective custody enrolled in a managed care plan who were prescribed at least one psychotropic medication.

For Objective 4, we selected a random, nonstatistical sample of 40 out of 299 children in DCF’s protective custody in an FFS plan who, based on Medicaid Management Information System (MMIS) data, received psychotropic medications that exceeded the FDA’s recommended maximum dosages.

Rogers Guardianship Orders and Department Consent

To determine whether DCF obtained required court approval for antipsychotic medications and documented its consent for psychotropic medications prescribed to children in its protective custody, we took the following actions:

We met with DCF officials to go through its iFamilyNet (iFN) system to gain an understanding of where all the Rogers guardianship orders should be located in iFN. Next, we reviewed the MMIS All Services Report¹² for each child listed in both our FFS and MCO samples and identified the claims for psychotropic medication(s) for these children to determine whether each child was prescribed an antipsychotic and/or another type of psychotropic medication. (As previously stated, any child prescribed an antipsychotic medication required a Rogers guardianship order, and any child prescribed another psychotropic medication required DCF consent. We determined whether Rogers guardianship orders and/or DCF consent forms were included in iFN. In addition, for the Rogers guardianship orders not documented in iFN, we requested and reviewed hard copies of the Rogers guardianship orders from DCF, who reached out to juvenile courts in order to provide them to us.

Based on the results of our testing, we determined that, during the audit period, DCF did not always obtain or renew required court approval before children in its protective custody were administered antipsychotic medications and did not properly maintain Rogers guardianship orders in iFN. See Findings 1 and 2c for more information.

Psychosocial Services and Follow-up Visits

To determine whether children in DCF’s protective custody who were prescribed psychotropic medications received follow-up doctor appointments and recommended psychosocial services, we took the following actions:

• We inspected the MMIS All Services Reports for the children in both our FFS and MCO samples and identified all psychosocial service claims and follow-up doctor appointment claims.
• We compared the services in the MMIS All Services Reports to what DCF documented in iFN by searching social worker notes, encounter forms, and medical passports for evidence that the children in our samples received psychosocial services and follow-up doctor appointments in conjunction with prescribed psychotropic medications.

Based on the results of our testing, we determined that, during the audit period, children did not always receive recommended psychosocial services in conjunction with their prescriptions for psychotropic medications and DCF did not properly maintain healthcare records in iFN. See Findings 2b and 3 for more information.

Medical Passports

To determine whether DCF maintained medical passports for children in its protective custody who received psychotropic medications, we took the following actions. We inspected the medical passport in iFN for each child in both our FFS and MCO samples to determine whether the medications and dosages listed on each medical passport matched the medications and dosages in each child’s MMIS All Services Report.

Based on the results of our testing, we determined that DCF did not properly maintain healthcare records in iFN during the audit period. See Finding 3 for more information. 

Maximum Dosages

To determine whether DCF provided oversight to children in its protective custody who received psychotropic medications that exceeded the FDA’s recommended maximum dosages, we reviewed each child’s full medical record in iFN for the sample of 40 and determined whether children were over the maximum dosage. We further determined whether DCF documented its consent for the psychotropic medication and the reasoning for the high dosage.

Based on the results of our testing, we determined that DCF should provide more oversight for children in its custody receiving psychotropic medication in amounts and dosages that exceed the FDA’s recommendations. See Other Matters for more information.

Internal Control Plan and Cybersecurity Awareness Training

To determine whether DCF updated its internal control plan to address the COVID-19 pandemic, as required by the Office of the Comptroller of the Commonwealth’s “COVID-19 Pandemic Response Internal Controls Guidance,” we reviewed DCF’s fiscal year 2020 internal control plan.

To determine whether DCF ensured that employees who had access to COVID-19 funds during the audit period completed annual refresher and/or initial cybersecurity awareness training in accordance with EOTSS’s Information Security Risk Management Standard IS.010, we obtained and inspected transcript reports of cybersecurity awareness training for all 10 employees who had access to COVID-19 funds.

Based on the results of our testing, we determined that, during the audit period, DCF updated its internal control plan to address the COVID-19 pandemic. However, DCF did not ensure that all employees with access to COVID-19 funds received annual refresher cybersecurity awareness training during the audit period. See Finding 4 for more information. 

John H. Chafee Foster Care Independence Program Grant

To determine whether DCF made outreach efforts to ensure that it reached eligible youths who aged out of DCF care in order to allocate funds from the John H. Chafee Foster Care Independence Program grant, we interviewed DCF management and requested and reviewed DCF’s plan for reaching out to youths eligible for these funds.

We noted no exceptions in our testing; therefore, we determined that, during the audit period, DCF made outreach efforts to ensure that it reached eligible youths who left DCF care to allocate funds from the John H. Chafee Foster Care Independence Program grant.

We used a combination of statistical and nonstatistical sampling methods for testing, and we did not project the results of our testing to any corresponding populations.

Data Reliability Assessment.

We received from DCF a list of children in DCF’s protective custody who were prescribed at least one psychotropic medication during the audit period and separated the list by children who were enrolled in an FFS plan and children who were enrolled in an MCO plan. We performed validity and integrity tests first on the list of children enrolled in an FFS plan, including (1) testing for blank fields, (2) scanning for duplicate records, and (3) tracing a sample of 20 children on the list to protective custody court documents. We also performed validity and integrity testing on the children enrolled in a MCO plan, including (1) testing for blank fields, (2) scanning for duplicate records, and (3) tracing a sample of 25 children on the list to their MCO plan from MMIS. To verify the completeness of the list provided by DCF, we attempted to extract a list from MMIS of all children in DCF’s protective custody who were prescribed at least one psychotropic medication. However, MMIS does not distinguish different levels of custody, and the focus of our audit objective was children in DCF’s protective custody. Therefore, we used the list provided by DCF as it was the only source of data available.

To determine the reliability of the data from MMIS, we relied on the work performed by OSA, completed in 2020, that tested certain information system controls in MMIS. As part of that work, OSA reviewed existing information, tested selected system controls, and interviewed knowledgeable agency officials about the data. As part of this audit, we performed validity and integrity tests on all MMIS data, including (1) testing for blank fields, (2) scanning for duplicate records, (3) looking for dates outside of the audit period, and (4) determining whether each child’s age on the date of service was between 0 and 17.

We also received from DCF a list of employees who had access to COVID-19 funds. We performed validity and integrity tests on this list, including (1) testing for duplicates, (2) testing for blank fields, and (3) tracing the list back to a list that we generated from the Human Resources Compensation Management System, which is the Commonwealth’s official payroll system.

Based on the results of the data reliability assessment procedures described above, we determined that the information we obtained was sufficiently reliable for the purposes of our audit.

Other Matters

During this audit, additional areas of concern that were outside the original scope of our objectives came to our attention. Given the high-risk nature of these areas, we examined them while we were still engaged with the auditee. These areas include human trafficking prevention measures, as well as DCF’s implementation of recommendations by the Massachusetts Commission on LGBTQ Youth.

We emailed DCF to determine what corrective measures the agency took, or was taking, to address the findings and concerns referenced in the US Department of Health and Human Services Office of Inspector General (HHS OIG) report that cited Massachusetts as one of five states where there was no evidence that children in foster care were screened for human trafficking after they had gone missing from, and later returned to, foster care. We also inquired about several related follow-up questions with DCF that addressed the detection and prevention of human trafficking.

In addition, we inquired about the Massachusetts Commission on LGBTQ Youth’s 2023 annual report with DCF and whether it had implemented the recommendations

The results of this work are included within the “Other Matters” section of this audit report.

7.    Auditors use statistical sampling to select items for audit testing when a population is large (usually over 1,000) and contains similar items. Auditors generally use a statistics software program to choose a random sample when statistical sampling is used. The results of testing using statistical sampling, unlike those from judgmental sampling, can usually be used to make conclusions or projections about entire populations.

8.    Confidence level is a mathematically based measure of the auditor’s assurance that the sample results (statistic) are representative of the population (parameter), expressed as a percentage.

9.    Expected error rate is the number of errors that are expected in the population, expressed as a percentage. It is based on the auditor’s knowledge of factors such as prior year results, the understanding of controls gained in planning, or a probe sample.

10.    Desired precision range is the range of likely values within which the true population value should lie; also called confidence interval. For example, if the interval is 90%, the auditor will set an upper confidence limit and a lower confidence where 90% of transactions fall within those limits.

11.    Auditors use judgmental (i.e., nonstatistical) sampling to select items for audit testing when a population is very small, the population items are not similar enough, or there are specific items in the population that the auditors want to review. Auditors use their knowledge and judgment to select the most appropriate sample. For example, an auditor might select items from areas of high risk. The results of testing using judgmental sampling cannot be used to make conclusions or projections about entire populations; however, they can be used to identify specific issues, risks, or weaknesses.

12.    Each MMIS All Services Report documents the healthcare services that MassHealth has paid for a specific member, including pharmacy services.

 

Date published:	November 7, 2024