• This page, Audit of the Department of Career Services Objectives, Scope, and Methodology, is   offered by
  • Office of the State Auditor

Audit of the Department of Career Services Objectives, Scope, and Methodology

An overview of the purpose and process of auditing the Department of Career Services

Table of Contents

Overview

In accordance with Section 12 of Chapter 11 of the Massachusetts General Laws, the Office of the State Auditor has conducted a performance audit of certain activities of the Department of Career Services (DCS) for the period July 1, 2014 through December 31, 2016.

We conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Below is a list of our audit objectives, indicating each question we intended our audit to answer and the conclusion we reached regarding each objective.

Objective

Conclusion

  1. Does DCS administer an effective job placement system? Specifically,

 

a. Do job placement reports accurately reflect jobs obtained, and is DCS meeting its established performance goals in this area?

Yes

b. Are users of the JobQuest system (job seekers and employers) able to navigate the system effectively?

Yes

 

We gained an understanding of DCS’s internal control environment related to our audit objectives by reviewing applicable laws, regulations, and agency policies and procedures, as well as conducting inquiries with agency management. We tested the operating effectiveness of controls over Massachusetts One-Stop Employment System (MOSES) access and data accuracy by comparing the dates when One-Stop Career Center employees received MOSES training to the dates when they were given access to MOSES. During our control testing, we identified an issue related to staff training on this system (Finding 1).

To achieve our objectives, we performed the following audit procedures:

  • We conducted two sets of One-Stop Career Center visits, visiting a total of eight centers, and met with DCS staff members at each center. While visiting the first group of six centers, we attended seminars and programs in which Department of Unemployment Assistance (DUA) benefit recipients were required to participate. We also observed a career counselor assisting job seekers, as well as the general interaction of DCS staff members with job seekers. During the second set of visits to six career centers, we interviewed managers and directors of the centers to obtain a better understanding of JobQuest usability and the controls associated with the validity of MOSES data. We reviewed the process by which data were entered in JobQuest to gain a better understanding of data entry, user access, and system navigability. We asked career center managers and directors how MOSES data were validated for accuracy, including how the managers and directors reviewed reports generated from MOSES to look for outliers in the data (such as people identified as employed when the staff knows that they are still seeking employment). We interviewed managers to obtain further details on the monitoring techniques performed by each office, to provide additional confirmation that the data in MOSES were valid and agreed with the data in performance reports.
  • We used MOSES to develop a geographic representation of jobs obtained to assess whether any regions appeared to be underperforming.
  • We filtered all MOSES transactions by user identification number to determine the number of individuals who obtained services. We used data analytics to filter, summarize, and classify various DCS measures to determine whether specific people were hired. We filtered by career center and result (e.g., “hired,” “not hired”); summarized by individual identification number and career center to obtain the number of occurrences; and classified by career center (and in some instances by individual identification number) to compare one career center to another.
  • We used data analytics to compute jobs attained as a percentage of all transactions in MOSES to corroborate the jobs listed as obtained in the Workforce Investment Act (WIA) Title I Performance Measures Report for the quarter that ended June 30, 2016.
  • We obtained an understanding of the extent to which JobQuest users across the state could effectively navigate the JobQuest portal; we did this by looking at the number of DCS employment services users needed to obtain before they found a job.
  • To validate the accuracy of the jobs listed as obtained in the WIA Title I Performance Measures Report for the quarter that ended June 30, 2016, we selected a nonstatistical sample of 35 out of 950 job seekers in the report and verified that they obtained employment. We did not project the results of this sample to the population. We verified that each of these people was a program exiter.3 We confirmed with DCS that each of the exiters was wage-matched4 by DUA by interviewing DUA’s supervisor of quality assurance to gain an understanding of the process DUA used to verify that a person had earned wages in the selected period. We also reviewed a sample of DUA’s internal annual audit reports to obtain assurance that the DUA wage-match process could be relied on.
  • We verified that all 78 of the supplemental employments5 that were listed in the WIA Title I Performance Measures Report were also listed in MOSES, to verify that each entry in the quarter after a person left the program indicated that the person had obtained employment.

We performed a data reliability analysis of the approximately 6,834,000 transactions recorded in MOSES during our audit period to verify that there were no duplicate transactions or gaps in service and that all dates of service fell within our audit period. We also selected a sample of 50 people with 20 or more transactions each and verified that the descriptions and results represented by the transactions were consistent with the types of service typically delivered by One-Stop Career Centers.

3.    An exiter is a job seeker who has not used any DCS services for 90 consecutive days.

4.    DCS sends requests for employment verification to DUA for job seekers who have exited DCS services to determine whether they have found employment. Once someone is identified as an exiter and an exit date is determined, DCS requests a wage match for the person from DUA to determine whether s/he earned wages in the quarter after exit, which would indicate that s/he had obtained employment.

5.    A supplemental employment in DCS’s performance report is an exiter for whom DUA was not able to verify wages in the quarter after exit. These people are credited in the report as having obtained employment if the MOSES database contains one of the four entries (“employed,” “hired,” “found employment,” and “obtained employment”) that would indicate that they have obtained employment.

Date published: September 26, 2018

Help Us Improve Mass.gov  with your feedback

Please do not include personal or contact information.
Feedback