Audit of the University of Massachusetts Chan Medical School Objectives, Scope, and Methodology

An overview of the purpose and process of auditing the University of Massachusetts Chan Medical School.

Overview

In accordance with Section 12 of Chapter 11 of the Massachusetts General Laws, the Office of the State Auditor has conducted a performance audit of certain activities of the University of Massachusetts (UMass) Chan Medical School (Chan) for the period July 1, 2021 through December 31, 2022.

We conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Below is a list of our audit objectives, indicating each question we intended our audit to answer; the conclusion we reached regarding each objective; and, if applicable, where each objective is discussed in the audit findings.

Objective	Conclusion
1. Did UMass Chan execute all bank card purchases in accordance with Sections IV(A) and (B) within Appendix C of the “University of Massachusetts Business and Travel Expense Policy” (document T92-031); Articles I and II of the “University of Massachusetts Administrative Standards for Business and Travel Expense Policy”; and Sections 2, 4–8, 11, 12, 15, and 21 of the UMass Bank Card Use Standard?	No; see Finding 1
2. Did UMass Chan adhere to its “Privacy and Security Training Policy” regarding cybersecurity awareness training?	Partially; see Finding 2

To accomplish our audit objectives, we gained an understanding of the aspects of UMass Chan’s internal control environment relevant to our objectives by reviewing applicable policies and procedures and by interviewing UMass Chan and UMass system management.

To obtain sufficient, appropriate evidence to address our audit objectives, we performed the procedures described below.

Bank Card Purchases

To determine whether UMass Chan executed all bank card purchases in accordance with Sections IV(A) and (B) within Appendix C of the “University of Massachusetts Business and Travel Expense Policy” (document T92-031); Articles I and II of the “University of Massachusetts Administrative Standards for Business and Travel Expense Policy”; and Sections 2, 4–8, 11, 12, 15, and 21 of the UMass Bank Card Use Standard, we performed the actions described below.

We distributed the total population of 41,848 bank card transactions made during the audit period, totaling $12,326,504, into the three categories described below.⁵

Category Number	Category Description	Number of Transactions	Total Dollar Value of Transactions
1	$7,500 or Higher	25	$      232,541
2	Online (i.e., Amazon, eBay, PayPal), Food, and Grocery Vendors	8,118	1,370,060
3	All Remaining Transactions*	33,705	10,723,903
	Total	41,848	$ 12,326,504

*      This includes transactions that did not fit into the two previous categories. Examples include laboratory materials, books, subscriptions, hardware, and marketing items.

The method we used to select our sample, which consisted of 128 bank card transactions totaling $281,771, is as follows:

• From category one, we selected all 25 transactions, which totaled $232,541.
• From categories two and three, we used a 95% confidence level,⁶ a 50% expected error rate,⁷ and a 20% desired precision range⁸ to determine that our sample should consist of, at a minimum, 103 transactions. We then randomly selected the following:
• From category two, we selected 20 transactions (out of 8,118 transactions), which totaled $10,102.
• From category three, we selected 83 transactions (out of 33,705 transactions), which totaled $39,129.

See the following sections for actions we took with our sample of 128 transactions.

Submission of Bank Card Transaction Documents

For each of the 128 transactions in our sample, we performed the actions described below.

To determine whether UMass Chan cardholders completed timely U.S. Bank statement reconciliations and uploaded the corresponding bank statements and any supporting documents into the UMass system’s online bank card transaction repository, we met with a UPST bank card manager and observed them locating all of the requisitions for the 128 transactions in our sample in the bank card transaction repository. We recorded the creation dates of the relevant requisitions. We then took screenshots of each bank statement and any supporting documents within the bank card transaction repository. If any of the transactions in our sample were missing bank statements or receipts, which were required to be submitted, the UPST member obtained those from the cardholders. Once documents related to our sample were provided to us, we recorded which documents were uploaded and, for those not uploaded, which documents were retrieved from the cardholder or were attempted to be retrieved but were still missing. By comparing each requisition’s creation date and the bank statement date, we determined whether the requisition was created within 30 days after the bank statement date.

See Finding 1 for information about the results of our testing regarding submissions of bank card transaction documents.

Information on Receipts and Bank Statements

For each of the 128 transactions in our sample, we performed the actions described below.

We inspected each receipt to ensure that it contained the vendor name, the description of the item or service purchased, the transaction date, the transaction total, and the last four digits of the bank card number used to make the purchase. We obtained screenshots of each requisition and inspected them for cardholder and supervisor signatures.

To determine whether each receipt and/or purchase log related to each transaction in our sample contained a documented business purpose, if not self-evident, we inspected each receipt and/or purchase log for a documented business purpose. When a transaction’s business purpose was not documented on either its corresponding receipt or purchase log, we used the Human Resources Compensation Management System (HR/CMS), which is the Commonwealth’s official payroll system, to identify the cardholder’s title. We inspected the relevant receipts and purchase logs for the type of item or service purchased. We then determined whether the description of the items or services purchased were typical purchases for that cardholder’s title and department. We also met with UMass system and UMass Chan management to ask about the business purposes for transactions that did not have documented business purposes on their corresponding receipts and/or purchase logs.

To determine whether each transaction in our sample was related to the goals and mission of UMass Chan, we inspected the bank statements and supporting documents to identify the types of purchases. We noted whether the purchases had a documented business purpose and was approved by the cardholder’s supervisor. We also met with UMass system and UMass Chan management to ask how the purchases related to the goals and mission of UMass Chan.

Additionally, we identified which transactions were travel-related by inspecting the supporting documents for vendor names and transaction descriptions related to travel (e.g., airlines, lodging, or car rental agencies). For out-of-state travel-related transactions in our sample, we inspected the supporting documents, confirming that each transaction was for out-of-state travel and was for a business-related purpose. We then inspected each travel-related receipt and bank statement for a travel authorization number, if applicable.

We identified which transactions were for subscriptions (e.g., marketing software or access to online news websites) by inspecting the receipts for descriptions of what was purchased. For those transactions in our sample that were for subscriptions, we inspected each receipt for subscription start date and end dates.

See Finding 1 for information about the results of our testing regarding information on bank statements and receipts.

Allowable Purchases

For each of the 128 transactions in our sample, we performed the actions described below.

To determine whether a transaction was for an allowable purchase, we inspected the supporting documents for the type of item(s) or service(s) purchased.

To determine whether a transaction was a foreign expense,⁹ we inspected each receipt for a vendor address outside of the United States and for any foreign expense fees. We noted that no transactions in our sample were for foreign expenses.

To determine whether a transaction was for out-of-state travel, we inspected the relevant supporting documents (i.e., receipts and invoices) for vendor addresses that were out-of-state and for any notations that the transaction was for travel or travel-related meals. We also inspected screenshots of preapprovals for the transactions in our sample that were for out-of-state overnight travel.

We also inspected each receipt to determine whether sales tax was charged. If sales tax was charged, we inspected the related bank statement and general ledger to determine whether the sales tax was refunded by the vendor to UMass Chan.

We noted no exceptions in our testing; therefore, we concluded that all 128 transactions were for allowable purchases. 

Cybersecurity Awareness Training

To determine whether UMass Chan followed its “Privacy and Security Training Policy” regarding cybersecurity awareness training, we performed the actions described below.

We obtained a list of workforce members who were employed at some point during the audit period, which UMass Chan provided to us as a Microsoft Excel spreadsheet. According to UMass Chan officials, this list contained all UMass Chan workforce members who had the potential to access UMass Chan’s computer network (meaning that the list included workforce members who did and did not have access to UMass Chan’s computer network). UMass Chan management explained that they were unable to provide a list of only the workforce members who had access to UMass Chan’s computer network because UMass Chan’s human resources system was unable to generate reports with that level of detail. Therefore, this list of 7,932 UMass Chan workforce members who were employed at some point during the audit period was the only source available to identify a population of UMass Chan workforce members who were required to complete cybersecurity security awareness training during the audit period.

UMass Chan had a population of 7,932 workforce members who were employed at some point during the audit period (which included individuals whose employment with UMass Chan ended at some point during the audit period). We distributed these 7,932 workforce members into the following two categories: 1,972 workforce members who were hired during the audit period (i.e., newly hired workforce members) and were required to complete initial cybersecurity awareness training and 5,960 workforce members who were hired before the audit period (i.e., existing workforce members) and were required to complete annual refresher cybersecurity awareness training. We selected a random, statistical¹⁰ sample of 60 newly hired workforce members from our population of 1,972 and another random, statistical sample of 60 existing workforce members from the population of 5,960, and, in both cases, used a 95% confidence level, a 0% expected error rate, and a 5% tolerable error rate.¹¹

To determine whether UMass Chan ensured that workforce members from our two samples completed cybersecurity awareness training—the initial training for our sample of 60 newly hired workforce members and the annual refresher training for our sample of 60 existing workforce members—in a timely manner, we took the following actions with each sample. We obtained evidence (e.g., certification of completion) from UMass Chan’s cybersecurity awareness training platform and inspected each assignment date and completion date recorded in the cybersecurity awareness training platform. In addition, for each of the workforce members in our samples whose employment with UMass Chan ended during the audit period (of which there were 12 newly hired workforce members and 12 existing workforce members), we obtained evidence (i.e., screenshots of users’ UMass Chan employment status, which we obtained from UMass Chan’s human resources system) showing the end date of each user’s employment status with UMass Chan, which would also deactivate each user’s access to UMass Chan’s computer network.

See Finding 2 for information about the results of our testing regarding cybersecurity awareness training.

We used statistical sampling methods for testing; however, we did not project the results of our testing to the corresponding population(s).

Data Reliability Assessment

Bank Card Purchases

To determine the reliability of the bank card transaction data, we interviewed UMass system officials who were knowledgeable about the data. We also tested the security management and access controls for UMass Chan’s computer network. To determine the completeness of the bank card transaction data, we observed the UPST bank card manager query the UMass system’s finance system and extract all 56,846 bank card transactions that were made during the audit period. The UPST bank card manager then provided these 56,846 bank card transactions to us in a Microsoft Excel spreadsheet. We ensured that the total number of transactions we observed within the finance system matched the total number of bank card transactions from the Excel spreadsheet. We inspected the bank card transaction data for hidden rows and columns, embedded data,¹² and invisible content. We also inspected the bank card transaction data to see whether a transaction number appeared more than once within the data. Because we did see transaction numbers occur in the data more than once, we then tested whether these transaction numbers were shared transaction numbers¹³ (which can occur because of acceptable business processes) or were duplicate transaction numbers (which should not occur). We did this by selecting a judgmental¹⁴ sample of five transaction numbers that appeared more than once and verified that these five transaction numbers were not duplicated in the general ledger. Then, from the 56,846 bank card transactions, we removed any transactions with non-unique transaction numbers, resulting in a total population of 41,848 unique bank card transactions.

To determine the completeness of the population of 41,848 unique bank card transactions, we judgmentally selected a sample of 20 transactions listed on bank statements and compared them to the 41,848 unique bank card transactions that were made during the audit period, which were listed in the UMass system’s finance system data. To determine the accuracy of this population, we selected a random sample of 20 bank card transactions from the 41,848 unique bank card transactions from the finance system that were made during the audit period and traced the cardholders’ names, the last four digits of the bank card numbers, the transaction dates, the vendor names, the dollar amounts of the transactions, and the transaction numbers to the 20 transactions listed on relevant bank statements. We then verified that all cardholders relevant to this population of 41,848 unique bank card transactions were UMass Chan employees by tracing their names to a list of individuals who were actively employed by UMass Chan during the audit period, which we generated independently from HR/CMS.

Cybersecurity Awareness Training

To determine the reliability of the list of 7,932 UMass Chan workforce members who were employed at some point during the audit period, we took the following actions. We conducted interviews with UMass Chan officials who were knowledgeable about the data. Also, during a remotely held meeting, we observed a UMass Chan information technology employee log into the cybersecurity awareness training platform and show us the cybersecurity awareness training process, starting with the assignment of the training to a workforce member and concluding with the workforce member’s receipt of the training completion certificate.

We inspected the list of 7,932 UMass Chan workforce members for hidden rows and columns, embedded data, invisible content, and duplicate information. In addition, we selected a random sample of 20 workforce members from the list and verified their employment status with UMass Chan by tracing employee information—i.e., employee identification number, name, pay status,¹⁵ department, job title, campus location, employee class,¹⁶ start date, rehire date (if applicable), and termination date (if applicable)—to the employee information in the electronic personnel files maintained by the UMass Chan Human Resources Department. To test the completeness of the list of 7,932 workforce members, we compared this list to a list of individuals who were actively employed by UMass Chan during the audit period, which we generated independently from HR/CMS.

Based on the results of the data reliability assessment procedures described above, we determined that the information obtained for the audit period was sufficiently reliable for the purposes of our audit.

5.    The average bank card transaction during the audit period was $295.

6.   Confidence level is a mathematically based measure of the auditor’s assurance that the sample results (statistic) are representative of the population (parameter), expressed as a percentage. A 95% confidence level means that 95 out of 100 times, the statistics accurately represent the larger population.

7.   Expected error rate is the number of errors that are expected in the population, expressed as a percentage. It is based on the auditor’s knowledge of factors such as prior audit results, the understanding of controls gained in planning, or a probe sample. In this case, based upon the implementation of the UPST system and understanding of controls gained in planning, we assume there are relatively frequent errors in the data the auditee provided to us.

8.   The desired precision range defines the area of likely values within which the true population value should lie. The lower or higher the precision range, the larger or smaller, respectively, the sample size would be. We chose a 20% desired precision range based on our understanding of the population of bank card transactions and the expected error rate of 50%.

9.   A foreign expense is a transaction made with a vendor or business that is outside of the United States.

10.   Auditors use statistical sampling to select items for audit testing when a population is large (usually over 1,000) and contains similar items. Auditors generally use a statistics software program to choose a random sample when statistical sampling is used. The results of testing using statistical sampling, unlike those from judgmental sampling, can usually be used to make conclusions or projections about entire populations.

11.   The tolerable error rate (which is expressed as a percentage) is the maximum error in the population that is acceptable while still using the sample to conclude that the results from the sample have achieved the objective.

12.   Embedded data is data within a Microsoft Excel worksheet that was added from another source and/or data that cannot be edited.

13.   Each UMass bank card transaction has a unique transaction number assigned to it by the bank during the transaction process. Transactions with shared transaction numbers can be attributed to various situations, such as splitting the cost of purchased items with multiple departments.

14.   Auditors use judgmental sampling to select items for audit testing when a population is very small, the population items are not similar enough, or there are specific items in the population that the auditors want to review. Auditors use their knowledge and judgment to select the most appropriate sample. For example, an auditor might select items from areas of high risk. The results of testing using judgmental sampling cannot be used to make conclusions or projections about entire populations; however, they can be used to identify specific issues, risks, or weaknesses.

15.   Pay status indicates an employee’s current employment situation (e.g., active, terminated, or on leave with pay).

16.   Employee class indicates an employee’s classification (e.g., dual-employed physician, staff member, or faculty member).

 

Date published:	September 6, 2024