Objectives, Scope, & Methodology
The objective of this audit was to determine:
- what steps the county has taken to ensure that vital services can continue safely and equitably during the pandemic,
- whether those steps are in line with CDC and other guidance to reduce health risks, and
- what improvements can be made moving forward.
With these objectives in mind, we examined the following areas:
- Higher risk congregate settings which included homeless shelters, detention settings, and adult care homes
- Overall countywide guidance and support to departments and employees
- Physical changes to county buildings
- Teleworking
To accomplish these overall objectives we:
- Focused our work on the guidance and practices in place as of June 1, 2020, and which continued to evolve through December 18, 2020.
- Conducted over 70 interviews with elected officials, department directors, division directors, county contractors, and county employees.
- Studied COVID-19 guidelines from the Centers for Disease Control and Prevention, Oregon Health Authority, Johns Hopkins University, Oregon Occupational Safety and Health (OSHA), and Multnomah County.
- Researched relevant literature as well as other jurisdictions’ audit reports.
- Conducted site visits of some county buildings, as well as of congregate and motel-based homeless shelters.
- Conducted surveys of county employees and adult care home personnel.
Internal Controls and Scope of Work
We obtained an understanding of internal controls over COVID-19 safety procedures for homeless shelters, detention settings, and adult care homes. We also examined internal controls for countywide COVID-19 guidance and support as well as any physical changes to buildings needed to ensure the safety of employees and the public. Because the county necessarily performs its work differently within the safety restrictions of COVID-19 guidance, we also analyzed teleworking internal controls.
An effective control structure in a COVID-19 environment provides reasonable assurance that the county designed its protective controls to ensure the safety of clients served, the public and employees. In a best-case scenario, we would have been able to evaluate the design, implementation and operational effectiveness of internal controls. However, in high-risk settings such as homeless shelters, detention settings, and adult care homes where being on-site or in close proximity would cause an additional risk of infection to others, we generally limited our procedures largely to the design of internal controls. We observed internal controls for homeless shelters and county facilities where possible to determine implementation and sought corroborating evidence supporting effective internal control implementation and operating effectiveness as described below for each section.
Homeless shelters
To obtain an understanding of homeless shelters’ internal control design we focused on whether shelters followed the Center for Disease Control, Oregon Health Authority and the county COVID-19 safety protocols. We also examined internal controls associated with Joint Office of Homeless Services responsibilities in the homeless shelter system. We conducted interviews, studied legal requirements and guidance, reviewed documents, and visited three sites. We also used comments from the employee survey to further our understanding.
Based on our understanding of internal controls and survey results, we assessed the effectiveness of internal control design and, to some degree, implementation. Our assessment identified concerns related to clarity of communication related to guidance on the public facing website and that not all new contracts or amendments have COVID-19 related clauses.
Detention settings
To obtain an understanding of detention setting internal control design we focused on whether detention settings (both adult and juvenile) followed the Center for Disease Control, Oregon Health Authority and the county COVID-19 safety protocols. We also examined internal controls associated with county responsibilities in detention settings. We conducted interviews and read policies and procedures.
Our office conducted a survey of all county employees about the county’s pandemic response. Approximately 300 Multnomah County Sheriff’s Office jail employees and 30 juvenile detention employees responded to the survey, giving us insight into the conditions within the county’s detention settings.
Based on our understanding of internal controls and survey results, our assessment identified concerns related to frequency of cloth mask exchange in the jails, reduced no-cost visiting options, and consistently enforcing face covering policies.
Adult Care Homes
To obtain an understanding of adult care homes’ (ACH) internal control design we focused on whether adult care homes followed the Center for Disease Control, Oregon Health Authority and the county COVID-19 safety protocols. We also examined internal controls associated with county responsibilities in the adult care home system. We conducted interviews, reviewed legal requirements, studied authoritative health literature, monitored county adult care home management reports as well as state restricted admissions reports.
We conducted a survey of adult care home operators, resident care managers, and caregivers to provide information about internal control implementation. Based on our understanding of internal controls and survey results, we assessed the effectiveness of internal control design and, to some degree, implementation.
Based on our understanding of internal controls and survey results, our assessment identified opportunity for the ACH Program to increase communication with ACHs about compliance with guidance and regulations regarding exposure, infection control, physical distancing and reporting.
Countywide Guidance and support
To obtain an understanding of internal control design for county guidance, we focused on whether the county’s guidance was in alignment with guidance from the Center for Disease Control, Oregon Health Authority and Johns Hopkins University. In addition to reviewing guidance, we also conducted interviews and analyzed the results of our employee survey. The employee survey provided valuable information on the extent to which guidance is being followed.
Based on our understanding of internal controls and survey results, we assessed the effectiveness of internal control design and, to some degree, implementation. Our assessment identified concerns related to oversight, consistent implementation, and opportunities for feedback.
Physical changes to county buildings
To obtain an understanding of internal controls we focused on whether the county’s practices for making physical modifications and engineering controls at county buildings was in alignment with guidance from the Centers for Disease Control, and Oregon OSHA. In addition to reviewing guidance, we conducted interviews, visited county locations, and analyzed the results of our employee survey. The employee survey provided valuable information on the extent to which guidance was implemented at county facilities.
Based on our understanding of the guidance, we assessed the design of internal controls related to physical modifications and engineering controls at county buildings, and assessed the CDC guidance for engineering and administrative controls against county practices.
Teleworking
We obtained an understanding of how the county designed telework internal controls. In addition to conducting interviews, reviewing literature and studying other jurisdiction practices and audits, we also reviewed telework policy, online resources, training, and payroll information. We analyzed specific risks of teleworking and determined whether teleworking policy addressed those risks.
Based on our understanding, we assessed the design of telework internal controls. We did not have the information we needed to evaluate telework eligibility or equity. Our assessment identified concerns related to telework agreements, equipment, security of information, timekeeping, and training.
Internal Control Components and Principles
The following are the internal control components and underlying principles that are significant to the audit objective. Management is responsible for all of these activities.
Risk Assessment
- Define objectives clearly to enable the identification of risks and define risk tolerances.
- Identify, analyze, and respond to risks related to achieving the defined objectives.
- Consider the potential for fraud when identifying, analyzing, and responding to risks.
- Identify, analyze, and respond to significant changes that could impact the internal control system.
Control Activities
- Design control activities to achieve objectives and respond to risks.
- Design the entity’s information system and related control activities to achieve objectives and respond to risks.
- Implement control activities through policies.
Information and Communication
- Use quality information to achieve the entity’s objectives.
- Internally communicate the necessary quality information to achieve the entity’s objectives.
- Externally communicate the necessary quality information to achieve the entity’s objectives.
Monitoring
- Establish and operate monitoring activities to monitor the internal control system and evaluate the results.
- Remediate identified internal control deficiencies on a timely basis.
Data Reliability
We used financial information for the time-period of January 1, 2019 to September 30, 2020 from Workday, the County’s current enterprise resource planning system to accomplish our audit objectives. We also used financial information for the time-period July 1, 2014 to December 31, 2018 from the prior enterprise resource planning system SAP. Based on the annual reviews of SAP and Workday by the County’s external auditor, our office has determined that the data were sufficiently reliable for the purposes of this report.
Survey of all County Employees
Our office conducted a survey among county employees about the county’s pandemic response to learn about employee experiences during the pandemic and employee perceptions related to safety measures in worksite environments. We utilized SurveyMonkey, an on-line survey software that helps create surveys and collect data, as the platform for our survey. To encourage participation, our office made the survey anonymous and was not linked to email addresses, name, employee ID, IP address or any other identifying information. Participation in the survey was voluntary and respondents had the right to withdraw from participation at any time.
The survey included open-ended and closed-ended questions about employee satisfaction and wellbeing, workplace safety, and telework. We also asked demographic questions. To ensure the quality and reliability of the survey, we pretested the questionnaire with auditors not assigned to this section and some county employees. We conducted the pretests to check (1) the clarity and flow of the questions, (2) the appropriateness of the terminology used, and (3) if the survey was comprehensive and unbiased. We revised the questionnaire based on feedback from the pretests. Our office determined the data were sufficiently reliable for the purposes of this report.
Survey of Adult Care Home Providers
We conducted a survey of adult care home (ACH) operators, resident care managers, and caregivers to provide information about internal control implementation. We utilized SurveyMonkey, an on-line survey software that helps create surveys and collect data, as the platform for our survey. Although verification through direct observation of controls is best, we used a survey to help assess controls because onsite visits would pose an unacceptable infection risk to adult care home residents and staff. We issued the survey to 2,364 unique email addresses for persons that work with/for the 618 homes licensed and monitored by the County’s Adult Care Home Program (ACH Program) as of August 19, 2020.
The survey was conducted between September 22nd and October 12th, 2020. The survey was anonymous and responses were not linked to email addresses, name, ACH license ID, IP address, or any other identifying information. While we did not collect any data on the responses by respondents - SurveyMonkey does track if a response was received or not but no other details were collected. We asked respondents to self-identify their role with the ACH (operator, resident manager, and/or caregiver – a respondent could select multiple types). We obtained a response rate of 33% of individual ACHs, 34% of operators, 32% of resident managers, and 8% of caregivers.
We analyzed results by provider type (e.g., caregiver only vs operator/resident manager) and by number of residents in a home (e.g., one resident vs more than one resident) and found little to no significant variation in results, in most cases. Most results are presented based on all responses provided unless specifically identified. The survey included mostly closed-ended questions about ACHs efforts made to ensure vital services could continue safely during the pandemic. We used federal, state and local requirements and guidelines as the criteria to develop a list of internal controls that we determined to be significant for ACHs to help ensure that vital services at ACHs can continue safely during the pandemic for vulnerable residents. These significant internal controls became the questions for the survey. To ensure the quality and reliability of the survey, we pretested the questionnaire with auditors not assigned to this section of the audit and ACH Program management. We conducted the pretests to check (1) the clarity and flow of the questions, (2) the appropriateness of the terminology used, and (3) if the survey was comprehensive and unbiased. We revised the questionnaire based on the pretests. Our office determined the data were sufficiently reliable for the purposes of this report.
Statement of Compliance with Government Auditing Standards
We conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.
Why We Conducted an Employee Survey
We conducted this survey to provide an opportunity for employees to communicate their experiences during the pandemic. This survey provides valuable evidence for our audit, across county departments. We wanted to provide an anonymous way for employees to communicate their thoughts and concerns. We hoped that, because the County Auditor is independently elected, employees would feel free to express concerns openly without fear of retribution. In addition to using the survey to inform this audit, we will issue a separate report with more detailed survey results in early 2021.
About the Survey
The Auditor’s Office sent a survey to all county employees who had a valid county email address as of the day we released the survey, September 22, 2020. The survey was open on Survey Monkey for three and a half weeks, closing on October 16, 2020. We ensured respondents’ anonymity by not collecting any identifying information. When the survey went out we provided employees with an FAQ on how we would conduct the survey, including how we would maintain the confidentiality of responses.
The survey had five main sections – employee satisfaction, safety in the work environment, telework experience, comment sections to give employees the opportunity to share their opinions, and a section on department and demographics. Employees were directed to the telework and work safety sections depending on their responses to work environment questions. The majority of questions on the survey were optional. Because most questions were optional, some respondents did not answer all the questions, and employees were directed only to sections that were relevant to them, the number of employees who answered any individual question varied. In total, 3,374 employees participated in the survey.
We provided an opportunity for comments, which were wide-ranging
Throughout the survey, we included several comment boxes for respondents to share any additional concerns or comments about their experience during the pandemic. We have included some selected comments in the report to highlight findings and provide insights into respondents’ perceptions. In some cases, comments helped explain why people answered the way they did. We edited responses for clarity and to remove identifying references.
About the Responses
More than half of the county workforce completed the survey. We asked respondents to identify their assigned department. Collecting department is important because each department has different practices, messages from leadership, and approaches to following policies, and we wanted to be able to illustrate these differing approaches in the data. This question was optional and many respondents did not provide a response or dropped off before that question. We received the highest number of responses from the Health Department, the largest county department. Response rates varied by department. Each department had at least a third of their workforce respond.
To better understand employees’ work experiences during the pandemic, we also asked respondents about the work settings where they spent the most time. Work setting is important because different work settings present different risks and county policies vary by setting. Many respondents had worked in more than one setting. We asked respondents to list the workplace settings that they worked in during the previous month. Three quarters of respondents named telework as one of their workplace settings.
Respondents reported working in a variety of county work settings.
Future survey report will provide additional information
Employees shared much more about their experiences than the focus of the audit on COVID-19 related safety. We are only reporting on a few select questions in this audit, and looking at responses by only department or worksite.
We believe that people will be interested in fuller survey results and publishing results are important for transparency. After this audit is complete, we will work on compiling a report of survey results with a goal of publishing that in early 2021. We will look at demographic responses with more analysis and depth in that report, in order to examine disparities experienced by groups of individuals based on their demographics. We will also look at the full range of survey questions.
