The Joint Office is now correctly using Housing Move-in Date for reporting housing placements
The limitations of HMIS lower data quality
Objectives, scope, & methodology
Report Highlights
Reporting was an accurate reflection of provider data most of the time, but providers would like a more user-friendly data system.
- What we found: The Joint Office is now correctly using Housing Move-in Date for reporting housing placements.
- Why this is important: The prior method counted when clients entered into a program or started receiving services, which is not an accurate count of housing placements.
- What we found: The Joint Office's housing placement data was an accurate reflection of homeless service provider data most of the time.
- Why this is important: Accurate data is an important way to measure if a program is successful.
- What we found: The limitations of the Joint Office's data management system lower data quality.
- Why this is important: Data entry continues to be time consuming, inefficient, and at risk of data entry error because providers must use a second, more functional system for most data.
Background
We started this audit because our office found (as reported in a 2021 letter to management) that the Joint Office was using the Project Start Date to count placements into housing, rather than the Housing Move-in Date. The Project Start Date is the date the client entered into a program or started receiving services and Housing Move-in Date is the date the client moved into permanent housing. The letter to management also said that "approximately 60% of clients were missing address data or had address data that were not actual addresses."
This prompted a new audit to explore the data discrepancies further. However, we encountered data access issues, so our first report for this audit focused on contract management, communication, and equity. This is the second report resulting from the audit, and it addresses the initial concern around data.
Our goals for this portion of the audit were to:
- Determine if the Joint Office is now using the Housing Move-in Date for reporting housing placements.
- Determine if homeless service providers used the Housing Move-in Date field to capture a housing placement, rather than a program enrollment.
- Determine why so many housing placements lack address information.
The Joint Office is a part of the U.S. Department of Housing and Urban Development's (HUD) Continuum of Care Program. Continuums of Care are responsible for creating a coordinated local system to assist people who are unhoused in getting housed. Continuums of Care are required to measure their performance as a coordinated system. HUD requires Continuums of Care to maintain a Homeless Management Information System (HMIS) and collect specific data points. An HMIS is a data collection application that facilitates the collection of information about people and families who use residential or other homeless assistance services, and it stores that data in an electronic format. It is designed to capture information over time to provide the most accurate description of homelessness and homeless services within a community. There are multiple vendors for these data collection applications.
The Joint Office's contracted providers are required to enter specific types of information into HMIS and provide quarterly reports on the data they enter. Continuums of Care can also choose to add data points, response categories, or maintain historical data beyond what is specified by HUD as long as it does not impact the ability to accurately collect and report on the required data elements.
The data entered into HMIS are important because funds are awarded to Continuums of Care based on performance measures entered into HMIS. HUD expects Continuums of Care to calculate and use system-level performance measures as the established selection criteria for awarding Continuums of Care Program projects and to evaluate system performance.
In addition to determining funding, data can highlight what's working and not working within the homeless service system. Continuums of Care can use data to:
- improve programs
- ensure a common understanding among stakeholders about how they're doing and what they're working towards
- communicate and advocate for community needs.
HMIS data is about the human beings who experience homelessness and access services. Each piece of data is about a person's life and their efforts to stay in or find housing. HMIS data can provide significant opportunities to improve the delivery and accessibility of services. Continuums of Care can use data to understand what disparities exist and reduce barriers to people who need services. When the Joint Office does not use all available data to assess how the system is working, there is a lost opportunity to address problems and make systems improvements. This increases the risk of causing harm by addressing issues without the full picture of what is causing the problem.
The Joint Office is now correctly using Housing Move-in Date for reporting housing placements
Our first goal for this portion of the audit was to determine if the Joint Office is now using the Housing Move-in Date for reporting housing placements. We asked Joint Office data staff to give us a list of all the people receiving either permanent supportive housing or rapid rehousing in fiscal year 2023. They gave us the following numbers:
- Funding type: Rapid Rehousing
- Definition: Short-term (up to three months) and medium-term (4-24 months) tenant-based rental assistance and supportive services.
- Number of people: 845
- Funding type: Rapid Rehousing
- Definition: Permanent housing in which housing assistance (e.g., long-term leasing or rental assistance) and supportive services are provided to assist households.
- Number of people: 525
- Total people: 1,370
Source: Joint Office of Homeless Services. This will not match public reports because some people are partially funded and the Joint Office prorates them in their reporting. For the purposes of this audit, we counted the number of people to test the data's accuracy and did not prorate them. Additionally, this does not contain people whose data is protected due to domestic violence.
We also asked for the calculation they used to determine these numbers. They stated that their calculation used Housing Move-in Date, and we also observed that those in the list of people housed had dates in the Housing Move-in Date field.
Homeless service providers used the Housing Move-in Date field to capture housing placements
We interviewed 11 homeless service providers to confirm if the Housing Move-in Date reflected a housing placement. All of the providers said they enter a date in the Housing Move-in Date when someone was housed. These 11 providers provided 60% of the Joint Office's housing placements for fiscal year 2023, giving us sufficient confidence about the majority of housing placement data provided to the Joint Office.
The Joint Office's housing placement data was an accurate reflection of homeless service provider data most of the time
For the 11 providers we spoke with, we randomly selected people who had been housed according to Joint Office records and asked the homeless service providers if this matched their records. Many homeless service providers use data systems other than HMIS, so we asked them to check their other systems for the status of their clients. These results are only applicable to the providers we spoke with.
In 96% of cases, their data indicated that the Joint Office's report about housing placements was an accurate reflection of the providers' data. For the 4% where the data were not accurate, the following occurred.
- Double counting: In three cases (1.4% of the sample), the Joint Office counted people as receiving both rapid rehousing and permanent supportive housing on the same day. The provider confirmed that only rapid rehousing should have been counted.
- Uncertainty: In two cases (0.9% of the sample), the provider was unsure if the person had received housing since their bookkeeping did not show paying a deposit for their housing.
- Wrong category: In two cases (0.9% of the sample), people received housing, but it was classified as permanent supportive housing, when it was actually rapid rehousing.
- Duplicate person: One person (0.5% of the sample) was housed, but was counted twice. The agency said that they do not have the permissions to remove duplicate people in HMIS.
In addition to the providers we spoke with, we also looked at the entire population of rapid rehousing and supportive housing placements (1,370) the Joint Office sent us. We observed the following potential errors.
- Double counting: In 18 cases (1.3% of the total placements), the people appeared to have already been counted in the housing data. Seventeen received both rapid rehousing and permanent supportive housing on the same day. One person was counted as receiving rapid rehousing by two different organizations on the same day. Three of these cases were in our sample mentioned above.
- Date outside of the report timeframe: In eight cases (0.6% of the total placements), people's housing move-in date was outside of fiscal year 2023. None of these cases were in our sample mentioned above.
The limitations of HMIS lower data quality
We wanted to find out why there were so few addresses in HMIS. During our conversations with providers, we learned that privacy concerns, such as domestic violence situations, are a potential reason, but there are two other reasons providers don't typically enter addresses into HMIS:
- HUD does not require addresses to be entered into HMIS.
- The current HMIS is not a user-friendly system for providers. Many providers use a second, more user-friendly system where they put the majority of information, including addresses.
For providers who use secondary systems, HMIS has become a system where many providers enter the minimum information necessary for reporting requirements. Entering data into multiple systems is time consuming, inefficient, and increases the risk of data entry error. And because HMIS was not the preferred system for most providers we spoke with, it means that valuable information isn't going into HMIS.
Many providers have data systems other than HMIS
Providers told us they enter most information into internal data systems. Some of the reasons include:
- The current HMIS is archaic/clunky/not very user friendly.
- It takes time to learn and can be difficult for new staff to navigate.
- It doesn't allow providers to track information that is useful in their day-to-day work, such as service plans, case notes and incident reports.
- It doesn't provide useful information and reports for providers.
- It can't upload information from providers' secondary systems.
A number of providers said that the current HMIS doesn't produce information and reports that are as useful as their secondary systems. For example, one provider said HMIS doesn't have fields for some of the information they're interested in tracking, such as pet and vehicle information for shelter participants, employer connections and employment referrals, and incident reports.
Given the challenges with the current HMIS, the local homeless services system would benefit from having a better-functioning, more user-friendly HMIS system where providers can either enter everything in one place or link their systems with HMIS and upload data.
Duplicate data systems increase risk and are inefficient
Using two systems increases the risk of data inaccuracy
When providers have to manually enter and maintain data in two different systems, it increases the risk that data are less accurate in both. The providers we spoke with who use secondary systems tended to minimize the amount of information they entered into HMIS, entering only what is required by HUD. This makes it unlikely that the current HMIS system has a robust amount of information to use and share back with stakeholders. However, if the information each provider is keeping in their internal systems could be put together, it could yield more useful and robust information for planning and for sharing with the community.
Maintaining two systems is time consuming and inefficient
Learning, using, and maintaining two systems is time-consuming and inefficient. In a field where the primary task is providing direct services to people experiencing homelessness, it can be helpful to minimize the amount of time providers perform unnecessary and time-consuming administrative work, like entering data twice. This is compounded by the fact that many providers donâ't have dedicated data staff.
Problems with HMIS occur in a context of regional reorganization
This audit took place during a time when the Joint Office is participating in the implementation of a regional reorganization of Continuums of Care as a result of the supportive housing services measure voters passed in 2020. This measure created funds for supportive housing services for people who experience or are at risk of experiencing homelessness.
As a result of the measure's passage, Metro, along with Multnomah, Washington, and Clackamas Counties and their respective housing authorities, are partnering to coordinate the Supportive Housing Service Program implementation. In this process, the Joint Office will assume the responsibility as the Tri-County HMIS system lead, which had previously been the City of Portland. Additionally, the City of Portland Housing Bureau is in the process of transferring HMIS leadership responsibilities to the Joint Office, which includes leading the implementation of HMIS for the tri-county area. This includes working together to standardize data collection and quality control.
The Multnomah County's guiding document, the 2020 Regional Implementation Plan, noted that there is a substantial need to strengthen and integrate existing data systems, in particular HMIS and healthcare data systems by investing in initiatives that focus on connecting disparate data.
As part of the Tri-County implementation of the Supportive Housing Services Program, the Joint Office data team has been meeting regularly with data managers from the tri-counties to coordinate, strategize, and learn from one another. There are three monthly meetings when coordination occurs. One meeting is a more detailed conversation about aligning Supportive Housing Services reporting and measures the Joint Office reports to Metro. Another monthly meeting is for data managers from the tri-counties to coordinate, strategize, and learn from one another. The third meeting is coordinating around the Regional HMIS implementation plan.
The Joint Office hired a third-party consultant to evaluate how to improve HMIS and how it's used internally within agencies. The objectives of that evaluation were to identify the current state of HMIS and the expectations for the future, identify gaps between current HMIS use and goals for the future, provide insights on potential software solutions, and provide recommendations to guide the Tri-County HMIS implementation. The report was shared with the Board of County Commissioners in April 2024.
Despite the fact that the Joint Office assumed the responsibilities of the HMIS lead in 2023, the City of Portland still had some admin control until recently. Some of the holdup has been waiting for HMIS system administration to transfer from the city to the county. This occurred on March 25, 2024.
Recommendations
We recommend that the Joint Office, no later than January 1, 2027:
- To improve efficiency, usefulness of data, and reduce the potential for errors in HMIS, implement a database compatible with an application programming interface (API). An API is a software intermediary that allows two applications to talk to each other. Using a database compatible with an API will allow homeless service providers to upload data from their internal systems.
Objectives, scope, & methodology
We started this audit because our office found (as reported in a 2021 letter to management) that the Joint Office was using the Project Start Date to count placements into housing, rather than the Housing Move-in Date. The "Project Start Date" is the date the client entered into the program or started receiving services and Housing Move-in Date is the date the client moved into permanent housing. The letter to management also said that "approximately 60% of clients were missing address data or had address data that were not actual addresses."
This prompted a new audit to explore the data discrepancies further. However, we encountered data access issues, so the first report for the new audit focused on contract management, communication, and equity. This second report addresses the initial concern around data.
The objectives of this portion of the audit were to:
- Determine if the Joint Office is now using the Housing Move-in Date for reporting housing placements.
- Determine if homeless service providers used the Housing Move-in Date field to capture a housing placement, rather than a program enrollment.
- Determine why so many housing placements lack address information.
To accomplish these objectives:
- We selected a random sample of people that the Joint Office reported to have been housed and spoke with homeless service providers to verify the data. The random sample included 223 housing placements by 11 different providers. These 11 providers made up 60% of the rapid rehousing and supportive housing placements JOHS reported in fiscal year 2023. For these 11 providers, we can be 95% confident that JOHS's data is an accurate reflection of the provider's data 96% of the time with a margin of error of 5.62%. We cannot provide assurance for the providers we did not speak with.
This report describes data accuracy only for the data we evaluated.
For this audit, we analyzed HMIS data from the Joint Office of Homeless Services for people housed in rapid rehousing or permanent supportive housing from fiscal year 2023. By verifying the data with homeless service providers, checking the data for double counting, and verifying the date range, our office has determined that the data were sufficiently reliable for the purposes of this report.
We conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objective. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.
Audit staff
Nicole Dewees, Audit Director
Mandi Hood, Management Auditor