Joint Office of Homeless Services Audit

This is the Multnomah County Auditor’s Office’s report on our audit of the Joint Office of Homeless Services.

Report Highlights

What We Found

Created in 2016, the Joint Office of Homeless Services (JOHS) consolidated Multnomah County and City of Portland governance, policy implementation, and funding to address local homelessness. We had two objectives in our audit of this new program: 1) identify where the consolidated dollars were targeted and if increased reliance on service providers exposes the County to additional risk, and 2) determine whether the system performance information provided to leadership, stakeholders, and the taxpayer is useful, transparent, and based on complete data.

We found Joint Office staff to be dedicated and hardworking, with years of social service experience and a passion for homeless services work. We also found issues to be addressed:

1. The County is reliant on a limited number of providers in certain homeless services areas.
2. The Joint Office currently lacks full evaluation and analytical capacity, and does not have full access to complete source data, which is critical to become the data-driven system they strive to be.
3. The Joint Office has not communicated with stakeholders and the public regarding system performance as frequently and as extensively as needed.

What We Recommend

To support its agenda, we recommend that the Joint Office should:

1. Work with the Homeless Management Information System (HMIS) administrator at the City of Portland to gain full access to HMIS reports, including source data for evaluation.
2. Build in capacity for analysis and evaluation, and conduct formal HMIS data analysis and program evaluation.
3. Regularly report to the public on system performance targets and results.
4. Work with service providers to improve data collection efforts, particularly at the shelter level.
5. Comply with Resolution 08-112 when considering future contracting relationships, or, in light of the partnership with the City of Portland, revisit the rule.

Background

Joint Office based on more than 10-years of City/County partnership.

In 2005 the City and County together adopted Home Again: A 10-Year Plan to End Homelessness in Portland and Multnomah County. In 2007 the City of Portland’s auditor performed a high-level review of the plan and then, in 2012, included a follow-up on the results of the plan within the Housing Bureau Risk audit.

The City Auditor’s report on Housing Bureau Risk identified a lack of collaboration among jurisdictions, potentially causing duplication and misalignment of housing and homeless services. This led the City Auditor to recommend more collaboration focusing on shared resources, alignment of services, and removal of duplication to produce measurable results. At the time, homeless services for single adults were primarily administered by the City, through the Portland Housing Bureau (PBH), while services for families and youth were primarily administered by the County, through the Department of County Human Services (DCHS). With two years remaining on the original 10-year plan, the plan was reset; a shared governance structure was recommended in order to oversee coordinated homeless service efforts. The recommended governance structure was established in 2014 – resulting in A Home for Everyone (AHFE), the current governing body. This governing body includes an executive committee made of top officials from Multnomah County, the City of Portland, the City of Gresham, and industry, and a coordinating board of local government, service providers, and business and community members. Informing the Board are five forums and workgroups, each focusing on a specific area within homeless services.

The new shared governing body first tackled ending veteran’s homelessness. Collaborative efforts were reported as successful in ending chronic homelessness for this specific population. In July 2016, the City and County established the Joint Office of Homeless Services (JOHS), formalizing a consolidation of governance, policy implementation, and funding to address local homelessness. The Joint Office and AHFE are somewhat intertwined, currently sharing the same executive director.

Most City and County housing assistance programs and homeless services are part of the Joint Office.

Joint Office Funding

The office is primarily funded with unrestricted general funding from the County and City, along with more restrictive funding from the State of Oregon and the federal government. For five years, the County and City have each committed to making annual investments of $15 million or more toward homeless housing services. Both entities have exceeded the minimum commitment set in the intergovernmental agreement forming the Joint Office.

County and City funding for homeless services has increased significantly.

Audit Objectives

We audited the Joint Office in its first year to support effectiveness and accountability, and to assess risk.

It is less common for auditors to audit a program when first launched. But auditing a program early in its development is an opportunity to identify areas for improvement that can help the program strengthen its foundation for success. We expected to see an organization in flux, and we understand that the organization will continue to build on the experience of its first year. While we recognize the tremendous challenge of the Joint Office and its providers to make a significant, enduring impact on the lives of homeless and vulnerable citizens in our County, we also know that millions of taxpayer dollars are making the work of the Joint Office possible. Given the challenges facing the Joint Office and the dollars involved, auditing in the first year seemed important to make sure the Joint Office was taking its first organizational steps in a manner that would support effectiveness and accountability.

We initially focused on identifying how funding was distributed and prioritized. After meeting with Joint Office staff and community providers, we pursued an additional focus on data and performance reporting.

In the survey phase of our audit, we spoke with Joint Office staff and with staff from 21 different community providers. We wanted to learn from them, their perspective on a number of issues, including how the transition to the Joint Office from County or City was going; how they use data; and any challenges. We learned a great deal from these interviews and used what we learned in those interviews to direct the work that followed.

Identifying funding and looking at provider capacity was the initial objective of the audit. Data and performance measurement came up often in discussions with providers. High performing homeless assistance systems are data-driven. This means these organizations rely on data and performance measurement when making decisions about how to best allocate scarce resources. In our 2015 audit of County homeless assistance programs, Housing Assistance for Vulnerable Families, we noted the lack of evaluation and analytical capacity with regard to homeless assistance programs.

After spending time learning about the Joint Office, and speaking with providers and staff, we focused on two audit objectives:

1. Identify where the combined Joint Office dollars were targeted and if increased reliance on service providers exposes the County to additional risk.
2. Determine whether the system performance information being provided to leadership, stakeholders, and taxpayers is useful, transparent, and based on complete data.

As auditors, we make every effort to obtain full access to data for programs or projects we are auditing. This allows us to validate data and work with raw data in our own analyses. We use our analyses to help build findings and fully report the conditions of the programs we audit. In this case, data on Joint Office clients are input by providers into a centralized database called the Homeless Management Information System (HMIS). In this audit, concerns over the privacy of client information and the perceived difficulties we would face in accessing the HMIS database, resulted in us auditing without direct access to data, limiting our analysis. The HMIS administrator fulfilled some individual data requests, and a number of the reports we received were useful in our analysis and evaluation of the data.

In evaluating our objectives, we found those working in the Joint Office to be dedicated and hardworking. Each person we interviewed brings years of social service experience and each is passionate about the homeless services work being done in the community. And, while not yet a system wide practice, we found evidence of promising evaluative work being performed in the homeless youth system.

Service Provider Funding

The Joint Office contracts with community partners to provide homeless services. We used payment data to illustrate how the Joint Office distributed and prioritized funding in FY2017.

Multnomah County passes a significant portion of human service funding to local nonprofit agencies via human service contracts, and this is certainly true with homeless services; nearly all homeless services are provided by community providers, under contract with the Joint Office. While advantageous in that those with the expertise are directly delivering services, there is a service delivery risk to the County should a provider be unable to perform.

As one of our objectives in this audit, we set out to identify how the Joint Office was prioritizing funding and distributing funding among providers. This analysis will be useful as a benchmark for the future, in terms of how the Joint Office prioritizes and funds homeless services, and indicates where the Joint Office is reliant on one or a handful of providers. We used FY2017 payment data from SAP (the County’s enterprise accounting system) to identify funding based on population served, type of service provided, spending area, and provider. Though performance data is collected separately, and is not easily integrated, we hope that in the future provider performance can be incorporated into similar analyses.

Provider Funding visualization

The County is reliant on a limited number of providers in certain homeless services areas.

We found that in some areas of contracting, the Joint Office is vulnerable to concentration risk in contracting. Board of County Commissioners (BOCC) resolution 08-112 came into existence in response to the financial failure of a large County human service provider in 2008. The County resolved to proactively guard against a recurrence in the future. The resolution directs the Department of County Management to, among other things, consider contractor financial stability, contractor reliance on County revenue, and County reliance on contractor services when determining risk mitigation activities. The resolution prescribes the actions needed should the contracting department decide to concentrate greater than $1 million and greater than 40% of a line of business to a single contractor, otherwise called a concentration risk.

We reviewed SAP payment data, and financial statements and contracts of the providers working with the Joint Office. We did this to establish a funding benchmark that would illustrate financial commitments to the providers, and to assess concentration risk to the County in outsourcing vital services. In FY2017, the process of managing concentration risk was complicated by the rapid transfer of City contracts to the newly formed Joint Office.

We found two Joint Office business lines (we considered homeless services for each subpopulation to be a line of business) of homeless services where each line has been concentrated greater than 40% with a single contractor: families and veterans; the adult system has a provider very near the 40% threshold. Two of the contractors have contracts exceeding $1 million which, when combined with the greater than 40% concentration, crosses the dual threshold set in the resolution.

Concentration Risk visualization

We also analyzed the risk associated with each contractor’s reliance on County contract revenue. We referred to the Department of County Management’s risk assessment tool, which applies a tiered approach to determining contractor reliance risk with greater than 40% of a contractor’s revenue coming from the County being considered high risk.

In FY2016, two Joint Office contractors relied on the County for more than 40% of their agency-wide revenue. One reported a 13% deficit (expenses exceeded revenue) in their annual financial statements and the other essentially broke even.

The Joint Office lacks evaluation and analytical capacity, and full access to complete source data.

We drew attention to a lack of evaluation and analysis of homelessness prevention programs in our audit, Housing Assistance for Vulnerable Families, released in January 2015. Resources dedicated to homeless services continue to increase, but we still find little evidence of program evaluation or analysis to indicate which programs and projects are performing most effectively.

In early FY2017, the Joint Office contracted the development of an evaluation framework, which outlines potential future program and system performance evaluation efforts. At the time of our audit, implementation of the framework had not begun, but it is a promising effort with great potential for future evaluation. We also found evidence of promising evaluative work being performed in the homeless youth system, and limited data analysis efforts in the family system. But the County and City did not build formal evaluative and analytical capacity into the Joint Office when they created it, which we view as a significant barrier to the implementation of the evaluation framework and other performance-related analyses. The Joint Office does have access to evaluation capacity in the Department of County Human Services (DCHS), but while they are available to work with the Joint Office, DCHS evaluators have done little work on Joint Office programs, and in fact have limited access to Homeless Management Information System (HMIS) data.

The Joint Office, as well, has limited access to complete HMIS data for reporting purposes, instead relying on aggregate reports from the HMIS administrator at the City of Portland. HMIS is the database which contains data about all clients that enter a homeless project. Individual service providers enter client data for homeless assistance programs into HMIS and the data helps inform system performance and patterns of use for services by those experiencing homelessness. Meaningful analysis of HMIS data is reliant on complete, accurate, and timely entry of data. Housing and Urban Development (HUD) requires the collection of specific universal data elements (UDE). The data collection standards for programs and projects under the Joint Office were established by A Home for Everyone (AHFE) and closely match HUD data expectations. While we found the completeness of HMIS data to be generally within the standards established by AHFE, data collection for emergency shelter programs fell short of the AHFE data expectation standards for some universal data elements.

Data Quality units in PHB and DCHS monitor data quality and work with providers to update missing data. These quality units retained the same responsibilities for data quality as before the creation of the Joint Office: PHB monitors data quality for singles programs, and County DCHS monitors data quality for family and youth programs.

Regular reporting of performance measures is needed to communicate system performance.

We found system performance difficult to interpret based on the measures we reviewed. In its performance reporting to the Board, the Joint Office has primarily focused on three figures:

1. number of homeless persons placed into permanent housing;
2. number of persons that received eviction prevention assistance; and
3. number of emergency shelter beds available in the community.

While it is encouraging to see increases year over year in these areas, these figures lack context, which makes them less effective in communicating system performance. To indicate impact, permanent placement figures require the context of demand; for example, while 251 families were placed into permanent housing through mobile housing last calendar year, 1,500 families remained on the waiting list at the time of our inquiry. Context about who is being placed is meaningful as well. Based on sample data, we found that about one third of permanent placements went to those staying with family or friends, already in a rental, or similar situation, rather than literally homeless. The prevention assistance numbers also benefit from additional context, particularly with regard to effectiveness. The reported prevention number indicates the number served, but does not answer the question of how many of those prevention efforts were effective. And while available emergency shelter beds have increased dramatically, transitional beds – another type of temporary shelter – have declined significantly over the last several years (primarily due to changes in federal guidance and funding).

Guidance from the National Alliance to End Homelessness suggests that the effectiveness of system strategies can be illuminated by focusing on HUD performance measures. There are three in particular to focus on: length of time homeless, returns to homelessness, and the number of people who become homeless for the first time. Longer episodes of homelessness mean more time and resources spent on each household and less time and resources to work with new entries. Returns to homelessness indicate original interventions were unsuccessful. Increases in the first time homeless may indicate ineffective prevention efforts.

Regularly reporting performance on the HUD measures alongside the output data may help decision-makers and the public see the progress of the system. For instance, the HEARTH Act set a national goal that no one will be homeless longer than 30 days. For our community in 2016, the average time homeless was 92 days. Setting targets toward the goal would be a data-driven practice.

Data-driven communities rely heavily on performance measurement and performance reporting. The National Alliance to End Homelessness cites the Community Shelter Board (CSB) in Columbus, Ohio as a model for data driven homelessness prevention systems. The CSB reports publicly and transparently each quarter and annually on program and project goals and measurements, including performance targets for individual providers, which helps hold providers and the system accountable.

Recommendations

We found the County is heavily reliant on a limited number of providers in certain homeless services areas. This may place the County at risk should a significant provider be unable to serve a unique homeless population. Having previously suffered the consequences of being over-reliant on a service provider, the County should comply with the Resolution adopted in 2008 to mitigate its contract concentration risk. This report highlights the areas of concentration risk within homeless services – both from the County’s reliance on providers’ services and from the providers’ reliance on County funding.

In FY2017, the process of evaluating concentration risk was complicated by the transfer of City contracts to the newly formed Joint Office at the County. Prior to renewing Joint Office contracts going forward, we recommend that the County evaluate the risk associated with concentrating homeless services with a limited number of providers and implement risk mitigation activities as needed.

We found that the Joint Office lacks two fundamentals of system performance measurement: access to complete data, and evaluation and analytical capacity. During the time of our audit complete data was not available for evaluation within the County. Data evaluation analysts were not evaluating system-wide data and, because of availability limitations, we too had to forgo some analysis. As the Joint Office begins its second year of operation, we believe it will benefit by building in capacity for data analysis and program evaluation, helping shed light on system performance, and performance at the program and project levels.

System performance is currently being communicated to stakeholders in the form of three measurements – number of permanent placements, number of eviction prevention placements, and number of shelter beds available - with limited context to truly understand system performance. Regular public reporting using HUD system performance measures will help decision-makers better understand the impact of their investment decisions and help taxpayers better understand how local investments are being used to reduce homelessness.

1. The Joint Office should work with the HMIS administrator at the City of Portland to gain full access to HMIS reports, including source data for evaluation.
2. The Joint Office should build in capacity for analysis and evaluation. The Joint Office should conduct formal HMIS data analysis and program evaluation.
3. The Joint Office should regularly report to the public on system performance targets and results.
4. The Joint Office should work with service providers to improve data collection efforts, particularly at the shelter level.
5. Comply with Resolution 08-112 when considering future contracting relationships, or, in light of the partnership with the City of Portland, revisit the rule.

Scope & Methodology

The objectives of this audit were:

1. Identify where the increased funding was targeted, and determine if increased reliance on these providers exposes the County to additional risk.
2. Determine whether the system performance information being provided to leadership, stakeholders, and taxpayers is useful, transparent, and based on complete data.

To accomplish these objectives we:

• Conducted over 40 interviews, including with providers and County staff.
• Reviewed best practices in homeless assistance.
• Reviewed resolutions, contracts, management reports, presentations, and other documents.
• Analyzed HMIS data contracts, SAP data.
• Attended AHFE board meetings.
• Employed the Department of County Management’s risk assessment tool in analysis of contractor reliance risk.
• Worked with Department of County Management Research and Evaluation to identify Joint Office funding of providers.

In this audit we analyzed FY2016 and 2017 financial data from SAP, the County’s enterprise resource planning system. Based on the annual review of SAP datasets by the County’s external auditor, our office has determined that the data were sufficiently reliable for the purposes of this report. We analyzed limited HMIS data from fiscal years 2015 – 2017 and sample HMIS data for fiscal year 2016, which included historical client data going back as far as the early 2000s. HMIS data is housed in software, Service Point, of Bowman Systems Inc. The steps taken by the HMIS administrator to verify the HMIS data gave us assurance that we could rely on the limited data provided. We validated summary data in permanent placement and prevention reports using the source data included, but relied on the assurance steps taken by the HMIS administrator when evaluating HMIS aggregate reports, primarily the “APR” report.

We conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings, and conclusions based on our audit objective. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions.

Related documents