Outcomes Matter: First Step House's Move to Data, Outcomes, and Evaluating Impact
NFF spoke with First Step House as well as the Community Foundation of Utah and Sorenson Impact Center about the Salt Lake County Pay for Success R.E.A.C.H Project; specifically about the service provider’s outcomes in action. This blog is part of an interview series with selected project partners from our Social Innovation Fund transaction structuring competition.
NFF: Before we speak about the Pay for Success projects, tell us about First Step House and how you became interested in tracking and delivering on outcomes. What was the original impetus? How did the shift come to fruition?
First Step House: First Step House (FSH) is a Salt Lake City based nonprofit substance abuse treatment center that has been providing affordable, effective treatment and housing for low-income individuals since 1958. The mission of FSH is to help people build lives of meaning, purpose, and recovery. We have a long history of working with the criminal justice system, adult probation and parole, and individuals entrenched in the legal system with significant behavioral health needs. A large majority of those served by our treatment and housing programs are engaged with the criminal justice system, with a significant number of referrals coming directly from associated departments (e.g., probation). We have continually sought to improve our evidence-based services for this population. The FSH leadership team has a strong background in data analysis and clinical research and we are constantly refining our larger data-driven management system. Overall, we have been interested in tracking outcomes and providing evidence-based services for over a decade.
As we have grown and our leadership teams have matured, our focus on outcomes has developed and intensified. First, we spent a significant amount of time getting our financial metrics and reports cleaned up and delivered on a regular basis. This allowed us to make well-informed decisions that led to increased growth and organizational development. Next, we intensified our focus on examining data related to our services. We looked at discharges, drug test results, admissions information, average length of stay, outcomes based funding sources, and more. Buried in this sea of information we began asking ourselves, “what data elements are most central to our primary mission?” This started an organization-wide dialogue that led to us developing a comprehensive data-driven management system. What this means is that we pull all the data- both financial and programmatic- that we believe is critical to the success of each of our programs, we feed that information to all relevant team members, and they set goals and take accountability for goal variances. Our teams then meet on a regular basis to come up with solutions to problems found in the data and to refine the data-driven management system. This has been an exciting and intense process that has led to significant improvement across a number of metrics.
NFF: What inspired you to take the delivery of outcomes to the next level in responding to the Pay for Success RFP? What were the biggest drivers? The things that gave you the most hesitation?
First Step House: When the initial Pay for Success Request for Information (RFI) was released, even before the RFP, our data-driven management system was in full swing. Because we had developed a culture focused on holding ourselves accountable for a range of outcomes, we immediately saw the Pay for Success project as a unique fit to what we were doing.
It was also clear that the Pay for Success structure offered many benefits not found with traditional funding mechanisms. From our perspective, the beauty of the Pay for Success model is: (1) the funding is significant enough that new, innovative programs can be developed, from the ground up if necessary, to solve social problems, (2) private investors put up the capital and take the risk if outcomes are not achieved, allowing smaller non-profits like us to enter into agreements that would otherwise be too risky, (3) outcome-based payment structures, coupled with early termination clauses if key metrics are not met along the way, are drivers of quality in ways not found in other social service payment structures, (4) the addition of independent evaluators significantly increases the level of expectation and further drives quality.
We have worked tirelessly to design the best program possible using our experience of what works, the experience of others, and drawing from the most cutting-edge research available. We were both excited and nervous to accept this challenge. A major impetus for us accepting an outcomes-based project like this was knowing that the concomitant accountability, expectations, and pressure would force us to grow and develop at a rapid pace. Many on our team were also extremely excited to work on a project that includes a randomized control trial in the design.
Our biggest hesitations revolved around the inherent risks associated with such a project, including project failure. Given the high visibility of these types of projects, failure to meet the projected outcomes and/or early project termination could significantly hurt our reputation, put a strain on relationships with community partners, and jeopardize future funding opportunities. However, during the development of the project, we spent a significant amount of time looking at external research on similar program models and scrutinizing our own internal data. We worked closely with project stakeholders to set outcome thresholds for the project that were both realistically achievable and would challenge us to achieve high levels of success through quality program implementation.
NFF: As you know, the shift towards outcomes can be challenging and the road to launching a PFS project is a long one! Can you share with us what the biggest challenge operating in an outcomes environment has been to date? How have you overcome this challenge?
First Step House: Shifting to outcomes, and for us ultimately PFS, is not a one time or one-step process; it’s a shift in organizational culture and attitudes to continually tackle challenges. This includes increasing capacity so outcomes can be gathered accurately and consistently, developing reliable tracking systems, operationalizing outcomes definitions effectively, and focusing on metrics that are the most central to achieving our desired outcomes. We are still working on overcoming these challenges, however, we have made progress by reporting outcomes consistently, increasing the accuracy of our data, hiring team members who are familiar with research methods and statistics, having a dedicated data manager, discussing data on a regular basis throughout the organization, enhancing our data-management systems, and holding people accountable for outcomes. Operating in an outcomes environment is ultimately about continuous improvement both for our organization internally and for the services we deliver to our clients.