When Pay for Success Doesn’t Launch: Lessons Learned from Hillside Children’s Center

In August 2013, the New York State Division of the Budget issued a Pay for Success (PFS) Request for Proposal (RFP). Hillside Children's Center, along with its intermediary Third Sector Capital Partners, responded to that RFP and was chosen as a finalist. After three years of exploration, planning, and significant steps towards project execution, Hillside learned that the State decided not to move forward on contracting for its proposed Pay for Success project.

In this post, Erika Barry, Director of Strategic Expansion at Hillside Family of Agencies, reflects on lessons learned and the path forward for the project.

A project connecting communities to provide alternatives to teen incarceration

Hillside Children's Center (HCC) works with children throughout New York State providing health, education, and human services programs and resources that are either unavailable or in short supply—but which are desperately needed to ensure the next generation is successful. Hillside knows what is at stake when a teen enters the criminal justice system: with the right supports, they might set on a productive path, but without intervention, they face sobering rates of repeated incarceration and unemployment.

As detailed in a prior blog post, HCC designed the Intensive Community Asset Program (ICAP) as a community-based alternative for youth in the juvenile justice system who have been unable to access or find success in existing services because of barriers such as limited English language proficiency, unstable housing, lack of family engagement, and mental health conditions. Because of the complexity of their needs, these youth are at high risk for long-term placement in correctional or residential institutions. By providing youth with six months of intensive services within their local communities, ICAP’s intent was to help these clients flourish in their communities and avoid time in detention and placement. The ICAP model sought to use collaborative partnerships and wraparound care management to engage and stabilize youth and their families by meeting immediate needs; fostering connections with “placed-based,” or accessible and neighborhood-based, community assets and natural supports; and developing and strengthening tools for coping with challenges. Hillside’s ICAP Pay for Success project would have served approximately 500 youth in ten counties across Western, Central, and Upstate New York throughout a five year period.

Pay for Success project assets and challenges

The proposed project had several significant assets, including HCC's experience with pay-for-performance contracts. ICAP represented the expansion and evolution of Hillside’s successful Re-investing in Youth (RIY) program in Monroe County. ICAP would be based on the existing program but would expand services from one county to multiple counties, better identify the youth most at risk of longterm placement in a juvenile justice facility, and collect more information about outcomes. ICAP would feature smaller caseloads and increased flex-fund resources. The RIY package of interventions and service model, including the creation of a Core Team, a team of formal and informal supports and service providers, and the signature subcontracted partnerships with Neighborhood-Based Organizations, would have remained consistent within ICAP.

Beginning in April 2014, Hillside and its intermediary partner, Third Sector Capital Partners, worked with key State stakeholders to structure ICAP as a Pay for Success program. Hillside also formally involved county level juvenile justice content experts in fine-tuning the design of ICAP via an Advisory Council. Hillside received funding from Nonprofit Finance Fund under a program of the Social Innovation Fund to support project development and participate in a knowledge sharing network of initiatives pursuing PFS projects.

Despite the significant experience of these partners and a willingness to collaborate, two insurmountable challenges emerged:

Due to limitations in state data, project partners had not yet been able to pinpoint uniform eligibility criteria that would have allowed Hillside to objectively identify all youth for whom ICAP would be appropriate. Because eligibility was predicated on a high risk of incarceration, Hillside was dependent upon historic state data to project future risk, and the resulting sample size could not support a sufficiently rigorous randomized control trial.
The rapidly changing policy landscape made it difficult to predict judicial behavior, complicating the referral process and increasing risk for New York State and potential funders. As a recent example, there was concern that the roll-out of the new pre-disposition sentencing tool (PDRAI) in spring 2017 and proposed Raise the Age legislation-to serve sixteen and seventeen year olds in the juvenile justice system-would change, in unpredictable ways, the number and characteristics of youth headed to placement facilities.

Hillside’s final program design, created in concert with Third Sector Capital Partners and Urban Institute, addressed both challenges through the use of a rate card. A rate card is a new innovation on Pay for Success models whereby a government outlines a list of desired outcomes and identifies a fixed amount it is willing to pay for each individual that verifiably achieves each outcome. However, this new innovation in the field diverged too radically from New York State’s original understanding of the confines of Pay for Success in the 2013 RFP, posing procurement challenges. That, along with other concerns and policy-related delays, led to a decision not to fund ICAP through Pay for Success.

Hillside's Erika Barry on PFS Lessons learned and ICAP today

Our exploration of Pay for Success will have lasting implications for our organization and clients. Not only have all of us at Hillside learned a tremendous amount about the application of the Pay for Success model, but it has strengthened our resolve to continue to pursue means to provide solutions to these youth in the juvenile justice system. While the current project will not move forward through PFS, we value the lessons reinforced throughout this process.

Lessons on the Pay for Success model and development process

Transparent and collaborative relationships with external partners are key. The process moved fastest and most successfully in areas in which we had strong, pre-existing relationships with judges, county and state officials, and partnering Neighborhood-Based Organizations. Building strong relationships with our intermediary and evaluation firms was also critical, as was clear communication and shared understandings of goals and timelines.
Inclusion, beginning at project conception, of those closest to the work is critical to building buy-in. Because county and nonprofit partners were invested in ICAP from its infancy and had been an active part of our ongoing contracting efforts, they were better able to withstand the ebbs and flows of the process and showed greater patience and flexibility than they might have otherwise. Of course, their investment of time and resources and relationships was willingly made in the context of a PFS project coming to fruition and being able to reach more youth in need. With ICAP no longer moving forward in a PFS model, costs are felt more acutely. We deeply value the relationships we have built through this process and the investment made by our partners. We will continue to invest significant time in “processing” the frustration associated with the termination of the effort and hope to identify additional ways to leverage the work that has been done.
This process takes longer than anticipated, at all points of development. Variance in staff resources, schedules, and competing priorities always exist when working with multiple state and local governments and multiple nonprofit agencies across multiple regions. To some extent, Hillside Children’s Center had anticipated that because of our strong partner relationships, we would be an exception to the time it was taking other agencies to bring this type of project to execution. We learned, however, that even with our organizational expertise and capacity we were not immune to many of the most common PFS challenges.
Pay for Success projects require an incredible depth and breadth of expertise and a serious investment of organizational time and resources during the project construction phase. One element that was helpful in navigating this unfamiliar PFS territory was being part of the Nonprofit Finance Fund subgrantee community and sharing in the lessons learned by our peers. We were also lucky to be able to draw, on a weekly basis, from internal staff with the necessary programmatic, marketing, fundraising, budgeting, HR, legal, and evaluation expertise. Even with all of this support, Hillside also hired additional staff, prior to contracting, in order to manage the PFS negotiations and lay the groundwork for program launch. These team members needed not only expertise in their field, but also required a great deal of optimism and flexibility around shifting timelines and program modifications. While ICAP was delayed, and when it eventually did not move forward, Hillside also had to build adaptability to accommodate the associated staff and manage morale for a project that was quite visible in the public eye.
Involve the independent voice of the external evaluators from the beginning. The crux of a PFS project hinges on accessing the pertinent historical data, understanding what the available data allows us to predict, and determining what kind of evaluation best fits the program and population. Since moving a project to contract quickly is in everyone’s best interest, it is important to begin talking concrete evaluation specifics as soon as possible. While all of the parties involved had significant research expertise, our partners at Urban Institute brought a new and different perspective that substantively advanced our thinking and planning and could collaborate to build the strongest research design removed from any hint of bias. Additionally, because data from disparate external state sources can be time intensive to compile and clean up, it would have been most efficient to have our evaluators closely involved in the process from the onset so that they fully understood the system constraints and were able to make their requests more focused and specific.
Despite the need for an intervention and a clear value proposition, external factors outside of an agency’s control can simply make a project’s timing unfavorable. Ultimately, limitations in data/ information systems, as well as a rapidly changing policy landscape, had the most impact on the timeline in which ICAP could have been operationalized and the degree of outcome consistency that could have been predicted. In retrospect, bringing the evaluation partners into the conversation earlier would have moved the work forward with enough speed to make some of these issues surmountable. Nonetheless, despite the skill and the commitment of all parties involved, we were not in a position to orchestrate or even confidently predict the timing or content of likely legislative changes, let alone the way in which individuals and systems might respond.

Where ICAP is today

Project partners agree that the exploration and collaboration over the past few years has led to a better understanding of community needs, of Hillside's unique expertise serving a highly challenging population, and about how to articulate and measure goals and data related to helping New York's youth secure better futures. There were many tangible and intangible benefits realized by Hillside, and we are optimistic that these gains will strengthen our ongoing work to bring the ICAP model to fruition in one or more counties of NYS. Specific lessons and benefits include:

ICAP project partners were able to analyze more data than ever before, giving us a deeper and more nuanced understanding of the youth and systems needs for future program development. In fact, we have heard from county partners that simply seeing the data that surfaced through the ICAP project led them to a change in procedure and investment strategy to better serve youth.
Furthermore, as a result of this process, Hillside also benefitted from strengthened partnerships with government and nonprofit partners and increased visibility with philanthropic funders and private investors.
Finally, as we have several other programs that are well-positioned for PFS, Hillside is now much better prepared to analyze the level of risk and investment that would be involved in pursuing future projects.

And so, while disappointed that ICAP will not move forward as a PFS project, Hillside continues to seek alternative funding for the ICAP program. We are currently in conversation with two counties who recognize the benefits of the ICAP approach, and are looking at how traditional streams of funding might be redirected to enable ICAP to work with this population. We may also seek demonstration or pilot funding from philanthropic organizations that recognize the deep needs that exist. Last, we remain in conversation with New York State Office of Children and Family Services about the ongoing challenge to provide meaningful interventions on behalf of this population of youth.