How do you measure the experience of a stay at Spring Lake Ranch? Some things simply defy measurement. The bedrock of the SLR program is formed on the belief that each person has something to contribute, and all are accepted. Our core values of hope, work for a sense of purpose, personal relationships as the currency of healing, and a connection to the land infuse the unique therapeutic approaches of work, community, and professional expertise. But does this approach work, and if so, can we measure it? Beginning in July 2022, Spring Lake Ranch initiated an Outcomes Advisory Group to begin to ask and answer some of these questions.
Witnessing Transformation
Initiated in July 2022, the Outcomes Advisory Group is a committee of professionals and staff whose aim is to oversee the design, development, and dissemination of Spring Lake Ranch’s outcomes program, the primary purpose of which is program evaluation and improvement of services. Through examination of data and recommendations for additional areas of focus, the group seeks to answer questions about the lasting benefits of the Spring Lake Ranch program and ways to increase satisfaction and effectiveness in the program delivery. The committee may also make recommendations on ways to use outcomes findings to inform continuous quality improvement.
Presently, the primary instruments used to evaluate the effectiveness of the program are satisfaction surveys and the DLA-20. Satisfaction survey data has been collected since 2010 and serves to provide feedback about how residents and families experience the Spring Lake Ranch program. The DLA-20 was initiated as an assessment tool in 2019. The results on this website represent the initial collection and analysis of DLA-20 data from 2019 – 2021 and satisfaction survey data from 2019-2023.
Staff were trained in administering and scoring the DLA-20, a 20-item observer-rated scale assessing 20 domains of functioning relevant to recovery from severe mental illness (SMI). The DLA-20 was designed to be a brief, and easy-to-administer rating scale suitable to administration by case managers, including both professional and paraprofessional raters (Scott & Presmanes, 2001).
What amount of improvement is meaningful? We calculated the standardized response mean (SRM) effect size.
The SRM represents the ratio of the mean change to the standard deviation of that change and is an effect size index useful for indicating responsivity to an intervention. An SRM of 0.4 or greater is considered the lower bound for clinically meaningful change in functioning (Revicki et al., 2006).
Thus, for this analysis, domains with SRM’s of at least 0.40 are considered to have demonstrated clinically meaningful improvement at discharge.
Results
An improvement in LOF (level of functioning) of at least 0.5 points from admission to discharge is considered meaningful.
For residents with complete data (n=29), the average degree of improvement was 1.21 points (std 1.19), with a range from -0.95 (decline in functioning) to 4.3 (large improvement).
75% of residents improved by at least 0.5 points, exceeding the recommended benchmark of at least 35% that indicates program success.
- Domains with the greatest magnitude of improvement:
- self-management of health conditions
- communication skills
- time management
- productivity
- money management
- problem solving
- leisure skills
- Several domains demonstrated lower degrees of improvement (residents were less impaired at admission, thus leaving less room for improvement):
- appropriate dress
- behavior norms
- personal hygiene
- appropriate sexual behavior
- substance use
Resident Satisfaction
Another important indicator of program success is resident satisfaction. In order to measure this, we administer a satisfaction survey at discharge.
Responses from 62 residents from September 2019 to July 2023:
Average age 29 years (8.27 std), range 18 to 61 years
Gender: Female: 14 (23%), Male: 43 (69%), Non-binary: 5 (8%)
Average length of stay: 223 days (145 std) [around 7 months], range 23 to 976 days
The Outcomes Advisory Group currently is working on developing and implementing a post-discharge survey. We are grateful for the expertise of former House Advisor, Kim Van Orden, PhD, in analyzing and presenting this data.
See full explanation of data and collection in the video below.
