Home  |   Library  |   Children and Youth Services Review  |   Welfare of Children  
Gateway to resources
  Library  |

Children and Youth Services Review   Volume: 24 (6-7) 2002  forthcoming

Previous issue  | Forthcoming

Challenges Implementing and Evaluating Child Welfare Demonstration Projects

Guest Editors : Devon Brooks, assisted by Leslie Wind (University of Southern California)

Devon Brooks                   379     Challenges Implementing and Evaluating

Leslie Wind                                Child Welfare Demonstration Projects

 In recent years, there has been an increasingly loud and vociferous demand for results-oriented or outcome data on the effectiveness of child welfare demonstration projects. Such data have profound implications for child welfare practice and policy, not the least of which is the impact on services that are funded, implemented and replicated. Yet, myriad and complex challenges exist in implementing and evaluating child welfare demonstration projects as a result of the unique nature of both child welfare and evaluation research. These challenges are routinely overlooked and unfortunately, stakeholders—including policymakers, federal agencies, funders, social work students, non-child welfare researchers, the media and the lay public—are left with the impression that child welfare interventions are clean and straightforward and that they lend themselves easily to implementation and evaluation. Indeed, child welfare researchers frequently are faced with the question, “Does it work?” If only it were that simple. The fact of the matter is that often, it is not even clear what “it” is, or what is meant by “work.”

Barbara Solomon              385     Accountability in Public Child Welfare:

                                                  Linking Program Theory, Program
                                                  Specification and Program Evaluation

 In public child welfare services, an increasing emphasis over the past decade on outcome accountability has not changed the prevailing perception of stakeholders that public child welfare agencies are not accountable. A host of demonstration projects have been required to measure and account for results. The evaluations of these projects have rarely provided evidence of program effectiveness or led to program enhancement. A major contributing factor has been the lack of fit between the model of causation that is the basis for most outcome-oriented evaluative research and the reality of how most large, complex public social welfare programs are developed and implemented. A realistic evaluation approach pioneered by British sociologists is more closely articulated to the reality in large, public child welfare agencies. The potential for transforming a large, public child welfare agency into a more accountable organization is illustrated with the developmental and realistic approach to program evaluation that has been initiated in Los Angeles County ’s Department of Children and Family Services (DCFS). This article describes the development of program theory, how program theory has been linked to program specification, how program specification has been linked to planning for program evaluation and how a realist approach solves some on-going problems in the evaluation of public child welfare programs.

E. Wayne Holden              409     Evaluation of the Connecticut Title IV-E

Susan Rousseau O’ Connell         Waiver Program:  Assessing the

Tim Connor                                Effectiveness, Implementation Fidelity,

Ana Maria Brannan                    and Cost/Benefits of a Continuum

E. Michael Foster                       of Care

Gary Blau

Heather Panciera

This paper presents the evaluation of the Connecticut Title IV-E Waiver demonstration program. Initially approved in September 1998, this demonstration program is designed to reduce escalating costs of out-of-home/residential care for children at moderate mental health acuity levels who are either in a residential placement or approved for residential placement within the child welfare/child mental health system. Eligible children are randomly assigned to services within community-based continua of care or treated as usual within the state system. The multi-level experimental evaluation is designed to evaluate outcomes, document implementation fidelity, and examine the net benefits of the demonstration program using a randomized experimental design. Implementation issues encountered during year 1 of the evaluation are discussed, and initial results from the first 8 months of the program are presented. Implications for the evaluation of child welfare demonstration projects and for the field of children’s mental health services more generally are highlighted.

E. Michael Foster              431     Benefit-Cost Analyses of the Child

E. Wayne Holden                       Welfare Demonstration Projects: 

                                                  Understanding the Resource Implications

                                                  of the IV-E Waivers

 This article describes the application of benefit-cost analysis to the evaluation of the Child Welfare Demonstration projects. First, it provides an overview of benefit-cost analysis and describes the steps involved in conducting such an analysis. The article then outlines a hypothetical benefit-cost analysis of a waiver. Tracing the waiver's effects through its impact on services provided, permanency and child outcomes, the article identifies potential costs and benefits for families, other members of society and government programs. The last includes Temporary Assistance to Needy Families, Medicaid, adoption assistance as well as child welfare services and programs. The article reviews data required to conduct such an analysis and describes the way in which an analyst could calculate and present net benefits.  It identifies the pitfalls associated with benefit-cost analysis and describes the key choices facing a researcher planning such an analysis. The article concludes by arguing that the time is right for the application of benefit-cost analysis to child welfare research.

Cassandra Simmel             455     The Shared Family Care Demonstration

Amy Price                                  Project: Challenges of Implementing and

                                                  Evaluating a Community-Based Project


Shared Family Care (SFC) is a demonstration program designed to assist families who are involved, or are at risk of involvement, with the child welfare system.  Reasons for being at risk of involvement may include homelessness,  substance abuse, domestic violence, and being a teen parent.  SFC can serve either single-parent or two-parent headed families.   The National Abandoned Infants Assistance Resource Center (AIARC) at The University of California at Berkeley has evaluated several SFC demonstration programs in California and Colorado since 1997.  This paper begins with an overview of SFC and then presents a case study of one county’s program to illustrate the challenges and benefits associated with program implementation and evaluation. Particular emphasis is given to the challenges and conflicts faced by the AIARC evaluation team, who are in a dual role of providing technical assistance while evaluating the programs.  Also discussed is the issue of multi-agency collaborations.  The paper concludes with recommendations for improving the implementation of demonstration projects and evaluating such programs.

Daniel Webster                 471     Data are Your Friends: Child Welfare

Barbara Needell                         Agency Self-Evaluation in Los Angeles

Judith Wildfire                            County with the Family to Family


The Family to Family Initiative was launched by the Annie E. Casey Foundation in response to a need to reform the child welfare system. Embedded in the goals of the initiative is the premise of self-evaluation—using data to guide the planning, implementation, and evaluation of child welfare policies, programs and procedures. Several crucial pieces comprise the process of fostering self-evaluation in Family to Family—attitude adjustment, creating self-evaluation teams, and harnessing technology to measure outcomes. This process helps participants move beyond skepticism of data to understand that information exists that can actually be helpful for their immediate planning and practice concerns. Since 1992, much has been accomplished in a number of sites aided by Foundation-funded evaluators and technical assistants. In 1997, researchers from the Center for Social Services Research at the University of California at Berkeley joined the technical assistance team. This paper discusses the challenges and triumphs encountered thus far in Los Angeles County as the CSSR staff, the child welfare agency administrators, front-line workers, and community partners work together to use data to improve their social work practice.

Sandra Ortega                  485     Methods and Practical Approaches for

Eric J. Mundy                             Evaluating Social Service Collaboratives:

Gwendolyn Perry-Burney            The Evaluative Coterie


Doing more with less and showing results are two themes that are driving social service agencies. Partnerships between institutions are one way to provide both services and evaluative skills that one agency alone cannot provide. Increasingly, such collaboratives are becoming commonplace. This article will focus on the formation and underlying dynamics of decision-making with respect to evaluation collaboratives, i.e., coteries of evaluation professionals that review collaborative social service initiatives. The article will outline the new concept of evaluative coteries from a theoretical perspective and describe, examine and offer practical ways of reconciling challenges and maximizing benefits in such collaborative evaluation efforts. Two examples that exemplify such collaborative efforts from the state of Ohio will be highlighted.

NSCAW                          513     Methodological Lessons from the

Research Group                          National Survey of Child and Adolescent

                                                  Well-Being: The First Three Years of the

                                                  USA ’s First National Probability Study

                                                  of Children and Families Investigated for

                                                  Abuse and Neglect

The National Survey of Child and Adolescent Well-Being is a national probability study of children investigated for child abuse and neglect. This core study is complemented with a national probability study of children who have been in foster care about one year. Plans and efforts to recruit 105 county agencies, more than 6,000 children ages 0-14, and a total of nearly 25,000 respondents associated with the child are described. Parents or other permanent caregivers, foster parents for children removed from the home, and the children themselves are interviewed. For children in out-of-home placement, the caregiver from whom the child was taken is also recruited for interviews. Investigative Child Welfare Workers at the baseline, and service workers in subsequent rounds if the family is receiving services, are also interviewed about the case. In addition, an annual teacher survey is conducted for children in grades K-12. Several advances in survey methodology help to manage the process in a cost-efficient and scientifically rigorous manner. Lessons from the planning stages and from the early weeks of fieldwork are presented. The sampling and instrumentation techniques are discussed alongside other methodological issues including agency recruitment, recruitment of families, human subject protection issues, FR attrition, and data release.

Leslie Wind                      545     Child Welfare Demonstration Projects: 

Devon Brooks                            A Model for Implementation and


In recent years important efforts have addressed the demand for accountability in child welfare services. However, influences both within and outside of the child welfare system have impaired the field’s ability to delineate a comprehensive description of child welfare services performance. Desperately needed is a child welfare evaluation knowledgebase that can guide the evaluation of complex and dynamic child welfare services. As a first step in the development of such a knowledgebase, a preliminary model of core components for effective child welfare implementation and evaluation is presented. The model is based on theory and empirical findings from both the child welfare and program evaluation literatures.

© Copyright 1999-2002, Elsevier Science, All rights reserved.