Children and Youth Services Review Volume: 24 (6-7) 2002
forthcoming
Previous issue | Forthcoming
Challenges
Implementing and Evaluating
Child Welfare Demonstration
Projects
Guest Editors
: Devon
Brooks,
assisted by Leslie
Wind (University
of Southern California)
Devon
Brooks
379 Challenges Implementing and Evaluating
Leslie
Wind
Child Welfare Demonstration Projects
In recent years, there has been an increasingly loud and vociferous
demand for results-oriented or outcome
data on the effectiveness of child welfare demonstration projects. Such data
have profound implications for child welfare practice and policy, not the least
of which is the impact on services that are funded, implemented and replicated.
Yet, myriad and complex challenges exist in implementing and evaluating child
welfare demonstration projects as a result of the unique nature of both child
welfare and evaluation research. These challenges are routinely overlooked and
unfortunately, stakeholders—including policymakers, federal agencies, funders,
social work students, non-child welfare researchers, the media and the lay
public—are left with the impression that child welfare interventions are clean
and straightforward and that they lend themselves easily to implementation and
evaluation. Indeed, child welfare researchers frequently are faced with the
question, “Does it work?” If only it were that simple. The fact of the
matter is that often, it is not even clear what “it” is, or what is meant by
“work.”
Barbara
Solomon
385 Accountability in Public Child Welfare:
Linking Program Theory, Program
Specification and Program Evaluation
In
public child welfare services, an increasing emphasis over the past decade on
outcome accountability has not changed the prevailing perception of stakeholders
that public child welfare agencies are not accountable. A host of demonstration
projects have been required to measure and account for results. The evaluations
of these projects have rarely provided evidence of program effectiveness or led
to program enhancement. A major contributing factor has been the lack of fit
between the model of causation that is the basis for most outcome-oriented
evaluative research and the reality of how most large, complex public social
welfare programs are developed and implemented. A realistic evaluation approach
pioneered by British sociologists is more closely articulated to the reality in
large, public child welfare agencies. The potential for transforming a large,
public child welfare agency into a more accountable organization is illustrated
with the developmental and realistic approach to program evaluation that has
been initiated in
Los Angeles
County
’s Department of
Children and Family Services (DCFS). This article describes the development of
program theory, how program theory has been linked to program specification, how
program specification has been linked to planning for program evaluation and how
a realist approach solves some on-going problems in the evaluation of public
child welfare programs.
E.
Wayne Holden
409 Evaluation of the
Connecticut
Title IV-E
Susan
Rousseau O’ Connell
Waiver Program: Assessing the
Tim Connor
Effectiveness, Implementation Fidelity,
Ana Maria Brannan
and Cost/Benefits of a Continuum
E.
Michael Foster
of Care
Gary
Blau
Heather
Panciera
This paper
presents the evaluation of the Connecticut Title IV-E Waiver demonstration
program. Initially approved in September 1998, this demonstration program is
designed to reduce escalating costs of out-of-home/residential care for children
at moderate mental health acuity levels who are either in a residential
placement or approved for residential placement within the child welfare/child
mental health system. Eligible children are randomly assigned to services within
community-based continua of care or treated as usual within the state system.
The multi-level experimental evaluation is designed to evaluate outcomes,
document implementation fidelity, and examine the net benefits of the
demonstration program using a randomized experimental design. Implementation
issues encountered during year 1 of the evaluation are discussed, and initial
results from the first 8 months of the program are presented. Implications for
the evaluation of child welfare demonstration projects and for the field of
children’s mental health services more generally are highlighted.
E.
Michael Foster
431 Benefit-Cost Analyses of the Child
E.
Wayne Holden
Welfare Demonstration Projects:
Understanding the Resource Implications
of the IV-E Waivers
This
article describes the application of benefit-cost analysis to the evaluation of
the Child Welfare Demonstration projects. First, it provides an overview of
benefit-cost analysis and describes the steps involved in conducting such an
analysis. The article then outlines a hypothetical benefit-cost analysis of a
waiver. Tracing the waiver's effects through its impact on services provided,
permanency and child outcomes, the article identifies potential costs and
benefits for families, other members of society and government programs. The
last includes Temporary Assistance to Needy Families, Medicaid, adoption
assistance as well as child welfare services and programs. The article reviews
data required to conduct such an analysis and describes the way in which an
analyst could calculate and present net benefits.
It identifies the pitfalls associated with benefit-cost analysis and
describes the key choices facing a researcher planning such an analysis. The
article concludes by arguing that the time is right for the application of
benefit-cost analysis to child welfare research.
Cassandra
Simmel
455 The
Shared Family Care Demonstration
Amy
Price
Project: Challenges of Implementing and
Evaluating a Community-Based Project
Shared
Family Care (SFC) is a demonstration program designed to assist families who are
involved, or are at risk of involvement, with the child welfare system.
Reasons for being at risk of involvement may include homelessness,
substance abuse, domestic violence, and being a teen parent.
SFC can serve either single-parent or two-parent headed families.
The National Abandoned Infants Assistance Resource Center (AIARC) at The
University of California at
Berkeley
has evaluated several SFC demonstration programs in
California
and
Colorado
since 1997. This paper begins with
an overview of SFC and then presents a case study of one county’s program to
illustrate the challenges and benefits associated with program implementation
and evaluation. Particular emphasis is given to the challenges and conflicts
faced by the AIARC evaluation team, who are in a dual role of providing
technical assistance while evaluating the programs.
Also discussed is the issue of multi-agency collaborations.
The paper concludes with recommendations for improving the implementation
of demonstration projects and evaluating such programs.
Daniel
Webster
471 Data are
Your Friends: Child Welfare
Barbara
Needell
Agency Self-Evaluation in
Los Angeles
Judith
Wildfire
County with the Family to Family
Initiative
The
Family to Family Initiative was launched by the Annie E. Casey Foundation in
response to a need to reform the child welfare system. Embedded in the goals of
the initiative is the premise of self-evaluation—using data to guide the
planning, implementation, and evaluation of child welfare policies, programs and
procedures. Several crucial pieces comprise the process of fostering
self-evaluation in Family to Family—attitude adjustment, creating
self-evaluation teams, and harnessing technology to measure outcomes. This
process helps participants move beyond skepticism of data to understand that
information exists that can actually be helpful for their immediate planning and
practice concerns. Since 1992, much has been accomplished in a number of sites
aided by Foundation-funded evaluators and technical assistants. In 1997,
researchers from the Center for Social Services Research at the
University
of
California
at
Berkeley
joined the technical assistance team. This paper discusses the challenges and
triumphs encountered thus far in
Los Angeles
County
as the CSSR staff, the child welfare agency administrators, front-line workers,
and community partners work together to use data to improve their social work
practice.
Sandra
Ortega
485 Methods and Practical Approaches for
Eric
J. Mundy
Evaluating Social Service Collaboratives:
Gwendolyn
Perry-Burney
The Evaluative Coterie
NSCAW
513 Methodological
Lessons from the
Research
Group
National Survey of Child and Adolescent
Well-Being: The First Three Years of the
USA
’s First National
Probability Study
of Children and Families Investigated for
Abuse and Neglect
The National
Survey of Child and Adolescent Well-Being is a national probability study of
children investigated for child abuse and neglect. This core study is
complemented with a national probability study of children who have been in
foster care about one year. Plans and efforts to recruit 105 county agencies,
more than 6,000 children ages 0-14, and a total of nearly 25,000 respondents
associated with the child are described. Parents or other permanent caregivers,
foster parents for children removed from the home, and the children themselves
are interviewed. For children in out-of-home placement, the caregiver from whom
the child was taken is also recruited for interviews. Investigative Child
Welfare Workers at the baseline, and service workers in subsequent rounds if the
family is receiving services, are also interviewed about the case. In addition,
an annual teacher survey is conducted for children
in grades K-12. Several advances in survey methodology help to manage the process
in a cost-efficient and scientifically rigorous manner. Lessons from the
planning stages and from the early weeks of fieldwork are presented. The
sampling and instrumentation techniques are discussed alongside other
methodological issues including agency recruitment, recruitment of families,
human subject protection issues, FR attrition, and data release.
Leslie
Wind
545 Child Welfare Demonstration Projects:
Devon
Brooks
A Model for Implementation and
Evaluation
In recent years important efforts have addressed the
demand for accountability in child welfare services. However, influences both
within and outside of the child welfare system have impaired the field’s
ability to delineate a comprehensive description of child welfare services
performance. Desperately needed is a child welfare evaluation knowledgebase that
can guide the evaluation of complex and dynamic child welfare services. As a
first step in the development of such a knowledgebase, a preliminary model of
core components for effective child welfare implementation and evaluation is
presented. The model is based on theory and empirical findings from both the
child welfare and program evaluation literatures.
© Copyright
1999-2002, Elsevier Science, All rights reserved.
