Impact and Evaluations
The child welfare certification requirements will be adjusted to include both knowledge and performance testing. In addition, there will be an evaluation of The Academy implementation, the simulation, and the Virtual Reality program. These evaluations will ensure that the model is implemented with fidelity and creates a positive impact on staff retention and child and family outcomes.
Child Welfare Certification Impact
The updated certification program standards require a knowledge test and performance assessments. New hires must pass a knowledge exam of child welfare fundamentals before beginning discipline-specific training. After passing the knowledge exam, new hires will complete discipline-specific training, including simulations and field work. During this phase, new hires will be required to demonstrate adequate performance of specific core job functions to hold provisional certification.
Provisional certification is awarded for a 12-month period when the following requirements are met:
1. Online certification account created
2. Online Child Welfare Fundamentals Exam application submitted
3. Child Welfare Fundamentals exam passed; retests are allowed every 10 calendar days.
4. Discipline-specific training, including simulations and field work completed
5. Performance assessments with a score of meets or exceeds performance requirements submitted
6. Provisional certification awarded for 12-month period.
The upgrade to full certification is earned when the following requirements are met:
1. Online Child Welfare Upgrade application submitted
2. 1,040 hours of discipline specific work experience completed during the provisional period submitted
3. 20 hours of individual or group supervision with supporting supervision logs submitted
4. Four to six performance assessments with a score of three or higher submitted
5. Three recommendations for certification submitted
Note
Florida Administrative Code requires any individual holding primary case responsibility to be provisionally or fully certified. This activity is a mechanism to ensure that the new staff member is provided with the training and feedback necessary to build skills before they are assigned to work more independently.
Learning Management System
The Department launched a new learning management system (LMS), My FL Families LEARN, for all Department employees and 140,000 external users, including all external child welfare professionals and foster parents in April 2023. The eModules for The Academy and skills tracking will be available in the LMS, allowing child welfare professionals across the system of care to train in a shared, virtual environment. The LMS will provide detailed reports and dashboards to easily track training progress.
Evaluation
The Department, in collaboration with the Florida Institute for Child Welfare (Institute), is committed to evaluating all aspects of The Academy implementation phases to ensure that the program is meeting its intended goals and objectives. This comprehensive evaluation process will involve collecting and analyzing data related to the effectiveness and efficiency of the program, including participant feedback, training outcomes, and the overall child welfare program impact. Evaluation information will be used to identify strengths and areas for improvement, and to make data-driven decisions to optimize the training’s effectiveness. In line with this effort, there are three evaluation efforts happening: full implementation evaluation, simulation evaluation, and virtual reality (VR) evaluation.
1. Full Implementation Evaluation:
In collaboration with the Capacity Building Center for States and the Institute, the Department created an evaluation workgroup to develop a robust evaluation plan for The Academy project. The goal is to ensure that all components of The Academy project are evaluated from initiation to activation to sustainability. The workgroup will coordinate with an independent organization to perform the evaluation.
2. Simulation Evaluation:
Currently, the Institute is supporting the Department with initial evaluations of the modernization aspects of The Academy project. Beginning in November 2022, the Institute began evaluating the Real Academy Simulation Train-the-Trainer program. The first cohort of trainers has completed the four-series training, and a second cohort has begun training. In general, initial evaluation efforts are focused on trainers' perceptions of the training, the vendor/trainers, support from the Department, and any barriers to statewide implementation. The Institute is sharing session-based reports, including recommendations for consideration, with the Department.
Preliminary feedback from the first cohort of trainers was generally positive for the first two sessions, which cover the use of critical reasoning tools and seeing how to implement a simulation training session. One participant shared, "It was a great training. The simulations were extremely realistic and stress producing even for experienced trainers."
Another said: "This has been one of the most impactful trainings I have been a part of during [my time] as a trainer. The REAL Academy trainers are phenomenal and truly believe in the work they are doing with simulation. I found the training to be a transformational addition to the pre-service curriculum that will better ready the workforce for the realities of the field. The way the [Good Call Tool] is interwoven into the simulations allows for trainees to exercise critical thinking, an opportunity that is often missed as we move through pre-service curriculum."
3. Virtual Reality Evaluation:
In addition, the Institute is currently developing an initial evaluation plan for the Accenture Virtual Reality Train-the-Trainer program, with the first cohort of trainers slated to undergo training in late March. Similar to the aforementioned simulation evaluation, the Institute will examine trainers’ perceptions of the training, the vendor/trainers, DCF support, self-efficacy, and any barriers to statewide implementation.
