Comprehensive Evaluation and Assessment of the "Creating a Culture of Safety" Program
Evaluation and outcomes assessment plans that use a multimethods approach to documenting evaluation and assessment data are an essential component of any educational program. 'Evaluation' refers to the systematic gathering and interpretation of evidence enabling judgment of program effectiveness and value and promoting improvement. 'Assessment' refers to the measurement of outcomes for individual learners or teams.
Evaluation data can be obtained through:
- focus groups
Assessment data for a program like this can be obtained through:
- attitude and perception surveys
- knowledge tests
- observable behaviors or skills (checklists)
- rubrics used to assess proposals and presentations
- number of presentations/abstracts/publications
- impact of completing the program on participant professional practice
The following describes the multimethods approach used to evaluate and assess the "Creating a Culture of Safety" program:
Evaluation Plan and Results
- Overall Program Evaluation: The "Creating a Culture of Safety" faculty met regularly to discuss objective ratings from student evaluations, qualitative feedback from student evaluation forms and informal student discussions, and comments from program faculty and clinician presenters. Continuous process improvement was implemented with changes in course sequencing and content for subsequent cohorts. The grant team followed a process similar to the Plan-Do-Study-Act cycle described on the Institute for Healthcare Improvement website.
- Student Evaluations of Program Activities (DNP/Resident PSQI Workshops, DNP Brownbag Seminars, Interprofessional Simulations, DNP Project Development Workshops, Expert Speaker Sessions): DNP students were given an anonymous evaluation form for each activity. Residents received the standard School of Medicine evaluation form.
Three cohorts of BSN-DNP students (30) and medical residents (72) completed the program. Data is available from evaluations for the first cohort (7 BSN-DNP and 24 medical residents). Due to pending academic journal publications, detailed data results are not included here, but are summarized below. The grant team welcomes inquires from those interested in specific data.
"Creating a Culture of Safety": Cohort 1 Evaluation Summary
- DNP/Resident PSQI Workshops: Student evaluations were consistently highly positive for these sessions. Sessions receiving the highest scores were Human Factors & Systems Thinking and Patient Safety/Error Disclosure/Peer Support.
- DNP Brownbag Seminars: These sessions were all well-regarded, with Relational Factors in Teamwork and Change Management scoring the highest on student evaluations.
- Interprofessional Simulations: Three out of four simulations received very positive scores from all participants, however, the Rapid Response simulations received low scores from the DNP students because it was structured so that the medical residents led the code with little opportunity for the nurses to step into the leadership role.
- DNP Project Development Workshops: These sessions were very highly rated.
- Expert Speaker Sessions: All sessions for which evaluations were obtained were very highly rated.
Outcomes Assessment Plan and Results
"Creating a Culture of Safety" learner outcomes were assessed using the following plan:
- Impact of Program on Attitudes Toward Interprofessional Teamwork: Each participating BSN-DNP student and Medical Resident was administered the TeamSTEPPS Teamwork Attitudes Questionnaire (TAQ) Assessment Tool at the start of Year 1 and again at the end of Year 3. Team STEPPS TAQ scores prior to the beginning of the program were high for both residents and DNP students. As a result, post-program TAQ scores did not show significant improvements for the DNP students. Four items related to emotional awareness and support of team members were significantly improved for the medical residents.
- Impact of Program on Self-Perceived Quality Improvement Skills: Each participating BSN-DNP student and Medical Resident was administered the Quality Assessment & Improvement Curriculum (QAIC) Assessment Tool at the start of Year 1 and again at the end of Year 3. Medical resident scores were highly significantly improved (p < 0.001) for all 12 items and total score. DNP student scores were significantly improved for 9 items and total score. These results highlight the effectiveness of the Creating a Culture of Safety program in building team-based quality improvement skills for both DNP students and medical residents.
- Impact of the Program on Development of an Effective IP PSQI Project Proposal: Each participating BSN-DNP student completed a written Objective Structured Clinical Examination (OSCE) during which they created an IP PSQI proposal for a mock case. The OSCE was scored using the Quality Improvement Project Assessment Tool (QIPAT) to assess their ability to create a PSQI project proposal before the project development workshops. Results were then compared with their final formal project proposal submission. Post program scores were highly significantly improved for all 7 QIPAT domains. Those domains for which the DNP students demonstrated the most significant improvements (p < 0.001) included Defining the Problem, Choice of QI Project, and Implementation/Evaluation.
- Assessment of IP PSQI Project Outcomes: All 7 of the BSN-DNP students in the first cohort successfully defended their IP PSQI projects and graduated on time. The projects presented by these students were given high scores by the DNP Capstone Project rubric.
- 5 students presented their projects at local and national conferences
- 5 students have submitted their projects for publication
- 3 students have had their project manuscripts accepted for publication