CAEP Annual Reporting Measures

 

Impact Measure #1: Impact on PK-12 learning and development

As part of its CAEP accreditation, Relay engaged in a study on the impact of its completers on PK-12 learning and development and benchmarked its completers’ performance against other teachers working in the same school district. Relay found that its completers had statistically significant higher average value added scores in math in 2016, 2017 and 2018 in NYC. Among teachers working in the same district who did not attend Relay, their average value-added score was -.01 in all three years compared with Relay completers’ scores that were .41, .66, and .41 in each of the three years, respectively. In ELA, Relay completers’ growth was greater than those of non-Relay teachers in 2016 and 2017 but not 2018; however, these findings across all three years were not statistically significant. When examining average ELA scores across the three years, Relay completers were able to bring their students to average expected growth in ELA with a weighted average value-added score of .05 (value-added scores of 0 or above indicate average expected growth).
 

Impact Measure #2: Indicators of teaching effectiveness

Relay engaged in a study of the teaching effectiveness of a sample of its completers using the Classroom Assessment Scoring System (CLASS) observation instrument. CLASS was designed by the University of Virginia’s Center for Advanced Studies in Teaching and Learning (CASTL) to assess the quality of teacher-student interactions that are associated with positive academic outcomes for PK-12 students. CLASS covers three domains: emotional support, classroom organization, and instructional support. These domains are further broken out into multiple dimensions. CASTL has validated the CLASS observation instrument in over 2,000 classrooms, and the observation can only be conducted by a certified CLASS observer who has completed CLASS inter-rater reliability training.

Relay found that the completers included in its study are applying the knowledge, skills, and dispositions that its program is designed to impact. Specifically, Relay’s completers in K-3, upper elementary, and secondary classrooms outperformed teachers in comparison studies on nearly all CLASS dimensions.

Tables 1 and 2 benchmark  Relay completer CLASS scores alongside CLASS scores collected from previous studies conducted by CASTL. Table 1 shows that on the K-3 CLASS rubric, Relay completer scores were comparable or higher than scores from the comparison studies for most CLASS dimensions. In particular, Relay completers outperformed teachers in the comparison studies on the Positive Climate, Teacher Sensitivity, Behavior Management, and Productivity dimensions. Table 2 shows that on the upper elementary and secondary CLASS rubric, Relay completer scores were higher or comparable than teacher scores in the comparison studies on every CLASS dimension.

Table 1: CLASS Scores on K-3 Rubric Across Four Studies

Source: Center for Advanced Study of Teaching and Learning. (2019). Classroom Assessment Scoring System. Retrieved April 2, 2019, from https://curry.virginia.edu/classroom-assessment-scoring-system.

Table 2: CLASS Scores on Upper Elementary and Secondary Rubrics Across Four Studies

Source: Center for Advanced Study of Teaching and Learning. (2019). Classroom Assessment Scoring System. Retrieved April 2, 2019, from https://curry.virginia.edu/classroom-assessment-scoring-system.

Impact Measure #3: Satisfaction of employers and employment milestones

Satisfaction of Employers

Findings from Relay’s Employer Survey demonstrate that the majority of principals are satisfied with the preparation their teachers received at Relay.

Initial level programs:

• 75% of surveyed principals responded that completers were moderately, mostly, or fully ready to meet the needs of students after completing their studies with Relay, with 58% stating that completers were mostly to fully ready.
• 75% of surveyed principals believe that completers contributed to an expected level of student learning growth in their classrooms.
• Principals provided the following feedback on specific aspects of completers’ performance in the classroom:
  • 75% of principals responded that completers were “Very Well” or “Well” prepared to implement well-structured lessons.
  • 75% of principals responded that completers were “Very Well” or “Well” prepared to maintain an academic learning environment where students are unafraid to academic risks.
  • 88% of principals responded that completers were “Very Well” or “Well” prepared to make adjustments to practice based on assessment data.
  • 75% of principals responded that completers were “Very Well” or “Well” prepared to meet the diverse needs of learners within the classroom including English Language Learners and students with special needs. 
  • 83% of principals responded that completers were “Very Well” or “Well” prepared to use self-reflection to improve practice.

Advanced level programs:

• 82% of surveyed principals responded that completers were mostly or fully ready to meet the needs of students after completing their studies with Relay.
• 73% of surveyed principals believe that completers contributed to an expected level of student learning growth in their classrooms.
• Principals provided the following feedback on specific aspects of completers’ performance in the classroom:
  • 73% of principals responded that completers were “Very Well” or “Well” prepared to consistently enforce high expectations for all students.
  • 100% of principals responded that completers were “Very Well” or “Well” prepared to implement well-structured lessons.
  • 73% of principals responded that completers were “Very Well” or “Well” prepared to maintain an academic learning environment where students are unafraid to academic risks.
  • 82% of principals responded that completers were “Very Well” or “Well” prepared to make adjustments to practice based on assessment data.
  • 80% of principals responded that completers were “Very Well” or “Well” prepared to meet the diverse needs of learners within the classroom including English Language Learners and students with special needs. 

Employment Milestones

Relay’s Alumni Survey is administered electronically in the fall of each academic year to all program completers. The survey includes a set of questions that are intended to measure completer satisfaction with the quality of teacher preparation they received, as well as completers’ current employment and employment milestones. What follows are highlights from the survey questions that focus on employment milestones.

For initial level and advanced level Programs, 232 alumni respondents who work in PK-12 school settings indicated that they had been promoted since completing Relay’s program. See here for the full list of roles to which completers have been promoted.

 

Impact Measure #4: Satisfaction of completers

What follows are highlights from the Alumni Survey which focuses on completer satisfaction. The survey questions were designed and validated in collaboration with Deans for Impact (DFI), a consortium of deans from schools of education that share best practices and data with each other. 
 

Initial Level Programs

Overall, 82%, 76%, and 77%  of completers responded “Agree” or “Strongly Agree” to the question, “Overall, I am satisfied with the preparation I received at Relay” in 2019-20, 2018-19, and 2017-18, respectively. Findings indicate that 80% or more (responding “Well” or “Very Well”) of completers were particularly satisfied with how well Relay prepared them to:

• Analyze student performance data (90% responded “Well” or “Very Well” in 2019-20);
• Create an environment of high expectations for all students (90% responded “Well” or “Very Well” in 2019-20);
• Adapt practice based on research and student performance data (84% responded “Well” or “Very Well” in 2019-20);
• Align instruction with state and/or professional standards (86% responded “Well” or “Very Well” in 2019-20);
• Develop positive and supportive relationships with students (82% responded “Well” or “Very Well” in 2019-20);
• Provide purposeful feedback to students to guide their learning (84% responded “Well” or “Very Well” in 2019-20);
• Improve my teaching based on feedback and reflection on practice (91% responded “Well” or “Very Well” in 2019-20);
• Maintain a purposeful and effective learning environment (85% responded “Well” or “Very Well” in 2019-20);
• Measure and analyze student growth and achievement (90% responded “Well” or “Very Well” in 2019-20);
• Set challenging and appropriate goals for student learning and performance (86% responded “Well” or “Very Well” in 2019-20);
• Use knowledge of student learning and curriculum to plan instruction (86% responded “Well” or “Very Well” in 2019-20).

 

Advanced Level Programs

Overall, 93%, 83%, and 80%  of completers responded “Agree” or “Strongly Agree” to the question, “Overall, I am very satisfied with the preparation I received at Relay” in 2019-20, 2018-19, and 2017-18, respectively. Findings indicate that 100% of respondents selected “Well” or “Very Well” in 2019-20 for how well Relay prepared them to:

• Adapt practice based on research and student performance data;
• Analyze student performance data;
• Differentiate instruction based on student needs;
• Improve my teaching based on feedback and reflection on practice;
• Measure and analyze student growth and achievement; 
• Provide purposeful feedback to students to guide their learning; and
• Set challenging and appropriate goals for student learning and performance.

Impact Measure #5: Graduation rates

*Data show that more candidates took a leave of absence in academic year 2019-20 than previous years. A potential cause could be disruptions caused by the COVID-19 pandemic.
 

Impact Measure #6: Ability of completers to meet licensing (certification) and any additional certification requirements

New York Campus Licensure Pass Rates, 2016-17, 2017-18, 2018-19; Source: Title II Reports

*Pass rate is not provided when there are less than 10 test takers

After analyzing our data, Relay’s Director of Certification and Deans have identified that the two types of exams our candidates struggle with the most are mathematics exams and multi-subject content exams. We believe that the underlying reasons for underperformance on multi-subject content exams is three-fold.

• Post-baccalaureate candidates in a multi-subject certification area are often tested in content areas in which they may have received minimal coursework prior to enrolling at Relay (e.g., an Elementary Education candidate whose undergraduate major was History or Philosophy has likely taken less mathematics coursework in their secondary and post-secondary schooling as compared to a candidate whose undergraduate major was Statistics or Calculus). 
• A number of candidates who enroll at Relay are not enrolling directly from their undergraduate institution -- many are career changers who have not attended an undergraduate or graduate program in several years. As a result, they likely have not taken coursework in or used much of the content they are being assessed on in several years and therefore need additional support.
• Finally, each multi-subject exam has a mathematics section. As noted previously, mathematics tends to be the most common content area that keeps candidates from passing licensure exams -- whether it’s a secondary mathematics exam or the mathematics portion of a multi-subject exam.

 

As a result, the Director of Certification offers targeted online synchronous workshops in early childhood/elementary math and middle school math to all enrolled and prospective candidates three times per semester. Faculty in our mathematics department have taken incremental steps to incorporate more “pure mathematics” instruction into coursework that is offered to candidates during their first year of the program (although this additional content is limited). We also offer a suite of content-specific resources to candidates, as listed above under question one. All state requirements, exam preparation resources, and Relay certification policies are made readily available to the leadership and faculty at each campus. The Director of Certification has worked closely with campuses to ensure these resources are made available to prospective candidates as early in the admissions and/or enrollment process as possible, so that candidates have an adequate amount of time to review material and access resources as needed.

Currently, the Director of Certification is working with our Academic Programs team and campus leaders to curate national content supports for mathematics, ELA, and Elementary Education content areas. Supports being created include additional Elementary Education Praxis, Elementary Math, Secondary Math, and ELA workshops, two 1-credit math courses, and a test in preparation seminar.

 

Impact Measure #7: Ability of completers to be hired in education positions for which they have prepared

At Relay, all candidates are full-time employees of a PK-12 school partner while they are enrolled with Relay. In order to enroll in Relay’s initial and advanced level programs, prospective candidates must demonstrate that they have secured a full-time classroom position at a partner PK-12 school. As a result, all of Relay’s candidates are hired in education positions while enrolled with Relay. Relay also tracks the ability of its completers to be hired in education positions through its Alumni Survey. Recent survey results are presented below.

Initial Level

• 88% of respondents to the 2019-20 survey are employed in a PK-12 role. The breakdown is as follows:
  • 69%: Classroom teacher
  • 11%: Other role in PK-12 education
  • 6%: Instructional leader
• 1%: School leader
• The majority of respondents (90%) indicated that they intend to remain in the classroom beyond this year.

 

Advanced Level

• 100% of respondents to the 2019-20 survey are employed in a PK-12 role. The breakdown is as follows: 67% are classroom teachers and 22% are instructional leaders.
   

Impact Measure #8: Student loan default rates

The loan default rates in the table below show that Relay’s default rate was 3.7% in FY 2015, 3.2% in FY 2016, and 6.8% in FY 2017, which is the most recent year for which data is available. Relay’s loan default rates are consistently considerably lower than the national default rate, which was 10.1% in FY 2016 and 9.7% in FY 2017.