Teacher Engagement Scale for Professional Development

Teacher professional development aims to bolster effective instructional practices that enhance


Introduction
Teacher professional development is critical for school improvement and student achievement (Desimone, 2009;Guskey, 2002;Jones, 2017).Significant resources, both time and money, are spent on professional development each year (Desimone, 2009).While researchers have identified certain features of effective professional development models affecting teacher change (Darling-Hammond et al., 2017), little research has examined the measures used to assess professional development interventions.Further, many professional development efforts lack a robust assessment component (Earley & Porritt, 2014).Ensuring the effectiveness of professional development is imperative.In addition, engaging teachers in professional development is a crucial component of professional development (Garet et al., 2001).Additional research and resources are needed to understand the components of professional development that impact teachers' willingness to engage in professional development.The current study fills a critical gap in the literature by developing an assessment of teacher engagement for professional development interventions.The development of the proposed measure will enable researchers and practitioners to identify teacher engagement behaviors that contribute to more effective classroom practice.Desimone's (2009) seminal work outlined five critical features of effective professional development: content focus, active learning, coherence, sustained attention, and collective participation.Additional studies have highlighted the importance of sustained attention in professional development interventions (Corcoran, 1995;Garet et al., 2001;Whitworth & Chiu, 2015).Professional development programs that are longer in duration can result in positive changes in teacher practice compared to shorter professional development programs or workshops (Whitworth & Chiu, 2015).Research suggests it is imperative to implement professional development that is school-based and job embedded to promote engagement among teachers (Fairman et al., 2020).While opportunities for teachers to engage in active learning are a hallmark of effective professional development (Garet et al., 2001), there is a lack of research about the assessment of the effectiveness of professional development.The few studies that have assessed specific aspects of professional development have limitations, such as focus on short-term changes in classroom practice, lack of follow-up, and reliance on teacher-report data (Palermo & Thomson, 2019;Palmer & Noltemeyer, 2019).
In order to support new behaviors that foster increased student learning, effective professional development must be ongoing and sustained over time (Yoon et al., 2007).The impact of professional development has largely focused on teachers' satisfaction; teachers' learning, use of new practices in the classroom environment, and student outcomes have not been the focus of extant literature (Rhodes et al., 2004;O'Sullivan, 2011).Recent studies have examined the impacts of professional development on teacher learning (Liang et al., 2020;Palermo & Thomson, 2019).Yet, additional research is needed to assess teacher engagement in professional development in order to understand the components of professional development that are most beneficial to teachers.Desimone and Garet (2015) found that changing teachers' classroom behavior was easier than improving content knowledge or inquiry-oriented instruction techniques.In addition, their results suggest that teachers vary in response to the same professional development, and leadership plays a key role in supporting and encouraging teachers to implement strategies and ideas they learned during professional development.A recent study suggests there is an association between perception of usefulness of professional development and positive changes to classroom instruction (Palermo & Thomson, 2019).Taken together, these studies highlight the need for professional development that impacts changes in classroom practices.However, assessing the effectiveness of professional development continues to be an understudied topic.

The Current Study
The aim of the current study was to develop an instrument that could help researchers identify teacher engagement behaviors with sustained professional development that lead to the greatest improvements in effective classroom practice and teacher beliefs about teaching and learning.The present study attempts to develop and validate a measure, Sustained Investment and Teacher Engagement Scale for Professional Development (SITES-PD).The goals of this study were to (1) assess content validity, (2) assess construct validity, and (3) assess predictive validity of the SITE-PD measure.We hypothesize that teacher engagement will serve as a stronger predictor of professional development outcomes than mere number of hours of participation in professional development.

Sample
Across six elementary schools, 108 teachers were enrolled in the study.Teachers self-reported teacher and classroom demographic characteristics.Descriptive characteristics of teachers and students are presented in Table 1.Teachers in the study were mostly female (98%).The majority of teachers identified as White (64%) or Black/African-American (20%).Many teachers in the study had a master's degree (41%); all teachers had at least a bachelor's degree (43%).Teachers' teaching experience ranged from zero to 45 years.Note.Several teachers held more than one certification, hence frequency count outweighs number of teachers.

Procedures
Researchers collaborated with school partners to create "Talent Development Academies" in rural Title I elementary schools with the aim of developing and identifying talent in traditionally underrepresented culturally and linguistically diverse populations.Highly trained teacher coaches held professional development (PD) sessions at specific intervals throughout the year; the teacher coaches also offered voluntary individualized coaching for teachers at each of the elementary schools.Coaches offered demonstrations, co-planning, and co-teaching as a part of their coaching repertoire.
Initial data collection (pre-test) occurred at the end of the academic year prior to initial professional development sessions and served as baseline data.The same data were collected at the end of the academic year when the intervention had been fully implemented (posttest).Data were collected from three sources.First, coaches recorded teachers' participation in mandatory and voluntary PD by number of hours and type of interaction.Coaches were additionally asked to rate teachers' engagement with professional development at the end of each academic year, using the SITES-PD form found in Appendix A. Second, teachers completed a series of questionnaires that gauged teacher perceptions of Teacher Self-Efficacy (Tschannen-Moran, & Woolfolk Hoy, 2001), Teacher Mindset (Gutshall, 2014), and Teacher's Observation of Potential in Students (Coleman et al., 2010).Third, trained and certified observers, blind to the conditions and objectives of the study, visited classrooms to assess the effectiveness of instructional interactions that occurred within the classroom using the Classroom Assessment Scoring System (CLASS; Pianta et al., 2008) and the Classroom Observation Scale-Revised (VanTassel-Baska et al., 2007).

Sustained Investment and Teacher Engagement with Professional Development
Items on the SITES-PD measure assessed purposefulness of teachers' communication, teachers' level of comfort with learning, self-reliance, flexible thinking, perception of students as learners, teachers' views of rules and authority, teachers' views of themselves as professionals, and teachers' assumption of a leadership role.The SITES-PD measure assessed change over time on each of these items by comparing the pre-test scores (assessed at baseline) to posttest scores (assessed after the professional development intervention).The measure was specifically created to assess the effectiveness and usefulness of the professional development intervention.
Throughout the year, two highly trained PD coaches documented all face-to-face and virtual teacher-coach interactions and assigned a value to capture the depth of interaction on a scale of 1 (rote) to 3 (reflective).An example of a rote interaction would be "Can you get me materials for X lesson?" and an example of reflective communication would be "I tried X lesson.I noticed a group of students seemed apprehensive.How can I adapt the next lesson to elicit more brainstorming?"If teachers did not seek out any additional coaching support throughout the academic year, they were assigned a 0. Researchers developed an 8-item global rating survey that implementers completed for each teacher at the end of the academic year.The SITES-PD also had a scale of 1 (rote) to 3 (reflective).Each teacher was rated once at the end of each academic year.See Appendix A for the full version of the SITES-PD measure.

Classroom Assessment Scoring System (CLASS)
The Classroom Assessment Scoring System (CLASS; Pianta et al., 2008) was used to measure the effectiveness of teachers' interactions with children.The CLASS, a standardized observational instrument, measures the effectiveness of classroom social interactions along 10 dimensions.Dimensions are scored on a Likert scale from 1 (low) to 7 (high).The 10 dimensions are aggregated into three domains: Emotional Support, Classroom Organization, and Instructional Support.The Emotional Support domain, which includes positive climate, negative climate (reverse coded), teacher sensitivity, and regard for student perspective dimensions, measures the degree of warmth present in the classroom and the degree to which the teacher is sensitive to the needs and perspectives of students (α = .70).The Classroom Organization domain, which includes behavior management, productivity, and instructional learning formats, measures managerial behaviors and interactions that provide structure and consistency to the learning environment (α = .74).The Instructional Support domain, which includes concept development, quality of feedback, and language modeling, measures the degree to which higher-order learning occurs and the presence of teacher behaviors that promote student understanding and learning through reciprocal interactions (α = .88).
Observers who were CLASS certified and blind to the goals of the study conducted classroom observations.The observers coordinated with teachers to conduct classroom observations at a time that was convenient for the teacher.Blind observers conducted two observation cycles in each classroom; the observation cycles lasted for twenty minutes with ten minutes of coding following the observation cycle.Thus, each teacher was observed for a total of 40 minutes.To ensure inter-rater reliability throughout the study, twenty percent of observations were dual-coded.During dual-coded observations, two observers independently coded the CLASS protocol following the observation cycles and later compared their scores to ensure reliability between coders.Research suggests there is moderate stability in CLASS observation ratings after two cycles (Curby et al., 2011).

Classroom Observation Scale-Revised (COS-R)
To gauge the effectiveness of general and differentiated teaching behaviors, certified and blind observers used the COS-R, a 50-item observation instrument, to assess classroom teaching.This instrument has strong technical adequacy and is highly reliable (α = .91 to .93), with the subscale reliability for all of the clusters averaged above .70(VanTassel-Baska et al., 2007).Observers used the COS-R to examine teaching in teacher instructional behaviors (23 items; α = .89)and student engagement with teacher instruction (23 items; α = .89)across a range of sub-categories, including general teaching/student behaviors (curriculum planning and delivery) and differentiated teaching behaviors (accommodations for individual differences, problem solving, critical thinking strategies, creative thinking strategies, and research strategies).A rating scale of effectiveness for each item is used in the COS-R, with 3 being effective, 2 being somewhat effective, and 1 being ineffective.
The Principal Investigator trained two COS-R observers, and 20% of observations were dual-coded to ensure inter-rater reliability.The COS-R provides evidence of the degree of differentiated strategies teachers are employing and helps project staff see where growth in desired teacher behaviors is occurring.Teachers were rated once per assessment window after scheduled observations that varied in length from 30-50 minutes (average 45 minutes).The COS-R and CLASS observations occurred simultaneously., 2001).Teachers responded to items, such as "How much can you do to get through to the most difficult students?" on a 9-point Likert scale where 1 = "nothing" and 9 = "a great deal."

Teacher Mindset Scale
Mindset refers to the degree to which a person believes talents, skills, or intelligence more broadly are inherent or unchangeable on the 'fixed' end of the spectrum or can be developed and strengthened with practice and effort on the 'growth' end of the spectrum.The Teacher Mindset scale assesses the degree to which teachers believe students can grow their talents and skills with effort across six prompts, for example "You can learn new things but you cannot change your basic intelligence."Teachers were asked to rate each prompt on a 6-point Likert scale where 1 = "strongly disagree" and 6 = "strongly agree" (Gutshall, 2014; α = .82).

Teacher's Observation of Potential in Students (TOPS)
Teachers were asked to provide a rating across nine indicators of talent for each student in their classroom.The nine indicators, including Learns Easily, Shows Advanced Skills, Displays Curiosity and Creativity, Has Strong Interests, Shows Advanced Reasoning and Problem Solving, Displays Spatial Skills, Shows Motivation, Shows Social Perceptiveness, and Displays Leadership, were assessed using a 4-point Likert scale (1 = never to 4 = always).A composite talent score was created using all nine indicators with strong internal consistency (α = .94;Coleman et al., 2010).For the purposes of this study, individual student ratings were aggregated at the teacher level.

Design
Exploratory factor analysis (EFA) rather than confirmatory factor analysis (CFA) was chosen to identify integral constructs underlying the SITES-PD.EFA was employed because of the novelty of the topic (previously unexplored themes of engagement with professional development) created theoretical uncertainty surrounding the underlying structure of the SITES-PD (Browne, 2001) and the potential for stronger structural evidence to emerge during later CFA replications (Goldberg & Velicer, 2006).Principal component factor analysis was employed using Varimax rotation with Kaiser Normalization, given its relative tolerance of multivariate nonnormality and its superior recovery of weak factors (Briggs & MacCallum, 2003;Cudeck, 2000;Fabrigar et al., 1999).Communalities were estimated through squared multiple correlations and were iterated to produce final communality estimates (Gorsuch, 2003).
Results from several investigations demonstrated that MAP and PA are the two best methods for determining the correct number of factors to accept and that the scree test is a useful adjunct (Buja & Eyuboglu, 1992;Glorfeld, 1995;Verlicer et al., 2000;Zwick & Velicer, 1986).

Content Validity
Table 2 provides descriptive information for teacher participation in PD as well as SITES-PD indicators.The last column reveals that hours of PD and the mean quality ratings of all face-to-face and virtual teacher-coach interactions were correlated with the SITES-PD measure.
As well, the SITES-PD indicators showed robust correlations with the summed composite (r = .70-.90).Correlations suggest that the SITES-PD measure reflects a teacher's sustained investment and engagement with professional development across the school year, rather than capturing a recency effect or other unforeseen subjective temporal influences.Note.F2F = face to face.

Construct Validity
Given the high correlations amongst SITES-PD indicators revealed in Table 2, it was anticipated that a one factor solution might emerge from exploratory factor analyses.Indeed, Table 3 indicates a one factor solution.One component was extracted using Principal Component Analysis.The first component explained 69.74% of variance.In keeping with the primary tenet of the American Psychological Association's Task Force on Statistical Inference (Wilkinson & Task Force on Statistical Inference, 1999), which is to employ minimally sufficient statistics when complex analyses are not necessary, we then compared the predictive validity of the factor score against a summed composite for SITES-PD with high internal consistency (α = .94).We ran our regression models for predictive validity twice; first we employed the extracted one factor solution and then we replaced it with the summed composite.
Both solutions arrived at identical findings when regressions were run to assess predictive validity, so the composite is presented below and recommended for future research.Note.Principal Component Analysis.

Predictive Validity
The aim of the study was to develop an instrument that could help researchers identify teacher engagement behaviors with sustained professional development that lead to the greatest improvements in effective classroom practice and teacher beliefs about teaching and learning.
A series of regressions were run with baseline scores covaried to assess whether (a) the total number of structured professional development hours or (b) the quality of sustained engagement (SITES-PD) was able to predict improvement across the academic year.Differentiating between mandatory and voluntary PD participation did not elucidate any interesting patterns (both forms of PD contributed to gains); thus, all PD participation hours were summed.In this set of analyses, baseline scores were compared to outcomes after the first year (but did not include subsequent years of intervention).
Table 4 includes findings from two data sources in order to enhance predictive validity assumptions.First, observers blind to the aims of the study conducted classroom observations using two measures that assess the effectiveness of teacher-student instructional interactions (CLASS and COS-R).Results indicate that the SITES-PD accounted for all variance that could have been attributed to hours of PD participation and provided some additional variance in terms of predicting improvement in teacher beliefs and behaviors across the academic year.Second, teachers rated their own perceptions of Teacher Self-Efficacy, Teacher Mindset, and Teachers' Observation of Potential in Students (TOPS).SITES-PD predicted more variance in score improvements than numbers of hours of PD for all measures with the exception of TOPS.

Discussion
Findings suggest that SITES-PD is a useful instrument for predicting whether the intervention has a positive impact on teachers' beliefs and observed classroom practices, above and beyond the total number of hours teachers participate in PD.The instrument holds promise for offering a deeper understanding of teacher engagement behaviors that may mediate the relation between professional development and classroom outcomes (operationalized here as observed effective instructional practices and teacher perceptions of the learning environment).When analyzed alongside total hours of participation in PD, SITES-PD accounted for variance in six of ten outcome variables and positive trends for additional three outcomes.Whereas, total hours of participation in PD only added unique variance for one of ten outcomes.
In the current study, using professional learning opportunities that combined both mandatory and voluntary sessions and support from coaches, as well as involving teachers in decision-making about content were explicitly built into the sustained professional development model to promote teacher buy-in and engagement.The development of the SITES-PD was inspired by the hypothesis that increased teacher engagement could improve the efficacy of the professional development intervention.Findings reveal teacher engagement, as assessed by SITES-PD, predicted the magnitude of desired outcomes, above and beyond PD quantity.Specifically, teachers who were engaged in the PD intervention throughout the year rated themselves more positively on self-efficacy and mindset, and blind observers noted significant increases in effective instructional practices.Our findings are important, because previous research has indicated an association between substantial professional development and student learning (Yoon et al., 2007).Additional studies have highlighted the need to support and strengthen teacher professional development in order to bolster student academic performance (De Simone, 2020; Gupta & Lee, 2020;Kinnucan-Welsch et al., 2006).
The findings from the current study support previous research that suggests teacher engagement in PD is crucial for changes in teachers' classroom practices (Darling-Hammond et al., 2017).The present study adds evidence that sustained professional development can improve the effectiveness of teacher instruction, and that the mechanism for change is teacher engagement with professional development, as assessed by SITES-PD.Findings reported here suggest the SITES-PD tool demonstrates adequate content, construct, and predictive validity.

Limitations and Future Directions
The present study has a few limitations and offers pathways for future research.First, although the SITES-PD was an effective tool for the current study, the instrument may not work in the same manner with other interventions that have different structures and goals.SITES-PD may need to be adapted for interventions that differ greatly from the intervention described in this study (for example, year-long coaching with teacher-driven content), which may in turn affect the reliability and validity of the measure.Second, the ratings of the SITES-PD were conducted by coaches, which may introduce bias or subjectivity.For the current study, the measure was implemented by observers who were invested in the outcomes of the project.The observers built a rapport with teachers over the course of the study, so remaining objective could have been difficult when conducting the SITES-PD.For our purposes, we collected data from two additional sources (teacher report and blind observers) to assess the reliability of the measure.
To bolster objectivity, coaches were asked to tally all teacher-coach interactions throughout the year and rank the depth of the interaction in the moment.Those moment-to-moment ratings averaged across the year were moderately correlated with the SITES-PD, adding some degree of comfort that the SIES-PD was not prone to recency effects.We further weighed the predictive validity of the SITES-PD against the quantity of PD received.In the context of this study, we were satisfied with the results but other projects that cannot collect data from multiple sources may not be able to ignore the possibility of rater bias.

Conclusion
The SITES-PD tool holds promise for informing future PD assessments and evaluations.SITES-PD can be implemented by other professional development interventions to measure the effectiveness of the professional development experience.While the indicators of the SITES-PD may need to be modified to fit the needs of specific interventions, the tool can assist coaches in identifying educators who could gain the most from professional development.The current tool may also help identify teachers who have not bought in early on, and coaches can work to target and strengthen those teacher-coach relationships.SITES-PD may be a useful formative evaluation tool that provides feedback on opportunities for PD implementation improvement in real time.
3. Did the teacher demonstrate self-reliance?Responsive; relies on one's own judgment and expertise within parameters; internal locus of control.
describes a belief in one's ability to perform at an expected level.Teacher self-efficacy more specifically addresses teachers' beliefs in their ability to reach and teach a variety of learners in their classroom.Teachers offered their perspective of their own teaching efficacy across three domains: (a) Efficacy in Student Engagement (8 items; α = .88),(b) Efficacy in Instructional Practices (8 items; α = .93),and (c) Efficacy in Classroom Management (8 items; α = .88)using the long form of the Teacher Sense of Efficacy Scale (Tschannen-Moran, & Woolfolk Hoy

Table 2 .
Descriptive statistics and correlations for Professional Development (PD) variables

Table 3 .
Exploratory factor analysis