Is professional learning effective?

I had the privilege of speaking on this topic at researchED Melbourne last year. This blogpost is an updated expansion.

Schools are as good as the teachers, support staff and leaders within them, and the administrators and policy makers that fund and oversee them. We know a bit about what works best in schools, and the New South Wales Centre for Education Statistics and Evaluation has recently updated their clear guidelines. Significant and sustained change takes time and hard work. Part of it involves investing in professional learning for staff.

I have written in previous posts about the inadequate preparation teachers receive during their training, and professional learning is an inherent requirement of the job. So, teachers start their careers with gaps in knowledge with respect to content (what to teach) and pedagogy (how to teach it) and need to complete 20 hours of professional learning per year to maintain their Victorian Institute of Teaching (VIT) registration. What could possibly go wrong?

What is the point of professional learning?

Most educational researchers in this space agree that professional learning should aim to sequentially and systematically achieve three key things:

- A change in knowledge

- A change in practice

- A change in student outcomes

Professional learning and professional learning agendas set by school leaders and/or staff need to pay attention to these steps, because professional learning does not guarantee:

- A change in teacher knowledge (or worse, it can instill information that is unhelpful or untrue)

- A change in practice

- A change in student outcomes

Equally, a change in knowledge does not mean a change in practice, and a change in practice does not guarantee a change in student outcomes. I’ll get to what we can do later, to try to ensure this process is a successful one.

Is professional learning effective?

Despite there being broad uptake, and it being viewed as essential, the literature consistently demonstrates that most professional learning programs are ineffective. Teachers and school leaders should be up in arms about this, with time and money often being poorly spent. Where does it go wrong? Here are a few of the explanations:

Knowledge building and quality

A lot of professional learning is not designed or delivered in a way that enhances teachers’ content or pedagogical knowledge. That is, specific learning intentions for the session, or details on what to teach and how to teach it are absent. As I mentioned before, a change in knowledge does not guarantee a change in practice, so if we fall at the first hurdle (change in knowledge), we are in dire straits. The quality of professional learning is variable and links to the evidence base are tenuous at best for many offerings.

Time

A focus on the number of days or hours spent on professional learning is not necessarily helpful. The amount of time spent engaged in professional learning is not related to student achievement in most studies. Rather, it depends on the quality of content delivered within the time spent as discussed above. Why? Doing ineffective things for longer does not make them more effective.

Collaboration

Collaboration (sometimes referred to collegiality in the literature) is a consistently reported factor in making or breaking professional learning effectiveness. Note that individuals can collaborate to block change as well as to enhance the process. For collaboration to be impactful, it needs to be structured, purposeful and guided, with clear goals for improving student learning. It usually isn’t.

Analysis

Despite student outcomes being paramount, less than half of the commonly reported ‘effective professional learning factors’ link content delivered to student data in any real way. Professional learning activities should be linked to student data or we have no way of knowing their effect. Of course, links to teacher practice matter too as the mid-point, and this can include expert support, peer observations and coaching.

Evaluation

Professional learning evaluation by presenters is abysmal. Often presenters only gather data on ‘customer’ satisfaction. How was the food? Did you have fun? Was the presenter engaging? Were you too hot, too cold, or just right? Did you learn a lot? For those of us who are presenting professional learning as part of our work, we must consider ways in which we can better measure change in knowledge and/or potential for change in practice. I have found using rating scales helpful as well as short answer and multiple-choice options. I am trying to incorporate these all the time now. Some examples I have used or am trialling are:

Rate your knowledge of ________ before the workshop.

0 1 2 3 4 5 (0 = no knowledge; 5 = expert)

Rate your knowledge of ________ after the workshop.

0 1 2 3 4 5 (0 = no knowledge; 5 = expert)

What did you learn today about ____________ ?

List what you know about _______ now that you didn’t know before today?

What are the first three steps when teaching ______________ ?

List three ways you can incorporate _________ in your teaching of ________ ?

How will you implement ________ when teaching _______ ?

How confident are you in your ability to implement _____ in your practice?

What are the potential barriers and facilitators?

How can you measure the impact of teaching _________ on student outcomes? How can you assess it?

How well did I answer your questions?

0 1 2 3 4 5 (0 = not at all; 5 = answered completely)

How knowledgeable was I about ________ ?

0 1 2 3 4 5 (0 = not at all; 5 = expert)

How clearly did I explain ______ ?

0 1 2 3 4 5 (0 = not at all; 5 = completely clearly)

What could I have done better or differently? (I leave A LOT of space after this one).

I am also including regular retrieval practice and comprehension checks during the workshops, and incorporating far more time for skill practice.

If we are presenting professional learning, we need to be clear about what our learning intentions are beforehand. Our content must be designed purposefully. What knowledge AND skills do we want attendees to leave our session with? And how will we ensure that happens? What does success look like for me and for the attendees?

Of course, still ask about the food and the room ambience. Word spreads quite quickly about an underdone scone, slim pickings, or a cold room.

Implementation support

Implementation support tends to be non-existent. Professional learning sessions are one-offs, job done. Except it isn’t.

There is a problem with the literature

Yoon et al (2007) analysed findings from 1343 studies which purported to examine the effect of professional learning on student outcomes. Only 9 met standards for credible evidence. 7 of 9 studies used standardised assessments of student achievement to track professional learning effectiveness, and 2 used non-standardised measures. What is the lesson? Be careful when reading articles about what makes for effective professional learning.

What were some of the recurrent factors across the nine studies?

They were based on workshops which had focused on the practical implementation of research-based instructional practices with active learning experiences for participants. They were able to show a relationship between the professional learning and student outcomes. Many workshops focus on unproven ideas and strategies, so running a workshop is insufficient on its own as the method of delivery. Content mattered. Content needs to be subject/topic related and pedagogy related. We need to cover what needs to be taught as well as how students will best acquire the knowledge.

They used outside expertise. In-house professional learning has demonstrated some learning consolidation, but the most effective professional learning tends to be delivered by outside experts. By experts, they meant researchers or program authors who presented ideas directly then helped facilitate implementation. Interestingly whole staff, on site, and/or in-house professional learning can be less effective when striving for significant change in practice. In the literature schools have tended to choose professional learning sessions or an agenda already aligned with what they were doing.

Time alone is not effective in terms of measurable impact on student outcomes. In this analysis time was a key factor but this was because the time was spent on quality content. Time in these studies varied from 5-100 hours, but 30 or more professional learning contact hours showed positive effects for student learning. Professional learning therefore requires considerable time and that time needs to be structured and purposeful and focused on both pedagogy and content.

All studies showed positive improvements in student learning when there were significant amounts of structured and sustained follow up/support after the professional learning had concluded.

In closing

Professional learning rarely changes practice on its own. Change is hard and gradual for all of us. Those responsible for planning and implementing professional learning must learn how to critically evaluate its effectiveness. Teachers must demand better evidence from professional learning providers. As many have said, professional learning is a process not an event, and providers, attendees, and school leaders should treat it as such.

Checklist for providers

· Am I an expert in this domain?

· Is the content based on the best available evidence?

· How will I ensure the content is based on the best available evidence?

· Is there opportunity for attendees to observe me demonstrating skills?

· Is there opportunity for attendees to practice new skills?

· Is there an opportunity for repeated practice in a sustained way?

· Is there opportunity for attendees to receive feedback on their practice?

· Is there a plan for how I will support and monitor attendees/schools’ progress after the professional learning session?

· How confident am I that my workshop will result in a change in knowledge and/or a change in practice?

· What data can I collect to evaluate how effective my teaching of content and pedagogy has been?

Checklist for attendees

· Is my presenter an expert in this domain?

· How can I ascertain prior to attendance that the content is based on the best available evidence?

· Does this professional learning session focus on what (content), why (rationales) and how (skill building)?

· Does this professional learning provider and/or my school leader have a plan for how we will implement new knowledge and practice, and how we will map it to student outcomes?

· What assessments am I already conducting in this domain?

· Will my current assessments be sufficient to capture any change to student outcomes following a change in my practice?

Checklist for school leaders

· What is our professional learning agenda?

· What are our current practices with respect to content and pedagogy?

· What do individual staff need, what is needed within learning areas, and what is needed across the whole school?

· How am I supporting my staff to choose evidence-based, effective professional learning? What processes do we have in place?

· How do the sessions my staff attend fit with our aims and agenda?

· How can I monitor and support a change in knowledge and practice in my staff?

· How are we linking professional learning to student outcomes?

· Who are the key staff responsible across the learning areas?

· What does success for staff, students and the school look like? How will we measure it?

A few key readings

Darling-Hammond et al (2017). Effective teacher professional development. Retrieved from https://www.yu.edu/sites/default/files/inline-files/Effective_Teacher_Professional_Development_REPORT.pdf

[This report discusses what makes professional learning effective as well as policy and practice considerations such as implementation of standards for professional development and conducting needs assessments for staff and schools.]

Desimone, L. M., Porter, A. C., Garet, M. S., Yoon, K. S., & Birman, B. F. (2002). Effects of professional development on teachers’ instruction: Results from a three-year longitudinal study. Educational evaluation and policy analysis, 24(2), 81-112.

[This study examined the impact of professional learning on instruction. 207 teachers in 30 schools participated. They found when professional learning focused on specific instructional practices, teachers increased their use of those practices in the classroom. They also found that active learning opportunities (practice) increased the impact of the professional learning on teacher instruction.]

Desimone, L. M., Smith, T. M., & Ueno, K. (2006). Are teachers who need sustained, content-focused professional development getting it? An administrator’s dilemma. Educational Administration Quarterly, 42(2), 179-215.

[The results from this study indicated that teachers with strong content knowledge were more likely to engage in sustained content-focused professional learning than teachers with weak knowledge. That is, effective professional learning is often attended by teachers with already well-developed content knowledge and instructional skill, rather than it addressing knowledge and practice gaps for less knowledgeable/skilled teachers.]

Desimone, L. M. (2009). Improving impact studies of teachers’ professional development: Toward better conceptualizations and measures. Educational Researcher, 38(3), 181-199.

[This study details how we can best assess professional learning effectiveness. Desimone discusses the core features and proposes a conceptual framework.]

Garet, M. S., Porter, A. C., Desimone, L., Birman, B. F., & Yoon, K. S. (2001). What makes professional development effective? Results from a national sample of teachers. American Educational Research Journal, 38(4), 915-945.

[This study looked at what made professional learning effective for 1027 maths and science teachers. They concluded that effective professional learning focused on building content knowledge, active learning (practising the ‘how’) and making connections with other learning activities, which is often called coherence in the literature.]

Guskey, T. R. (2002). Professional development and teacher change. Teachers and teaching, 8(3), 381-391.

[This paper discusses Guskey’s model of teacher change, within which he proposes that sometimes a change in practice and a subsequent change in student outcomes is required in order to change teacher knowledge, attitudes or beliefs.]

Guskey, T. R., & Yoon, K. S. (2009). What works in professional development? Phi Delta Kappan, 90(7), 495-500.

[This paper discusses the difficulty that often arises in translating professional development into student achievement gains despite the intuitive and logical connection. They posit that professional learning planners and providers must critically assess and evaluate what they do.]

Ingvarson, L., Meiers, M., & Beavis, A. (2005). Factors affecting the impact of professional development programs on teachers' knowledge, practice, student outcomes & efficacy. Retrieved from https://research.acer.edu.au/professional_dev/1

[This research report followed a survey of 3250 teachers. Key factors were a content focus, active learning, follow-up on knowledge, and professional community.]

Kang, H. S., Cha, J., & Ha, B. W. (2013). What should we consider in teachers’ professional development impact studies? Based on the conceptual framework of Desimone. Creative education, 4(04), 11.

[This paper discusses Desimone’s conceptual framework for professional learning within which she proposes:

Effective professional learning is content focused, uses active learning, there is coherence, duration, and collective participation.

Effective professional learning affects teachers’ knowledge, their practice, and finally students’ learning.

Contextual factors such as student characteristics, teacher characteristics, and school characteristics influence the effectiveness of professional learning.]