red Supported Site Towson University (Elementary): Assessment

Successes

  • Towson’s PhysTEC team adopted a multi-pronged approach to project assessment. Observations of the interns’ science lessons gave us the most relevant data concerning our project successes, since these observations allowed us to look at the interns’ actual classroom practice – thereby allowing us to gauge the degree to which we have been successful at fostering inquiry-based instruction at the field experience school sites. The observational data was complemented by survey data that allowed us to obtain both multiple-choice and open-ended written responses concerning the interns’ attitudes, beliefs, course activities, and suggestions for improvement.
  • To conduct the teaching observations, the project team created an observation instrument based on the National Science Education Standards. This instrument can be found in Appendix H.

Challenges

  • As is generally true, survey data can be difficult to interpret. To fully understand all of the interns’ survey responses, we would have needed to conduct interviews. However, our main goal was measured through the observation data, which had the potential for fewer ambiguities.
  • The observation protocol that we developed and used as our primary assessment has the potential to provide rich data for others interested in assessing inquiry instruction. It is difficult to use, however, since the observer must have a fairly sophisticated understanding of inquiry to measure the often subtle nuances of inquiry teaching. Making the protocol more “user-proof” would lose that sensitivity.
  • During our first three years of using our observation instrument, we encountered some difficulties with coding consistency. The first year the two observers (one PI and the TIR) were able to code with a high rate of inter-rater-reliability. In the second year, the new TIR, after training with the PI, coded at a high rate of reliability the first semester; however, the TIR’s second semester codes were quite disparate and had to be revised after a retraining session with the PI. This also occurred with the two new TIRs in the first semester of the third year. Some of this was due to a misinterpretation of the instrument itself rather than a less-than sophisticated understanding of inquiry. These discussions helped us to embed more scaffolding and clarification into the instrument so that novice users can more easily use it.

Sustainability/Physics Department Buy-In

  • The course reforms will continue as long as funding and personnel are available for the course instructor and mentor teacher workshops and any follow-up coordination and communication.

Lessons Learned

  • There was often a mismatch between the interns’ stated orientations toward science teaching (in surveys) and what was observed in classroom practice. Often, interns with stated orientations that appeared to be aligned with inquiry would implement extremely traditional science lessons. This survey/observation mismatch underlined for the project team the importance of relying on teaching observations rather than surveys to obtain valid course impact data.

Assessments and Results

  • The project team adopted a multi-pronged approach to project assessment. In almost every case, the team combined multiple choice and free-response surveys with both informal and formal observations. The observations provided us the most relevant data in that they allowed us to examine actual practice and assess our final goal of inquiry teaching practice by the interns, while the survey data allowed us to obtain information from a larger number of respondents and get accurate statistics on quantitative measures.
  • The PhysTEC team had not attempted to make any substantial changes to the field experience course in Fall 2004, and so the Fall 2004 results reflect the state of the course before any PhysTEC-related course improvements were instituted (i.e., the Fall 2004 data represents baseline data).
  • Pre- and post-surveys were administered to the field experience interns every semester. The pre-survey was a multiple-choice instrument that elicited information about the interns’ orientation and attitudes towards science, science teaching, and inquiry. The post-survey repeated the pre-survey questions, and also included additional multiple choice questions about course experiences (e.g., how often the interns taught during the semester) and some free-response questions about teaching principles and strategies and suggestions for improving the course.
  • The surveys revealed that, as a result of our project activities, when compared to baseline data, the field experience interns spent more time teaching (and less time observing), the interns more frequently taught modified science lessons (rather than teaching the official lessons as-is), and the interns’ attitudes and beliefs about science and science teaching shifted in a more positive direction.
  • To assess the inquiry teaching of the interns, the project team developed an observation protocol based on the National Science Education Standards to document whether the field experience interns’ science lessons had an inquiry focus. More specifically, the protocal was used in conjunction with class observation notes to code various aspects of the interns’ teaching (e.g., focusing on right answers rather than reasoning) as to whether each aspect emphasized traditional teaching, inquiry teaching, or a mixture of both. Intent and success were coded separately to allow for the fact that the interns were novices. The observation protocol was used to evaluate approximately two lessons per course section each semester.
  • The biggest success of the project lay in the shift of the field experience interns’ science lessons toward inquiry. Our observation data revealed that, compared to the Fall 2004 semester, the subsequent semesters’ teaching focused much more frequently on the investigation and analysis of science content, public communication of science ideas, scientific discussion and debate, the use of evidence, and the selection and modification of science activities.