INTIME Evaluation Overview

Two parallel evaluation efforts are designed to address the project effectiveness both formatively and summatively. The main categories of the formative stage of the evaluation process are as follows:

a)      Teacher Reflective Practice

b)      Website Resources

c)      Project Progress

a) The teacher reflective practice takes into account project goals, indicators, benchmarks, and measures as they relate to classroom practice. The following categorization of evaluation data sources lends itself to understanding the complexity of the project:

  • Stages of Concern Questionnaire (SoCQ) data collected from the participating faculty. The first two sets were administered at the beginning and the end of a preparation workshop offered before implementing the INTIME project. All the subsequent sets have been administered at the beginning and the end of each semester during which INTIME was used in the methods classes taught by the participating faculty.

  • Revised Syllabi and Technology Integration Action Plans collected from all participating faculty as evidence of their project-based planning of classroom instruction. The accommodations made to integrate INTIME into the methods classes taught by the participating faculty are also included in the Final Course Revision Reports that have been submitted to the project management team. An in-depth analysis of all these data reveals the fact that the INTIME experienced users, as indicated by the SoCQ positive evolution pattern, have been creative and effective in the overall use of the project materials and technology.

  • INTIME WebCT postings in separate fora for participating faculty and the students enrolled in their methods classes. The rationale for using this method to assess the project effectiveness relies on facilitating reflective thinking and encouraging an exchange of ideas about the integration of technology and components of quality education.

  • Technology Competency Pre- and Post-test data collected from students enrolled in the classes taught by the INTIME participating faculty. The pre-test has been administered at the beginning of the semester, prior to exposing students to the INTIMEproject, while the post-test has been administered at the end of the same semester.

  • Phone interviews with participating faculty, based on the formative evaluation of their to-date use of the project-developed resources. The INTIME management team constantly takes into account the input from all participants and focuses on possible ways to improve certain aspects of the project. The next step of the formative evaluation effort is to encourage the INTIME faculty to reflect on the facilitators and hindrances in the employment of INTIME resources.

  • PT3 Baseline Survey administered to all project participants. This survey is a 35-page formal evaluation instrument used to assess the impact of PT3 grant activities on teacher preparation programs at recipient institutions of higher education.

b) Web Resources

  • Website Usability Questionnaire data collected from the participating faculty. This particular instrument was created by the INTIME management team in an attempt to capitalize on the input from participants regarding their use of the project in the college classroom.

  • Website server report that breaks the overall number of hits and visits over a given period of time down to top pages visited, geographic location of visitors, search engines used, etc.

c) Project Progress

  • Weekly/quarterly/annually project progress reports sent to project staff and the granting PT3 central office. All these reports offer details concerning the status of different project-related tasks and their assigned INTIME staff members, upcoming deadlines, project presentations at various local/national/international conferences, etc.These reports are used to assess the project progress and adapt any further developments to the needs of the participants and other users.

At the end of the third year into the project, a professional evaluation team will examine summatively the effective impact of new learning resources and new standards developed.

Teacher Reflective Practice

Extensive data have been collected in relation to the movement of individual faculty toward integrating technology into their methods courses. These changes have been charted using the Stages of Concern Questionnaire (SoCQ) from the Concerns-Based Adoption Model created by Hall, George, & Rutherford (1977).

[The model] assumes change to be a highly personal and lengthy process, one that affects individuals differently. The model hypothesizes two dimensions along which individuals grow as they become more familiar with and sophisticated in using innovations: Stages of Concern about the Innovation (SoC) and Levels of Use of the Innovation (LoU) (Loucks, Newlove, & Hall, 1998, p.1).    

Data for each of the measures are collected and used by methods faculty involved in course revision to determine the effectiveness of their strategies to integrate technology.

            The following different stages of concern in the questionnaire assess several major factors that influence the adoption process of any given instructional innovation: Awareness, informational, personal, management, consequence, collaboration, and refocusing.

The first stage is related to the degree a user is aware of the innovation. A little over half of the INTIME participating faculty who have completed at least four SoC Questionnaires have high scores on the awareness stage. A possible explanation is that while they have become comfortable with the innovation, they are currently concerned with other issues not related to the project. 

The second stage of the questionnaire focuses on the innovation-centered information that the user has acquired or has yet to acquire. Most of the INTIME respondents’ scores are constantly decreasing, in the average to low region. This indicates that the faculty have acquired content-specific knowledge that enables them to implement the innovation effectively.

The third stage of the questionnaire emphasizes user’s possible personal concerns about the innovation and its consequences. While these concerns reflect uneasiness regarding the innovation, they do not necessarily indicate resistance. The vast majority of the INTIME participating faculty who have completed at least four SoC Questionnaires have fluctuating average to low scores. This indicates increasing comfort using the innovation.

The next stage is related to the degree to which managing the innovation poses problems to the user. Once again, the vast majority of the INTIME respondents display average to low scores that indicate that logistics, time, and management are not hindering issues in using the innovation.

            The fifth stage of the questionnaire represents possible concerns a user may have regarding the consequences of the innovation upon students. Almost all the respondents have average to low scores. This trend indicates not only increasing comfort using the innovation, but also demonstrate the effective communication between the faculty and their students during the implementation process.

The sixth stage deals with concerns a user may have about collaborating with others on the innovation. Half of the INTIME faculty who have completed at least four SoC Questionnaires display high scores on this particular stage. The most likely interpretation is indicative of possible impediments in sharing knowledge and information about the innovation.

The seventh stage of the questionnaire focuses on the various ideas and/or suggestions that a user might have gained from using the innovation. Most of the INTIME respondents have fluctuating average to low scores.  This interpretation also indicates faculty comfort in using the innovation.  However, the results graphs of more than half of the respondents show they have ideas/suggestions as to how the innovation could be improved or implemented differently, thus pointing out refocusing in using the INTIME project.

            Overall, the graphs based on the scores for at least four SoC Questionnaires indicate increasing familiarity with the innovation. The individual differences in scores are related to the specifics of the contexts in which each participating faculty uses INTIME. These data can also be tied with the information gathered from the Website Usability Questionnaire administered to all participating faculty. Their increasing familiarity with the innovation is to be found in the degree to which they find the INTIME online resources accurate content-wise, effective, and helpful in their professional practice.

Revised Syllabi and Technology Integration Action Plans have been collected from all participating faculty as evidence of their planning process centered on INTIME. The project staff identified some of the commonalities among all of these revised syllabi. The focus of this overlap is on the goals that the faculty revisited and adapted to integrate technology into their methods classes. It should also be noted that the more positive the SoCQ evolution pattern (as demonstrated by the graphs mentioned above), the more accommodating to INTIME these revised syllabi are, as follows:

a)      Students are to identify the ways in which technology can support student learning.

b)      Students are to view and critique the INTIME online video vignettes.

c)      Students are to engage in online chat using the INTIME WebCT.

d)      Students are to take a set of pre- and post-tests designed to assess their technology competencies as pre-service teachers.

e)      Instructors are to model the use of varied instructional technologies in alignment with those demonstrated in the INTIME online video vignettes that students are to watch as a class requirement.

INTIME WebCT Postings have been collected from their different fora (by institution) in an attempt to use this kind of qualitative data as a way to assess formatively the project effectiveness. Taking into account the input from participating faculty and the students enrolled in the methods classes they taught at the five universities, the INTIME staff could offer assistance in adapting the project to the various needs of the users. For instance, a team of methods faculty created their own preliminary technology assessment tool based on the INTIME Technology Competencies Test.

            The students who have been exposed to the project in the methods classes taught by the participating faculty also had the opportunity to voice their opinions and concerns, if the case, related to the use of technology in the classroom, as demonstrated by the INTIME online video vignettes. In addition to meeting the requirements of their class by critiquing the video vignettes, these students also share their insights by relating the topic to their experiences either as students or student teachers. The following examples show the wealth of qualitative data the project staff have had to incorporate into their evaluation efforts:

I found that the video "Giving 'Em the Business" presented a new and entertaining way to incorporate mathematics into the elementary school classroom.  Children of this age (or any, for that matter) have a much easier time grasping concepts presented to them through real world scenarios rather then just learning off of a chalkboard.  The video seemed to show the students showing genuine enthusiasm for the projects as opposed to barely staying awake, which is a common occurrence in many math classrooms.  On top of being a great educational opportunity for math, this project also incorporates many other areas such as technology and language arts, two very important fields of study for the student of today.

Overall, this was a very effective, hands-on approach to learning, which involved the kids "doing" instead of just watching.  Plus, planning a business, marketing the product, guessing sell prices, estimating income, is all part of the "real world", so this was a very practical problem. Even though it was a math lesson, it incorporated many other ideas as well.  Technology planned a vital role as well.  Not only were they using a variety of software programs on the computer to design pictures for their business and calculating profit, loss, spending, etc. they also were incorporating other technology, such as the digital camera. It was used effectively, but was not the only thing they used.  They worked technology into the project, but did not over use it, which I thought was important.

I agree that it was important that technology was incorporated, but not overused.  You want for your students to have a solid background in technology but you do not want it to overshadow the basic principles that you are trying to teach.

I agree with what you said. I had the same feelings about technology as you did.  I liked how you also said that they used many different software programs. I had forgotten about them when I wrote my response. The things presented in this lesson do allow the students to feel like they are apart of the "real world."  I had commented that they felt like adults.

            We have also been collecting data on the evolution of technology competencies for students enrolled in the classes taught by the INTIME participating faculty prior and after implementing the project. While implementing the new online learning resources and standards, methods faculty could model the use of instructional technology to their students. Therefore, the technology competencies of pre-service teachers would change over time either positively or negatively. These competencies have been evaluated formally by administering a set of Technology Competencies pre- and post-tests at the beginning and toward the end of a semester when the INTIME project would be used in a methods class.

A negative change on the post-test compared to the pre-test would point out the fact that students could have over-rated their technological skills before using the INTIME project in class. Under these circumstances, the lower post-test scores would be interpreted as students’ better self-reflection regarding their skillfulness in using technology in the classroom as demonstrated by the INTIME video case studies they analyzed in class.

            A positive change could rely on the learning curve that students go through while using the INTIME project in class, as well as on the methods faculty’s modeling of appropriate uses of instructional technology.

            No change in technology competencies could indicate various hindrances that affected the learning process assisted by the INTIME project. In this case, the methods faculty’s insights on the dynamics of the class would reveal the possible causes for the zero difference between the technology competencies pre- and post-test.

For instance, the data collected for the Fall 2001 semester during which the INTIME project was implemented in various methods classes at all five participating universities revealed no change in technology competencies. 269 students took both the pre- and post-tests. A little over half of those students showed no change in their technology competencies. An accurate analysis of the figures would have to take into account the specifics of each class using the INTIME project.

            The PT3 Baseline Survey represents yet another set of data that helps in assessing the effectiveness of the INTIME project as a PT3 grant. An initial look at what a PT3 grant represents in today’s communication era would put things into the right perspective:

Federal, state and local agencies are investing billions to equip schools with computers and modern communication networks, but only one-third of our nation's teachers feel well prepared to use computers and the Internet in their teaching.1

With 98 percent of our schools connected to the Internet and the powerful learning that the Internet makes possible, profound changes in the way future teachers are taught are necessary if we are to meet the demand for teachers prepared to educate 21st century learners.2

In response to this need, the Preparing Tomorrow's Teachers to Use Technology (PT3) initiative awards grants to consortium partners that are working to transform teacher preparation programs.

The challenge of transformation is daunting. PT3 grantees are developing models, tools, support and incentives to help faculty make the change to technology-infused teaching, both within schools of education and throughout the campus. (available at www.pt3.org)

       Under these circumstances, the impact of the INTIME project on the five participating colleges of education is of substantial importance in assessing the effectiveness of this particular PT3 grant.  Based on the data collected from our INTIME faculty, some major differences occurred between the period before and after project implementation:

  • Faculty and students in the participating teacher preparation programs shifted from inadequate to adequate access to technology, including a substantial increase in the availability of hardware, software, computer labs, and supporting staff.

  • Participating faculty engaged in new activities as a result of the project. These primarily included: a) providing professional development to help faculty integrate technology into the curriculum; b) redesigning the curriculum to incorporate technology; c) developing online learning environments; d) obtaining hardware and/or software; identifying model school/college of education programs at other institutions of higher education for preparing teachers to use technology; e) addressing state technology standards for students in their teacher education programs; and f) collaborating with P-12 schools.

  • Participating faculty now more often improve their technology knowledge and skills, expand their curriculum to include more technology-focused classes, and integrate more technology in the curriculum of non-technology classes.

  • Participating faculty changed their curricula to include: a) more online courses; b) more student time spent observing P-12 teachers via electronic means; c) technology courses are completed earlier in the teacher preparation program (as a pre-requisite for both student teaching and for other courses in the program); and d) more technology-focused courses are required.

In addition, faculty have been using more technology in instruction in the following areas: technology basics, using the Internet to provide essential course information to students, using multiple media to communicate information in the classroom, and requiring students to develop lessons and make in-class presentation using technology.

Faculty also offered more core courses in which students are instructed on basic technology skills, classroom information management, ethical issues related to technology, and ways to integrate technology into the curriculum. As a result, faculty perceive their students as more proficient in terms of technology operation and concepts, planning and designing technology-enhanced learning environments and experiences, teaching, learning, and curriculum, assessment and evaluation, and productivity and professional practice. 

Evaluation of Project Developed Web Resources

The Website Usability Questionnaire is an instrument designed by INTIME staff members to evaluate usability of online resources developed by the project and administered among faculty-participants demonstrates that for the vast majority of respondents the overall resources were (1) easy to use, (2) useful for instruction, (3) supportive of teaching and learning styles, and (4) content accurate and free of errors.  When asked to comment on each of the resource components, the faculty-participants expressed overall satisfaction with the format and content of video examples and teacher insights available on the site.  In regards to the theoretical framework of the project –TFQE Model – the faculty also found it to be useful and easy to use for them and their students.  In spite of its later addition to the site, the Probing Questions feature became quite popular among faculty-participants, according to their responses to the instrument.  The majority of them started to use it as soon as it became available and reported that it was user-friendly and beneficial.  Based on the data collected using this questionnaire, the majority of faculty participants used project developed online resources quite frequently in their classes and found them to be supportive of teaching and learning. One example along these lines is offered by the comment one participant makes with regard to the overall usefulness of the INTIME Web site:

This Web site is helpful to visual, auditory, and tactile learning. Students like to see the videos and read the lesson plans, articles, and video captioning. Others like the auditory features of the videos. Tactile learners like the computer aspect. Summary: TELL ME and I’ll forget. SHOW ME and I may remember. INVOLVE ME and I’ll understand. This Web site does all three. 

Server Reports data collected using Webtrends Inc., software is another way to assess the impact of the project and its resources on a regular basis.  These reports provide specific information on visits to the INTIME site such as: who, from where, how often, for how long, pages most frequently hit, etc. The following statistics reflect the most recent evolution in the life of the Web site: a) number of successful hits to date: 4,124,125; b) number of visits to date: 114, 964; c) average number of hits per day: 5,412; d) top geographic region: North America (82,963 visits); e) most active countries: US (81,873), UK (1,136), and Australia (1,071).

 

Project Progress Reports

            The Weekly/quarterly/annually project progress reports sent to project staff and the granting PT3 central office provide stakeholders with specific information related to the various tasks within the project, past and upcoming formal presentations of INTIME at different local/national/international conferences and workshops, as well as the status of the online resources production. Therefore, the granting organization as well the project personnel would be able at any given time to assess how the project goals and objectives are constantly being met. In addition, these reports also provide information related to new directions in which the project is going.

The instruments of both formal and informal evaluation show that the INTIME project has had a major positive impact on the participating faculty and their students. The long-term effects of the project on the teacher education programs involved will be displayed once these students go out into real classrooms and apply their knowledge and skills. Therefore, the INTIME project represents an invaluable resource for all those interested in improving our schools by preparing better teachers and it may serve as a model for future similar projects.

References

Callahan, W.P., & Switzer, T.J.  (2001).  Technology as facilitator of quality education:  A model. In W.Heineke & J. Willis (Eds.), Methods of evaluating educational technology (pp. 215-235).  Greenwich, CT:  Information Age Publishing, Inc.

Hall, G. E., George, A. A., & Rutherford, W. L.  (1998).  Measuring stages of concern about the innovation: A manual for the use of the SoC Questionnaire.  Austin, TX:  Southwest Educational Development Laboratory. 

International Society for Technology in Education (ISTE).  (2000).  National educational technology standards for teachers. [Online]  Available:  http://cnets.iste.org/index3.html [2002, March 15]    

Krueger, K., Hansen, L., Smaldino, S. (2000, April). Preservice teacher technology competencies. Tech Trends 44 (3) 47-50.

 

Loucks, S. F., Newlove, B. W., & Hall, G. E. (1998). Measuring levels of use of the innovation: A manual for trainers, interviewers, and raters. Austin, TX:  Southwest Educational Development Laboratory.