What did they try?

The Education Department at Frostburg State University focused on two areas in the implementation of their PT3 grant: candidate performance and data collection and analysis.

Candidate Performance
To improve candidate performance on the Standard V performance task, they worked with school technology representatives to scaffold activities in the task for teacher candidates. They did this by refining and revising the Maryland Teacher Technology Standard (MTTS)V materials, adding explicit instructions via various forms and tools, and developing materials that could be used as examples in the classroom to share with candidates.

They also revised the peer review process by having candidates make more formal presentations and by including questioning and reflection as part of the review. In addition, they added classroom teachers to the review groups and included training before the review process and a debriefing at the conclusion of the process.

Data Collection and Analysis
TaskStream was purchased to facilitate:

• design and development of student templates, such as a lesson plan format for candidates; portfolio templates for undergraduate and graduate students; and a shared folio containing documents needed by the candidates.
• design and development of Directed Response Folio (DRF) for MTTS performance assessments, for course assessments by phase and program, and for providing reports based on analysis of assessment rubrics.

Training on TaskStream was provided to candidates in technology courses and to faculty in special workshops.

In addition, because the assessment course in the Education Department was dropped, they undertook the process of refining assessment outcomes for programs, mapping out what students need to know about assessment across courses. Over a one-year period, an assessment focus group met to review syllabi, determine what was being taught regarding assessment and what should be taught, and design a scope and sequence for the programs. As part of this process, they conducted face-to-face meetings with P - 12 classroom teachers to get feedback on assessment expectations.

They also undertook the process of refining intern documents. Additions / modifications were made to the internship handbook regarding technology assessments, and modifications were made to the field assessment instrument regarding assessment of technology use.

What worked and why?

Candidate Performance
The forms and procedures developed by the school tech reps gave a good organizational structure to the activities for candidates to learn about/practice the integration of technology in instruction. The integration examples provided by the tech reps were used in instruction in the technology course.

Revisions to the peer review process added a more professional tone to the process and reduced the assessment time needed after the review of each candidate's performance. The participation of classroom teachers as reviewers provided a more authentic and relevant environment for the process. The debriefing after completing the process helped in determining what modifications needed to be made to the process to make it more effective.

Data Collection and Analysis
The selection of TaskStream proved to be effective for students and faculty. Students now are able to

• use the lesson planning template for all of their lessons and find it to be a good time-saving tool.
• use the portfolio template for storage of their documents;
• use the Technology Assessments DRF for submission of their performance assessments and other assessments with very little difficulty;
• store their documents in folders within the system, which has helped them to organize materials and not lose documents;
• access their work from anywhere and easily and quickly create web pages; and
• use the shared folios to access various documents, including phase application forms and the resume form.

Faculty have used TaskStream to

• design rubrics;
• submit assessments, such as the MTTS assessments, and evaluate them through the system; and
• generate reports, such as portions of the Specialized Professional Association (SPA) reports.

The process for refinement of assessment outcomes for programs was successful. There is now a scope and sequence for assessment knowledge and skills across courses. Meeting with P-12 teachers was very informative and helped to give direction to development of a plan.

The process of refining intern documents was also productive. MTTS V and the Technology Environment Survey are now included in the Intern I Handbook that is given to every candidate, supervisor, and mentor teacher. MTTS VII is included in the Intern II Handbook. The Field Assessment Instrument includes indicators relating to the use of technology in the clinical experience, which are things they can do to include in lessons.

What didn't work and why?

Candidate Performance
Even with the scaffolding of activities, candidates still have questions about the 'technology' and want to use hardware that is not appropriate. Some candidates were not able to complete the hardware/software list, and some did not complete the lesson overview summary before developing the lesson. There also was confusion about when to submit the lesson overview summary and who should approve it.

In the peer review process, candidates were not satisfied with the 'points' given for the presentation of their lessons and also had some difficulty adequately reflecting on their lesson preparation and implementation. There also were some technical problems encountered in uploading videos to TaskStream.

Data Collection and Analysis
Related to work with TaskStream, problems were encountered in the following areas:

• Completing TaskStream coordinator work in the time allotted;
• Content and quality of the candidates lesson plans;
• Ever changing ideas for the design of the portfolio and the constant redesign of the resource folio used for the portfolio;
Faculty designing rubrics in TaskStream and using it for assessment of candidate work;
• Faculty lack of familiarity with tools that candidates are using;
• Candidates submitting assessments for evaluation through TaskStream and remembering other procedures given only oral instructions;
• Faculty attendance at workshops.

Related to the refinement of assessment outcomes for programs, two challenges were

• course syllabi not containing enough information to determine what was taught regarding assessment;
• faculty willingness to include the designated assessment information from the scope and sequence in their courses.

Related to refinement of intern documents, the one problem had to do with candidates are not using technology during Internship II without a specific requirement.

What will they do next?

Candidate Performance
Some next steps for improving candidates' performance on MTTS V task will include:

• Finding and using more relevant integration examples for candidates;
• Using more video examples of teaching with technology;
• Providing more workshops for the intern supervisors and mentor teachers relating to MTTS V and the University's expectations.

In the peer review process, next steps will be

• Limiting the types of media that can be used by the candidates for the videos;
• Asking candidates to bring all supplemental materials and examples of student work to the presentation;
• Assigning more 'points' to the presentation and using a rubric to assess the presentation;
• Working on development of reflections; and
• Changing the time and format to more closely resemble the portfolio presentations.

Data Collection and Analysis
Related to the use of TaskStream, next steps include the following:

• Conducting meetings regarding lesson plan formats and revising the lesson plan format based on results from candidates using the current format for three semesters;
• Using the portfolio templates in undergraduate and graduate technology classes to assess usefulness and ease of use;
• Working with a committee of faculty members to define a process for review of electronic portfolios and a timeline for implementation;
• Examining the structure of the current DRFs and moving toward more competency-based structures;
• Seeking agreement on major assessments to be evaluated through TaskStream;
• Examining the reporting structures for TaskStream and make better use of the reporting system;
• Offering more workshops, especially on creating faculty portfolios so that faculty learn how to use many of the TaskStream tools that they are not currently using;
• Creating a TaskStream Committee to help with management of the system.

Related to the refinement of assessment outcomes for programs, the following are some next steps:

• Seeking support for inclusion of assessment in courses based on the scope and sequence plan;
• Seeking support for development of candidate assessments of their knowledge, skills, and dispositions regarding assessment.

Related to the refinement of intern documents, some next steps are:

• Including more specific requirements for Intern II candidates regarding use of technology during their internship;
• Giving feedback to committee members who are redesigning the field assessment instrument to make certain that use of technology is assessed.