2020 Update! The MAEIA Collaborative Scoring System is once again offered to art educators in Michigan in a completely virtual format. Recently “rebooted” through the “Better Together In the Arts” professional development initiative, visual and performing arts teachers are invited to participate in a one hour a month program that combines the previously established MI-CSS as well as teacher social-emotional education. This teacher learning and support group meets once a month virtually via Zoom on Sunday afternoons. MI-CSS allows you to collaborate safely from your home or classroom. Want to join us? It’s not too late and we welcome a growing community of practice! Please visit the MAEIA website to learn more, including information about the availability of scholarships to defray costs of the program. For more information, email Heather Vaughan-Southard at hvsouthard@gmail. We look forward to having you!
In early 2019, I and an eager group of other K-12 Arts Educators volunteered to embark on a new phase of the MAEIA learning community. This new phase was an opportunity to re-visit the process of a Collaborative Scoring Pilot. While an earlier pilot was done in 2018, results were narrowed as the original software revealed a great deal of limitations. The 2019 pilot was shaped based on feedback from the original pilot and new and improved software created by MZ Development. The new software generated a smoother process that yielded more teacher suggestions and even more ways to obtain reliable student assessment data.
The overall purpose of the Michigan-Collaborative Scoring System (MI-CSS) software is to provide a platform that allows arts teachers a way to reliably score responses of their students to teacher selected MAEIA assessments, as well as comparable responses from other teachers’ students. When the MAEIA assessments are scored reliably it can allow teachers to better monitor and improve student learning in the arts which can in turn support and improve teacher effectiveness. When arts assessments and the manner in which they are scored are reliable, we are able to advance the arts as a core element of public education.
Why try collaborative scoring? To quote assessment expert Ed Roeber,
“Results from teacher self-scored assessments may not be considered trustworthy sources of information for educator use in demonstrating their effectiveness. Yet, teacher scoring of student work is some of the strongest professional learning educators participate in. Central scoring (via an independent vendor) produces much more trustworthy scores, but is very expensive. Assessments, such as MAEIA, would be incredibly expensive to centrally score (and funds are not available to do so). The goal of this project is to provide independent scoring of student work by Michigan’s teachers at lower cost than central scoring – which is both a process and a technical issue.”
The collaborative scoring process blends independent scoring and comparative scoring, is less of a technical issue and can be done at a lower cost. Field testing collaborative scoring helps answer the questions: Can it be done? Can it be done reliably? Can it produce viable data to further improve teacher practice?
Teacher participants received a day of in-person training for the MI-CSS software. We then selected and administered two or three MAEIA assessments in our discipline (Dance, Music, Theatre, or Visual Arts) and grade range (K-2, 3-5, 6-8, or high school) to no more than 40 students for each assessment. We used MI-CSS to upload students’ responses (scans of written work, digital images, and/or video or audio files) for each assessment we chose to use. We uploaded work from 25-30 students after each took no more than two or three assessments.
After evidence was uploaded, Teacher Scoring, Second Scoring, Resolution Scoring (done by a Team Lead), and Notating of Scoring could begin. Here is a breakdown of our instructions on each step of the collaboration:
- Teacher Scoring: As the work of each of your students is uploaded, use the Teacher Scoring Rubric(s) provided within the software system (the same as those found in the Teacher Booklets) to score each student on each scoring dimension.
- Second Scoring: View and score the responses from other teachers’ students, using the same processes as you will have used to score the work of your own students. The Second scorer does not see the scores given by the teacher scorer and thus scores independently.
- Resolution Scoring: We are asking Team Leads to resolve any differences in scoring between the teacher who first scored the work and that of the second scoring. The Resolution scorer does not see the scores given by the teacher or the second scorer, but rather is asked to score a third time independently as a “tie breaker” if the first two scorers did not agree.
- Notate scoring as desired: One new feature of the MI-CSS system is for the teacher (first scorer), other participating teachers (second scorers), and resolution scorer (Team Leads) to send notes to one another anonymously. Such notes might include how the assessment was modified or administered, how particular students responded to it, challenges in scoring the work of students, and/or how one or more of the students were scored. This is an optional feature of MI-CSS
I teach K-12 Visual Arts. In this field test, I concentrated on the assessment of my middle school students. I was grouped with a middle school visual arts Team Lead and fellow middle school art teachers. The ability to network and connect with the other teachers in my group was a major benefit of this program. I also increased my competency in the use of technology as well as my ability to plan and properly administer MAEIA assessments. It was interesting to view, analyze and score the artwork created in my colleagues’ classrooms. Without knowing their students, I was able to score them objectively. They, in turn, did the same for me.
The process was not without frustration or flaw, which is expectedly uncovered in any pilot. Confusing wording or point values in some MAEIA rubrics were sometimes a challenge. Meshing the timing of the workflow of all involved was a challenge as all teachers are simply very busy. The ability of the software to accept Google Documents or work from Chromebook style computers was also a hurdle. Software techs made improvements and tweaks along the way.
Future plans and improved capabilities of MI-CSS include making a variety of score reports available such as: individual student reports, classroom or school reports, scoring accuracy reports for the teacher. Also, it is hoped that in the future teachers will select or voluntarily be assigned a scoring group with which to collaborate based on one or more of the following: school district, arts discipline, region of the state, and/or grade level assigned.
Are you interested in learning more? Keep following MAEIA to be updated on the evolution of the MI-CSS!
Carrie Jeruzal is a MAEIA Leadership Fellow and Visual Arts Educator in Pentwater, MI. She was honored by the National Art Education Association as the 2017 Western Region Middle Level Art Teacher of the Year.Click here for a Printer friendly version of this article.