Global and Cultural Awareness Team Updates
2018-19 Scoring Year Activities (reverse chronological order)
Meeting with Team Leads and IERP for review and planning for next steps. Slight adjustments made to summary report which will be posted soon.
Plenary session will be devoted to feedback from evaluators and constributing instructors (will send personal invite). Early fall will be devoted to slight adjustments to areas of the rubric where areas of concern were raised during norming: Skills (Communication) regarding whether the paper itself is the communication and Cultural Diversity regarding the definition of subcultures.
Plan to collect artifacts from fall 2019 and score again in spring 2020.
4/8/2019 - 4/15/2019
Summary Report emailed to Team Leads for review. Individual Instructor Reports emailed to appropriate constributing instructors.
3/1/2019 - 3/22/2019
Scoring window opened/closed
Norming session - 10 am
Members present: J. Alonso, M. McCamish, A. Sanders, M. Spear, A. Kaufman, R. Aslinger, A. Wood, M. McCarthy, N. Mansito, T. Williams
An aggregated score sheet for each artifact was shared with team members. Discussion began with ANTH1230 and ENGL1010_norm2 artifacts because 1) between the 2 artifacts, all areas of the rubric were evaluated and 2) there was disagreement among team members on student performance levels for each artifact.
Team members were reminded that evaluation using the rubric is not the same as grading the assignment and that in order for performance to land in a particular performance level, all descriptors of the performance level must be evident. There are no between scores such as 1.5 or 2.5 - these scores round down to the highest level at which all descriptors of performance are evident.
Members discussed specific examples from each artifact and agreed on a common viewpoint for evaluating those areas of greatest disagreement. The final normed scores were agreed and will serve as a resource for evaluators during the scoring window (along with the normed artifacts)
Email went to scoring team members (12) with access to normed artifacts and resources as well as individually assigned artifact packets for scoring via individual Office 365 shared folders. Each evaluator will score 12 artifacts by March 22, 2019.
Added Michele McCarthy as scorer; email with access to materials sent. Evaluator count at 12. Artifact count still at 274.
Organizational session for norming - 10 am
Members present: J. Alonso, M. McCamish, A. Kaufman, A. Wood, N. Mansito, T. Williams
Discussed timeline for norming process and scoring artifacts. Evaluator count is at 11 currently.Artifact count is 274 currently. Sampling plan discussed.
Donna Seagle (Director, CARE Center) discussed a grant opportunity with AAC&U to be a part of the VALUE Institute and attend a summer conference for the team's part (100 student artifacts submitted and scored by colleagues in AAC&U and feedback given) and the potential for this process feeding into continuous improvement planning. Melanie Ribaric (HIPs Specialist) discussed the potential for developing a GCA HIPs code for courses. She is working with TBR for guidance on the process and minimum criteria.
Email went to scoring members at 4:00 pm with access to norming artifacts and resources. A poll for the norming session will be sent and will be scheduled so the most scorers can attend (either Feb 27 or Mar 1). Method is single shared Office 365 folder.
8/ /2018 - 1/30/2019
Recruitment of contributing instructors
Plenary session -
Team members discussed 1) results from 2016-17 scoring and the work of the review year 2017-18; 2) the recruitment of instructors to contribute artifacts and 3) the use of the revised and expanded rubric for instructional and assignment design. T. Williams shared a form that thsoe contributing artifacts need to complete which allows for 1) indication of to which area(s) of the rubric the submitted assignment aligns and 2) indication of how the instructor thinks the students SHOULD perform on the area(s) - to be used for follow-up once results are in and distributed back to instructor. Team members will go back to respective divisions and departments to recruit volunteers.
2017-2018 Review Year Activities (reverse chronological order)
March 22, 2018
Summary in email from J. Alonso dated 3/23/2018
- We all agreed it is much easier to grade assignments with the new rubric. Everyone appreciates particularly the examples for each score.
- We also agreed the document needs a new format (the current one is very cumbersome). We decided to lighten the rubric, and create separate documents with the details for each indicator. Traci offered to create a PDF that would allow you to review the rubric and access the detail of any indicator by clicking on it. Find attached a sample of how the rubric might look like, and a sample of the detail information for three of the six indicators. Rubric is published now the Rubrics link.
- The score norming activity went according to plan, and we found only one major discrepancy from “Rater 7”. This rater found many “not applicable” where the other raters actually gave scores. Exploring this discrepancy further is recommended.
- Moving forward, now it is time to recruit faculty willing to design a global and cultural awareness assignments for their fall courses. We will collect student artifacts from this assignments at the end of the fall, and work on scoring in the Spring of 2019. Here are a few basic points we discussed regarding recruitment:
- Each one of us in the committee who teach in the HFA and SBS divisions should attempt to design and contribute a global and cultural awareness assignment
- Each one of us in these divisions should try to recruit at least one other person to do the same
- April is the best month to recruit, as it is ePortfolio goal-setting month. We can remind people developing this assignment is a great eportfolio goal.
- Anyone contributing an assignment in 2018-2019 must be ready to contribute to the norming discussion in 2019-2020 (two year ePortfolio goal!)
- When talking to prospective recruits, make sure you share the new rubric materials with them.
- Offer prospective recruits our help to design and review the assignment: it is important the request sounds like an offer of help and collaboration, and not an imposition.
Feb 22, 2018
Team members met to finalize the process of "testing" the revised rubric. Finalized version of rubric shared with team members. Evaluation of the rubric using student artifacts was spread among team members. Artifacts and rubric were shared via a shared folder in OneDrive.
|Global Self-awareness||Michael McAmish||Michael Holsomback||Traci Williams|
|Cultural diversity, understanding differences|
|Knowledge, Cultural Self Awareness||Theresa Lemon||Juan Antonio Alonso||Angie Wood|
|Knowledge, Knowledge of cultural worldview frameworks|
|Skills, Empathy||Katheryn Thompson||Rebecca Aslinger||Nicolas Manisto|
|Skills,Verbal and nonverbal communication|
Dec 8, 2017 - Jan 5, 2018
Team leads finalized the rubric. Discussion of the process to "test" the rubric using student artifacts. T. Williams will collect student artifacts for sharing with team members.
Dec 8, 2017
Team leads (J. Alonso and M. McCamish) met with T. Williams to discuss next steps and finalization of the rubric. The following assignments were distributed:
- Speak to Maria to come up with examples for indicator 6 (JA)
- Explain the type of assignment that would fit each indicator (JA)
- Bring up the definitions of key words to where they are relevant (JA)
- Work out the examples for indicator 1 (JA)
- Work out the empathy benchmark clarifications (JA)
- Cultural Self Awareness indicator (MM)
- Examples for Empathy (MM)
- Collaborate with the type of assignment that would fit each indicator (MM)
Nov 29, 2017
Team members were asked to review the lastest additions to the rubric by a visiitng team member (M. Spear) for communcation (verbal & nonverbal) as well as to offer suggestions to other areas of the rubric. (email from J. Alonso dated 11/29/2017)
Nov 1, 2017
Summary from email from J. Alonso dated 11/6/2017-
This is what we decided in our meeting last Wednesday:
- We have now a description of each benchmark in the onedrive rubric document
- One exception is the “Oral and Written Communication” indicator. We are going to reach out to Speech faculty to have them help us define the benchmarks for that indicator
- We will have another meeting at the end of the month. In the meantime, we are to independently work in the following tasks:
- Come up with INDICATOR definitions (last page of the onedrive document). We believe they are as important as the benchmark descriptions
- Read everything in the document and leave your COMMENTS. Make them as detailed as possible, and do not forget to initial them
- At the very end of the document, we are starting to collect a list of keywords that members of the team feel need further clarification or definition. If you find a term you will like to have clarified, write it at the end of the document. If you feel like you can clarify any items in that list, feel free to write a definition
- The purpose of this meeting is to review all this work and decide on a first draft version of our new rubric
- Once we have this draft version, we will receive some student artifacts and we will regrade them independently (through December and early January)
- We will meet again once the Winter break is over to compare our results and bring up any issues we have encountered with the rubric draft while grading
Oct 10 - 30, 2017
Team members edited a rubric document (shared folder in OneDrive).
Sept 28, 2017
Team members discussed an extensive review of the rubric descriptors as well as the student performance descriptors for each criterion of the rubric. In order to maximize time, team members chose a criterion of the rubric to review before the next meeting. Using student performance scores, sel;ected student samples, and individual evaluator experiences during scoring in 2016-17, members were asked to offer suggestions for clarifying language within the student performance levels. Materials were shared with team members via shared folderr in OneDrive.
From a reminder email from J. Alonso dated 10/2/2017 -
Individuals or pairs are going to take each indicator in the rubric and try to clarify in plain language what each score in that indicator implies and how it would look in a student assignment. The idea is to get back together and contrast and combine these descriptions to create a simple guide to both teach and grade global and cultural awareness. The work groups are going to be:
- Global Self-Awareness: Liz Norell, Rebecca Aslinger
- Cultural Diversity/Understanding differences: Michael Holsomback
- Knowledge I/ Cultural self-awareness: Katheryn Thompson
- Knowledge II/Knowledge of cultural worldview frameworks: Juan Antonio Alonso, Nick Mansito
- Skills/Empathy: Michael McCamish, Angie Wood
- Skills/Verbal and nonverbal communication: Traci Williams