Ryerson University’s teaching evaluations can no longer affect an instructor’s promotion, tenure or advancement—and it may impact Queen’s.

On June 28, Ryerson reached a decision through arbitrator William Kaplan, who found that student-opinion surveys are affected by student bias towards race, age or even accent.

He awarded that on the surveys, response rates must be clear to show statistical reliability. Individuals assessing the surveys must be educated on the bias that can impact the survey results. 

The arbitration comes as Queen’s approaches its third year of revising its own teaching evaluations. 

Some of the same issues are on the table: the goal for revising USATs includes developing instructions about the ways the assessments should be interpreted.

As Queen’s prepares its revisions, Ryerson’s arbitration decision holds weight—but it’s no court ruling, according to Kingston mediator, arbitrator and workplace investigator John Curtis. 

“They would be wise to take a very careful look at what that decision could imply if a similar [case] was made involving Queen’s,” Curtis told The Journal.

According to Curtis, arbitration can sometimes set a kind of informal precedent. He said it’s highly persuasive to follow the decision of an arbitrator if any “similar facts cases” came up. 

In Queen’s case, Curtis said the University wouldn’t be bound to Ryerson’s decision, but it would it be of value to the decision-making process regardless. 

Revisions, Revisions, Revisions

Every student knows that moment at the end of the course where they can finally share their feedback. Completed anonymously without an instructor present, the USAT is their chance to share their voice. 

According to Queen’s University Registrar’s website, the surveys take questions from the university-level down to the department, teacher and course.

Flipping the page, any thoughts the student has—from course pace to lecture style—goes in the student comment space. 

In the wake of Ryerson’s decision, there’s now legal history to suggest these comments and ratings can be biased.

To address this, Queen’s University Faculty Association (QUFA) and the University are working to update USATs.

On the ground, the Teaching Assessment Implementation Committee (TAIC) is brainstorming changes for the USAT. It’s a sub-committee of the Joint Committee on the Administration of the Agreement (JCAA), created out of a collective agreement between QUFA and the University. 

Both committees have members from the University and QUFA. The co-chairs of the TAIC are Professor Elizabeth Hanson  for QUFA and Professor John Pierce for the University. 

Hanson told to The Journal via email the TAIC is working towards providing the JCAA with recommendations, and the JCAA then has the final say on implementing them.

She went on to explain, “[T]he TAIC has developed a new survey which it will pilot in selected classes at the end of the Fall Term 2018; the majority of classes will use the existing USAT. The TAIC will then analyze the results and submit a report to the JCAA.”

Hanson said that Kaplan’s decision in Ryerson’s case is “based on solid research and reasoning, and addresses practices that are applicable to Queen’s. Therefore QUFA is taking it very seriously.”

Pierce, representing the University, concurred with his co-chair.

QUFA President Kayll Lake said the committee is focused on developing an online version of the survey, to be completed in class. However, there is concern that online surveys lead to lower participation rates. 

For Lake, this would be the worst outcome. 

“Then we would have nothing,” he said.

Lake added that while the revision process still has work to be done, the sub-committee’s recommendations for the JCAA are in line with the Ryerson decision.

“Whatever we do, in order to avoid problems, we’ll have to do what they say. When an arbitrator rules, everyone pays attention,” Lake said. 

Andrew Monkhouse, a Toronto labour and employment lawyer agreed. The decision is likely to be considered as highly important in similar disputes in higher education.

“It’s about how the weight is attributed. Arbitral decisions technically hold the same weight to other arbitrations,” Monkhouse told The Journal. “That weight would be persuasive but not binding on the arbitrator.”

While the Ryerson decision limits the power of teaching assessments, they still remain an important tool to monitor the student experience. 

Eleftherios Soleas, the SGPS representative on the sub-committee, said student voices are still important at Queen’s even if they’re heard through a mechanism other than the USAT. 

“This committee is not looking to discount student voices. They matter an awful lot and were listened to in that room quite a bit,” Soleas said.

He said every decision has been reached by consensus and has been open to student explanations. 

Tom Harris, the Interim Provost and Vice-Principal (Academic) at Queen’s, told The Journal via email that the University has been pleased with the USAT revision process. 

“This is an exciting next step because for the first time graduate and online courses will be included in the survey,” Harris said, discussing the pilot program.

One anonymous member of the TAIC said the new format for USAT’s will be online, and is tentatively being called the Queen’s Student Evaluation of Teaching (QSET). Choosing a vendor for the online platform has delayed the process—on top of turnover and slow-going procedural rules. 

“Often, the committee comes to a point where we’ve reached the end of our mandate, and we need to consult back with the JCAA to receive clearance to continue on, and that can sometimes take a month or so,” the member said.

In the classroom

Meanwhile, instructors are the ones in the middle of the lengthy update process and the changing legal landscape around the evaluations.

Susanne Soederberg, a professor in the Global Development Studies department, said the USATs are a useful, yet imperfect, assessment of teaching.

“USATs are a good barometer in gauging teaching quality. But, like all forms of measurement, it has limits,” Soederberg said.

According to Soederberg, the USATs would provide a more accurate picture of teaching success if they were supplemented with additional documentation, such as letters provided by both undergraduate and graduate students.

A professor in the Chemistry department, Dr. Michael Momborquette, expressed his issues with the USATs in a 2017 Journal article. Ultimately, he believed some students can let their biases muddy the waters of the assessments.

Discussing the Ryerson decision, Momborquette said it addressed all of his concerns.

Momborquette pointed out that the design of the USAT as a factor in promotion and tenure could incentivize instructors to “teach to the USATs,” focusing on higher grades that might translate into higher ratings. He believes this would lead to less deep learning.

Momborquette did recognize that the student comments section is very valuable. He reviews it each time in order to make improvements to his teaching.

However, Momborquette doesn’t agree with the current system of sending these evaluations to the Deans, and the effect they can have on promotion and salary—as negative ratings did for him. “My salary is now going to take a new track,” he said.

As an alternative, Momborquette instead suggested a third-party evaluator attending multiple lectures to accurately evaluate instructors, similar to the process of peer-reviewing a research paper.

“You need to throw the data away, and go sit in [their] lecture,” Momborquette said.