USU studies student evaluations
The online IDEA course evaluations students receive at the end of the semester are here to stay, but university officials are trying to make them more convenient for students to take.
Michael Torrens, director of analysis, assessment, and accreditation at USU, held a workshop last week for faculty on the IDEA rating system as part of the Provost’s Series on Instructional Excellence.
Emails are sent out to students at the end of the semester asking them to evaluate their courses. The information is used by professors to improve their own methods, by the university in the retention and promotion of faculty and for accreditation purposes, according to Torrens.
Torrens said he’s talked to students who say they don’t see the point of the evaluations, but he wants students to know the data is actively used and he encourages students to participate.
In order for the data to be accurate, professors need a response rate above 65 percent from their classes.
Poor response rates are due to a “self-selecting bias,” Torrens said. Often the students who are most motivated to respond are those with strong negative feelings toward the class and professors, which reflects badly in the results.
In the past, the emails asking students to respond went out every three days and often in the middle of the night.
Damon Cann, a political science professor, said late-night emails are bad for the survey results.
“I know the bandwidth is cheaper, but if you annoy the students right before they respond, that’s going to affect the way they respond to the survey,” Cann said.
The university receives complaints about getting too many late-night emails. Torrens said the university lowered the rate of emails being sent to every five days because he got so many messages about it. If students take more surveys with the lowered rate, the reminders will keep dropping.
According to Torrens, students only participate in course evaluations when it’s meaningful to them.
If it’s something extra to the class, response rate tends to be lower. If the evaluation is seen as integral to the course, response rates tend to be higher, Torrens said.
It is up to the professor to decide how to motivate their students to participate.
“If you have a university where the faculty consistently as a group value this and communicate that to the students, and administration communicates that to the students, and the students communicate that to each other about why it’s important, then we’ll have a high response rate and we won’t have to work as hard,” Torrens said.
Torrens said it is controversial about whether it’s fair to offer extra credit or make the evaluation a graded assignment.
Cann sometimes hears of professors offering to bring donuts for the class. But the professors don’t get information on which students respond, so they can’t reward individuals, only classes for reaching certain response levels, he said.
“Some faculty members work really hard to get students to respond, but it does put an additional responsib
ility on the faculty member,” Cann said.
It’s less important whether extra credit is offered or not, but it is a signal to the class that course evaluations are important, Torrens said.
In the old system, the evaluation was on paper and handed out to everyone in the class at once, which meant response rates were high.
“We leave it to students to decide whether they’re going to complete the evaluation form or not, and you will only get students who feel motivated to respond,” Cann said.
For himself, Cann’s response rates have dropped since they switched to the new system. To motivate students, Cann said he tells them he’s interested in hearing what they have to say, reminds them it is important and he uses what they tell him.
Evaluations can be done in three minutes, according to Torrens.
“It’s not a huge burden to do course evaluations, and it’s so important,” he said.
Landon Stauffer, a sophomore majoring in statistics, said it didn’t take him more than five minutes to fill out the evaluation. His professors didn’t push him to take the assignments, but Stauffer feels it’s important for the professors to do better.
“If it had been an assignment, I would have done it no matter what, but (my professors) didn’t push us and I still did it,” Stauffer said.
During the workshop, Torrens went over the process of IDEA and discussed how professors should interpret the results. A week before emails go out to students, professors get their own email telling them to set up their evaluation.
Professors are encouraged to choose three to five objectives as either important or essential from a list of 12. Objectives chosen as essential are weighted twice against the final score, while important objectives are weighted once. Professors ask three questions when choosing objectives: “Is it a significant part of the course?”, “Is something specific done to help students reach that objective?” and “Does it affect their grade in the class?”
Even though professors select three to five to be weighted, students are asked rate all twelve objectives on a scale of one to five and the scores for each objective are reported to the professor.
Because not all objectives are as easy to teach, some objectives are weighted differently than others to make the final averages a more accurate measure of the professor’s success.
“I think that’s a strength of the IDEA system, that faculty members can say these are the things I really want my students to do,” Cann said.
Torrens said three things will lower a professor’s overall score: larger classes, student work habits and student motivation. IDEA asks the student about their work habits and motivation and adjusts the score to account for them.
The following semester, reports from IDEA are mailed in paper form to individual departments and distributed to faculty, but the results are also available online.
Janis Boettinger, USU vice provost, is in charge of organizing the monthly Provost’s Series on Instructional Excellence. She said these last few years, they’ve focused on students ratings.
“It’s so important that we get good data so that ultimately we can improve instructional excellence,” said Boettinger.
A recording of the workshop will be posted on the USU website under the Provost’s Series.
Twitter: @mburnett