LENS (new SEI)

Adam Driscoll, Sociology (2018-2022)
Nabamita Dutta, Economics (2019-2022)
Jennifer Klein (chair), Biology (2018-2022)
Bryan Kopp, CATL and English (2021-2022)
Tesia Marshik, Psychology (2018-2022)
Alysa Remsburg, Environmental Studies (2019-2022)
Leslie Rogers, Educational Studies (2021-2022)
Alder Yu, Biology (2018-2022)
Kristin Koepke (Ex-Officio), CATL (2020-2022)
Williem Vanroosenbeek, LGBTQA Services (2018-2019)
Marco Vriens, Marketing (2018-2019)
Samuel Ott, undergrad student (2019-2020)
Andre Rowe, graduate student (2018-2019)

Consultant: Kristin Koepke (CATL)

In response to concerns that have been raised in both local and national arenas regarding the validity of student evaluations of instruction (SEIs) and the problematic use of SEIs in personnel decisions, the University of Wisconsin-La Crosse (UWL) Faculty Senate Executive Committee (SEC) established an ad-hoc committee to investigate the use of SEIs on UWL’s campus.  ​

Executive Summary of 2019-2020 Report to faculty Senate

In 2018-2020, the UWL SEI Committee completed (1) an in-depth literature review of SEI tools, focusing on validity and bias, and (2) an investigation of SEI use at UWL. The major findings were described in the 2019-2020 report and are summarized here:

• Multiple recent high-quality meta-analyses have shown that SEI scores generally fail to correlate with student learning.
• Numerous studies indicate that SEI scores are highly sensitive to implicit bias against instructors of color and female instructors. Using SEI scores in personnel decisions could be a discriminatory practice at UWL.
• Many UWL instructors report stress and anxiety due to both the use of SEIs in personnel decisions and in receiving low scores and/or negative student comments.
• Most UWL instructors (>70%) do not find SEIs useful for teaching development.
• Female instructors and instructors of color are unfairly burdened with striving to achieve higher SEI scores by devoting excessive time to altering their teaching or presentation, or in dealing with anxiety related to low SEI scores.
• Resources devoted to improving student perceptions of learning could detract instructors and the institution from higher impact practices.
• UWL students are dissatisfied with the UWL SEI because they feel their voice is not heard on high priority items.

Recommendations to faculty Senate regarding SEI methodology and the use of SEIs in personnel decisions

The SEI Committee has made the following recommendations to Faculty Senate to mitigate bias in the SEI tool and to lessen the impact of biases in personnel decisions:

• Evaluation of teaching effectiveness should include student feedback, but only as a part of a holistic evaluation that includes peer observations, review of teaching materials, instructor development activities, and instructor self-reflection.
• Questions on SEIs should focus on student experiences and the instrument should be framed as an opportunity for student feedback, and not a formal rating of instructor effectiveness.
• Numerical SEI scores should never be used to compare instructors to each other or to a department average (ranked lists are particularly problematic). As part of a holistic assessment, numerical scores can be used to document patterns for an individual instructor member over time.
• Evaluators (all instructors) should be trained in using student ratings as part of a holistic teaching evaluation. Students should also be trained in rating their experiences in a First Year Seminar.

Charge to test and pilot an alternative SEI tool

In 2020-2021 the committee was tasked with determining what a productive student feedback instrument might look like that is less discriminatory, less stressful for instructors, and helps instructors to improve their teaching. The committee searched for existing instruments that adhered to the above recommendations and found just a few. The most fully developed tool was created by Dr. Zinta Byrne, an organizational psychologist at Colorado State University (CSU) and was implemented at CSU for the first time in 2018-2019. We’ve adapted that tool for UWL in collaboration with Dr. Byrne and her colleagues.

Here, we share with you a vision for a revised SEI tool that we will now refer to as the UW-La Crosse Learning Environment Survey (UWL-LENS).

Focus on student experience of learning environment

Students aren’t qualified to formally evaluate their own learning or the effectiveness of their instructor, as they haven’t achieved mastery of the discipline, nor are they experienced teachers. However, students are uniquely situated to describe the course learning environment, which includes, but is not limited to, instructors, peers, course activities, inclusivity of activities and discussions, online platforms, and other technology. The UW System Regent Policy Document 20-2: Student Evaluation of Instruction suggests that in interpreting SEIs “emphasis may to good effect be placed on description of what happens in the classroom.” Knowledge of what happens in the classroom, as the students experience it, is a critical aspect of teaching development and evaluation.

The student learning environment shapes student success, both in terms of academic achievement and the general sense of fulfillment that accompanies learning. With knowledge of how students perceive key aspects of their learning environment, it is possible for instructors to reshape the environment to promote student success. The UWL-LENS therefore focuses on the student’s experience of the learning environment and strictly avoids questions that students are not qualified to respond to.

Based on Research in undergraduate education

The UWL-LENS survey items are based on the Seven Principles For ​Good Practice in Undergraduate Education, originally described by Chickering and Gamson in 1987 (AAHE bulletin 3 (1987): 7):

• Encourages contact between students and instructors
• Develops reciprocity and cooperation among students
• Encourages active learning
• Gives prompt feedback
• Emphasizes time on task
• Communicates high expectations
• Respects diverse talents and ways of learning

These principles of effective teaching have been applied to modern teaching modes, including online education (New directions for teaching and learning 75 (1999): 81; The technology source 30.5 (2001): 50). The LENS questions were created by considering the student experience associated with each principle of effective teaching. For example, effective teaching includes providing prompt feedback. A student can be reasonably asked whether the feedback they received (from the instructor or peers or elsewhere) was useful and timely enough to monitor their own learning and make progress. This contrasts a typical SEI, which might ask students to rate the quality of an instructor’s feedback, which a student isn’t qualified to evaluate. The shift in focus from evaluating the instructor to describing the learning environment is a subtle, but important distinction.

Facilitates evidence-based improvement in teaching

Students are generally eager to provide feedback and desire that instructors will use this feedback to improve the course experience. Since LENS questions are aligned with established principles of effective teaching, survey items can be mapped to resources for instructor development. Student responses are reported as frequency distributions (and ample comments), which allows the instructor to monitor the percentage of students who experience the learning environment in a particular way. When used in a process of reflection, purposeful change, and re-assessment, LENS data can be used as evidence of teaching development in formal teaching evaluations. Of course, instructors drive their own development, choosing focus areas. The LENS will promote movement away from evaluating instructors using a potentially biased and statistically mistreated number and toward a holistic evaluation that focuses on teaching development.

The LENS survey was designed to be given both mid-semester and at the end-of-the-semester so that instructors can respond to student feedback at a point in the semester where changes could be introduced.

Colorado State University has carefully mapped the LENS questions to their framework for effective teaching developed by their center for teaching and learning (https://tilt.colostate.edu/prodev/teaching-effectiveness/tef/). CATL is working to similarly align LENS survey items with their teaching framework.

Reduced bias against instructors

Student evaluations of instruction often include questions that trigger student biases related to the instructor’s gender, race, nationality, and age. Numerous studies have indicated that SEI scores more sensitively reflect these biases than the quality of instruction. The LENS redirects student focus from the instructor to the learning environment to lessen the probability of triggering implicit bias against the instructor. A preliminary pilot of LENS questions at Colorado State University generally supports its effectiveness, but a larger study is needed to clearly establish whether these questions trigger the same biases as the traditional SEI tool at UW-La Crosse. It is important to note that no tool will be free of bias, as implicit bias can occur for survey items seemingly unrelated to gender norms or stereotypes.

The UWL LENS asks no direct questions about the instructor.  All questions are framed to prompt the student to reflect on their experience of the learning environment, which the instructor plays a large role in shaping.

No instructor comparisons

No matter the instrument used to collect student feedback, it is a best practice to avoid comparing instructor scores for the purpose of formal evaluation​s used in personnel decisions, as no tool will be entirely free of bias.

The UWL-LENS was designed so that it doesn't make sense to ascertain an aggregate numerical 'score' that could be used as the basis for comparison among instructors. ​The UWL-LENS presents students with just a few categorical options rather than a numerical Likert-scale. Frequency distributions and student comments are reported. By replacing the Likert-scale with categorical options (that don't map sensibly to numbers), we’ve decreased the probability that an aggregate score could be derived from multiple survey items.  ​

For example, the LENS report would communicate that 42% of students felt feedback ‘occurs early enough to make progress,’ rather than communicating that students rated instructor feedback as an average of 2.8 on a 5-point Likert scale. The current SEI survey uses Likert scales to make it easy to average scores from multiple survey items. Note that this is an inappropriate mathematical manipulation of categorical data (like averaging the responses ‘blue’, ‘green,’ and ‘red’). The resulting aggregate score made for easy comparisons among instructors, but the practice is both statistically and ethically inappropriate. For the LENS, student responses for every question must be considered independently of other questions, which lends itself to focused teaching development.

It would be practical for a Chair or Dean to focus on a couple of key questions on instructor LENS reports for quick insight into how students experience an instructor's classroom. Departments and colleges could choose which LENS items would be particularly helpful for this purpose.

The UW System Regent Policy Document 20-2 encourages system institutions to allow a wide variety of methods for using SEI results for teaching improvement and requires each institution to develop “systematic and firm procedure(s)” guiding the use of SEI results for personnel decisions (e.g. tenure or promotion). It would be reasonable for a subset of LENS results to be included in promotion, retention, and tenure materials, perhaps within the context of teaching development.

Richer data for personnel decisions

The UWL-LENS does not ask students to evaluate the instructor, and shouldn't be used directly for evaluations related to personnel decisions. Instead, the LENS produces a rich dataset that could be used by instructors to reshape the learning environment to maximize student success. ​

A component of a formal teaching evaluation might focus on how an instructor has reflected on student responses, made efforts to reshape the learning environment for the success of all students, and measured the impact of these changes. Evidence of teaching development based on LENS data will result in a more accurate and useful evaluation than the SEI score. ​

Some LENS items might be useful to departments and promotion committees for insight into an instructor’s efficacy in particular domains of teaching. It is valuable to monitor an instructor’s progress along a domain of teaching, particularly if they’ve indicated they have focused development efforts in that area.  It might be reasonable for an instructor to choose one focus area per year and document progress within that domain.

Summary

In summary, the LENS was developed to (1) gather student perspectives and experiences related to their learning environment so that feedback can be used to enhance their course experience, (2) redirect students from evaluating the instructor to describing their learning environment to avoid triggering implicit biases, and (3) create a reliable and informative tool for instructor development.

Learning Environment Survey (LENS) Items

All UWL instructors were invited to participate in a LENS pilot study in Spring 2021.  One hundred UWL instructors chose to allow us to administer the LENS to students in one or more of their classes.  In total, 7,300 students were invited to take the LENS and 2,800 actually completed the survey, for a response rate of 38%.

In summary, the LENS pilot study revealed the following:

• • • A large majority of students (over 80%) felt that the LENS provided sufficient opportunity to address overall feedback and specific concerns. 
    • A large majority of instructors (80-90%) perceived that the LENS provided valuable insight into the student's experience that would be useful in improving student learning; a minority of instructors perceived the SEI this way. 
    • Three times as many instructors felt that the SEI elicited student comments about instructor appearance, gender, etc. than the LENS.
    • Twice as many instructors felt that the SEI elicited stress and anxiety than the LENS.
    • While 74% of instructors clearly preferred the LENS, only 1% of instructors clearly preferred the SEI.   

Full Report on LENS Pilot Study (Spring 2021) 

• LENS Items 
• Faculty Senate LENS Policy (under review by Faculty Senate in March 2022). For the most updated copy of the Faculty Senate LENS Policy approved by the Faculty Senate click here
• Mockup LENS Full Report and Summary Report (preliminary, not yet under review)
• DRAFT LENS Departmental Bylaw Template (preliminary, not yet under review)