The Rat Race: An Exploration of Learning versus Performance

My name is Rob Wickham, and I’m a fifth-year graduate student in the Interdepartmental Neuroscience Program and a first-year teaching fellow in the Center for Teaching and Learning here at Yale. My teaching experience includes formerly being a lab instructor at the University of Minnesota for a zoology course for two years, a Teaching Fellow at Yale in psychology and an adjunct professor at Albertus Magnus College.  

A big part of my research focuses on the neural mechanisms of learning, which in part, requires me to assess on a day-to-day basis whether or not the subjects in my experiments learn.  This got me thinking: "How do I know my students are learning?" or better yet…"What are my students learning?" Let's do a quick experiment to demonstrate why these questions matter.  

Figure 1: The rat can solve this problem in multiple, often unexpected, ways.

The experiment

Let's put a rat in a simple two tunnel maze (Figure 1). Each tunnel has a marking on the floor: triangle or star. The goal is to get the rat to learn that the path marked with the triangle leads to the cheese. Over multiple maze runs, he eventually starts going down the triangle path more and more often.  Mission accomplished...right? 

Let’s revisit our goal. Our goal was for the rat to learn that the triangle path leads to the cheese.  The rat performed the maze perfectly--he got the cheese, didn't he? But how do we know he completed the maze by using the symbols on the path? Maybe he guessed. Maybe he learned to choose "left", not "triangle". Maybe he smelled the cheese. Or maybe he did learn that the triangle path led to the cheese.  This is a classic example of the "learning versus performance" (LVP) problem.

The LVP problem, in its essence, is the task of determining "what" is learned by examining the performance, or behavior, of an organism.  Even for the simple maze, we cannot tell "what" the animal learned by simply examining its behavior.  We have to do a few tricks in order to tease this information out:
·         We have to counterbalance (i.e. switch the location of ) which path has the triangle on it.
·         We have to make the paths long enough so the animal can't smell the cheese (or mask the cheese smell with another smell).
·         We have to make sure the animal is not just going down the path that is "not the star". One way to do this would be to add a third symbol (e.g. square), in place of the star. The animal should then learn to avoid this path.
If we do all these things and the rat is still able to get the cheese, then we can probably assume that the rat learned that the triangle path leads to the cheese, and thus, his choosing of the triangle path reflects this learning.

The validity of an assessment
In the classroom, the LVP problem is analogous to assessment validity. Assessment validity is the ability of an educational assessment to predict the degree of learning in a student.  Assessments usually come in two flavors: formative and summative. Formative assessment is generally more informal, such as calling on a student to answer a question, and is used by the instructor to gauge student progress.  Summative assessment is generally more formal and is typically used to generate a grade (e.g. exams, presentations, papers). 

Whether formative or summative, a valid assessment permits good scores (performance) when the student has learned the material and poor scores when the student has not (Figure 2).  An invalid assessment can produce two distinct populations of students: overperformers and underperformers. An overperformer performs better than they should given how much they have learned, whereas an underperformer performs less than their potential given how much they have learned. I know producing an overperformer may sound like a good thing, but like the case of our rat friend, you do not know what your students are taking away from the class. Having a student ace a test, despite learning nothing, isn't much better than having a student fail a test, despite knowing everything.

Figure 2: Valid assessments occur when the degree of learning matches the degree of performance. Invalid assessments can occur either if a student performs below what is expected given the amount of learning achieved (underperformer) or if the student exceeds what is expected given the amount of learning achieved (overperformer).


The valid assessment has a particular relationship between learning and performance. Let's imagine we had perfect information about what our students learned and had the perfect assessment tool (Figure 3; blue dotted line).  Optimally, the relationship would be some variation of a line.  Thus, if a student learns X information they will have Y performance: no less, no greater. If the assessment is invalid, however, deviations from this line will occur. Overperformers will achieve higher scores (Figure 3; green dotted line) and underperformers will achieve lower scores (Figure 3; red dotted line), relative to what they should receive based on the degree of learning. 

Figure 3: Theoretical depiction of the relationship between learning and performance. Blue dotted line: maximum possible performance given the amount of learning achieved. Red dotted line: A student who underperforms.  For underperforming students higher degrees of learning don't produce comparable increases in performance, indicative of a failure to translate and express their knowledge. Green dotted line: A student who overperforms. For overperforming students, lower levels of learning produce higher than achievable performance, suggesting student is using strategies and skills other than the assessments tests.

 

Figure 4: Short-list of potential reasons for underperformance and overpeformance.

Underperformers and overperformers

A valid assessment minimizes the chances of creating underperformers and overperformers. One way to approach the creation of a valid assessment is to understand what makes an individual underperform and overperform. The reasons why a student may underperform are many and varied (Figure 4). Remember--they know the material, but they just simply cannot translate this knowledge into performance. Overperformers, on the other hand, are somehow achieving levels of performance that cannot be explained by their knowledge of the material. One caveat is that overperformers may be integrating information from outside the classroom to achieve these levels of performance. While not necessarily a bad thing, having students use skills outside of the ones taught in the classroom can be a detriment if they need the class skills for a future course or future learning.   

Making a valid assessment

Remember the goal of an assessment is to determine "what" is learned. Learning, unfortunately, cannot be directly measured, and we must use a person's performance to infer learning. Thus, assessment by its nature is a noisy process. That is OK, we are not seeking perfection.  However, there are some simple heuristics to get as close as possible! These tips will be placed in the context of minimizing creating underperformers and overperformers (Figure 4).

(1)  Generate specific learning objectives: The more specific your learning objectives, the more control you have over your students' learning. For example, the learning objective: "learn to integrate mathematical functions" is far too vague. There are numerous ways you can integrate mathematical functions and get the correct answer on an assessment. An alternative version, "learn to integrate functions by u-substitution," is far more precise. Now, the only way to get the correct answer is by using the approach outlined by your learning objective. Then, for the assessment, you can ask students to use this specific approach to answer a problem.  This approach minimizes the likelihood of generating overperformers.

The other benefit of generating specific learning objectives is that it is easier to match assessments to the objective.  Therefore, there will be less of a chance for an assessment that is incompatible (Figure 4: underpeformers) with the content.  For example, if you want students to "organize all the presidents of the United States in order of entering office,” you may want to have an assessment that does exactly that, such as providing a list of presidents and having students organize them. A less compatible assessment would be to have students list the presidents by recalling them (no list available) and then organize the presidents. This type of assessment would be incompatible with the learning objective, since the objective was to "organize" and not "list and organize."  Thus, specific and clear learning objectives can also aid underperformers as well. 

(2) Have students show their work: Permit students the opportunity to show what process they are using in their head.  Most people think of "show your work" for quantitative type problems. For the example above, having students show each step of performing u-substitution (demonstrating learning of the process) is fairly easy to ask. However, this approach can also be used for more qualitative subjects and assessments. For example, in writing composition courses, permit the student to comment on the margins (such as in Microsoft Word) to explain their writing process. Furthermore, the instructor can assign questions about the student's writing process . This also minimizes the likelihood of generating overperformers. Moreover, since showing work permits students to operate outside the bounds of an assignment, there are more opportunities for expressing their ideas. This will also help prevent underperformance.

(3) Use a variety of assessment tools: This approach is highly tailored to help potential underperformers translate their knowledge. In order to perform, a student must access the information they have learned.  Even if they can access the information, they must then translate this information into some observable form. Some students can more easily translate their ideas in writing versus speaking, or vice versa.  Moreover, some students can translate ideas more efficiently if asked a very specific question versus a general, open-ended one, or vice versa. Permitting more translation options for students will minimize underperformance.

(4) Match formative and summative assessments where appropriate:  Asking a student to do something on an exam that they have never done before will generate underperformers. This is not to say you shouldn't ask them to solve a novel problem; rather, the skill or approach required to answer the problem should not be novel. One way to prevent this is by matching the form of assessment in class and during the exam. For example, if you have students do a matching assignment to align the base pairs of DNA in class, also have them do it for a different sequence on an exam. The problem is different, but they are applying the skills and knowledge you want them to learn, and hence, creating a valid assessment.

Let's hear from you!

These heuristics are ones that I have come up with in my experiences as an instructor. Hopefully, they will be valuable to you, but I write them knowing that there are alternative approaches and strategies for creating valid assessments. Please comment below with any other suggestions/heuristics, comments, praise, or disagreements.  I would love to hear your thoughts!