From students seeking cheat codes to embracing dice rolls

Teaching students industry processes isn’t about guaranteeing success, it’s about improving odds. Instructors need to pay attention to both how they teach any given process, and the desperation students typically have to use processes as shortcuts to success.

In the past few years of teaching many different problem-solving and UX design processes, students always find a way to ask a question like “how do I guarantee I am making the right decision?” By nature humans want to find cheat codes. We want a process that will always guarantee a good outcome. 

Instead, students would benefit from flipping that mindset to “how do I improve the odds of making a good decision?” To keep with the games analogy, this would be like learning about strategies that would add dice to your roll before you make a move, rather than hoping for a “get out of jail free” card.

Shifting the mindset to probability can feel at odds with current educational paradigms. It communicates that even if you do everything you are taught, you might still fail. This failure might come from internal and external forces. Internally, your cognitive biases might have had a negative impact on your creative work, or you misinterpreted what stakeholders communicated. Externally, a stakeholder might have changed their expectations, or some environmental disaster altered the conditions that you had started with. 

There’s another challenge with communicating the realities of probability: someone might perform well but not because they did anything particularly smart or good. They might have followed the process incorrectly but externalities changed to favour what was previously inadequate behaviour.

However, both these circumstances are some of the biggest challenges and disruptors outside of the classroom. Ignoring them and requiring students to check boxes to get an A is increasingly absurd in the face of the rapid change we’re seeing in technology. If these factors are communicated correctly by instructors and students are asked to reflect more deeply, I believe it can lead to impressive growth in critical thinking. 

You can immediately see how modern assessments and exams wouldn’t work. Lots of modern marking guides and rubrics give students clear paths to success. In the past I’ve spoken about how simulations can be a great way to learn about empathy, but perhaps there’s more we take from them. Can we modernize assessments to create conditions that make coping with odds of success more educational?

This would have to be more than just adding a random success calculator to an exam or creating some fake externalities. That would be ridiculous! Imagine grading someone’s designs, giving them an A, running the Reality Randomness Generator, and the A becomes an F because the generated fake user got confused and exit the app. “Too bad Billy… it’s just life as a designer!”

Instead, what if we focus on the metacognition that happens after a simulation is failed despite “following all the right steps”. I decided to search for inspiration from NASA, the masters of this kind of analysis, and came across a document titled Techniques For Improving Pilot Recovery From System Failures.

It’s chockfull of insights but there are three ideas I summarized from it that I want to address with a question for each.

Understanding the Limits of Systems and Processes

Research from the documents highlights how more advanced systems don’t always result in better outcomes. Pilots tended to perform better when they were given just enough information, rather than being bombarded. 

This immediately made my mind start thinking about how students believe that if they know more, they will do better in any given circumstance. Yet, the context of the knowledge is more important than its sophistication. 

Over-Reliance on Systems (or Processes)

The report also showed that pilots sometimes over-relied on alerting systems, which could lead to disastrous outcomes if the system provided incorrect information (Just ask the Boeing engineers about MCAS and the 787 Max-8).

Similarly, our assessments can give students the sense that they are doing either better or worse than they actually are. Consider the student that can easily get A’s because they have mastered how to study, but are unable to cope with feedback in the workplace. Inversely, many students are failed because they approached a problem laterally and that simply didn’t fit into the grading criteria.

The Importance of Context and Adaptation

In part of the study, they describe how the effectiveness of any system or process is specific to the context of the emergency and the pilot’s ability to adapt. 

Yet, we continue to test students with little more context than “study everything you’ve learned until now for the upcoming exam.” We provide static testing conditions in the name of standardization rather than creating dynamism so students can practice adapting to new contexts. 

Closing thoughts

I don’t have good answers to these questions at this moment—and there likely won’t be any single good answer that will apply to all contexts. However, I do feel that in creative work, we need to carefully exam what we tell our students to focus on. Creative work and the research required to inspire creativity both come with high degrees of failure. They are naturally riddled with situations where the process doesn’t always lead to expected outcomes. The point is to divert attention away from being correct and towards acting in the way that makes the most sense in a given situation.