What to Do with Cranky Course Feedback

It’s that course again. You slaved over that course for months. You did a thorough task analysis, you consulted with SMEs, you chased down stakeholders. You action mapped, you prototyped, you iterated like no one has iterated before! And though it seems like you finally launched your masterpiece just last week, a year has already passed, and the course up for revision.

Where do you start?

You’ve collected course evaluations over the last year, so you decide to take a look at those, even though you know that “awesome,” “helpful,” and “the best course I ever took” isn’t very actionable feedback.

But wait. What’s this? Your learners couldn’t tell how awesome your course is! Those ungrateful so-and-sos actually said the course was confusing? Frustrating?! Dumb?!!! Who do they think they are? They wouldn’t know the zone of proximal development if they walked right into it!

Should you dismiss these ignorant, unjustified, just plain cranky complaints? Maybe not just yet…

Extracting the nuggets

Let’s face it: every course you build is not going to be an award-winning work of art. But that doesn’t mean that the course evaluations are right and the whole thing is garbage. It may just mean that unlike you —an experienced training professional grounded in adult education theory and learning models —your learners don’t have the the time, or the vocabulary, or the expertise required to properly articulate exactly what its flaws are.

So—being the professional that you are—you dig into those subtle, nuanced critiques (that was sarcasm), in order to extract the nugget of wisdom within each.

“Make the Sale” gets a makeover
To help you through this, we gave one of our own sample courses a thorough critique that didn’t pull any punches. Then, we revised the course to try to address some of the biggest issues. Before you read any further, we invite you to take the original course as if you were one of the target audience: a new sales rep at a travel company that sells cruise packages. This will take you just five to ten minutes.

original_course

Original Course

[Elevator music]

All done? Well, what did you think? Was it confusing? Frustrating? Dumb? Go ahead, don’t hold back. We can take it.

What to do if your course is “confusing”

Why is the course confusing?

Once we asked ourselves this question, we found several reasons to justify this complaint.

  • Prior knowledge is assumed. The first challenge for learners is to establish the customer’s “profile,” but there is no description of each profile that would aid the learner in matching. For example, there isn’t any obvious connection between this customer’s answers and the correct profile, Fun in the Sun.

fun-in-the-sun-profile

Fun in the Sun

  • Learners don’t find out what they did wrong. If you correctly identify the customer’s profile, but haven’t asked all of the “relevant” questions, you get feedback that you may have missed valuable information about the customer’s needs. But only one cruise seems to match each profile, so it’s unclear what the other relevant information might be. (And by the way, did you notice how it is possible to guess the correct profile without asking any questions? If not – try it. If there is a way to “game” a course, learners will find it!)

How did this happen, and how can it be fixed?

When you have a vision of how the course will flow in your head, you can lose sight of the fact that others don’t share the exact same vision. You have to spell it out for them. This can be like having to explain the punchline of a sophisticated joke so many times that it’s not even funny anymore. But sometimes you to have to help learners “get it,” by providing basic instructions.

Sometimes it’s appropriate to have knowledge pre-requisites for an elearning course, but in most cases, all the information that the learner needs to succeed should be provided. Otherwise the learner can feel tricked. You don’t have to add pages of content that the learner is required to click through; it can be an optional resource.

How we addressed these issues in “Make the Sale”

  • We introduced a “coach” character who gives the learner basic instructions before taking calls. coach_speaking
  • We removed the unnecessary step of having learners match customers to profiles, and instead gave them all the information about the cruise packages up front. cruise_binder

What to do if your course is “frustrating”

Why is the course “frustrating”?

There’s a lot of overlap between “confusing” and “frustrating.” Here are a couple of reasons you might have been frustrated.

  • The rules are inconsistent or unclear. There are certain questions that one customer reacts negatively to that other customers don’t. For example, the key to identifying Mr. Sullivan’s profile is that you must ask if he will be bringing children. But if you ask Ms. Kivman the same question, she is flabbergasted. It turns out that this response makes sense only if you ask questions in the order they are listed. Then you would already know that only she and her boyfriend are going on the cruise.
  • Learners lose all progress after an incorrect guess. If you select the wrong profile after asking a few questions, the customer’s previous answers disappear from the screen. That’s like if all of the pellets Pac-Man already ate came back after he got eaten by a ghost! In this case, it’s frustrating because it’s inconsistent with real life – if a sales rep makes an offer that the customer doesn’t appreciate, they don’t start the call over and pretend that no questions were ever asked.

How did this happen, and how can it be fixed?

We’d be willing to bet that if you got similar complaints about one of your courses, it’s because you didn’t prototype and/or test the course with representative learners. It’s an investment to build a truly interactive course, and you can easily waste that investment if you skip this step, and build your course without learner input.

You are too close to the material to judge it objectively. You must test each major interaction in order to confirm that learners understand what they’re supposed to do and why. You don’t have to act on every nitpicky thing that your testers point out, but you will almost always gain valuable information about how learners engage with the material.

How we addressed these issues in “Make the Sale”

  • We added logic to make customers’ responses dependent on which questions had already been asked. For example, if you ask “How old are your children?” the customer doesn’t get offended unless you’ve already asked if they are bringing children, and they said no.
  • We added a separate “Notes” screen that the learner can review any time. For the first caller, Coach helps with the notes. But after that, the learner has to input the notes. This is more realistic – a real sales rep would have to remember or write down what the customer says.
  • We tested and tweaked before we showed the course to representative learners, and then we tweaked and tested some more. We learned valuable information from our testers. For example – it turned out that learners didn’t often use the cruise binder or the notes features. We left them in anyway because they reflect the real work environment, but the learner isn’t required to use them to succeed.

What to do if your course is “dumb”

Why is the course “dumb”?

Well, of course, there are infinite interpretations of this word, but we think the most common cause of this kind of feedback is context.

The interaction lacks realistic context. Why go to the trouble of simulating a conversation with a customer, and then include such unrealistic elements as giving the learner the ability to read customers’ thoughts? Having the customer verbalize objections would be more realistic.

mrsullivan_quote

How did this happen, and how can it be fixed?

Courses lack realistic context when they are designed without directly consulting the people who actually do the job. Deadlines always seem tight, but a good instructional designer will do process and task analysis in consultation with SMEs and target audience members, prior to designing learning activities. That way, the activities can be designed to replicate the environment and the task as closely as possible. (We’ve yet to come across a real-life job that requires telepathy – have you?)

How we addressed this issue in “Make the Sale”

  • We gave each customer a different “question tolerance,” meaning that for each call, there is a different number of questions the rep can ask before the customer gets impatient. The customer gives verbal cues when they are running out of patience, allowing the rep to wrap things up. If the rep is really careless and asks the same question more than once, the caller will just hang up.

caller_hangs_up

    • Instead of the disembodied tips that pop up in typical eLearning courses, we had the coach “monitor the call for quality customer service,” and provide advice via chat. This is very common in call centers.
    • We redesigned the entire graphics and interface of the course to tell the story of the sales rep’s first day on the job. The coach gives rapid instructions while showing the rep to her cubicle, and the call interaction takes place from the rep’s point of view, while sitting at her desk.

illustrated_introduction

  • We enlisted the help of an awesome graphic designer to illustrate the course in a graphic-novel style that we felt would better resonate with the target audience.

Prioritize to manage scope

Just like when you first designed and developed the course, there might be a long list of features you would like to add, but you won’t have time to do everything. Some other enhancements we considered but ultimately tabled when we revised “Make the Sale” were:

  • Add a timer to the call interaction
  • Make the tutorial self-directed instead of led by coach
  • Add a transcript of the entire conversation

Maybe next time!

Follow your proven design and development process

Apply the same rigor and discipline to your revisions that you do when you create a course from scratch. At SmartBuilder, our iterative design and development process includes action mapping, prototyping, and lots of testing. This more time up front, but it saves time and budget later on, when changes are harder to make.

Learn more about our process here.

SmartBuilder’s new and improved course

You can see our revised course here. Let us know what you think, by leaving a comment below.


make_the_sale_course

Revised Course

For help interpreting your cranky feedback, or to discuss giving one of your courses a makeover, please get in touch.


FacebooktwitterlinkedinmailFacebooktwitterlinkedinmail

4 Comments

  1. Eleanor Ferris :

    Nice one. I really like the revamped course. I can see a lot of thought went into the realism. I found I had to actually pay attention to what the customer said in order to suggest the best cruise. Imagine that, e-learning where you can’t just guess!

    • Hi Eleanor, thank you for so much for reading and commenting. You’re exactly right: we imagined e-learning where you can’t just guess, and where there are realistic consequences for mistakes. If you have other applications of these concepts in your projects, please share!

  2. Really like the revamped course. It was more realistic, and required me to think & match information to options. As always, SmartBuilder excellence! Thank you for providing this.

    • Hi Suzanne, thank you so much for reading and commenting. Glad to hear you found the revamped course realistic and that it required you to think things through! If this inspired changes or a different approach to anything you’re working on, please let us know!


Leave a Reply

*