What essential data do you need to measure eLearning ROI?
This article is part of a series on tracking the right metrics using the right methodology to visually display and quantify the investment in learning and prove ROI. What you are going to read is a fable. The company, AshCom, is fictitious, but the learning challenges that Kathryn, AshCom’s CLO, and her team face are real and are widely shared by learning teams in large organizations. It is our hope that you will be able to connect with the characters, their challenges and the solutions they discover. We also invite you to read the first e-book in the series.
e-book release
The Learning Counting Map: How to Measure L&D Impact and Prove ROI
This e-book introduces a system to track the right metrics using the right methodology to visually display and quantify the investment in learning and prove ROI.
The Chief Learning Officer’s challenges
Kathryn crumpled up old sticky notes and put them in her trash can. After fixing two stacks of papers, she put them in files on her desk. She tore off the top page of her daily calendar. It was a new day.
Kathryn, AshCom’s Chief Learning Officer, faced two challenges. The first came from Kurtis, the company’s chief financial officer. After losing money for the first time in the company’s history in the last quarter, Kurtis and his financial team explored every line item in their budget looking for opportunities to reduce their expenses. They called their effort “Defend the Spend”. This applied to Kathryn’s learning team. Kathryn knew that learning notoriety was weak in demonstrating learning returns on investment. AshCom’s sales, operations, and human resources teams were all able to show impressive ROI statistics on well-designed dashboards that gave the financial team’s performance results in real time.
The learning team had nothing comparable. This caused Kathryn considerable tension, especially when budget cuts were on the table for everyone in the organization.
Kurtis asked Kathryn and her team to build a set of new learning experiences focused on preventative maintenance. As Kurtis researched expenses and performance, he quickly realized that AshCom was losing a lot of money because its machinery was not properly maintained until problems became apparent. Machines were far more down for repairs than they should have been. Parts produced per hour were not reached. Scrap rates soared. All of this was related to a lack of a solid preventative maintenance program.
Building the ROL system
Kurtis combined this immediate need with a return-to-learning system he asked Kathryn to build. Kurtis did not need a system at this time to monitor the impact of all learning. It would come later. For now, Kathryn’s team will be expected to build new preventative maintenance training while also building a system that will demonstrate how learning improves performance.
Kathryn was driven to track learning impact by a second factor, more internal and related to her desire for continuous improvement. She wanted her team to understand how their learning experiences work for learners. Too often, she believed her team simply moved on to building the next set of ladders without looking back at what works and what doesn’t.
To achieve both goals, Kathryn arranged a series of meetings with Amy, a trusted learning consultant for the learning team. Amy has more than 20 years of experience in teaching as both an instructional designer and a consultant for some of the largest companies in the United States.
Amy has drawn up a schedule to dive deep into what to consider when building an impact learning structure. They began a review of Kirkpatrick’s model for levels of evaluation and added a fifth level that included the learning team’s thoughts on the experiences they were creating.
Amy showed Kathryn how MindSpring’s The Learning Scorecard software was able to collect the data and weigh it properly. Not every measure was equally important and so every data point was weighed at every level, which provided a net score.
What was needed next was a dashboard to show results in real time. Kathryn knew that her reports to the financial and operating team would not be effective if they consisted of page-to-page spreadsheets. She needed a dashboard that would demonstrate the impact at a glance and match the characteristics of those used by AshCom’s sales, operations, human resources and marketing teams.
The Breakfast Meeting
An overview of dashboards was the topic of the next breakfast meeting between Kathryn and Amy.
“I thought about it a bit,” Kathryn said at the beginning of the meeting, “and I think I know what I want included on our learning dashboard.”
Amy smiled. “Well, like my grandfather always said, ‘Let me say that.’ We need to slow down a bit and talk about what dashboards are and why they can be so useful to business leaders. ”
“I should have expected it,” Kathryn says, giving the smile. “I am under a lot of pressure to get it done.”
“I know you are,” Amy replied. “We will not take much time, but we really have to think through every step of this.”
“I will follow your example,” Kathryn said with a slight hint of impatience.
“I appreciate your trust,” says Amy. “Let’s talk about the origin of dashboards. People like your CEO, CFO and Chief Operating Officer have to make countless business decisions every month. Their ability to make the right decisions is directly linked to the amount of information they have at their disposal. They need to see what works and what does not work. They constantly choose where to spend resources to give them the best business outcomes. Bad decisions are mostly caused by bad information. ”
“And it has to happen fast,” Kathryn says. “I got it. They cannot spend an enormous amount of time digging into every bit of information they need. They rely on people like me and other leaders here to provide that information. ”
“Exactly,” Amy said. “They need the right data, and they need it fast. Most learning teams I work with struggle to provide the data that decision makers want in the time they want it. ”
Collect ROI statistics
“That’s why I’m stressed out by this process,” Kathryn said. “I do not have a good record on this. My reports are usually about how many people followed the training and how they did with knowledge checks. I know it’s a long way from the ROI stats they want, but that’s about all I have most of the time.
“That’s why we’re going through this exercise,” Amy said. “However, do not overlook the benefits of a dashboard for you and your team. You can learn a lot about what works, what doesn’t work, and what you want to do differently next time. ”
“It’s a concern,” Kathryn says. “I’m not sure a dashboard that is meaningful to our CFO would be so beneficial to my learning team.”
“You bring an excellent point,” Amy says. “There are different types of dashboards. Some are tactical, for use with people like you and others in management. Others are strategic control screens used by people like your CFO and chief operating officer. Still others are operational dashboards that will be used by you and your team to make their learning more effective and continually improve. ”
“Does that mean we need three separate systems?” asked Kathryn. “I think it just got a lot more complicated.”
“It doesn’t have to be,” says Amy. “Ideally, your system is collecting, weighing and tracking all the data you need for each dashboard. The only difference is that the dashboard draws different criteria for each specific audience. That’s why I recommend The Learning Scorecard “You can customize multiple control screens to suit your needs.”
“It will make things a lot easier,” Kathryn said.
Go beyond the quantitative data
“I want to add something else to consider,” Amy said. “People hear ‘dashboard’ and they think they can only report numbers. In other words, dashboards are only good for providing quantitative information. ”
“It would be a pity,” says Kathryn. “I would admit that we usually report qualitative data — things like success stories and testimonials. They do not provide the return on investment as our CFO would like to see, but they are an important part of what really happens in learning. ”
“I totally agree,” Amy says. “You do not want to lose it. The Learning Scorecard has a feature that allows these stories and testimonials to be part of the overall narrative for each audience. The ideal mixture is both qualitative and quantitative. Numbers for numbers of people, and stories for story people. ”
“I like it,” Kathryn said.
“Another piece of advice on dashboards,” Amy said. “Make sure you get clarity from each audience about what kind of dashboard they want. It will be worth an hour to spend with Kurtis the chief financial officer to determine what he wants to see in his version. You will have to think about what you want to see and also what your team wants to see in theirs. ”
“So, do we need different levels of access?” asked Kathryn.
“For sure,” Amy said. “Or at least reports that take people to the correct version. I’m not sure if any of your learning outcomes should be confidential, but it should be clarified with Kurtis before you introduce anything. ”
“I feel even more optimistic,” Kathryn said. “For preventive maintenance learning, we have identified what we want to measure and how we intend to measure it. We know what our model is going to be, and we have a way of making the results visible to various audiences within AshCom. ”
“That was the goal,” Amy said. “I created a demonstration control screen in The Learning Scorecard that I thought might be useful to you.”
“Of course you did,” Kathryn replied. “I really appreciate how you led me through all this.”
“You’ll love our next session a lot,” Amy said. “We are going to look at how The Learning Scorecard can not only tell you how your learning works, but it can also help you predict how well new learning products will work in the future. You and your team will love it. ”
“We mostly went on instinct and a few additional data points to decide what to do next,” Kathryn said. “It would be nice to have something stronger on which we can base our decisions.”
“Experience, intuition and stories are all still part of deciding what type of learning experiences connect with your learners,” Amy said. “But it will be a powerful tool for you and your team.”
“I’m looking forward to that conversation,” Kathryn said as Amy stood to leave. “Until next time.”
“Until next time,” Amy says, smiling. “I’ll see you in a few days.”
Closure
Download the e-book The Learning Scorecard: How to Measure L&D Impact and Prove ROI to dig into the data and discover what key metrics your L&D team should consider. You can also join the webinar to discover a whole new approach to measuring ROI.
Dear reader, if you would like to see a demonstration of MindSpring’s Learning Map, please click here to schedule a convenient time for you and the learners at MindSpring will be happy to guide you through it.