Stephen Lowe
July 2009
Frame Of Reference
This article reflects my own, perhaps idiosyncratic, view of the eLearning problem. I certainly don't claim that my ideas might provide solutions for everybody, only for somebody. My focus is on the provinces, not on the big centres. In the provinces human resources are naturally more general, and teachers are good all-rounders before they are specialists.
Introduction
An article on the Times Higher Education website states the problem very well: "Seeking new customers and offering them new products has become the Holy Grail for managers in higher education, but the policy can expose institutions to unforeseen risks" (Utley, 2001). The eight years since Alison Utley wrote those words have certainly been marked by quests for the Holy Grail. There has been a continual push by institution managers to find new markets, and to deliver to them in new ways. In some luxurious and maybe fantasy past, academic staff were shielded from fiscal worries, now they share them with management, and are even held up as scapegoats when projects fail. The silver bullet (Brooks, 1995, p179) of "delivery method: online" runs every chance of being a dud jammed in the breech. People charged with doing the work may have no prior knowledge: not the foundations on which it is built; not the pedagogies (or andragogies) that may work; not how to implement them; not how to maintain them; and certainly and demonstrably not how to estimate and budget for them. Teachers are highly resourceful people, but in many cases too much is being expected of them at the technological frontier. Some institutions can boast success, most need to confess to horrible early failures that did not meet expectation either educationally or fiscally. This article seeks to explore the risk management aspects of this problem, such that managers and academic staff can proceed with better-placed confidence in the future.
Foundations :: Classes of People
If you solicit people off a network to enter your course domain that is a totally different thing to asking them to file through a doorway and sit in rows in a class. These are two entirely different classes of people, and they have very different needs. The industrial era classroom model endures because of who-knows-what deeply ingrained social mores. Many adult students still like this model, preferring it over more modern methods, and for this reason alone it may endure for some time to come. However, it is valid to separate the two things: the classroom model, and the networked model. It is important to separate them not to become muddled in what we do. Muddle increases risk.
Foundations :: Education and Technology
More than ten years ago Ben Shneiderman noted "Novel patterns of teaching/learning" emerging in University of Maryland's electronic classrooms, "from its unidirectional information flow to a more collaborative activity" (Schneiderman, 1998). Combine this with established andragogy like Knowles to put together a methodology for contemporary online courses: "adults need to know why they need to learn something; adults need to learn experientially; adults approach learning as problem-solving; and adults learn best when the topic is of immediate value" (Kearsley, 2009). We can skip a million words to home in on the essence of the change in education—empowered by email, by the bulletin board, by the cellphone, and by YouTube—Oleg Liber (then Director of eLearning at University of Wolverhampton) used the phrase, "Moving the locus of control" (Liber, 2005). In a modern eLearning environment faculty must be prepared to relinquish some measure of control to the student.
Foundations :: Learning Design
Some learning design methodologies are complicated like LD2003 (Koper & Tattersall, 2005), and some are vague like connectivism. The bleeding edge of eLearning thinking is not where you should be to ensure an educationally and fiscally succesful new programme. Working with simple clear learning design patterns that everyone can understand reduces risk. Shneiderman is known for his "four-phase genex framework for generating excellence:
It is a sound framework, and I suggest it should be in every learning designer's toolbox. For example you treat the genex as a macroworld design tool, then add the microworld design tool of the Read-Talk-Do Triangle—you can enter at any point, you can go around in any direction, you must complete all three activities before exiting—and you have something robust you can get started with today. Oleg Liber's Colloquia had individual users constructing activities and inviting friends to participate (Wikipedia) and models like this—personal learning environments—may shape the future of adult learning. Liber made it sound pretty straightforward when he said: "People doing activities using resources" (Liber, 2005).
Foundations :: Maze vs Small World Theory
It has to be realised that Moodle's main function is as a Course Management System (CMS), and not primarily as a learning community. If success is measured in terms of student satisfaction using Moodle does not in itself automatically reduce risk. It could be said that Moodle is an eToy for faculty. Like The Sims without graphics, it lets coordinators assign people to rooms to perform prescribed activities. Many teachers wear pained expressions when first they face the prospect of anybody learning anything in such a maze. To be fair, Moodle does ship with a number of useful "Activities" that Dougimas did conceive in pedagogical terms. To this extent Moodle is also a Learning Management System (LMS). If you are a marathon runner you may have the endurance necessary to create a meaningful course in it. Imagine now some heaven where faculty doesn't exist, and students can connect with ideas and with teachers in a more streamlined model. That could be something as simple as a blog, why not? But if bespoke is on the menu, imagine a learning walled garden (world, microworld) where an evolving small world network (Buchanan, 2002, p53) reduces the separation between the ideas and the people (Lowe, 2008) —now that might be called a Learning Facilitation System (LFS). In the figure below think of Moodle as being on the left of the diagram and my mythical Learning Facilitation System (LFS) in the middle and (dare I say it?) connectivism is on the right.
Figure 1: Small World Networks (reproduced from Buchanan)
Foundations :: Instruments
We are looking at the confluence of three great rivers of risk:
Is our CMS capable of giving answers to all these questions in a sufficiently fine-grained way to inform decision making for the next round? The answer may be no, not all out of one box. In practice we probably cobble it all together from various instruments used by each actor in the process: student, teacher, coordinator, and manager. Imagine a CMS that had plug-in modules that fulfilled the role of LMS, and LFS. Each module could report up, displaying the final outcomes on one dashboard.
Figure 2: Three instruments one dashboard
Foundations :: Risk Management
Whatever Quality system is in force, I suggest CMMI is the best model for the purposes of identifying, managing and mitigating risk in this context. Mapping CMMI to whatever QA system is in place institutionally should be a simple matter of two columns and some connecting arrows. The CMMI is "a merger of process improvement models for systems engineering, software engineering, integrated product development, and software acquisition" (Kulpa & Johnson, 2003, p1). Seen in that context CMMI is indeed a good choice of quality system to apply to eLearning projects. The very name Capability Maturity Model suggests we can start now and gain capability and maturity along the way. Like the Unified Modeling Language (UML) we don't have to buy into the whole thing to use one part of it. In fact all we may need is one diagram. Working with just this one simple diagram will be hugely better than working with no risk management system at all.
Figure 3: CMMI Risk Management (Reproduced from Persse)
Risk Management :: Risk Management Policy Document
The CMMI asks us to: prepare for risk management; identify and analyse risks; mitigate risks. To prepare for risk management we need a risk management policy. That means a working group (small, perhaps three people) to pick relevant stuff out of the CMMI and apply it to the context. The result should be a short document that lays out the guidelines, the expectation.
Risk Management :: Business Risks
In the faculty office the managers need to identify the business risks. The list might look something like this:
Risk Management :: Learning Design Risks
At the coal face the learning designer (and learning advisors, and academic foobars) need to identify the risks. The list might look something like this, where the students:
Risk Management :: Network Risks
In the eLearning suite the Mac operators need to be creating their list of why students may become disenfranchised:
I don't mean these lists to be exhaustive, simply examples of the kind of items one finds on lists like these.
Risk Management :: Example
I'll expand on one from each category:
| ID | Description | Status | Possible causes | Possible mitigations |
|---|---|---|---|---|
| 1.3 | Student peaks early then falls off. | 1|2 |
|
|
| 2.3 | One or two key people become indispensable then quit. | 3|3 |
|
|
| 3.1 - 3.10 | All points grouped as "disfunctional online environment" | 2|3 |
|
|
The column marked Status has two integer values in the range 1 to 3 separated by a pipe. The first number indicates the likelihood of the risk occurring, the second number indicates the weight of the impact on the project. For example a risk with status 1|3 is unlikely to occur, but if it does it will have a serious negative impact on the project. Status changes as a project proceeds and regular reviews can repay the time taken.
Risk Management :: Tools
My own experience of eLearning development was that neither managers, learning designers or techies had any risk management tools. Nobody analysed the risks even in the simple way shown above (it is possible they did, and I just didn't see it). The negative outcomes we experienced could have been avoided had some risk management been done at the outset. The table above, simple as it is, could rightly be described as a tool. It can be implemented in any spreadsheet application, or in a word processor, or on a whiteboard. One such tool is the 1-minute Risk Assessment Tool (Tiwana & Keil, 2004). It takes just one minute to run this, and yet it could save months of misdirected work and associated costs.
| Project Characteristic Question | rating |
x |
weight |
= |
|
| Fit between methodology and type of project | x |
3.0 |
= |
||
| Level of customer involvement | x |
1.9 |
= |
||
| Use of formal project management practices | x |
1.7 |
= |
||
| Similarity to previous projects | x |
1.5 |
= |
||
| Project simplicity (lack of complexity) | x |
1.1 |
= |
||
| Stability of project requirements | x |
0.8 |
= |
||
Overall project risk score |
| Overall risk score | 10 - 28 |
29 - 46 |
47 - 64 |
65 - 82 |
83 - 100 |
| Project risk level | High |
Moderately high |
Medium |
Moderately low |
Low |
In the ratings column enter integer values in the range 0 to 10 where 0 indicates none and 10 indicates strong. Example: A really simple project might score a rating of 9.
Risk Management :: Risk Drivers
Let us look at each element in turn, relating it to an eLearning project from the perspectives of the three main divisions: management, learning designers and techies.
Risk Management :: Risk Driver 1 of 6
Fit between methodology and type of project: [Management] Is this course being set up for online delivery because that's the best way to deliver it? Or because the online learning unit is short of work? Or because some funding has just become available for online courses? [Learning Designers] This time are you going to be allowed to create an instance aligned to your teaching/learning philosophy? Was this course designed to be taught online, or is it a classroom course being crudely repurposed? Has someone suggested converting some old notes into PDFs and whacking them into Moodle? [Techies] Has Moodle been chosen because that's where all the courses go? Or has consideration been given to exactly what environment would suit these materials? Should this course be delivered through animated interactive media [Flash] or are simple images going to be sufficient? Has anybody got the time to create this stuff in Flash?
Risk Management :: Risk Driver 2 of 6
Level of customer involvement: [Management] Did this come out of a drinks party with the CEO of the ITO, or was it a meeting in your office with the Training Manager? Is it political, strategic or tactical? Has anybody surveyed 100 auto workshops to see if they actually want a course in laser alignment techniques? [Learning Designers] Have you met with some typical students? Have you developed, or will you develop a persona called Daniel, and base your designs on Daniel's learning needs? Will you test your storyboards, your early designs, and each unit on a sample of students with similar educational backgrounds to the target market? Will you feed your findings back into the final design? [Techies] Have you done some usability tests of Moodle/Blackboard/First Class on five young farmers pulled at random from the target market group? Have you tested the throughputs on a dial-up at the back of Albury?
Risk Management :: Risk Driver 3 of 6
Use of formal project management practices: [Management] Did anybody do a SWOT analysis on this idea, or a CBA? Has anybody presented a business case and satisfactorily answered the tough questions that were asked at the end? Has a Gannt Chart been set up so we can start to see who will be away on maternity leave at a crucial point of the development? Has a critical path been charted through this project, all the way to the day the students will enrol? [Learning Designers] Are you defining and documenting your processes such that if a key person goes sick they can be replaced by an agency worker? Are you measuring and estimating your time, so that next time around your estimates will be better? Are you creating reusable objects so that next time around you can estimate shorter lead times? [Techies] Do you have security and privacy policies in place that are more than mere small print? Do you have a level-of-service agreement with your students and with the institution? Do you have a disaster recovery plan? Have you run it?
Risk Management :: Risk Driver 4 of 6
Similarity to previous projects: [Management] If you documented the last one, this is going to be your biggest safety asset. Get the people who worked on the last one in and consult with them. Don't make the same mistakes again. [Learning Designers] Now you can roll out those procedures and put some junior staff on the job, they're busting to be given some responsibility, and will rise to the challenge. Now is the time to trot out your reusable objects and feed content into them, winning back your investment of time five-fold. [Techies] Look through the logs and remind yourselves of what had you working weekends. All that stuff can now be averted before it happens, wiser for all that. Save it as, and you're nearly ready to go. The risks of the first time around have just evaporated.
Risk Management :: Risk Driver 5 of 6
Project simplicity (lack of complexity): [Management] If they gives us the exclusive rights to that, and we give them a 40% involvement in this, and then we share the funding on a sliding scale based on volume we'll all be rich. Yeah, right. [Learning Designers] Do we actually need XML to achieve this? Will this schema ever give us payback? Is there not something commercial off the shelf (COTS) that would replace this big ugly monster we're creating? Do we have a rationale for creating anything more complicated than a blog? [Techies] All we want to happen is that the students log in, engage with the programme, and log out again. Developer gold-plating is No.30 on McConnell's list of Classic Mistakes (McConnell, 1996, p43).
Risk Management :: Risk Driver 6 of 6
Stability of project requirements: [Management] Earlier I asked if this was political, strategic or tactical? Let's hope it's tactical with a clear purpose. Nothing is going to be worse for the learning designers, the course, and ultimately the students if the goal posts move half way through development. [Learning Designers] You can only work with what you are given. Maybe there need to be some frank talks with management before work starts, especially if there is the least hint of suspicion that some alliance may change after you have started. [Techies] Please tell us that you are not changing to (or from) Linux in the middle of this particular project. A stable environment politically, socially, and technically is one of the most valuable assets any project can have.
The paragraphs above may state the obvious for people who have experience of these types of projects in the types of institutions in which they usually occur. They also explain the hidden power in the deceptively simple 1-minute Risk Assessment Tool. If the user of the tool is bringing thinking like this to bear—and the more cynical and battle-scarred the approach the better—then the scoring will be good, and the result will be a remarkably accurate prediction of the final outcome.
Procedure
Here's the procedure:
PROCEDURE FOR ESTABLISHING PROJECT RISK MANAGEMENT
CONDITION: You want to motivate and inform your eLearning team in the project risks, their identification, severity, causes and mitigations.
TOOLS & MATERIALS: Big room, tables, chairs, whiteboard. Paper, pens, whiteboard markers. Photocopies of 1-minute tool. Refreshments.
REMARKS: Allow enough time for people to discuss just about every identified risk.
1. Hold a meeting with everybody. Don't let the managers preside over the meeting, pick someone from lower down the food-chain to be MC.
2. Separate into three groups and divide people up not by job but by hair colour, or creed, or anything but their departmental orientation. Brainstorm risks, causes, mitigations and n|n status scores.
3. Bring the groups back together. Paste up all the listed risks, eliminating duplications by general agreement.
4. Create the first draft of Table 1 of this article. Now everybody owns it.
5. Now back into groups, but this time by department. Each department runs Table 2 of this article.
6. Three representatives come to the front and paste up their scores.
7. The lowest score is perceived as the overall project risk.
8. Appoint a Risk Assessor for the duration, they will run the 1-minute tool say weekly as the project unfolds.
9. Drinks, finger food, soothing music, then home.
Steve Lowe July 2009
This article has described some very simple ideas and tools that can be used immediately and quickly to assess, manage and mitigate eLearning project risk. Knowing the differences between conventional classroom-based and network-based students is key. A good working knowledge of learning theories is important, so that you assemble methodologies that will work for your team, for the course, and for the students. Networked learning always needs to take a collaborative approach, otherwise students become isolated. When students become isolated they soon become disenfranchised. There are three broad categories of eLearning project risk: business risks, learning design risks, network delivery risks. Three instruments are needed: CMS, LMS, and LFS. The risks don't cancel each other out, they combine to create greater risk. Risk management is not a big deal to do, but it can make all the difference between success and failure.
Assuming you did get together and run the procedure, and that you ran Step 8 appointing a Risk Assessor, that person may be interested to see a more comprehensive project survival tool. I adapted it from an example in the Software Project Survival Guide (McConnell, 1998) to suit the context of eLearning projects. You score against 33 elements in 5 categories to produce a magic number. The objective is to steadily raise the magic number (reduce the risk) as the project progresses.
Brooks, F.P. (1995) The Mythical Man-Month, essays on software engineering. Anniversary Ed. Boston: Addison-Wesley.
Buchanan, M. (2002) Nexus: Small Worlds and the groundbreaking science of networks. New York: Norton.
Kearsley, G. (2009) Explorations in Learning & Instruction: The Theory Into Practice Database. (Internet) TIP.
Koper, R. & Tattersall, C. (2005) Learning Design. Berlin: Springer. ISBN 3-540-22814-4
Liber, O. (2005) Personal Learning Environments. Oracle Podcast [Online]. Available from: http://www.bath.ac.uk/e-learning/Download/podcasts/20051215-ple.mp3 (Accessed: 21st December 2007). Note: this link is broken, and we need to find where this podcast has gone.
Lowe, S. (2008) Student-user modeling in connectivist learning environments. (Online) Available from: http://itmaru.org.nz/stevelowe/tws/examples/studentusermodeling.pdf Accessed: 2009-07-15.
McConnell, S. (1998) Software Project Survival Guide. Redmond: Microsoft Press.
McConnell, S. (1996) Rapid Development, taming wild software schedules. Redmond: Microsoft Press.
Persse, J. (2006) Process Improvement Essentials: CMMI, ISO 9001, Six Sigma. Sebastopol: O'Reilly.
Shneiderman, B., Borkowski, E.Y., Alavi, M., Norman, K. (1998)
"Emergent Patterns of Teaching/Learning in Electronic Classrooms"
Educational Technology Research and Development 46, 4 (1998, 23-42)
CS-TR-3889 , UMIACS-TR-98-21 Papers of Dr. Ben Schneiderman: Articles.
Shneiderman, B. (February 1999) "Creating Creativity for Everyone: User Interfaces for Supporting Innovation"
CS-TR-3988 , UMIACS-TR-99-10, ISR-TR-99-4 Papers of Dr. Ben Schneiderman: Articles.
Tiwana, A. and Keil, M. "The One Minute Risk Assessment Tool." Communications of the ACM, Vol. 47, No. 11, pp. 73-77, 2004. Available at SSRN: http://ssrn.com/abstract=934013
Utley, A. (2001) Project seeks to cut risky business. (Internet) Times Higher Education.