Wednesday 20 May 2009

How To Write Multiple Choice Questions (MCQs)

Multiple Choice questions have been used by educationalists since the early 1900s. Constructed properly and written well, MCQs can be an effective tool for assessing skills and knowledge in your students. Written badly, they can be confusing, demotivating and result in good students being rated poorly and less able students losing the plot.

We might think that multiple choice questions are easy to write , but that is not the case. Considerable skill, care and practice needs to be applied when writing them if you are to avoid confusing your students by asking unanswerable questions with poorly chosen alternative answers.

This post offers some simple tips on how to write effective MCQs painlessly. First, some useful terms: we use the term 'stem' to refer to the question you are asking; we use 'distractor' to describe a possible answer; 'feedback' is used to describe the comments you make on the learners' answers.

How to Write An Effective Stem

Keep the stem as short as you can, avoid complex punctuation and overlong questions.

Try to write your stem as one sentence only - this will be hard at first, but it does get easier with practice.

Try not to test more than one thing per question (you'll know you're doing this if your stem gets very long or you have to use more than one sentence to make the question). It's better to write two well targeted questions rather than one vague one.

Keep instructions for answering the question brief - such as 'Which of the following is a capital city?' or 'Select the season when hurricanes are most likely to occur.'

How To Write Effective Distractors

The first thing to do is to write the expected correct answer. Then think of possible answers a student might give that are feasible, but incorrect. This is easier in some subjects than others. For example you might want learners to identify a magnetic material from a list and use iron, gold, aluminium, silver as distractors. It gets more difficult, though if you want learners to identify the correct parts of speech in an English test as this is less clear cut. Again, through practice, this will get easier.

Avoid including a seemingly ridiculous distractor - using 'hair' in the first example above might lead even very able students away from the point of the question, which might be about choosing components for a small motor.

Try to keep your distractors similar in length as many students become skilled at spotting that the correct answer is always the longest or shortest distractor. If you are new writing MCQs, you will find that you fall into this trap, but keep at it as it does get easier and quicker.

Check the grammar and syntax of your distractors as incorrect usage or clumsy writing means the answer is wrong because the student realises that the writer spent more time on writing the correct answer and just dashed off the distractors.

It is part of received student wisdom to think that distractor B or C is always correct. There is some truth in this - look through some MCQ tests online or examine ones you've written yourself and you will find that many correct answers are in fact B/C or 2/3. To overcome this, review your questions and make sure you have a spread of correct answers in the A,B,C,D 1,2,3,4 range. If you have a radomising software then all the better as it will save you time.

It is often difficult to think of enough distractors to make the offering up to four or five. To get over this some writers often deploy the deadly option of 'All of the above' or 'None of the above.' Please try to avoid doing this as students again know that this is often the correct answer.

If you find you are struggling to write distractors, then consider using a different question type such as True/False, Fill in the blank (also called a cloze test) as some topics just aren't suited to the MCQ approach.

Remember that distractors do not always have to be rendered as text - you can use pictures, diagrams, photos and the like. In some disciplines it is often more sensible to ask students to identify a graphic of something as written distractors can get too wordy and confusing.


How To Write Effective Feedback

We set questions either as formative or summative exercises. Formative questions, where the student can learn from the questions they got wrong should have feedback as you want the students to learn from their mistakes. Summative questions, often used in formal tests do not require feedback, but would need some visual indication of what was right and what was wrong either after each question or at the end of the test as part of a review.

Writing feedback message is a skill that gets better with practice. Here are some tips to help you:

Keep it friendly avoid: 'No! Wrong! Incorrect!

Feedback more than 'Yes, well done' or 'Sorry, that's not correct.' You need to explain why the chosen answer was correct or incorrect. This is a lot of work because to write really useful feedback, you need to write a tailor made response to each distractor. So, that 50 question formative test/exercise can suddenly involve you in many hours of work. Of course, you can always reuse it.


Rob Alton

How To Work With Learning Technologists

Some of our learning objects are created by lecturers themselves without much input from learning technologists. Making materials yourself can be very satisfying, but it is time consuming. Larger projects can often get waylaid by more pressing matters such as teaching and research and sometimes projects never get finished. So, you might need to enlist the services of your Learning Technology (LT) department to develop the object for you. When you involve third parties in your project work, a range of communication, expectation and technical constraints come into play and they need to be managed by you, the client.

This post offers some guidelines on commissioning and overseeing projects so that they are made on time/budget and do the job they were designed to do. By following these suggestions, you will be able to commission and manage projects effectively:


1. Know What You Want

You need to know what you want from the e-learning in terms of the change it will bring about - is it, for example, to help students become more numerate, or to be able to use laboratory equipment, or understand how to sequence DNA? Think about what you might need to get the content across - is it video scenarios, simulations, certification tests or just a simple tutorial? Discuss this with your LT people, get their input and then write a simple specification. By all mean take along examples of similar solutions you've seen online as these stimulate discussion and give your LT people an idea of the complexity of your requirement. You might end up with something like this:

Target population - First Year Biology students who are struggling with the Maths that is required on the course.
Expected solution - simple tutorials that review and test knowledge of equations and scientific notation.
Treatment - A lecturer is filmed a she describes and demonstrates how to solve a range of problems. A test with random questions is used to check understanding
VLE - The material is accessed via the VLE and it should record scores that learners acquire on simple post tests.


2. Expressing What You Want To Your Learning Technologists

E-learning developers will often talk in the language of technology and objectives - things they can measure their solution against. Once you know what you want, expressing it becomes easy - you just need to specify a terminal (end) objective and then break that down into the components that contribute towards that end. It's always a good idea to ask your LT people to 'talk back' the brief to you at the close of the meeting as you can check understanding and clarify any potential sticking points.


3. Having Buy-In From Your Colleagues and Senior Academics

Believe me things go a lot smoother if you have the interest and support of senior colleagues. For a start you’ll probably get a budget and a date on which to complete. Having these will give you confidence to commission within the cost and time constraints - a budget of £3K and a time limit of two weeks won’t get you a complex interactive video based game for instance.


Behaviour of the Effective LT Client

LT clients come from many disciplines - some are ICT savvy, others are not, some are used to working with 3rd parties, some are a tad more isolated. Here are guidelines to follow that will ensure that you have a friendly and productive relationship with your LT supplier:

1. Only Ask For Quotations When You Have The Funding

You can save a lot of time and heartache for yourself and LT staff if you only ask for quotations/pitches when you are sure you have the budget. Sure, projects can get cancelled and LT teams are used to this, but asking for pitches for non-existent projects will result in you attending unnecessary meetings, responding to emails about imaginary projects and wasting everyone's valuable time.


2. Only Ask For What You Can Afford

Client/LT relationships can become strained if the client isn’t aware of how much things cost. Asking for video when the LT team knows that you barely have enough budget for an audio treatment is asking for trouble. Asking a rapid developer to deliver complex simulations for £2K also tends to curdle relations. It’s OK to ask about costs, but not to change your mind about an agreed solution.


3. Honour Sign-Offs

A sign-off triggers the next phase of a project’s life cycle. If the next phase starts and the sign-off is reneged, then unnecessary effort is expended and someone needs to pay for it. The likely consequence is that timescales will slip. Let your colleagues know about the critical role that sign-offs play in the production process. If colleagues are known for letting things slip or are very busy, then let the LT team know and try to work around it - you might need to add additional time to the life-cycle. Try not to change your mind, but if you do, let people know as soon as you can.


4. Apply Common Sense To Deadlines

It is not unusual for timescales to slip on projects. Slippages can sometimes be down to the client - the content isn’t ready in time, sign-offs take too long, changes are asked for at the last minute. Despite slippage on their part, some clients still insist on the original deadline being met - this is unreasonable and creates bad feeling. Be prepared to apply some give and take.


5. They Have Set Aside Time to Work With The LT Team

It is likely that you hold the information required to produce the e-learning in your head - you are a subject matter expert and are key to the success or failure of your project. Try to be available during the analysis and design phases of your project. Ask your LT team what time commitment is expected from you and when it will be required. Try to ensure that you will be available.

Rob Alton

Thursday 7 May 2009

Why We Write Questions

This is the first in series of posts about how to write effective questions for your learning objects.

In this post, I look at why we ask questions, the types of question that can be set and why it's important to provide feedback. In future posts, I'll offer tips on how to write effective question types.

Why Do We Need to Ask Questions?


We ask questions because we want to know if the learner has understood the material presented to them. Questions and their associated feedback can be used to correct misconceptions and build the learner's confidence. So learners know their strengths and weaknesses and tutors can gauge the effectiveness of the learning object - did the learner learn anything? Did they retain anything?
.

Questions though have other uses:

1) They change the pace. E-learning, particularly lengthy tutorial based e-learning, is one paced and this can lead to learners switching off. Injecting a quiz, test or game changes the role of the learner from a passive consumer into a doer. Just as a lecture or seminar benefits from a change in pace, so does e-learning.

2) They encourage practice and rehearsal in the learner. For deep learning to occur, learners need to do something with the information/skills they are acquiring and questions give them the opportunity to try 'what if?' scenarios for example.


Classification of Questions


There are three types of questions:

Diagnostic - assesses skill level of learner so that appropriate activities can be matched to their requirements.
Formative - checks what has just been taught.
Summative - given at the end of a course to test mastery of the overall concept

Feedback


Feedback tells the learner how they're doing - it reinforces learning. You should always try to feed back to learners because it can:

Instill confidence
Dispel misconceptions

When writing feedback, bear these tips in mind:

Provide the answer to the question in the feedback
Provide more than just correct/incorrect - add an explanation
Say why their answer was right/wrong

Tips for Writing Good Questions

Don't make them too easy
Don't ask trick questions
Don't make them too wordy
Provide clear instructions if it's a complex question.

Rob Alton

Tips for Working With Subject Matter Experts (SMEs)

If you are a learning technologist or learning designer, you will probably be working on projects where the subject matter is unfamiliar to you. To learn about the subject in order to produce a design you will need to consult people who know the subject inside out - the subject mater expert (SME).

This post looks at how you can work with an SME effectively.

What is an SME?

Celebrity Chef, Knighted Emeritus Professor, Army Instructor, IT Manager, UFOlogist, Booker Prize Winner, Clinical Practice Manager and a Pharmacologist. What do they have in common? Not much you’d think, but they have all been subject matter experts (aka SMEs) on learning projects I’ve produced in the last few years.

We know that SMEs are a keystone in learning projects, so how is it possible to work with them to get usable content on time and in scope?

Some Tips

The first thing to understand is that your contact isn’t called an SME in their place of work - that’s not a job title, it’s a role. Your SME will have other things to do - their job. They may or may not see helping you as part of their job and you’ll have to deal with this whatever happens. Many people, of course, are only too wiling to help, some might even be over helpful and that brings its own challenges.

So, what can you do to ensure you get their time, commitment and content?

1 - Meet Your SME

Meet your SME and find out how much they know about your project and fill them in if they don’t know very much. This seems obvious, but their understanding of the project may be different from yours. Explain and iron out any misunderstandings. Keep management in the loop as scope may not have been described (or costed) accurately and some risk management may need to be applied. This meeting need not take long, but it is crucially important to the success of your project.

Let your SME know how much time you’ll need from them as they may not have been informed about the level of commitment required. Don’t underestimate the time needed for review cycles and fact checking - projects can go seriously awry if the content is wrong or is inaccurately rendered. Imagine what would happen if you told apprentice electricians in the UK that the brown wire means earth and the green/yellow was live. That nearly happened to me 20 years ago I am ashamed to say.

2 - Find Out How Much They Know About E-learning

Find out how much your SME knows about e-learning - it could be a lot more than you expect, it may be nothing, or it may be informed or uninformed negativity. Once you know levels of understanding, you can adapt your approach accordingly. Deal with the sceptics as soon as you possible. Do not dismiss their concerns. Counter their objections with calm, informed, polite but firm professionalism.

3 - Lose the Jargon

Don’t talk about learning theory in any great depth as most people don’t know or want to know about it - they want a piece of learning that will do something such as reducing the number of accidents in the workplace or increasing sales of cameras. They are paying you to change something, so concentrate on that and talk about the project in those terms - the resulting change and how your design will bring it about. Of course, you need to be sensible - if you are working with lecturers in an education faculty you might need to demonstrate that you know your constructivism from your behaviourism!

4 - Organise a Workshop

Kick off the project with a Workshop with all key stakeholders. Confirm that you are all clear and agreed on the key goals and scope of the job. Describe your proposed solution. Ask if they are happy that this is still within their perceived scope. If it’s not, let your project manager know as soon as possible. Show examples of the kinds of things that can be done for this kind of project and budget. This can get difficult as many people can’t resist putting everything bar the kitchen sink into a production and others might want overkill - asking for a full scope simulation when the budget can only stretch to a part task sim. Take notes during the meeting and issue them as Minutes with actions.

5 Write Things Down

When meeting your SME to get the content, take notes, use sketching techniques, if you have an interactive whiteboard write notes on that and save them. Record your conversations with SMEs (ask first though) and transcribe after. If you have been fact gathering then send the notes in for checking/sign off.

6 - Get Sign-Off

Many designers are frustrated by the length of time it takes to get sign-off from SMEs. Lengthy sign-offs can result the project losing momentum and the timescales slipping, but often the delivery deadline does not slip in parallel with it. Be assertive and politely pester your SME and the project stakeholders - if you don’t make your point early on and get it logged in the project risk log, then it will be you who makes up the time as the deadline looms. Ask any designer what comes last and too late - it’s nearly always the content.

There are many more things you can do. These are just a few. If you want more, then please contact me. Happy designing.

Rob Alton

Adobe Presenter Review

Adobe Presenter is a tool that allows rapid development of e-learning and presentation materials. It can also be used to host online meetings. It is a plug-in to PowerPoint.

Once Presenter has been installed, it can be accessed from the tool bar in PowerPoint. The basic method of operation is to set up a presentation in PowerPoint and then add to it using the features contained in Presenter.

These features include:

* A variety of pre-designed skins & navigation buttons
* Record a voiceover track and key it into slide/slide animations
* Import sound files (must be .flv)
* Import video (must be .flv)
* Import Flash files
* Sync PowerPoint animations with video and audio
* Set MCQ, True/False, Cloze, Short Answer, Matching, Lickert
* Set up and deliver Surveys
* Track user responses and performance
* Manage student groups

So, it is fairly quick and easy to develop sophisticated looking e-learning. Of course, there is a trade off as you lose some of the finesse and control that you often get with products developed in a more programming reliant environment such as Flash.

Presenter - Some Advantages

* Based on the popular PowerPoint product - instantly familiar
* Rapid development with no requirement for coding
* No need to build a user interface as Presenter supplies its own
* Built in question templates and student reporting
* Can be run from a VLE or as a raw file
* Control of media elements (video, audio)
* Presenter outputs to SCORM/AICC/IMS standards

Presenter - Some Disadvantages

* Manual, rather than code based authoring - lots of mouse work - may bore programmers.
* What you see isn’t necessarily what you get. Breeze reserves some screen space for its interface, so you may need to re-format existing PPT presentations, particularly text objects. This may have been solved in later versions, but this has not been tried yet.
* Possible usability and accessibility issues - can’t re-size text on the fly, some Presenter controls are a little small as is the Notes feature.
* Presenter outputs lots of files when a presentation is published - you need to manage your files and folders, particularly if you hot desk.
* Syncing animations with voice or video is fiddly and hit and miss for complex build-ups. Needs finer controls.
* Formatting applied to quiz questions after they have been created can be lost when adding new question items. This also happens if you edit quiz settings. It’s better to format after authoring the whole quiz.
* Button that allows users to clear question answers if they change their mind doesn’t seem to work consistently.
* You can use .flv (Flash 8 movie) integrated using the Presenter editor, but you cannot load an .flv in PowerPoint. Therefore, you cannot use video full screen, only in the Presenter window. A workaround for this might be simple to write.

Nice to Have Improvements

* Global editing of question formatting.
* Safe area marking on PowerPoint slide to indicate what will show on a Breeze screen.
* A table to allow finer control when syncing animations/objects with video/audio.

Prerequisites for using Presenter Effectively

* Basic ability in authoring presentations in PowerPoint - you must be familiar with the PPT interface and the package’s basic features.
* The ability to set up and manage a PowerPoint Master Page is crucial.
* A Storyboard that describes what you want to do. If you don’t have a plan, things become confusing.
* Basic layout and design skills.

Would I Use It?

Definitely. But only for fairly simple types of e-learning, where information is presented and then understanding is checked. The package is great for re-purposing the outputs of presentations/lectures and chunking this up into modules.

Presenter would also great for prototyping/visualising a new project.

Rob Alton