RSC MASHe

Listen to this:

Download link

Originally uploaded by Dan's Photos

Audio feedback can be defined as formative messages, recorded and distributed as digital audio to individual students or student groups in response to both ongoing and submitted work, allowing each student to develop their knowledge and the way they learn. (Middleton, A. 2008)

Why do I think audio feedback is worth exploring? It is clear from the National Student Survey (NSS) that feedback is a particular area if dissatisfaction. There is a growing pool of evidence that students perceive audio feedback as a positive to their learning experience (although I’m not aware of research on actual learning gains).

Existing practitioners/projects

In my research for the QMU workshop I came across a number of projects and practitioners exploring this area, which I believe are worth sharing in this post:

• Andrew Middleton – Sheffield Hallam University
• audiofeedback – University of Sunderland
• Audio Supported Enhanced Learning (ASEL) – managed by Will Stewart
• Clare McCullagh – University of Reading
• Derek France – University of Chester
• Podcasting for Pedagogic Purposes SIG
• Sounds Good 2 project – led by Bob Rotheram

[If I’ve missed anyone off please use the comments to highlight them and their work]

In researching audio feedback it was clear while there were distinct benefits but there were some reoccurring themes in terms of limitations. First is scalability. It’s all well and good providing individual feedback to up to 50 students but with figures beyond this it just becomes to onerous.

Audio feedback models

A solution to this problem is to explore other audio feedback models. Andrew Middleton at Sheffield Hallam University has identified a number of alternative models which include:

• Personal tutor monologue – tutor feedback to individual student
• Personal feedback conversations – recording tutor/student meetings
• Broadcast feedback – generic feedback to the class
• Peer audio feedback – student generated audio feedback
• Tutor conversations – recording teaching staff conversations

More information on these models is available in a presentation made by Andrew at the Blended Learning Conference or in a forthcoming paper entitled ‘Audio Feedback design: principles and emerging practice’.

Don’t expect to save time

A finding from the Sounds Good 2 project is that providing individual student feedback is unlikely to save any staff time (there is a messy debate about whether high quality feedback offers long term gains in terms of how much additional feedback is required further down the line). Sounds Good have however circumstances where time can be saved (taken from the Sounds Good Final Report):

• The assessor is comfortable with the technology.
• The assessor writes or types slowly but records their speech quickly.
• A substantial amount of feedback is given.
• A quick and easy method of delivering the audio file to the student is available.

Technology

In terms of the technology there are a number of solutions which various projects propose. These include:

• using a digital voice recorder which saves the files directly into MP3 format
• using you Mac/PC using free audio recording software like Audacity (a portable version of Audacity on EduApps) or Wavosaur (this also can be used without installing any software)
• using a mobile phone – if this feature is available

I also recently posted about Using Google Talk for Audio Feedback. This solution appears to be more troublesome than its worth but was useful as Joe Dale, via Andrew Middleton, highlighted the Vocaroo web service which look incredibly easy to use and worth a look at.

General advice

So what advice would I give to anyone thinking about using audio feedback? There are some very good recommendations from the Sounds Good project (final report) which are worth highlighting grouped under 4 themes: saving time; technical matters; administration; and structure:

Saving time

• Don’t expect immediate savings in time, if any. Think of the long term returns for you and your students (there is anecdotal evidence that audio feedback reduces the need for follow up face-to-face sessions)
• except minor mistakes. You are not looking to produce broadcast quality audio

Technical matters

• optimise your files to minimise download size. Recommended mp3, mono and if possible reduce the bitrate to 32-40kbit/s
• check files can be played on campus computers
• make sure the audio is loud enough
• have clear guidance on how to play files (I wouldn’t bother with guidance on playing files on portable media devices as the majority of students appear to prefer the convenience of playing them from a desktop computer)
• have a backup of files in a secure location
• if using shared devices or computers make sure files are deleted once backed up

Administration

• if audio feedback is particularly being used as part of summative assessment make sure you have a conversation with quality assurance
• make sure audio files are securely stored and distributed

Structure

• try to personalise the feedback by introducing yourself, the assignment you are giving feedback on and refer to the student my name
• keep focused – a 2 minute piece of feedback can be as, if not more, beneficial than a 10 minute ramble (the clip at the beginning of this post is just over a minute and conveys a lot of information)
• I would recommend using audio feedback as supplemental to written feedback.

In terms of a procedure for creating audio feedback Bob Rotheram from the Sounds Good project recommends this procedure and general structure:

Feedback Procedure

• Have the assignment details and assessment criteria with me.
• Read the assignment, making written comments on it as I go along. If it’s on paper, I jot things in the margin. If it’s in an electronic format (e.g. Word), I use the ‘Track changes’ facility to annotate the document.
• Read it again, more quickly this time, perhaps making a few more comments along the way.
• Jot down (on scrap paper) the main summary points I wish to make.
• Start the MP3 recorder.
• Don’t bother to erase and re-record ‘misspeaks’; just correct them immediately, as in conversation.

General structure

• Introduce yourself to the student in a friendly manner.
• Say which assignment you’re giving feedback on. Outline the main elements of the comments which you’ll be giving (see below).
• Work steadily through the assignment, amplifying and explaining notes you’ve put in the margins and, especially at the end, making more general points.
• Refer to the assessment criteria.
• Explain your thought processes as you move towards allocating a mark.
• Offer a few, reasonably attainable, suggestions for improvement, even if the work is excellent.
• Invite comments back from the student, including on the method of giving feedback.
• Round things off in a friendly way.

Bob expands on these in his ‘Practice tips on using digital audio for assessment feedback’

This post is based on material compiled and presented by Jim Sharp and Susi Peacock at Queen Margaret University

Originally uploaded by Dan's Photos

For the next post in my ALT-C series I’m going to highlight a session I didn’t actually attend but immediately regretted when comments started filtering in on twitter.

The session was based around the paper by Rodway-Dyer, Dunne and Newcombe from University of Exeter which summaries a study of audio and visual feedback used in two 1st year undergraduate classes. Click here for the paper and abstract.

Comments I picked up on this paper via twitter appeared to show audio feedback was not well received. Issues highlighted were:

• the finding that “76% of students wanted face-to-face from a tutor in addition to other forms of feedback” [@adamread, @JackieCarter]
• students found that receiving negative audio comments was harder than when written [@adamread, @ali818, @narcomarco]. Although this is still open to debate as @gillysalmon said that “duckling project at Leicester has found human voice easier to give negative feedback by audio than text”

Obviously there are issues with making assumptions based on a few 140 character tweets and it should be noted that the authors conclude that “overall, it seems that "there is considerable potential in using audio and screen visual feedback to support learning”, although students did express concerns in a number of areas.

Having had a chance to digest the paper the question I’m left with is how much of the negative experiences were a result of the wider assessment design rather than the use of audio feedback in itself. For example, reading the focus group discussions for audio feedback in geography I noted that:

• students were not notified that they would be receiving audio feedback;
• that despite the tutors best attempts students hadn’t engaged with assessment criteria; and
• that this was the first essay students submitted at university level and they were unclear of the expected standards.

Similar issues to these were addressed in the Re-Engineering Assessment Practices (REAP) project, which produced an evolving set of assessment principles. Principles which could be successfully applied to the geography example might be:

Help clarify what good performance is – this could be achieved in a number of ways including creating an opportunity for the tutor to discuss criteria with students, or perhaps providing a exemplar of previous submissions with associated audio feedback.

Providing opportunities to act on feedback – as this was the students first submission providing feedback on a draft version of their essay not only allows students to act on feedback (it’s not surprising when students ignore feedback if they have no opportunity to use it).

Facilitates self-assessment and reflection – One of the redesigns piloted during REAP was the Foundation Pharmacy class, in which students submitted a draft using a pro-forma similar to that used by tutors to grade their final submission. Students were required to reflect on distinct sections of their essay, which again also allowed them to engage with the assessment criteria.

Encourage positive motivational beliefs – using the staged feedback described above would perhaps also address the issue of students becoming disillusioned.

Talking to a friend during the lunch break the research methodology used by the authors was also mentioned, in particular the use of ‘stimulated recall’. For this the authors played back examples of audio feedback to the tutor asking him to explain his thought processes and reflect on how his students would have responded to his comments. This methodology seems particularly appropriate to evaluate the use of audio feedback, and is something I want to take a closer look at.

A moment of serendipity

Whilst searching the twitter feed for comments on the session I noticed a tweet by @newmediac which was promoting a free webinar in which  “Phil Ice shares research on benefits of audio feedback” (here’s the full tweet). The session has already passed  but the recording for this event is here.

Moment of serendipity

The presenter, Phil Ice, has been working on audio feedback in the US for a number of years and has a number of interesting findings (and research methodologies) I haven’t seen in the UK.

For example, Ice and his team report:

students used content for which audio feedback was received approximately 3 times more often than content for which text-based feedback [was] received”

and that

students were 5 to 6 times more likely to apply content for which audio feedback was received at the higher levels of Bloom’s Taxonomy then content for which text-based feedback was received”.

These results were from a small scale study of approximately 30 students so aren’t conclusive. Ice has also conducted a larger studies with over 2,000 students which used the Community of Inquiry Framework Survey. Positive differences were found across a number of indicators including excessive use of audio to address feedback at lower levels is perceived as a barrier by students.

Ice has also conducted studies which breaks audio feedback into four types: global – overall quality; mid level – clarity of thought/argument; micro – word choice/grammar/punctuation; and other – scholarly advice. The study indicates that students prefer a combination of audio and text for global and mid-level comments.

Findings from Ice have been submitted for publication in the Journal of Educational Computing Research (which will soon feature a special issue on ‘Technology-Mediated Feedback for Teaching and Learning’).

Screenshot showing inline audio comments

Finally, I would like to mention the method Ice uses for audio feedback. He uses the audio comment tool within Acrobat Pro 8 to record comments ‘inline’. This appears to be particularly useful for students to relate comments to particular sections of their submitted work. Click here for a sample PDF document with audio feedback (this isn’t compatible with all PDF readers – I’ve tested on Acrobat Reader and Foxit Reader).

Hopefully this post has not only stimulated some ideas in the use of audio feedback, but also highlight a range of methodologies to effectively evaluate it.