MASHe

For the next post in my ALT-C series I’m going to highlight a session I didn’t actually attend but immediately regretted when comments started filtering in on twitter.

The session was based around the paper by Rodway-Dyer, Dunne and Newcombe from University of Exeter which summaries a study of audio and visual feedback used in two 1st year undergraduate classes. Click here for the paper and abstract.

Comments I picked up on this paper via twitter appeared to show audio feedback was not well received. Issues highlighted were:

• the finding that “76% of students wanted face-to-face from a tutor in addition to other forms of feedback” [@adamread, @JackieCarter]
• students found that receiving negative audio comments was harder than when written [@adamread, @ali818, @narcomarco]. Although this is still open to debate as @gillysalmon said that “duckling project at Leicester has found human voice easier to give negative feedback by audio than text”

Obviously there are issues with making assumptions based on a few 140 character tweets and it should be noted that the authors conclude that “overall, it seems that "there is considerable potential in using audio and screen visual feedback to support learning”, although students did express concerns in a number of areas.

Having had a chance to digest the paper the question I’m left with is how much of the negative experiences were a result of the wider assessment design rather than the use of audio feedback in itself. For example, reading the focus group discussions for audio feedback in geography I noted that:

• students were not notified that they would be receiving audio feedback;
• that despite the tutors best attempts students hadn’t engaged with assessment criteria; and
• that this was the first essay students submitted at university level and they were unclear of the expected standards.

Similar issues to these were addressed in the Re-Engineering Assessment Practices (REAP) project, which produced an evolving set of assessment principles. Principles which could be successfully applied to the geography example might be:

Help clarify what good performance is – this could be achieved in a number of ways including creating an opportunity for the tutor to discuss criteria with students, or perhaps providing a exemplar of previous submissions with associated audio feedback.

Providing opportunities to act on feedback – as this was the students first submission providing feedback on a draft version of their essay not only allows students to act on feedback (it’s not surprising when students ignore feedback if they have no opportunity to use it).

Facilitates self-assessment and reflection – One of the redesigns piloted during REAP was the Foundation Pharmacy class, in which students submitted a draft using a pro-forma similar to that used by tutors to grade their final submission. Students were required to reflect on distinct sections of their essay, which again also allowed them to engage with the assessment criteria.

Encourage positive motivational beliefs – using the staged feedback described above would perhaps also address the issue of students becoming disillusioned.

Talking to a friend during the lunch break the research methodology used by the authors was also mentioned, in particular the use of ‘stimulated recall’. For this the authors played back examples of audio feedback to the tutor asking him to explain his thought processes and reflect on how his students would have responded to his comments. This methodology seems particularly appropriate to evaluate the use of audio feedback, and is something I want to take a closer look at.

A moment of serendipity

Whilst searching the twitter feed for comments on the session I noticed a tweet by @newmediac which was promoting a free webinar in which  “Phil Ice shares research on benefits of audio feedback” (here’s the full tweet). The session has already passed  but the recording for this event is here.

Moment of serendipity

The presenter, Phil Ice, has been working on audio feedback in the US for a number of years and has a number of interesting findings (and research methodologies) I haven’t seen in the UK.

For example, Ice and his team report:

students used content for which audio feedback was received approximately 3 times more often than content for which text-based feedback [was] received”

and that

students were 5 to 6 times more likely to apply content for which audio feedback was received at the higher levels of Bloom’s Taxonomy then content for which text-based feedback was received”.

These results were from a small scale study of approximately 30 students so aren’t conclusive. Ice has also conducted a larger studies with over 2,000 students which used the Community of Inquiry Framework Survey. Positive differences were found across a number of indicators including excessive use of audio to address feedback at lower levels is perceived as a barrier by students.

Ice has also conducted studies which breaks audio feedback into four types: global – overall quality; mid level – clarity of thought/argument; micro – word choice/grammar/punctuation; and other – scholarly advice. The study indicates that students prefer a combination of audio and text for global and mid-level comments.

Findings from Ice have been submitted for publication in the Journal of Educational Computing Research (which will soon feature a special issue on ‘Technology-Mediated Feedback for Teaching and Learning’).

Finally, I would like to mention the method Ice uses for audio feedback. He uses the audio comment tool within Acrobat Pro 8 to record comments ‘inline’. This appears to be particularly useful for students to relate comments to particular sections of their submitted work. Click here for a sample PDF document with audio feedback (this isn’t compatible with all PDF readers – I’ve tested on Acrobat Reader and Foxit Reader).

Hopefully this post has not only stimulated some ideas in the use of audio feedback, but also highlight a range of methodologies to effectively evaluate it.

The National Student Survey results has been published by HEFCE which has no doubt left school/department managers burning the midnight oil to see how they have faired. Feedback remains to be a talking point with only just over half of Scottish students agreeing or strongly agreeing that feedback has been prompt, detailed and helpful.

But what about the students who neither agree or disagree? If you turn the question around and ask what proportion of students disagree or strongly disagree with the level of feedback they receive then you are looking at approximately a quarter of students. Obviously this is still a substantial number and still makes feedback the worst performing area, but if you are drilling down into course level performance perhaps it is worth bearing in mind.

Table 1 below shows the results for the percentage of Scottish students who responded disagree or strongly disagree to the NSS questions.

Looking at how this analysis effects the overall satisfaction with Scottish HEIs the most notable changes are University of Stirling and Robert Gordon University who (by my calculations*) jump 2 rankings. Below (#) denotes rank.

University of St Andrews	91% (1)	3% (1)
University of Glasgow	91% (1)	5% (2)
University of Aberdeen	89% (3)	6% (4)
University of Stirling	88% (4)	5% (2)
University of Dundee	88% (4)	7% (5)
University of Strathclyde	87% (6)	7% (5)
Robert Gordon University	84% (7)	7% (5)
Glasgow Caledonian University	84% (7)	8% (8)
University of Edinburgh	82% (9)	9% (9)
Napier University	81% (10)	9% (9)
Heriot-Watt University	81% (10)	10% (11)
Glasgow School of Art	69% (12)	22% (12)

*Data provided by the NSS is susceptible to rounding errors. For example University of St. Andrews has an overall percentage agree for Q22 of 92% yet the percentage breakdown is 35% agree and 56% strongly agree, which equals 91%. To allow comparison with the percentage of disagreement, the sum of percentage of responses for agree and strongly agree have been used.

Table 1: Unofficial National Unsatisfied Student Survey (UNUSS)
Provisional sector results for Full-time students – Scotland Registered HEI (% of students who disagree/strongly disagree) extracted from HEFCE NSS 2009 Data

Question	2008	2009
The teaching on my course
1 – Staff are good at explaining things.	4	4
2 – Staff have made the subject interesting.	6	6
3 – Staff are enthusiastic about what they are teaching.	4	4
4 – The course is intellectually stimulating.	5	5
Assessment and feedback
5 – The criteria used in marking have been clear in advance.	17	15
6 – Assessment arrangements and marking have been fair.	11	10
7 – Feedback on my work has been prompt.	28	27
8 – I have received detailed comments on my work.	29	27
9 – Feedback on my work has helped me clarify things I did not understand.	27	25
Academic support
10 – I have received sufficient advice and support with my studies.	12	11
11 – I have been able to contact staff when I needed to.	6	7
12 – Good advice was available when I needed to make study choices.	13	12
Organisation and management
13 – The timetable works efficiently as far as my activities are concerned.	10	11
14 – Any changes in the course or teaching have been communicated effectively.	12	15
15 – The course is well organised and is running smoothly.	11	13
Learning resources
16 – The library resources and services are good enough for my needs.	12	11
17 – I have been able to access general IT resources when I needed to.	6	6
18 – I have been able to access specialised equipment, facilities or room when I needed to.	6	6
Personal development
19 – The course has helped me present myself with confidence.	7	6
20 – My communication skills have improved.	5	4
21 – As a result of the course, I feel confident in tackling unfamiliar problems.	5	5
Overall satisfaction
22 – Overall, I am satisfied with the quality of the course.	7	7

As I’ve recently revisited on generating audio feedback it seemed timely, particularly with a request from UHI coming into my inbox, to also have another look at video feedback. Russell Stannard recently won a Times Higher Education Award partly for his work in this particular area. In Russell’s work he uses screen capture software to record feedback on electronic submissions of student work. More information on this technique is available in a case study Russell prepared for the Higher Education Academy English Subject Centre on Using Screen Capture Software in Student Feedback. An example of using this technique is also available – click here for a short example of video feedback.

In my original post I highlighted Using Tokbox for Live and Recorded Video Feedback as a possible solution to distribute video feedback. At the time I felt there were two niggling issues with using Tokbox. First there was the requirement to install the ManyCams software to allow you to display your desktop and secondly Tokbox was very slow in uploading video you had recorded. For live video feedback Tokbox might still be worth considering, but shortly after publishing the post I discovered ScreenToaster.

ScreenToaster allows you to record your desktop without installing any software. It’s very easy to setup and the videos you create can be immediately uploaded allowing you to decides how you want to distribute and share them. The following video shows you how easy it is to setup and highlights some of the useful features. Even if you are not interested in delivering video feedback to students this is still a great site to record other material like demonstrations of software.

Example of using ScreenToaster to deliver video feedback on student submitted work from Martin Hawksey on Vimeo

Listen to this:

Download link

I captured this clip of students from the University of Chester and Sheffield Hallam talking about their experiences of receiving audio feedback at the last Podcasting for Pedagogic Purposes SIG at Glasgow Caledonian University. This event was very fortuitous as at the time I was helping source material for a Queen Margaret University (QMU) staff workshop on this very topic. If you are still unclear as to what audio feedback is here is a nice description from Andrew Middleton:

Audio feedback can be defined as formative messages, recorded and distributed as digital audio to individual students or student groups in response to both ongoing and submitted work, allowing each student to develop their knowledge and the way they learn. (Middleton, A. 2008)

Why do I think audio feedback is worth exploring? It is clear from the National Student Survey (NSS) that feedback is a particular area if dissatisfaction. There is a growing pool of evidence that students perceive audio feedback as a positive to their learning experience (although I’m not aware of research on actual learning gains).

Existing practitioners/projects

In my research for the QMU workshop I came across a number of projects and practitioners exploring this area, which I believe are worth sharing in this post:

• Andrew Middleton – Sheffield Hallam University
• audiofeedback – University of Sunderland
• Audio Supported Enhanced Learning (ASEL) – managed by Will Stewart
• Clare McCullagh – University of Reading
• Derek France – University of Chester
• Podcasting for Pedagogic Purposes SIG
• Sounds Good 2 project – led by Bob Rotheram

[If I’ve missed anyone off please use the comments to highlight them and their work]

In researching audio feedback it was clear while there were distinct benefits but there were some reoccurring themes in terms of limitations. First is scalability. It’s all well and good providing individual feedback to up to 50 students but with figures beyond this it just becomes to onerous.

Audio feedback models

A solution to this problem is to explore other audio feedback models. Andrew Middleton at Sheffield Hallam University has identified a number of alternative models which include:

• Personal tutor monologue – tutor feedback to individual student
• Personal feedback conversations – recording tutor/student meetings
• Broadcast feedback – generic feedback to the class
• Peer audio feedback – student generated audio feedback
• Tutor conversations – recording teaching staff conversations

More information on these models is available in a presentation made by Andrew at the Blended Learning Conference or in a forthcoming paper entitled ‘Audio Feedback design: principles and emerging practice’.

Don’t expect to save time

A finding from the Sounds Good 2 project is that providing individual student feedback is unlikely to save any staff time (there is a messy debate about whether high quality feedback offers long term gains in terms of how much additional feedback is required further down the line). Sounds Good have however circumstances where time can be saved (taken from the Sounds Good Final Report):

• The assessor is comfortable with the technology.
• The assessor writes or types slowly but records their speech quickly.
• A substantial amount of feedback is given.
• A quick and easy method of delivering the audio file to the student is available.

Technology

In terms of the technology there are a number of solutions which various projects propose. These include:

• using a digital voice recorder which saves the files directly into MP3 format
• using you Mac/PC using free audio recording software like Audacity (a portable version of Audacity on EduApps) or Wavosaur (this also can be used without installing any software)
• using a mobile phone – if this feature is available

I also recently posted about Using Google Talk for Audio Feedback. This solution appears to be more troublesome than its worth but was useful as Joe Dale, via Andrew Middleton, highlighted the Vocaroo web service which look incredibly easy to use and worth a look at.

General advice

So what advice would I give to anyone thinking about using audio feedback? There are some very good recommendations from the Sounds Good project (final report) which are worth highlighting grouped under 4 themes: saving time; technical matters; administration; and structure:

Saving time

• Don’t expect immediate savings in time, if any. Think of the long term returns for you and your students (there is anecdotal evidence that audio feedback reduces the need for follow up face-to-face sessions)
• except minor mistakes. You are not looking to produce broadcast quality audio

Technical matters

• optimise your files to minimise download size. Recommended mp3, mono and if possible reduce the bitrate to 32-40kbit/s
• check files can be played on campus computers
• make sure the audio is loud enough
• have clear guidance on how to play files (I wouldn’t bother with guidance on playing files on portable media devices as the majority of students appear to prefer the convenience of playing them from a desktop computer)
• have a backup of files in a secure location
• if using shared devices or computers make sure files are deleted once backed up

Administration

• if audio feedback is particularly being used as part of summative assessment make sure you have a conversation with quality assurance
• make sure audio files are securely stored and distributed

Structure

• try to personalise the feedback by introducing yourself, the assignment you are giving feedback on and refer to the student my name
• keep focused – a 2 minute piece of feedback can be as, if not more, beneficial than a 10 minute ramble (the clip at the beginning of this post is just over a minute and conveys a lot of information)
• I would recommend using audio feedback as supplemental to written feedback.

In terms of a procedure for creating audio feedback Bob Rotheram from the Sounds Good project recommends this procedure and general structure:

Feedback Procedure

• Have the assignment details and assessment criteria with me.
• Read the assignment, making written comments on it as I go along. If it’s on paper, I jot things in the margin. If it’s in an electronic format (e.g. Word), I use the ‘Track changes’ facility to annotate the document.
• Read it again, more quickly this time, perhaps making a few more comments along the way.
• Jot down (on scrap paper) the main summary points I wish to make.
• Start the MP3 recorder.
• Don’t bother to erase and re-record ‘misspeaks’; just correct them immediately, as in conversation.

General structure

• Introduce yourself to the student in a friendly manner.
• Say which assignment you’re giving feedback on. Outline the main elements of the comments which you’ll be giving (see below).
• Work steadily through the assignment, amplifying and explaining notes you’ve put in the margins and, especially at the end, making more general points.
• Refer to the assessment criteria.
• Explain your thought processes as you move towards allocating a mark.
• Offer a few, reasonably attainable, suggestions for improvement, even if the work is excellent.
• Invite comments back from the student, including on the method of giving feedback.
• Round things off in a friendly way.

Bob expands on these in his ‘Practice tips on using digital audio for assessment feedback’

This post is based on material compiled and presented by Jim Sharp and Susi Peacock at Queen Margaret University