Archive for the 'Feedback' Category

Generating Student Video Feedback using screenr

On the 28th May 2009  I wrote a post on Generating Student Video Feedback using ScreenToaster. As ScreenToaster is now ‘toast’ I thought I’d repost highlighting screenr instead. As the process for using ScreenToaster/screenr is so similar I haven’t re-recorded the demo video, but hopefully you get the idea (I’m glad I downloaded the original and put it on vimeo ;)
video feedback
video feedback
Originally uploaded by erikadotnet

As I’ve recently revisited on generating audio feedback it seemed timely, particularly with a request from UHI coming into my inbox, to also have another look at video feedback. Russell Stannard recently won a Times Higher Education Award partly for his work in this particular area. In Russell’s work he uses screen capture software to record feedback on electronic submissions of student work. More information on this technique is available in a case study Russell prepared for the Higher Education Academy English Subject Centre on Using Screen Capture Software in Student Feedback. An example of using this technique is also available - click here for a short example of video feedback.

In my original post I highlighted Using Tokbox for Live and Recorded Video Feedback as a possible solution to distribute video feedback. At the time I felt there were two niggling issues with using Tokbox. First there was the requirement to install the ManyCams software to allow you to display your desktop and secondly Tokbox was very slow in uploading video you had recorded. For live video feedback Tokbox might still be worth considering, but shortly after publishing the post I discovered ScreenToaster., but for recorded feedback you might do better with screenr.

ScreenToaster Screenr allows you to record your desktop without installing any software. It’s very easy to setup and the videos you create can be immediately uploaded allowing you to decides how you want to distribute and share them [You can also publish them directly to YouTube and/or download the video in MP4 format. The following video shows you how easy it is to setup and highlights some of the useful features. Even if you are not interested in delivering video feedback to students this is still a great site to record other material like demonstrations of software.

ScreenToaster Screencast 
Example of using ScreenToaster to deliver video feedback on student submitted work from Martin Hawksey on Vimeo

Using YouTube for audio/video feedback for students

A long, long, long time ago I wrote a post Using Tokbox for Live and Recorded Video Feedback in which I demonstrated how the free ManyCam software could be used to turn your desktop into a virtual webcam to provide feedback on students work in a Russell Stannard styley. Recently my colleague Kenji Lamb was showing me how you could directly record your webcam using YouTube, so I thought I would revisit this idea.

This time instead of focusing on the use of the visual element as a tool to direct students attention to a specific part of a assessment submission (e.g. highlight and talking about parts of a word document), I thought it would be interesting to demonstrate it in a more abstract way using images to reinforcing audio comments (e.g. you did good – happy face; you did bad – sad face).

When previously looking at audio feedback I’ve been very aware that reducing as much of the administrative burden is very important. Online form filling whether it be through the VLE, other systems or in the YouTube example, can be a bit of a chore so in this demonstration I also touch upon using bookmarklets to remove some of the burden. Here is a link to the bookmarklet I created for student feedback on YouTube (YouTube Feedback Template – you should be able drag and drop this to your bookmark toolbar but if you are reading this through an RSS reader it might get stripped out).

Having this link in you toolbar means when you get to the video settings you can click it to populate the form. Bookmarklets are a nice tool to have in the chest so I’ve covered them in more detail in Bookmarklets: Auto form filling and more. This post also shows you how you can create your own custom filling bookmarklet using Benjamin Keen’s Bookmarklet Generator.

So here it his a quick overview of using YouTube for recording student feedback:


Comments powered by Twitter Click here to add yours ...

ALT-C 2009 II: Audio and screen visual feedback to support student learning (and research methodologies)



Originally uploaded by Dan's Photos

For the next post in my ALT-C series I’m going to highlight a session I didn’t actually attend but immediately regretted when comments started filtering in on twitter.

The session was based around the paper by Rodway-Dyer, Dunne and Newcombe from University of Exeter which summaries a study of audio and visual feedback used in two 1st year undergraduate classes. Click here for the paper and abstract.

Comments I picked up on this paper via twitter appeared to show audio feedback was not well received. Issues highlighted were:

  • the finding that “76% of students wanted face-to-face from a tutor in addition to other forms of feedback” [@adamread, @JackieCarter]
  • students found that receiving negative audio comments was harder than when written [@adamread, @ali818, @narcomarco]. Although this is still open to debate as @gillysalmon said that “duckling project at Leicester has found human voice easier to give negative feedback by audio than text”

Obviously there are issues with making assumptions based on a few 140 character tweets and it should be noted that the authors conclude that “overall, it seems that "there is considerable potential in using audio and screen visual feedback to support learning”, although students did express concerns in a number of areas.

Having had a chance to digest the paper the question I’m left with is how much of the negative experiences were a result of the wider assessment design rather than the use of audio feedback in itself. For example, reading the focus group discussions for audio feedback in geography I noted that:

  • students were not notified that they would be receiving audio feedback;
  • that despite the tutors best attempts students hadn’t engaged with assessment criteria; and
  • that this was the first essay students submitted at university level and they were unclear of the expected standards.

Similar issues to these were addressed in the Re-Engineering Assessment Practices (REAP) project, which produced an evolving set of assessment principles. Principles which could be successfully applied to the geography example might be:

Help clarify what good performance is – this could be achieved in a number of ways including creating an opportunity for the tutor to discuss criteria with students, or perhaps providing a exemplar of previous submissions with associated audio feedback.

Providing opportunities to act on feedback – as this was the students first submission providing feedback on a draft version of their essay not only allows students to act on feedback (it’s not surprising when students ignore feedback if they have no opportunity to use it).

Facilitates self-assessment and reflection - One of the redesigns piloted during REAP was the Foundation Pharmacy class, in which students submitted a draft using a pro-forma similar to that used by tutors to grade their final submission. Students were required to reflect on distinct sections of their essay, which again also allowed them to engage with the assessment criteria.

Encourage positive motivational beliefs – using the staged feedback described above would perhaps also address the issue of students becoming disillusioned.

Talking to a friend during the lunch break the research methodology used by the authors was also mentioned, in particular the use of ‘stimulated recall’. For this the authors played back examples of audio feedback to the tutor asking him to explain his thought processes and reflect on how his students would have responded to his comments. This methodology seems particularly appropriate to evaluate the use of audio feedback, and is something I want to take a closer look at.

A moment of serendipity

Whilst searching the twitter feed for comments on the session I noticed a tweet by @newmediac which was promoting a free webinar in which  “Phil Ice shares research on benefits of audio feedback” (here’s the full tweet). The session has already passed  but the recording for this event is here.

Tweets - Moment of serendipity
Moment of serendipity

The presenter, Phil Ice, has been working on audio feedback in the US for a number of years and has a number of interesting findings (and research methodologies) I haven’t seen in the UK.

For example, Ice and his team report:

students used content for which audio feedback was received approximately 3 times more often than content for which text-based feedback [was] received”

and that

students were 5 to 6 times more likely to apply content for which audio feedback was received at the higher levels of Bloom’s Taxonomy then content for which text-based feedback was received”.

These results were from a small scale study of approximately 30 students so aren’t conclusive. Ice has also conducted a larger studies with over 2,000 students which used the Community of Inquiry Framework Survey. Positive differences were found across a number of indicators including excessive use of audio to address feedback at lower levels is perceived as a barrier by students.

Ice has also conducted studies which breaks audio feedback into four types: global – overall quality; mid level – clarity of thought/argument; micro – word choice/grammar/punctuation; and other – scholarly advice. The study indicates that students prefer a combination of audio and text for global and mid-level comments.

Findings from Ice have been submitted for publication in the Journal of Educational Computing Research (which will soon feature a special issue on ‘Technology-Mediated Feedback for Teaching and Learning’).

Screenshot showing inline audio comments
Screenshot showing inline audio comments

Finally, I would like to mention the method Ice uses for audio feedback. He uses the audio comment tool within Acrobat Pro 8 to record comments ‘inline’. This appears to be particularly useful for students to relate comments to particular sections of their submitted work. Click here for a sample PDF document with audio feedback (this isn’t compatible with all PDF readers - I’ve tested on Acrobat Reader and Foxit Reader).

Hopefully this post has not only stimulated some ideas in the use of audio feedback, but also highlight a range of methodologies to effectively evaluate it.

National (Unsatisfied) Student Survey 2009 – The Scottish Picture

The National Student Survey results has been published by HEFCE which has no doubt left school/department managers burning the midnight oil to see how they have faired. Feedback remains to be a talking point with only just over half of Scottish students agreeing or strongly agreeing that feedback has been prompt, detailed and helpful.

But what about the students who neither agree or disagree? If you turn the question around and ask what proportion of students disagree or strongly disagree with the level of feedback they receive then you are looking at approximately a quarter of students. Obviously this is still a substantial number and still makes feedback the worst performing area, but if you are drilling down into course level performance perhaps it is worth bearing in mind.

Table 1 below shows the results for the percentage of Scottish students who responded disagree or strongly disagree to the NSS questions.

Looking at how this analysis effects the overall satisfaction with Scottish HEIs the most notable changes are University of Stirling and Robert Gordon University who (by my calculations*) jump 2 rankings. Below (#) denotes rank.

University of St Andrews91% (1)3% (1)
University of Glasgow91% (1)5% (2)
University of Aberdeen89% (3)6% (4)
University of Stirling88% (4)5% (2)
University of Dundee88% (4)7% (5)
University of Strathclyde87% (6)7% (5)
Robert Gordon University84% (7)7% (5)
Glasgow Caledonian University84% (7)8% (8)
University of Edinburgh82% (9)9% (9)
Napier University81% (10)9% (9)
Heriot-Watt University81% (10)10% (11)
Glasgow School of Art69% (12)22% (12)

*Data provided by the NSS is susceptible to rounding errors. For example University of St. Andrews has an overall percentage agree for Q22 of 92% yet the percentage breakdown is 35% agree and 56% strongly agree, which equals 91%. To allow comparison with the percentage of disagreement, the sum of percentage of responses for agree and strongly agree have been used.

Table 1: Unofficial National Unsatisfied Student Survey (UNUSS)
Provisional sector results for Full-time students - Scotland Registered HEI (% of students who disagree/strongly disagree) extracted from HEFCE NSS 2009 Data

Question20082009
The teaching on my course
1 - Staff are good at explaining things.44
2 - Staff have made the subject interesting.66
3 - Staff are enthusiastic about what they are teaching.44
4 - The course is intellectually stimulating.55
Assessment and feedback
5 - The criteria used in marking have been clear in advance.1715
6 - Assessment arrangements and marking have been fair.1110
7 - Feedback on my work has been prompt.2827
8 - I have received detailed comments on my work.2927
9 - Feedback on my work has helped me clarify things I did not understand.2725
Academic support
10 - I have received sufficient advice and support with my studies.1211
11 - I have been able to contact staff when I needed to.67
12 - Good advice was available when I needed to make study choices.1312
Organisation and management
13 - The timetable works efficiently as far as my activities are concerned.1011
14 - Any changes in the course or teaching have been communicated effectively.1215
15 - The course is well organised and is running smoothly.1113
Learning resources
16 - The library resources and services are good enough for my needs.1211
17 - I have been able to access general IT resources when I needed to.66
18 - I have been able to access specialised equipment, facilities or room when I needed to.66
Personal development
19 - The course has helped me present myself with confidence.76
20 - My communication skills have improved.54
21 - As a result of the course, I feel confident in tackling unfamiliar problems.55
Overall satisfaction
22 - Overall, I am satisfied with the quality of the course.77

Generating Student Video Feedback using ScreenToaster

video feedback
video feedback
Originally uploaded by theLaika

As I’ve recently revisited on generating audio feedback it seemed timely, particularly with a request from UHI coming into my inbox, to also have another look at video feedback. Russell Stannard recently won a Times Higher Education Award partly for his work in this particular area. In Russell’s work he uses screen capture software to record feedback on electronic submissions of student work. More information on this technique is available in a case study Russell prepared for the Higher Education Academy English Subject Centre on Using Screen Capture Software in Student Feedback. An example of using this technique is also available - click here for a short example of video feedback.

In my original post I highlighted Using Tokbox for Live and Recorded Video Feedback as a possible solution to distribute video feedback. At the time I felt there were two niggling issues with using Tokbox. First there was the requirement to install the ManyCams software to allow you to display your desktop and secondly Tokbox was very slow in uploading video you had recorded. For live video feedback Tokbox might still be worth considering, but shortly after publishing the post I discovered ScreenToaster.

ScreenToaster allows you to record your desktop without installing any software. It’s very easy to setup and the videos you create can be immediately uploaded allowing you to decides how you want to distribute and share them. The following video shows you how easy it is to setup and highlights some of the useful features. Even if you are not interested in delivering video feedback to students this is still a great site to record other material like demonstrations of software.

ScreenToaster Screencast
Example of using ScreenToaster to deliver video feedback on student submitted work from Martin Hawksey on Vimeo

Student Audio Feedback: What, why and how

Listen to this:

Download link



Originally uploaded by Dan's Photos
I captured this clip of students from the University of Chester and Sheffield Hallam talking about their experiences of receiving audio feedback at the last Podcasting for Pedagogic Purposes SIG at Glasgow Caledonian University. This event was very fortuitous as at the time I was helping source material for a Queen Margaret University (QMU) staff workshop on this very topic. If you are still unclear as to what audio feedback is here is a nice description from Andrew Middleton:

Audio feedback can be defined as formative messages, recorded and distributed as digital audio to individual students or student groups in response to both ongoing and submitted work, allowing each student to develop their knowledge and the way they learn. (Middleton, A. 2008)

Why do I think audio feedback is worth exploring? It is clear from the National Student Survey (NSS) that feedback is a particular area if dissatisfaction. There is a growing pool of evidence that students perceive audio feedback as a positive to their learning experience (although I’m not aware of research on actual learning gains).

Existing practitioners/projects

In my research for the QMU workshop I came across a number of projects and practitioners exploring this area, which I believe are worth sharing in this post:

[If I’ve missed anyone off please use the comments to highlight them and their work]

In researching audio feedback it was clear while there were distinct benefits but there were some reoccurring themes in terms of limitations. First is scalability. It’s all well and good providing individual feedback to up to 50 students but with figures beyond this it just becomes to onerous.

Audio feedback models

A solution to this problem is to explore other audio feedback models. Andrew Middleton at Sheffield Hallam University has identified a number of alternative models which include:

  • Personal tutor monologue – tutor feedback to individual student
  • Personal feedback conversations – recording tutor/student meetings
  • Broadcast feedback – generic feedback to the class
  • Peer audio feedback – student generated audio feedback
  • Tutor conversations – recording teaching staff conversations

More information on these models is available in a presentation made by Andrew at the Blended Learning Conference or in a forthcoming paper entitled ‘Audio Feedback design: principles and emerging practice’.

Don’t expect to save time

A finding from the Sounds Good 2 project is that providing individual student feedback is unlikely to save any staff time (there is a messy debate about whether high quality feedback offers long term gains in terms of how much additional feedback is required further down the line). Sounds Good have however circumstances where time can be saved (taken from the Sounds Good Final Report):

  • The assessor is comfortable with the technology.
  • The assessor writes or types slowly but records their speech quickly.
  • A substantial amount of feedback is given.
  • A quick and easy method of delivering the audio file to the student is available.

Technology

In terms of the technology there are a number of solutions which various projects propose. These include:

  • using a digital voice recorder which saves the files directly into MP3 format
  • using you Mac/PC using free audio recording software like Audacity (a portable version of Audacity on EduApps) or Wavosaur (this also can be used without installing any software)
  • using a mobile phone – if this feature is available

I also recently posted about Using Google Talk for Audio Feedback. This solution appears to be more troublesome than its worth but was useful as Joe Dale, via Andrew Middleton, highlighted the Vocaroo web service which look incredibly easy to use and worth a look at.

General advice

So what advice would I give to anyone thinking about using audio feedback? There are some very good recommendations from the Sounds Good project (final report) which are worth highlighting grouped under 4 themes: saving time; technical matters; administration; and structure:

Saving time

  • Don’t expect immediate savings in time, if any. Think of the long term returns for you and your students (there is anecdotal evidence that audio feedback reduces the need for follow up face-to-face sessions)
  • except minor mistakes. You are not looking to produce broadcast quality audio

Technical matters

  • optimise your files to minimise download size. Recommended mp3, mono and if possible reduce the bitrate to 32-40kbit/s
  • check files can be played on campus computers
  • make sure the audio is loud enough
  • have clear guidance on how to play files (I wouldn’t bother with guidance on playing files on portable media devices as the majority of students appear to prefer the convenience of playing them from a desktop computer)
  • have a backup of files in a secure location
  • if using shared devices or computers make sure files are deleted once backed up

Administration

  • if audio feedback is particularly being used as part of summative assessment make sure you have a conversation with quality assurance
  • make sure audio files are securely stored and distributed

Structure

  • try to personalise the feedback by introducing yourself, the assignment you are giving feedback on and refer to the student my name
  • keep focused – a 2 minute piece of feedback can be as, if not more, beneficial than a 10 minute ramble (the clip at the beginning of this post is just over a minute and conveys a lot of information)
  • I would recommend using audio feedback as supplemental to written feedback.

In terms of a procedure for creating audio feedback Bob Rotheram from the Sounds Good project recommends this procedure and general structure:

Feedback Procedure

  • Have the assignment details and assessment criteria with me.
  • Read the assignment, making written comments on it as I go along. If it’s on paper, I jot things in the margin. If it’s in an electronic format (e.g. Word), I use the ‘Track changes’ facility to annotate the document.
  • Read it again, more quickly this time, perhaps making a few more comments along the way.
  • Jot down (on scrap paper) the main summary points I wish to make.
  • Start the MP3 recorder.
  • Don’t bother to erase and re-record ‘misspeaks’; just correct them immediately, as in conversation.

General structure

  • Introduce yourself to the student in a friendly manner.
  • Say which assignment you’re giving feedback on. Outline the main elements of the comments which you’ll be giving (see below).
  • Work steadily through the assignment, amplifying and explaining notes you’ve put in the margins and, especially at the end, making more general points.
  • Refer to the assessment criteria.
  • Explain your thought processes as you move towards allocating a mark.
  • Offer a few, reasonably attainable, suggestions for improvement, even if the work is excellent.
  • Invite comments back from the student, including on the method of giving feedback.
  • Round things off in a friendly way.

Bob expands on these in his ‘Practice tips on using digital audio for assessment feedback

This post is based on material compiled and presented by Jim Sharp and Susi Peacock at Queen Margaret University

About

This blog is authored by Martin Hawksey e-Learning Advisor (Higher Education) at the JISC RSC Scotland N&E.

mhawksey [at] rsc-ne-scotland.ac.uk | 0131 559 4112 | @mhawksey

JISC RSC Scotland North & East logo

If you would like to subscribe to my monthly digest please enter your email address in the box below (other ways are available to subscribe from the button below):

Subscribe to MASHe to monthly email updates

Loading...Loading...


The MASHezine (tabloid)

It's back! A tabloid edition of the latest posts in PDF format (complete with QR Codes). Click here to view the MASHezine

Preview powered by:
Bluga.net Webthumb

The MASHezine (eBook)

MASHe is also available in ebook and can be downloaded in the following formats:

Powered by NEWSTOEBOOK.COM

Archives

Opinions expressed in this blog are not necessarily those of the JISC RSC Scotland North & East.

JISC Advance Logo

JISC Advance is a new organisation that brings together the collective expertise of established JISC services:

For further information visit www.jiscadvance.ac.uk

Creative Commons Licence
Unless otherwise stated this work is licensed under a Creative Commons Attribution-ShareAlike 2.5 UK: Scotland License