Shaw Trust User Testing

The testers: Linda, Steve, Malcolm, Jamie and MarkAccessibility was a prominent feature of ClacksWeb's development plan, reflecting its status as a local authority site and my increasing awareness of it accessibility as a critical issue. As work on the site progressed in early 2005 I undertook regular checks to make sure it was on track to deliver, both by checking against the WCAG and by using very simple tests like increasing the text size, browsing the site with Lynx, and navigating with the keyboard. Yesterday it became very clear to me just how inadequate that sort of testing was in truly determining whether or not the site was accessible.

I had the pleasure of spending the day at the Shaw Trust's regional headquarters in Llandarcy, Wales, experiencing first-hand some of the user testing of ClacksWeb. Accompanied by Andrea Kennedy, the Trust's Web Accessibility Services Officer, and Grant Broome of CDSM (External link), who co-devised the audit programme and conducts the technical audit, I met, observed and learned from 6 users with various disabilities and levels of web experience. They were:

The users are provided with scripts by Andrea - basically a series of tasks, for example yesterday the users were required to register with MyClacksWeb (External link) - and are asked to record their positive and negative experiences while working their way through them. They all use PCs running Windows XP. The atmosphere is relaxed and informal - this isn't a sterile, clinical testing lab, but somewhere the testers seem to come to socialise as well as make their services available to the Trust. They are all volunteers, although there are plans to create a social enterprise offering user testing on a commercial basis - an excellent idea in my opinion.

I spent some time with each user, observing and asking questions, or in the case of Mark listening in to his use of Jaws. This was an invaluable experience for me, seeing and hearing how these users navigated the site, what barriers they were facing and what strategies they each used to overcome them. It only took a couple of minutes of observing Mark, and having him talk me through his perception of the site, to discover a serious problem with the site's contextual navigation menus. Basically they are placed just above the destination of the site's "skip to content" link, so he never knew that contextual navigation existed, and was forced to use alternative methods such as search or the A to Z to complete the tasks.

Another serious problem was encountered by Steve, the keyboard user - I hadn't specified any focus styling on links, so when tabbing around the site it wasn't obvious which link had the focus. The browser default, faint dotted border just wasn't enough for Steve to perceive the current focus. These two examples show just how important pan-disability user testing is. Neither would have been discovered by automated testing (99% of pages on the site are valid XHTML and satisfy WCAG to at least AA), and neither user was affected by the problem the other discovered.

More testing was scheduled for today, by another group of users with different disabilities, with the technical audit to follow soon thereafter. I expect to receive the initial report late next week, and based on my experiences yesterday have no doubt that it will prove to be an extremely valuable resource.


Excellent Dan,

Thank you for sharing your experiences with us, nicely written! as well I may add.

This is definitely going to be of value to myself and others.

Well done!!

Posted by: Richard Meredith at October 21, 2005 11:59 AM

Thanks for the kind words Richard, I hope it continues to live up to those expectations.

Posted by: Dan at October 22, 2005 6:24 AM

I certainly hadn't thought about the tabbing through links issue. Sounds like a really useful thing to do for web-developers. If only every web-developer could go the web would be a better place!

Posted by: Dave at October 24, 2005 8:29 PM

Dan - thanks for the review of the Shaw Trust facilities - very useful. Grant from CDSM has previously suggested that we (FTSE30 company) do some testing with Shaw Trust but I wasn't sure. Looks like a rethink for me! - Dai

Posted by: Dai Banker at October 26, 2005 12:16 PM

I'd recommend it highly Dai, even if you don't go for the full audit. The short time I spent observing user testing was enough to convince me that it's an essential part of web accessibility testing. You just can't simulate these users' experiences.

Of course some of the findings will be subjective, since disabilities aren't conveniently consistent, and every user will have their own strengths and weaknesses, preferences and peeves, but it's infinitely better than anything I could manage by myself.

And no, I'm not on commission...!

Posted by: Dan at October 26, 2005 6:00 PM

Post a comment

Personal information