Developing and Evaluating a Thai Website Accessibility Checker

Abstract

This research addresses the lack of a method to help with the evaluation of the accessibility of Thai websites and web applications by developing and evaluating an online tool with developers, experts and disabled users. The results suggest it is reliable and valid. Future work will extend the evaluation criteria for mobile accessibility.

1 Introduction

The motivation for this research is the lack of a method to help users, managers and developers with the evaluation of the accessibility of Thai websites and web applications. The National Statistics Office (NSO) estimated the number of disabled people in Thailand as over 1 million in 1996¹ and a majority of these could benefit from accessible websites: a report commissioned by Microsoft in 2003 estimated that 62% of people in the US of working age could benefit from accessible technologies². Research in 2006 reported that ninety seven percent of websites in Argentina, Australia, Brazil, Canada, Chile, China, France, Germany, India, Japan, Kenya, Mexico, Morocco, Russia, Singapore, South Africa, Spain, United Arab Emirates, United Kingdom and United States of America did not provide even minimum levels of accessibility³ and while there are no published figures for Thai websites it is very unlikely that they are more accessible as many of the countries tested have web accessibility legislation. Another benefit for making websites accessible to disabled people is that they are then also more likely to be usable on mobile devices.⁴

2 Literature Review

Tim Berners-Lee, W3C Director and inventor of the World Wide Web stated “The power of the Web is in its universality. Access by everyone regardless of disability is an essential aspect.”⁵ The Web was therefore invented to be used by everyone, irrespective of their ability, technology used, or culture and has the potential to remove barriers for people with disabilities, but only if websites are designed accessibly. Access to information through the Web is a basic human right according to The UN Convention on the Rights of Persons with Disabilities.⁶ Making the web accessible can help elderly people and those in poorer countries as well as people with disabilities in many aspects of their lives including education, employment, health care and social lives. Accessible websites can provide social, technical, financial, and legal benefits for companies, government and education [1].

Corporation benefits include:

• financial gains and cost savings due to increased potential market share, search engine optimization (SEO), and usability⁷ .

• reducing risk of legal action, high legal expenses, and negative image⁸.

• public relations demonstrating corporate social responsibility (CSR)

• an inclusive workplace that supports employees with disabilities

• increased productivity supporting and retaining older experienced employees

Government ministry or agency benefits include:

• laws and policies requiring public services available to all

• provision of information and services that are accessible to all citizens

• savings from improved server performance and decreased site maintenance

• enabling people with disabilities and older users to interact with them online

An educational institution benefits from:

• students, faculty, or staff with disabilities

• students with different learning styles, older computer equipment, or low bandwidth Internet connections

• increasing percentage of older employees with age-related impairments

• legal or policy requirements

When web pages are not designed accessibly, many people cannot use the Web. For example, people who cannot use a mouse need keyboard access and people who are blind need alternative text for images and this affects many people as they get older. There are estimated to be about two million people in Thailand, the majority in rural areas with 65 per cent unemployed and over 50 per cent working in agriculture and fishing and although there are anti-discrimination laws and guidance for disability development practice there are no specific laws or regulations for website accessibility.⁹ There has been a great deal of international research on the accessibility of websites resulting in the web accessibility guidelines¹⁰ which have been adopted in some countries.¹¹ However these guidelines were developed for English and simply translating them would not address all the localisation issues of Thai Language and context. Web2Access [2] was developed with the view that check-lists and tests for usability and accessibility are not an ideal way to address the issue of how easy it may be to use on-line learning materials or software in general, and that a more holistic approach is needed.¹² The Web 2.0 Services checks were developed based on the work of W3C¹³, Web Accessibility Group University of Washington¹⁴ and WebAIM¹⁵. A variety of tools were used for evaluation: AIS Web Accessibility Toolbar¹⁶ for Internet Explorer and Web Accessibility Toolbar¹⁷ - for checking web site accessibility: document structure, colours, HTML, CSS, links, images, Mozilla Firefox with WebAIM WAVE toolbar¹⁸, Web Developer Toolbar¹⁹, Accessibar Project toolbar²⁰, Illinois Firefox Accessibility Extension²¹, WebbIE text-only browser²², Zoom features in major browsers, Colour Contrast Analyser²³, VisCheck²⁴, Thunder²⁵ or NVDA²⁶ screen reader. Documentation included Testing Forms²⁷, and Criteria for Tests²⁸. The Web 2.0 Service Tests included²⁹:

• 1. Accessible Login, Signup and Other Forms: covering all aspects of registering with a service or site, then returning to sign-in and finally to work with forms.

• 2. Image ALT Attribute: so that a screen reader user can hear about the image.

• 3. Link Target Definitions: which need to be understandable when used without a surrounding sentence or button.

• 4. Frame Titles and Layout: if the frames do not have a title the screen reader user may not know where they are in the page or which piece of content to read next.

• 5. Removal of Stylesheet: as it is important to check how a site looks with and without style sheets.

• 6. Audio/Video Features: for those who have sensory disabilities such as deafness or a hearing additional text transcripts, captioning, and sign language can be very helpful.

• 7. Video/animations - audio descriptions: for those who have visual impairments offering alternatives for animations or videos where there are long scenes with no descriptive dialogue is essential.

• 8. Appropriate use of Tables: the order of content within the table and the use of row and column headers is important.

• 9. Tab Orderings Correct and Logical: when you cannot use the mouse the order in which the main navigational elements and links appear in a webpage is very important.

• 10. Page Functionality with Keyboard: after log-in.

• 11. Accessibility of Text Editors: many of the sites that allow users to contribute text, images and other multimedia also provide an editor that allows users to change the look and feel of their text as they would in a wordprocesser application.

• 12. Appropriate Feedback with Forms: once a user has submitted text or an answer to a question or multiple choice items it is important that correct feedback is received to prevent confusion.

• 13. Contrast and Colour Check: for everyone to have an enjoyable experience when reading web sites content should have good levels of colour contrast and no distracting elements.

• 14. Page Integrity when Zooming: allowing text and images to be enlarged through a zoom feature or text-resize.

• 15. Text size, style, blinking elements and Readability: avoiding items that flash or blink at a rate that can cause seizures and small text and serif fonts and complex language that can make text harder to read for some people.

3 Research Methodology

There is no official translation of the Web Accessibility Guidelines into Thai and no research into whether the guidelines require any localisation for the Thai language and culture. An interactive Thai website WebThai2Access was therefore developed to help manually test any Thai Web 2.0 site using a checklist based on Web Content Accessibility Guidelines (WCAG 2.0). In addition it explains the tools which can be used for the evaluation. It has been designed to be easier to use and score as the checklists for Web2access are all developed from the WCAG 2.0 guidelines and have been summarised and compressed into 15 criteria. The scoring has 4 levels corresponding to the WCAG 2.0 conformance levels, where 0% would be fail condition, 33% would be equivalent to an A, 67% would be equivalent to an AA, and 100% would be equivalent to AAA. The phases of the research were:

• Phase 1: A literature review was conducted to identify where Thai accessibility guidelines differ from English guidelines and tools to evaluate Thai Websites. The results of this activity helped identify changes to web accessibility guidelines for Thai tools that could be used to evaluate Thai websites.

• Phase 2: The guidelines, tests, tools and documentation were localized into Thai for the Thai Language and Culture to develop Thai guidelines, tests, tools and documentation. An expert review was conducted and validation pilot study of the tests and guidelines and tools and documentation involving accessibility experts to validate the Thai guidelines, tests and tools and documentation.

• Phase 3: Based on the results of the expert validation and review a Thai version of Web2Access was built and tested. A user evaluation pilot study of WebThai2Access was carried out and based on the results the experimental design was finalized and a user evaluation of WebThai2Access conducted with 30 developers and groups of 30 Visually Impaired, Elderly and Hearing Impaired users.

• Phase 4: The results were analysed for how well developers could evaluate Thai websites to predict how disabled users will use the websites.

4 Results and Analysis

Based on research [3] criteria 15’s text size, style, blinking elements and readability, was changed from sans-serif fonts to serif and 14–16px instead of 10–12px to suit Thai websites. Three experts followed instructions to evaluate the website http://www.tab.or.th by using a screen reader program such as NVDA, JAWS or Voiceover and also a speech Thai synthesizer program (Tatip, VAJA) by inserting an Outcome (%) which they believe to be appropriate in the ‘Evaluation’ box and in addition to this, answer all the questions in the ‘Technique’ box. An Example for the 1st test is as follows:

• 1. Login, signup, and other forms accessible, such as contact us, feedback form and help form.

• Check the process for the signup form, if there is access to the website or not, check how accessible the forms are, and if they can be accessed through the use of a keyboard and screen reader (NVDA, JAWS, and Voiceover) and check if the labelling has a meaningful name which can be understood by the users.

• References: (W3C WCAG 2.0 2.1, W3C WCAG 2.0 2.4, CAPTCHAW3C WCAG 2.0 1.1 and W3C WCAG 2.0 3.3).

• Target Audiences: Those with blind and severe visual impairment.

• Technique

  • 1.1 Check if it is possible to access any forms through the use of tab key and screen readers (NVDA, JAWS, Voiceover).

  • 1.2 Once you have access to a form, check if the label is given a meaningful name by using WAVE look at “Features” and “Form Label”.

  • 1.3 Check if it is possible to access the input aspect of the form through a logical order through the use of tab key and screen reader. If the inputted information is incorrect, such as type wrong password, then check to see if the screen reader reads the error message or not.

  • 1.4 Check CAPTCHA (W3C WCAG 1.1.1) if there is an option to change the captcha i.e. the option to change from text to sound or from image to sound or text. Check if these are able to be changed through the use of keyboard or not and also check if the screen reader is able to read the changes.

  • 1.5 Check if there are time limits (W3C WCAG 2.2.1) in the form.

  • 1.6 Check sending the form whilst pressing the button to send the form, to see if the screen reader reads the send button.

  • 1.7 Check if it possible to exit the form through the use of a keyboard and screen reader.

One of 4 ratings are possible:

• 0%: Unable to access the form and CAPTCHA through the use of keyboard and screen reader. Unable to access the form in time, and there is no label.

• 33%: Hard to access the CAPTCHA, the majority of the forms can be accessed by the use of a keyboard and the screen reader program can read the some of the form. There are a few labels used, the form has a time limit.

• 67%: The majority of the form can be access through the use of keyboard and screen reader, however there are some errors i.e. does not read the label or feedback and label identified by screen reader is not the same as displayed on the website. There is no time limit and there is an option for an alternative CAPTCHA.

• 100%: Forms can be accessed easily through the use of keyboard and screen reader, clear labels, no CAPTCHA, and there is no time limit.

The WebThai2Access website has tabs in the navigation linked to the following pages:

• products reviewed and approved by the system administrator

• list of disabilities with descriptions and associated tests

• list and short description of the 15 evaluation criteria. Selecting each criteria displays a page with further details

• Entered review information reviewer’s name, email, platform and website. If the website that they want to evaluate is not already listed as having been reviewed they will be required to add website name, URL and short description

Six developers were asked to use the WebThai2Access system at http://138.68.21.192/ and answer questions using a 5 point Likert scale and the average scores are as follows:

• 1. The content in the main page is easy to understand: 4.67

• 2. The links from the main page to the products page are all functioning: 5.00

• 3. All the links in the products page are functional: 5.00

• 4. The calculations for the products is correct: 4.33

• 5. The contents for the disability page is easy to understand: 4.33

• 6. All the links in the disabilities are functioning: 4.67

• 7. The user is able to enter their name and email in the evaluation form: 5.00

• 8. The user is able to select the platforms option: 5.00

• 9. The user can select a website which has been evaluated: 4.3

• 10. The user can add the name and details of a website which has not already been evaluated: 4.50

• 11. The system will warn the user if they do not enter the all the required details: 4.33

The average rating of 4.64 showed participants found WebThai2Access very usable. The participants suggested having more text space by increasing the character limit to more than 255, having multiple text boxes corresponding to the techniques and changing the word ‘test’ to ‘testing’ in the disability tab since when translated into Thai it is confusing for the user. The WebThai2Access prototype was modified based on the evaluation results and feedback and the 30 developers evaluated it, with an average score of 99%. Twelve criteria were rated 100% whereas criteria 8, 12, 15 were rated 98%, 95% and 99% respectively. The 30 Thai web developers also evaluated 3 websites (www.pantip.com, www.YouTube.com, http://tabgroup.tab.or.th) and the results were compared with how well 30 visually impaired, 30 hearing impaired and 30 elderly Thai People were able to carry out tasks related to their disabilities using the same 3 websites. This comparison was used to determine how well evaluations using WebThai2Access predicts the accessibility of websites for disabled users. The 30 visually impaired users’ ages ranged from 13–23 with an average age of 19. Nineteen were blind since birth and 7 became blind later on and 4 had severe visual impairment. Twenty-nine used Jaws and 1 used NVDA screen reader. Ten had 5 years screen reader experience and 20 had 1–2 years’ experience and all used the Windows operating system versions 7, 8 or 10. Eighteen used the Google Chrome browser, 9 used Internet Explorer and 3 used Firefox. All 30 hearing impaired users had been deaf since birth and were aged between 12 and 50 with an average age of 20. Twenty-five used a computer and 5 used mobile devices. The 30 elderly users’ ages ranged from 60–89, with an average age of 64.5. Twenty-four had 1–2 years experience using websites, 2 had 3years experience while 4 had more than 3years experience. Fourteen used a tablet, 7 used a smartphone and 9 used a computer. The 30 developers all had experience of HTML and developing websites and were trained to use a screen reader for the experiment and WebThai2acess. Analysis of the results suggested that using the test criteria was reliable for evaluating websites as for the 15 criteria the average 95% upper and lower confidence limits of the developer scores were plus or minus 10% for both www.pantip.com and www.YouTube.com websites and plus or minus 3% for http://tabgroup.tab.or.th and they did not overlap the rating levels of 33% or 67%. Analysis of the results for the disabled users suggested that using the test criteria was reliable for evaluating websites as for the 15 criteria the average 95% upper and lower confidence limits were plus or minus 0% for the visually impaired, plus or minus 2% for the elderly and plus or minus 5% for the hearing impaired and they did not overlap the rating levels of 33% or 67%. Comparing the average scores of the developers and experts the average difference was 18% (ignoring the direction of the difference) and 2% when the sign of the difference was considered. The mode ratings were the same for the developers and experts for 11 criteria on YouTube, 9 criteria on Pantip and 13 on tabgroup and for all websites for criteria 3, 5, 7, 8, 13, 14. Comparing the scores of the 3 groups of 30 elderly, blind and hearing impaired users with those of the developers showed that the average difference for the blind users was 26% and for the elderly was 17% and for the hearing impaired was 8% (ignoring the direction of the difference) and −1%, −17%, −7% when the sign of the difference was considered. The greatest difference between developers and blind users were 54%, 52%, −58% respectively for criteria 1, 2, (www.pantip.com) and 4 (http://tabgroup.tab.or.th). Looking at the mode values there was agreement between the developers and the blind users apart from criteria 1, 2, 4 where the mode ratings for blind users were 33%, 0%, 100% and for the developers were 100%, 67%, 33% respectively. The expert ratings were the same as the mode ratings by the blind users for all criteria suggesting that the experts were better that the developers at predicting how the blind users would perform. This might be because the developers were not experienced at using a screen reader. The greatest difference between developers and elderly users were −23% and −32% respectively for criteria 12, and 13 (www.pantip.com) and the only difference in the mode ratings were for criteria 13 where the elderly mode was 100% and the developer mode was 67%. The expert rating for criteria 13 was also 67% suggesting that the experts were not better than the developers at predicting how the elderly users would perform on criteria 13. The average differences between developers and the hearing impaired people were −12%, −12%, 2% for criteria 1, 12, 15 respectively and the mode ratings were the same, suggesting the developers predicted the hearing impaired people’s performance quite well.

5 Conclusion and Future Work

The results showed that WebThai2Access was very accessible and could be used reliably by developers and their evaluations predicted the accessibility of websites for disabled users reasonably well. Future work will investigate how to improve these predictions and also develop criteria and techniques for evaluating accessibility on mobile devices.