Keywords

1 Introduction

The traditional face-to-face patient interaction with a health services provider is be-coming less common, replaced gradually by frequent interactions with the respective health-sector web portals. It is thus increasingly important for these organizations to have an effective web presence. Furthermore, patients demand an effortlessly usable, gateway to initiate interaction, making an aptly organised portal crucial feature of the modern health care organization [1].

With patients taking over more responsibility for their own health care decision, web is an appropriate media to facilitate information exchange between patients and health-services providers. Increasingly, hospital websites are beginning to operate as extension of hospital services, offering access to a range of information and applications [2]. Therefore, in an effort to facilitate the public’s access to reliable information and to useful services from hospital websites, we consider that it is crucial to be able to assess health-sector organisations’ portals.

Health sector institutions’ websites evaluation contribute to maximize the exploitation of invested resources by organizations in the development of user-perceived quality websites. Evaluation on websites related to medical health has recently become a hot topic in the studies of health informatics and information management. Reviewing relative literature, it can be found that there are several studies related to evaluation on health-sector websites, each one assessing a variety of elements. But there is not yet an unequivocal definition of the concept of health sector website quality and the discourse about health sector institutions websites’ quality evaluation remains open [3].

The overarching aim of the present study is to review and analyse existing literature research efforts in the area of hospital website assessment and based on the extracted results, to propose an assessment framework that can integrate the identified aspects.

This paper is organized in six sections. Next section presents relative background information. The third section introduces the methodology applied, while section four reviews and analyses existing website quality assessment efforts in health sector. Section five analyses the proposed health sector website assessment instrument. Finally, section six presents the conclusions and possible future research steps.

2 Background Information

Compared to other areas of eGovernment, where assessment has been conducted more systematically for longer period (i.e. municipality services), the assessment of eHealth systems deployment is lagging behind. Hospital portals and web based systems provide patients more information, and more involvement in their healthcare, they improve access to health advice and treatment and can make healthcare systems more efficient if the patient-centred care aim is to be achieved [4].

Eighty percent of Internet users, or about 93 million Americans, report using the Internet as a resource for researching and making health care decisions [5]. A 2010 survey [6] of public, private and university hospitals in Europe showed that 81% have one or more electronic patient records systems in place, but only 4% grant patients online access to their health information. 71% use online eBooking systems for patients’ appointments with medical staff but only 8% offer patients the opportunity to book their own hospital appointment online. Only 30% use ePrescription for medicines, 8% telemonitor patients at home, 5% have some form of electronic exchange of clinical care information with healthcare providers in other EU countries.

Research in the area of health sector website assessment appears essential in order to identify the gaps and improve their overall performance. Most research on this field is focused on information context, software quality and usability issues. Nevertheless, hospital web sites should fulfil objectives beyond the delivery of accurate information and state of the art software solutions.

3 Methodology

In our literature review, we selected to conduct an exploratory study approach since it helps to acquire insight into the available literature by identifying the conceptual content of the field and by contributing to theory development towards formulating our conceptual framework [4]. The research methodology encompasses three phases.

3.1 Material Collection

In the present study, six well-known academic online databases, Science Direct, EB-SCOHost, Google Scholar, Web of Science, Scopus and Wiley Online Library were selected to search for relevant studies. The literature search was carried out in article titles from 2000 to February 2017 (time of the final search). The search for related publications was mainly conducted as a structured keyword search. The resulting search equation was defined using the Boolean operators “AND” and “OR”. The searching process was based on following keywords (hospital OR health sector) AND (web site OR website) AND (quality OR evaluation OR assessment). At the end of the database search, 45 published articles were found. Final selection of articles was carried out according to compliance with inclusion and exclusion criteria. Inclusion criteria were: the documents should be original articles published in peer reviewed journals or conferences. Only articles where the complete text was available for retrieval were included. Exclusion criteria comprised: studies which did not contain at least one health sector website evaluation aspect were excluded. Also, excluded were those which did not make specific references to website characteristics. Among these papers, 16 were determined as the suitable ones and were selected. In addition, a secondary search was carried out in article abstracts to locate possible relative resources which do not include some of the keywords in their titles. Four relative articles have been found.

3.2 Content Analysis

Content analysis was based on exploratory study of the selected articles carried out using systematic check techniques, on existing health sector web presence assessment studies. This type of research was chosen because it can provide significant insight into a given situation, facilitating the identification and structuring of new problems. The different assessment approaches have been analysed, extracting the significant elements of which they consist.

3.3 Conceptual Framework Synthesis

In designing the assessment framework, the following steps have been followed:

  1. I.

    Gleaning the main assessment elements from the selected literature.

  2. II.

    Propose framework’s main assessment criteria.

  3. III.

    Allocate the identified elements of content analysis to the proposed criteria.

  4. IV.

    Propose indicators to assess each criterion.

4 Related Works

Apart from research concerning general approaches of website evaluation and evaluation on websites subjected to commerce, government and education, there are several studies focusing on quality assessment of health-sector organisations’ websites. In this section, we review the existing literature on latter ones.

Llinás et al. [5] evaluate and compare the user-orientation of Spanish, American and British hospital websites. In their descriptive study, they evaluate websites according to readability, accessibility and the quality of information provided. Lewiecki et al. [6] develop and evaluate measurement tools to determine the quality of osteoporosis websites for patients. They use indicators in the categories of content, credibility, navigability, currency, and readability. Moreno et al. [3] present a qualitative and user-oriented methodology for assessing quality of health-related websites based on a 2-tuple fuzzy linguistic approach. To identify the quality criteria set, a qualitative research has been carried out using the focus groups technique. According to the qualitative research results they define five quality dimensions, credibility, content, usability, external links and interactivity services. Huerta et al. [1] and Huerta et al. [7] assess the web presence of hospitals and their health systems based on five dimensions, accessibility, content, marketing, technology, and usability. Tsai and Chai [8] developed an evaluation questionnaire for nursing websites covering overall impression, download and switch speed, accessibility and convenience, web page content, and compatibility with common browsers. Randeree and Rao [9] consider the following factors for evaluating health sector websites: access/usability, audience, accuracy, timeliness, content, authority, and security. Guardiola-Wanden-Berghe et al. [10] conducted an observational, descriptive and cross-sectional study carried out using systematic check techniques, on assessment of documentary and content quality assessment of eating disorder websites. Rezniczek et al. [11] evaluate the quality of websites of Obstetrics and Gynecology departments in German-speaking countries using Google search rank, technical aspects, navigation and content as objective criteria. Maifredi et al. [12] explored the characteristics of the contents and the user-orientation of Italian hospital websites. The analysis considered Italian hospitals with a working website assessing technical characteristics, hospital information and facilities, medical services, interactive on-line services and external activities. Bilsel et al. [13] present a quality evaluation model which consists of seven major e-service quality dimensions, including tangibles, reliability, responsiveness, confidence, empathy, quality of information, and integration of communication issues of websites. Moslehifar et al. [14] study focus in four different categories such as general information, accessibility of websites, functionality of websites, and facilities information provided in websites. Patsioura et al. [15] proposed framework focuses on three main criteria, information, communication and electronic services. Norum [16] evaluates the quality of Norwegian cancer hospitals’ Websites according to general information, hospital details and technical aspects. Calvo [17] assesses the quality and describe characteristics of websites of large Spanish hospitals evaluating the global quality, accessibility, usability, interactivity, updating, quality model and information. Liu et al. [18], focus on the evaluation of quality of hospital websites in China using a pre-defined objective criterion based on content, function, design, and management & usage. Garcıa-Lacalle et al. [19] determine which factors have an influence on website adoption and level of development over time. The used checklist includes elements such as general information, contacting information, web linkage, quality of care, information for patients, information about resources and performance, site navigation and usability, health information, services provided to professionals and facilitating transactions. Gruca and Wakefield [20] evaluate the status of US hospital websites examining the following features: electronic documents, providing decision aids, linkages to partners, building trust via external verification, facilitating transactions, multiparty targeting, self-service information and discussion forums. The study conducted by Mira et al. [21] on the readability and accessibility of Spanish hospital websites concludes that they need to be more patient oriented because the websites visited did not fulfil even half of the readability and accessibility attributes required by widely used standards. Mancini et al. [22] found that the enforcement of accessibility regulations has helped to significantly improve hospital website accessibility in Italy.

5 Assessment Framework

Based on the analysis of the above evaluation studies, we propose four fundamental health sector website assessment criteria – Content, Technology, Services and Participation – which cover the whole spectrum of the identified assessment elements of our literature review.

Table 1 classifies the identified assessment elements found in literature into each of the four proposed assessment criteria.

Table 1. Significant identified elements assigned to the proposed criteria
Full size table

Health sector website evaluators must be able to clearly identify whether specific goals or targets have been met and where adaptations to institution’s website strategy appear to be necessary. Progress toward achieving health institutions web presence goals can be tracked by selecting specific indicators that correspond and evaluate each of these criteria (Table 2). The performance indicators enable measurement of progress towards the achievement of the key objectives for each criterion, which in turn permits the ongoing evaluation of success in implementing the hospital’s website aimed strategy.

Table 2. Criteria and indicators allocated to each of them
Full size table

5.1 Content

Content criterion evaluates the presence of information relevant to the user. It evaluates the quality, availability, relevancy, completeness and concise representation of specific information that it is expected to be provided in a health’s sector institution website. Thereinafter the proposed indicators are analysed.

Hospital Information

Most of the hospitals provide general health information [10, 18]. The simplest health sector websites consist of electronic versions of their printed materials. Using these capabilities, a hospital website can provide up-to-date information in a cost-effective and involving manner. Hospital designation and logo on the home page are usually included in the home page [11, 14, 18, 20]. Almost all sites include information such as a general phone number for the hospital, fax number, postal address, e-mail address, VAT number, a map or directions to the hospital, parking information, transportation information and a history of the institution [5, 12, 13, 15, 16, 19, 20]. Additional elements are illustration of complementary services (press, cafeteria, Wi-Fi, telephone etc.), phone directory of the institution and emergency information [16]. Few take advantage of the available technology to provide a virtual tour of their facilities [5].

Quality Metrics

Public reporting of hospital quality data, empowers patients, referring physicians, and purchasers of health care with the information needed to make informed decisions regarding their care [20]. It also encourages hospitals and physicians to participate in continuous performance improvement by creating a healthy and competitive environment for better patient outcomes. Consequently, more and more hospitals are considering reporting their organizational quality metrics on their websites. Quality elements include the waiting list, the number of available beds, the admissions number report, the nosocomial infection rate, the inpatient mortality rate and the surgical mortality rate [14, 16, 17, 19].

Organisational Structure

The organisation chart depicts institution’s structure, it defines the hierarchy and the different roles that are involved [5]. Emphasizing on openness and accountability and attempting to make the provided services more patient-centred, lead hospitals to publish their services charter. Essential information is the list of clinical services avail-able at the hospital, the list of outpatient hospital services available (consultation, diagnostic services), the list of departments or units providing patient services, their relative working hours, their locations and their contact details [5, 9, 11, 12, 13, 14, 15, 16, 18, 19].

Medical Information

Hospital physicians should have their own place on a hospital’s website given their importance to the success of a hospital. Clearly, there is an incentive for hospitals to link website visitors with doctors having an existing relationship with the hospital. For potential patients, an electronic version of doctors printed directory is essential [18]. Apart from the list of employed doctors, sites should include doctor’s phone number, email address, picture, education/certification and relative practice information [6, 12, 18,19,20].

In this section health-disease specific information and relative treatment information is included [19]. It should also be provided the possibility to read online or to download health-care booklets and a medical glossary [12].

Patient Information

A clear description of patient’s rights and obligations is essential. Information that should be adequately addressed is the related indications for hospital admission and discharge. The website contains different types of admission, information and rules to be followed on admission, during hospitalisation and discharge as well as information to obtain a copy of the medical documents [5, 12, 19]. It also provides information for visitors [5]. Details of how to pay prescription charges, about private consultations/services and fees and information for foreigners is provided in this section [14].

Research and Teaching

Many hospitals have a teaching mission. Those institutions include in their website, information about graduate medical education in general and information for medical students, undergraduate or postgraduate courses that are held at the hospital, schedule of activities that take place at the hospital (courses, workshops and conferences), scientific studies that the hospital promotes or is involved in and publications of the hospital itself [5, 20].

Hospital libraries represent the most accessible source for medical information and services. Doctors, nurses, and other health professionals request information from hospital libraries related to a current case or clinical situation. The ability of hospital’s website to provide relative information about the library presence, address, working hours, publications catalogue and available services (reading, loans, copies) is important [5, 9].

5.2 Technology

This criterion appears to be a mixture of, mainly technical, items that relates to easy navigation, website quality, visual appeal, functionality and reliability. The technology criterion is related to how the content and services are assembled and made available on a website. Technology criterion is analysed in the following indicators.

Navigability

Navigability indicator examines the easiness that the user finds the required piece of information by moving through the website. Elements that are evaluated include effective use of hyperlinks and the degree to which the interface helps the user orient himself within the website [3, 5,6,7, 9, 12, 14].

Accessibility

Accessibility indicator refers to the practice of removing barriers that prevent inter-action with, or access to website, by people with disabilities or people with restricted computer literacy [1, 5, 7, 14, 17]. Elements that should be addressed include semantically meaningful HTML tags, textual equivalents provided for images, links named meaningfully, text and images that are large or enlargeable, flashing effects which are avoided or made optional, content that is written in plain language, compliance with WCAG W3C guidelines, compatibility with different browsers and access from various devices [1, 3, 5, 8, 11].

Usability/Readability

Usability indicator evaluates the ease of use of the website. Information should be presented concisely, without ambiguity and each item should be placed in the appropriate area [6, 8, 13]. Some of the common aspects of usability are simplicity, consistency, familiarity, clarity and relevancy [3, 8, 13, 19]. For prospective and current patients to effectively use the information available at a hospital’s website, they must have a search tool [5, 8, 12, 14, 16]. A search engine allows a patient to locate information without knowing how the hospital has organized website’s content. Other essential features include website map, content in foreign languages, quick load time, graphics that open conveniently, website pages that can be printed, individual sub-pages that have specific and meaningful titles [1, 5,6,7,8,9, 12, 14, 17].

Credibility

Because of the critical role of hospital websites in human’s health, credibility indicator is critical. Elements that should be evaluated include author and date of the provided information and the text quality which should be grammatically and spelling correct [1, 3, 7, 9]. Interest conflict declaration, date of last website update, HON (Health on the Net) foundation code certification, webmaster characteristics and sources and references should be clearly listed [3, 5, 6, 8, 9, 10, 13, 15].

Privacy/Security

Health sector website privacy holds profound implications since service delivery impacts human life, legality and social policy. Related information presentation and dissemination has raised privacy concerns among both consumers and providers. A privacy policy describing the website’s information practices should be easily accessible on the site [13]. Issues regarding patient confidentiality, copyright notice and terms of use, must be specifically addressed to become widely available [9, 18].

Inclusion of trust symbols (e.g. Verisign) allow a hospital website to stand out from the increasingly crowded internet marketplace. Security management tools and usage is an important part of the website. Other elements included in this indicator are general disclaimers, ownership of the site and provision of a secure website using encryption techniques (e.g. HTTPS) [3].

5.3 Services

The growth of consumerism and the proliferation of internet accessible sources of health-related information have modified the traditional roles of provider and patient. The trend towards creating individual patient profiles personalising the provided electronic services can bring many benefits to both hospital and patient. Personalised content can be provided during interactions with all users and this might improve loyalty to a particular hospital.

This criterion includes electronic healthcare scheduling, prescription renewal or drug acquisition, automation of hospital’s back-office procedures, forms availability on website, electronic completion of administrative transactions and on-line appointments.

Administration Procedures

Health institutions can use online forms or provide standardised documents for downloading and uploading, to their users [5, 19]. In this way, they simplify and optimise the administrative interaction with their customers. Taking this notion one step further, they can establish the use of digitally signed documents enabling the full electronic administration cycle.

Experiences in other e-commerce areas create high expectations to hospital customers for what is possible. Hospital websites are expected to facilitate interaction between visitors and the hospital staff [15, 17]. In order to achieve cost savings and streamline the treatment, hospitals allow visitors to submit e-mail requests for general health information [3, 5, 17, 19, 20]. Some of them provide the capability for referring doctors to use e-mail referral forms or furthermore enable interactive communication applications [17].

Appointments

Translating visitor’s interest in a hospital into action is one of the most important purposes of a hospital website. Online appointments and user membership registration are functions that should be included [14, 18]. Some hospitals enable their customers to interactively schedule appointments via web forms or via e-mail [12, 20]. These forms include the patient’s phone numbers, address, reason for appointment, best time to reach and preferred location for appointment. Some websites include a printable checklist of items to bring to the hospital in the appointment [20].

Patient Care

Features evaluated in this indicator provide an important link between patients and hospitals. Supporting professional practice, asynchronous communication between the patient and the physician is implemented through email or through web-based message exchange systems [8, 13]. Some hospitals offer real-time chat sessions between doctors and patients, providing in this way the opportunity to the patient to pose follow-up questions [18]. Through their websites, hospitals provide access to patient’s medical records system that creates and maintains all patient data electronically [9]. The system captures patient data, such as patient personal data, requests, lab orders, medications, diagnoses and procedures, at its source at the time of entry.

Inter-hospital Communication

Ubiquitous, secure electronic exchange of patient’s clinical data and patient’s record among hospitals/laboratories, through appropriate web interfaces, helps lessen the disruption from parallel electronic and paper-based medical record systems, thereby decreasing physician time costs and optimising service provision to the patient [23, 24].

Communication with Others

Electronic exchange of data and documents with other organisations, especially with public administration authorities, exploit the existing possibilities to automate bureaucratic procedures completion [3, 15].

5.4 Participation

Participation criterion is used to describe the interaction between hospital, patients and online communities on the web. Online communities often involve members to provide content to the website and contribute in some way. Examples of such include forums, complaints forms, interaction with the media and hospital’s marketing activities. Hospital sites can host patient support groups, interact with community organisations and become a portal for physician organisations and private medical offices.

Community Interaction

Hospital websites are aim principally to communicate with existing or prospective patients. While many visitors to a hospital’s website may have similar generic health questions or medical service needs, there is a significant heterogeneity across the entire visitor spectrum. Each patient has unique needs based on his health conditions. At the same time, the hospital must find ways to treat these widely-varied conditions efficiently. If hospitals can effectively meet patients’ widely varying information needs by using internet technology rather than more personnel, they further their twin goals of better health for patients and higher efficiency [14]. One such technology is a threaded discussion forum (e.g. diseases, allergies, treatments etc.) where visitors can post questions, and receive answers that other visitors may also access easily [3, 8, 12, 19, 20]. They often use these tools to build a community of users to strengthen the relationship with their potential and current patients [13, 18]. In order to be effective, hospitals must make a commitment to moderate the forums and provide timely as well as accurate feedback to participants.

Media

Many hospitals exploit the immediacy of the web to report current news about the institution, press releases and internal announcements [5, 14, 17]. In addition to general health information, many hospitals also inform the community about health events [14, 20]. Using internet is more cost effective than printing and distributing calendars through postal mail. It is expected hospitals to allow visitors to sign-up for newsletter or e-mail notices of community health events of interest.

Marketing/Advertising

A hospital’s website is one of its public faces [25]. Some hospitals use their websites to promote their work, and keep in touch with the different types of stakeholders [10, 14, 17, 19]. Hospitals can use their website to expand the reach of their medical practices to anyone with Internet access and advertise the international availability of their services [1, 7, 11].

Website sponsors and investors should be also clearly disclosed and possible advertising material should be differentiated form other content [6]. Social media applications can be included in this category (Facebook, Twitter, LinkedIn etc.) [14].

Financial information, including insurance details, can be included in their websites [14]. Hospital websites can be a convenient way for health care providers to analytically inform patients of their liability regarding insurance issues.

6 Conclusions and Future Work

Our review of related research has shown that hospital website assessment process can be based in four criteria, content, technical, services and participation. We identified 18 evaluation indicators which can be used to assess the above criteria.

Our framework has been designed to focus on how a specific health sector institution website applies its goals and objectives. The framework could help hospital management, health sector officials and website managers to understand causal links that show “how” and “where” a website is consistent with its strategy. This study should also be of interest to technology practitioners and researchers, as the findings shed light on the further development of performance measurements for hospital websites. To fulfil a strategic evaluation, we recommend that domain experts have a better understanding of the website’s aims and evaluate the site according to those.

Next step of our research will be to determine specific metrics and relative weights for each indicator in order to implement a concrete assessment instrument for health sector institutions web presence. Hospital websites assessment instrument, apart for health institutions’ managers, will allow patients to search for hospitals and compare them based on their performance on various quality measures.

In terms of practical application, we plan to use it in Portuguese hospital’s website assessment and discuss the results with hospitals’ management and health sector authorities. This will complement views expressed in individual discussions and group workshops, to assess practical acceptability in a better way.

Health-sector websites are the public face of most hospitals, integrating the hospital, the citizen, the physician, and the patient [9]. Website visitors will expect to complete their transactions with the hospital via the web. If they do not take advantage of the available technology to serve and interact effectively with their patients, then hospitals will have a greatly reduced role in many future health care decisions [20].