Developing A Method to Evaluate Emergency Response Medical Information Systems

Abstract

Emergency response medical information systems (ERMIS) are a specific type of medical information system used for communication and decision making during a crisis. Yet given the dependence on ERMIS during a crisis, these information systems are rarely evaluated to ascertain if the system is indeed successful. This research develops a method to evaluate the success of an ERMIS using a well-established research model as a guiding framework. We explain this method in the context of an ERMIS used in the diagnosis of pathogens in hospitals and state public health laboratories. We describe the insights obtained when using this method to evaluate emergency response medical information systems.

Appendices

Appendix A: Measures Identified in Survey Development Process

This appendix identifies the measures that were identified and considered as part of the survey development process. Most items were identified from validated instruments in the literature, while other items were developed by the authors specific to the ERMIS being evaluated.

In each table, the item and source of the item is identified. The “X” denotes which phases the items were considered during the survey development process. When evaluating an ERMIS system using the method described in this paper, these items in the following tables could be considered in Phase 1 (Creating the Question Pool). Unnecessary items to evaluate a specific ERMIS can then be deleted in Phase 2 (Narrowing the Question Pool). Finally, the measures can be piloted and further refined as part of Phase 3 (Reviewing the Question Pool).

9.1.1 System Quality Measures

Item	Source	Creating question pool	Narrowing question pool	Reviewing question pool
ERMIS is user friendly	Doll et al. (2004)	X
ERMIS is easy to use		X	X	X
ERMIS is easy to learn	Gable et al. (2008)		X
It is easy to get ERMIS to do what I want it to do	Wixom and Todd (2005)	X
ERMIS is easy to operate		X
Learning to use ERMIS is easy for me	Davis (1989)	X
It is easy for me to become skillful at using ERMIS		X
I am knowledgeable on how to use ERMIS	Authors			X
ERMIS requires only the minimum number of fields and screens to achieve a task	Gable et al. (2008)	X	X
ERMIS operates reliably	Wixom and Todd (2005)	X
ERMIS performs reliably		X
ERMIS is available when I need it	Authors			X
The operation of ERMIS is dependable	Wixom and Todd (2005)	X
The downtime of ERMIS is minimal	Chang and King (2005)	X
The ERMIS is always up-and-running as necessary	Gable et al. (2008)	X	X
The ERMIS responds quickly enough to commands.		X	X	X
ERMIS allows information to be readily accessible to me	Wixom and Todd (2005)	X	X	X
ERMIS makes information very accessible		X
ERMIS makes information easy to access		X
It is often difficult to access information that is in ERMIS	Gable et al. (2008)		X
ERMIS can be adapted to meet a variety of needs	Wixom and Todd (2005)	X
ERMIS can flexibly adjust to new demands or conditions		X
ERMIS is versatile in addressing needs as they arise		X
The ERMIS user interface can be easily adapted to one’s personal approach	Gable et al. (2008)	X	X
ERMIS can be easily modified, corrected or improved.		X
ERMIS effectively integrates data from different areas of the company	Wixom and Todd (2005)	X
ERMIS pulls together information that used to come from different places in the company		X
ERMIS effectively combines data from different areas of the company		X
ERMIS provides me appropriate information about hardware availability	Authors			X
All data within ERMIS is fully integrated and consistent	Gable et al. (2008)	X
It takes too long for ERMIS to respond to my requests.	Wixom and Todd (2005)	X
ERMIS provides information in a timely fashion		X
ERMIS returns answers to my requests quickly		X
ERMIS has all of the features that I need to do my job	Sedera and Gable (2004)	X	X	X
The system functionality of ERMIS is complete		X
ERMIS is sufficiently sophisticated to meet my needs		X
ERMIS meets the requirements of my organization	Gable et al. (2008)	X	X
ERMIS includes necessary features and functions		X	X
ERMIS always does what it should		X	X	X
In terms of system quality, I would rate ERMIS highly	Wixom and Todd (2005)	X	X
Overall, ERMIS is of high quality		X
Overall, I would give the quality of ERMIS a high rating		X	X	X
ERMIS meets your expectations	Chang and King (2005)	X

9.1.2 Information Quality Measures

Item	Source	Creating question pool	Narrowing question pool	Reviewing question pool
ERMIS provides me with a complete set of information	Wixom and Todd (2005)	X	X
ERMIS produces comprehensive information		X
ERMIS provides me with all of the information that I need		X
Information from ERMIS is unavailable elsewhere	Gable et al. (2008)	X	X
The information provided by ERMIS is well formatted	Wixom and Todd (2005)	X
The information provided by ERMIS is well laid out		X
The information provided by ERMIS is clearly presented on the screen		X
Information from ERMIS is in a form that is readily usable	Gable et al. (2008)	X	X	X
Information from ERMIS appears readable, clear and well formatted		X	X
Information from ERMIS is concise			X
ERMIS produces correct information	Wixom and Todd (2005)	X
There are few errors in the information I obtain from ERMIS		X
The information provided by ERMIS is accurate		X	X
ERMIS provides the precise information you need	Doll et al. (2004)	X
ERMIS provides output that seems to be exactly what is needed	Gable et al. (2008)		X
The resolution of the images captured and stored in ERMIS meets my standards	Authors			X
The images captured and stored in ERMIS accurately reflect the actual sample	Authors			X
It is easy to identify errors in information from ERMIS		X
Though data from ERMIS may be accurate, outputs sometimes are not	Gable et al. (2008)	X
ERMIS provides me with the most recent information	Wixom and Todd (2005)	X
ERMIS produces the most current information		X
The information from ERMIS is always up to date.		X
Data from ERMIS is current enough	Gable et al. (2008)	X
Information from ERMIS is always timely		X	X
ERMIS provides reliable information	Ives et al. (1983)	X
The information provided by ERMIS is consistent	Authors	X	X
ERMIS produces information that I can depend on	Authors	X
ERMIS has information that can be easily maintained	Chang and King (2005)	X
The information in ERMIS can be easily changed		X
I can easily update the information from ERMIS		X	X
The information from ERMIS can be easily compared to past information		X
The information from ERMIS can be easily integrated		X
The information from ERMIS can be used for multiple purposes		X
ERMIS contains information that is important		X	X	X
The information from ERMIS is clear	Doll et al. (2004)	X
ERMIS provides information that is interpretable		X
I can understand the information from ERMIS	Chang and King (2005)	X
Information from ERMIS is easy to understand	Gable et al. (2008)	X	X	X
The information from ERMIS is relevant	Ives et al. (1983)	X
ERMIS provides information that meets my needs	Authors	X
ERMIS produces information that is related to my task	Authors	X
Information available from ERMIS is important	Gable et al. (2008)	X
Overall, I would give the information from ERMIS high marks	Wixom and Todd (2005)	X
Overall, I would give the information provided by ERMIS a high rating in terms of quality		X	X
In general, ERMIS provides me with high-quality information		X	X	X

9.1.3 Service Quality Measures

Item	Source	Creating question pool	Narrowing question pool	Reviewing question pool
ERMIS support staff has up-to-date hardware and software	Jiang et al. (2002)	X
ERMIS support staff physical facilities are visually appealing		X
ERMIS support staff employees are well-dressed and neat in appearance		X	X
The appearance of the physical facilities of the ERMIS support staff is in keeping with the kind of services provided		X
When ERMIS support staff promises to do something by a certain time, it does so		X	X
When users have a problem, ERMIS support staff shows a sincere interest in solving it		X	X	X
ERMIS support staff is dependable		X	X	X
ERMIS support staff provides its services at the times it promises to do so		X	X
ERMIS support staff insists on error-free records		X	X
ERMIS support staff tells users exactly when services will be performed		X	X
ERMIS support staff employees give prompt service to users		X	X	X
ERMIS support staff employees are always willing to help others		X	X
ERMIS support staff employees are never be too busy to respond to users’ requests		X	X
ERMIS support staff employees are available when I need them	Authors		X	X
The behavior of ERMIS support staff employees instills confidence in users	Jiang et al. (2002)	X	X
Users feel safe in their transactions with ERMIS support staff		X	X
ERMIS support staff employees are consistently courteous with users		X	X	X
ERMIS support staff employees have the knowledge to do their job well		X	X	X
ERMIS support staff gives users individual attention		X	X
ERMIS support staff has operation hours convenient to all their users		X	X
ERMIS support staff has employees who give users personal attention		X	X
ERMIS support staff has the users’ best interest at heart		X	X
Support staff of ERMIS understand the specific needs of their users		X	X	X
Overall, I would rate ERMIS support staff highly in terms of their ability to provide quality service	Authors	X	X	X
In general, ERMIS support staff provides me with high-quality service	Authors	X	X

9.1.4 User Satisfaction Measures

Item	Source	Creating question pool	Narrowing question pool	Reviewing question pool
How adequate do you feel ERMIS meets the information processing needs of your area of responsibility?	Seddon and Yip (1992)	X
How efficient do you feel ERMIS is?		X
How effective do you feel ERMIS is?		X
Overall, I am satisfied with ERMIS	Seddon and Yip (1992) and Briggs et al. (2008)	X	X	X
I like having ERMIS available	Briggs et al. (2008)		X	X

9.1.5 Use Measures

Item	Source	Creating question pool	Narrowing question pool	Reviewing question pool
How many times have you used ERMIS (for both training and for other purposes)?	Authors		X	X
How many times have you used ERMIS for purposes other than training?	Authors		X	X
When were you first introduced to ERMIS?	Authors		X	X
How often do you believe your organization uses ERMIS (weekly, monthly, quarterly, yearly, never)?	Authors		X	X
I often use ERMIS to capture microscopic images for consultation	Authors		X	X
I often use ERMIS to capture gross images for consultation	Authors		X	X
I often use ERMIS to store microscopic and gross images for consultation	Authors		X	X
I often use ERMIS to review images from the Image History for consultation	Authors		X	X
I often use ERMIS the Electronic Textbook capability	Authors		X	X
When ERMIS is used for consultation, I am the person that usually uses ERMIS	Authors			X

9.1.6 Intention to Use Measures

Item	Source	Creating question pool	Narrowing question pool	Reviewing question pool
I am likely to use ERMIS in an emergency	Davis (1989)			X
I intend to use ERMIS in the future				X
Should a situation arise, I plan to use ERMIS				X

9.1.7 Net Benefits Measures

Item	Source	Creating question pool	Narrowing question pool	Reviewing question pool
I have learnt much through the presence of ERMIS	Gable et al. (2008)	X	X
ERMIS enhances my awareness and recall of job related information		X
ERMIS enhances my effectiveness in the job		X	X	X
ERMIS has resulted in overall productivity improvement for consultations		X	X
Using ERMIS in my job enables me to accomplish tasks more quickly	Davis (1989)	X	X
Using ERMIS improves my job performance		X
Using ERMIS in my job increases my productivity	X
Using ERMIS enhances my effectiveness on the job		X	X
Using ERMIS makes it easier to do my job		X
I find ERMIS useful for my job		X	X	X
Using ERMIS improves my decisions	Chang and King (2005)	X	X	X
Using ERMIS gives me confidence to accomplish my job		X	X	X
Using ERMIS increases my participation in decisions		X	X
Using ERMIS increases my awareness of job-related information		X
Using ERMIS improves the quality of my work product		X
Using ERMIS enhances my problem-solving ability		X	X
ERMIS is useful	Authors	X
Using ERMIS speeds up service delivery	Authors	X
Using ERMIS streamlines work processes	Authors	X	X	X
Using ERMIS reduces cycle times	Authors	X	X
Overall, I believe ERMIS is useful to me	Authors		X
In general, ERMIS is a positive impact on my work	Authors		X	X
Using ERMIS helps me to address patient needs	Authors		X
Using ERMIS helps me better work with physicians	Authors		X
Using ERMIS improves our organization’s care to patients	Authors		X	X
ERMIS has resulted in improved outcomes or outputs	Gable et al. (2008)		X
ERMIS has resulted in an increased capacity to manage a growing volume of activity (e.g. population growth, epidemics, bioterrorism attack, etc.)			X
ERMIS has resulted in overall quality improvement for consultations	Authors			X
Overall, ERMIS provides value to our organization	Authors		X	X
In general, ERMIS is a positive addition to our organization	Authors		X

Appendix B: STATPACK™ Evaluation Results

The following table contains the measures used to evaluate STATPack™. In addition, the number of responses, the mean, and standard deviation of each item is also included.

System quality	N	Mean	Std Dev
The STATPack™ system is easy to use	62	5.69	0.97
I am knowledgeable on how to use the STATPack™ system	63	5.44	1.30
The STATPack™ system has all of the features that I need for remote public health microbiology laboratory consultations and interactions	62	5.39	1.28
The STATPack™ system provides me appropriate information about camera, microscope and network availability	57	5.35	1.25
The STATPack™ system is available when I need it	61	6.20	0.75
The STATPack™ software always does what I expect it to do	61	5.48	1.15
The STATPack™ software performs quickly enough to commands	62	5.68	1.10
The STATPack™ system makes information readily accessible to me	60	5.38	1.14
In terms of overall system quality, I would rate the STATPack™ system highly	62	5.95	1.00

Information quality	N	Mean	Std Dev
Information from STATPack™ is in a form that is readily usable	61	5.61	1.16
The information presented by STATPack™ is easy to understand	61	5.80	0.98
The resolution of the images captured and stored in STATPack™ meets my standards	62	5.60	1.36
Information available from STATPack™ is important	60	5.78	1.04
The images captured and stored in STATPack™ accurately reflect the actual sample	62	6.06	0.81
In general, STATPack™ provides me with high-quality information (text and images)	61	5.72	1.05

Service quality	N	Mean	Std Dev
The technical support staff for the STATPack™ system is consistently courteous with users	57	6.39	0.88
The technical support staff for the STATPack™ system has the knowledge to do their job well	53	6.32	0.92
When users have a problem, the technical support staff for the STATPack™ system shows a sincere interest in solving it	54	6.33	0.87

Service quality	N	Mean	Std Dev
The technical support staff for the STATPack™ system is dependable	58	6.33	0.89
The technical support staff for the STATPack™ system gives prompt service to users	55	6.20	0.89
The technical support staff for the STATPack™ system is available when I need them	54	6.13	0.89
The technical support staff for the STATPack™ system understands the specific needs of their users	57	5.95	1.17
Overall, I would rate the technical support staff for the STATPack™ system highly in terms of their ability to provide quality service	57	6.28	0.84

User satisfaction	N	Mean	Std Dev
Overall, I am satisfied with STATPack™	62	5.82	1.02
I like having the STATPack™ system available	63	6.17	0.89

Use	N	Mean	Std Dev
I often use the STATPack™ capability to capture microscopic images for consultation	56	4.14	1.60
I often use the STATPack™ capability to capture gross images for consultation	55	3.93	1.56
I often use the STATPack™ capability to store microscopic and gross images locally	56	4.70	1.80
I often use the STATPack™ capability to review images from the Image History.	60	4.45	1.48
I often use the STATPack™ Electronic Textbook capability	59	3.69	1.50
When the STATPack™ system is used for consultation, I am the person in the lab who usually does this	58	5.07	1.50
When were you first introduced to/trained on STATPack™ (i.e., how long have you been using STATPack™)?	59	2.88	1.76
How many times have you used the STATPack™ system (considering both the times when you were trained on the system and for other purposes)?	62	32.94	128.00
How many times have you used STATPack™ for purposes other than being trained?	62	27.42	127.39
How often do you believe your organization uses the STATPack™ system? (1 = weekly, 2 = monthly, 3 = quarterly, 4 = yearly, 5 = never)	62	2.74	0.83

Intention to use	N	Mean	Std Dev
I am likely to use the STATPack™ system in an emergency	63	6.21	0.85
I intend to use the STATPack™ system in the future	62	6.21	0.89
Should a situation arise, I plan to use the STATPack™ system	63	6.37	0.81

Net benefits	N	Mean	Std Dev
The STATPack™ system enhances my effectiveness as a medical technologist	56	5.36	1.05
I find the STATPack™ system useful for my job	61	5.31	1.15
Using the STATPack™ system improves my decisions	54	5.17	1.22
Using the STATPack™ system gives me confidence to accomplish my job	60	5.22	1.17
In general, STATPack™ is a positive impact on my work	60	5.40	1.20
The STATPack™ system streamlines consultation and other work processes	52	5.60	1.21
Using the STATPack™ system improves our organization’s care to patients	56	5.21	1.25
The STATPack™ system has resulted in overall quality improvement for consultations	53	5.36	1.33
Overall, the STATPack™ system provides value to our organization	63	5.86	1.05

Demographic variables	N	Mean	Std Dev
I believe I am qualified to evaluate the STATPack™ system. (7 point Likert scale)	62	4.97	1.64
Age	61	3.82	1.09
Gender	60	1.77	0.43
Organization Type (1 = State Public Health Laboratory; 2 = Other)	61	1.90	0.30
Length of time in Organization (in years)	60	18.2	12.1
Have you encountered the Technical Support Staff for STATPack™? (1 = yes, 2 = no)	60	1.40	0.64
How many times have you encountered the technical support staff?	44	8.86	20.6
How would you rate your experience with STATPack™? (1 = novice; 2 = intermediate; 3 = advanced)	60	1.73	0.61

Open ended questions
Are there features or functionalities that the STATPack™ system is missing? If so, please explain
What feature do you find the most valuable? Why?
Does STATPack™ provide value to you, the organization, or public health? If so, how? If not, please explain your thoughts
What do you like the most about the STATPack™ system?
What do you like the least about the STATPack™ system?