Keywords

1 Introduction

A Naval Innovative Science and Engineering (NISE) project titled User Interface Prototyping Toolkit (UIPT), is exploring the rapid production of high-fidelity user interfaces (UI) prototypes to be utilized by end users, researchers, designers, and engineers working together to develop and validate, new concepts of operations (CONOPS) and emerging technologies.

The objective is to examine the ability to rapidly develop a specific human-machine interface prototype for future Navy information systems that address emerging operational and tactical threats and is currently targeting advanced UI’s designs specifically for a large multitouch display device. UIPT is a software development effort to explore rapid prototyping of information systems while standardizing forward looking UI interactions, such as multitouch and augmented reality (AR).

To date, software development and technology growth has escalated to meet the business demand. Often a holistic or a “full stack” approach is taken to design and implement an entire software system. This means the frontend or user interface is tightly coupled to the backend or server-side components of the software system. UIPT is taking a split stack approach, separating the frontend from the backend allowing a more focused approach to the creation of the user interface. The UIPT project took the concentrated approach to focus just on the user interface and apply a methodology to mold the interface to specific use cases in an interactive and rapid manner.

1.1 Motivation

The US Navy is increasingly dependent on producing superior information systems in order to create and maintain a decisive advantage over the Nation’s adversaries.

As new technologies such as Artificial Intelligence, Machine Learning, and Autonomous Systems grow in importance to the Navy, the ways in which end users will use and control these technologies to maximize user performance gains is largely unknown. More emphasis on the design of future information system’s user interfaces in the DoD is critical.

For decades, the lack of emphasis on User Experience (UX) design in software development has resulted in software user interfaces which are non-intuitive, difficult to use, non-task oriented, and have a general lack of form and function required for knowledge and decision support end use. Fortunately, this trend has shifted. Software companies like Apple, Google, Facebook and Amazon have been investing in internal design systems that speed the design process, allowing for product iteration and optimization in early stage prototyping before the commitment to full scale production. As examples, Google’s ‘Material Design’ [1] addresses design style, while tools from companies such as InVision and Figma have been addressing UI prototyping. Currently, UI prototypes which focus on innovative Naval CONOPS development are limited but work in such areas are continuing to increase through efforts such as User-Centered Design [2].

The UIPT toolkit concept when finalized, will allow the rapid production of high-fidelity UI prototypes to be utilized by end users, researchers, designers, and engineers working together to develop and validate CONOPS and new technologies through the use of a scenario driven interactive prototype. This process allows all participants to experience high fidelity user interfaces, that are representative of the desired end product, well before large financial commitments are approved.

1.2 Technical Background

The UIPT technical approach is based on years of experience developing user interfaces for the Navy. Examples range from the 1990s with the Multi-modal Watch Station [3, 4] (see Fig. 1), a touch-screen and task-based application which supported a 3 to 1 plus manning reduction through efficient tasking; to today with a “55” multi-touch workstations [5,6,7] for the NIWC Pacific Command Center of the Future VIP demonstration room.

Fig. 1.
figure 1

Multi-modal watch station (MMWS), 1990s

Full size image

Other NIWC Pacific programs involving mission planning, task-managed designs, decision support aids and visualization have provided a wealth of experience in design and development of user interfaces. This included many user engagements and knowledge elicitation sessions to validate UI concepts and requirements, all of which provided the foundational basis for the UIPT toolkit.

UIPT focuses on UI design and development using a split-stack approach to concentrate on the front-end versus the back-end of the system. Integration of back-end systems is often the focus of DoD software development efforts in lieu of UI design and application. The complexity and effort required for integration can overtake the UI work, therefore UIPT purposely focuses on the front-end while striving to approximate the back-end by simulation. End user and subject matter expert sessions are used to illicit knowledge, data and scenarios, to drive the front end, while the simulation provides enough realistic operational context to process the desired scenario.

UIPT grew out of a mockup demonstration of a future Navy information system as depicted in a NIWC Pacific Vision video, produced in 2015. NIWC Pacific contracted a multi-touch display manufacturer to work with NIWC designers and subject matter experts to design and manufacture the TouchTable hardware and code the mockup application. This mockup (see Fig. 2) was specifically developed for demonstrations to DoD VIP visitors striving to understand the future direction of C4ISR for the U.S. Navy. The mockup provided the foundation for the start of the UIPT toolkit from its first instantiation to the current implementation, which is a rapid prototyping tool for information systems development. The value of a high-fidelity mockup over a PowerPoint presentation was clearly evident and validated by the many VIPs visiting NIWC Pacific.

Fig. 2.
figure 2

Advanced user interface prototype design, 2015

Full size image

1.3 User Interface Display Hardware

User interface designs for UIPT were targeted towards a large 55-in. Touch Table device which is multi-touch enabled. The devices were scaled and positioned ergonomically to be within reach so the user can interact with the system via touch to make a compelling digital interaction for the end user. This device is ideal for 4 to 5 persons to gather around the display and interact with the information via the user interface. UIPT design and development efforts center around this Touch Table Device. User Interface elements within the system were specifically designed to be interactive in a touch environment. Multiple touch commands are recognized by the system to perform standard tasks such as zoom in and out on the map, change the aspect of the map, rotate the map, call up and dismiss data when needed, respond to alerts, initiate commands and set display options. Additionally, users can interact with the system through a series of voice driven commands (Fig. 3).

Fig. 3.
figure 3

Touch table used by UIPT

Full size image

1.4 Interfacing to Information Systems

The ability to link an existing or legacy information system to an advanced user interface prototyping system, such as UIPT, in most cases requires a significant amount of work to feed the data from the system itself. The question of sensitive or classified information also arises which further complicates such an integration. Often the use of simulated data to fill the display is sufficient for said design purposes. The idea here is not to build the UI as the system is being developed, but to minimize the risk of poor design by doing the design work in front of the development. The UIPT effort seeks to correct this design deficiency by simulating enough data to make a realistic scenario to drive user interactions and converge on an efficient and effective design before software and engineering development starts. This way the end state of the user interaction is already known, and the work focuses on engineering the solution.

2 Objectives

By providing a rapid User Interface Prototyping Tool that allows users to explore the impact of new technology on emerging operational concepts before large program investments are made, the Navy can significantly reduce the risks associated with pursuing revolutionary technology. The Navy is currently investing in new technologies, such as, autonomous vehicles, artificial intelligence, distributed sensing and integrated fires, but the way in which Fleet users will work with and “team” with these new capabilities is largely unknown.

The objective of the UIPT project is to provide users, researchers, designers and developers with a UI toolkit to explore, analyze, develop and test new CONOPs and emerging technology that maximize the human interaction efficiency. Currently, there are no tools dedicated to rapidly testing of CONOPs and technology and the effects on C4ISR decision making through the user interface (UI). The UIPT will enable acceleration of relevant and well-designed capabilities to the Fleet by testing novel CONOPS based on rapid UI testing. Testing UIs before development will save money as Fleet users will already have been informed of and validated the concept.

2.1 Development of the User Interface Prototyping Tool

The UIPT software application allows the quick development of UIs to collaboratively explore and validate new CONOPS and the impact of new technology on the end user decision making process. UIPT also supports the exploration and analysis of improving the usability of deployed, and currently under development software systems used by the targeted audience. This exploration and development targets C4ISR and Cyber mission areas for novel CONOPs and UIs. This results in meaningful requirements and design input into programs of record as well as other projects.

A conversion of the original touch table software from JavaScript was required based on the intended use and objectives for the project. Initial efforts during the first year of UIPT focused on utilizing agile development for development with HTML5, NodeJS, ReactJS, and the REST Framework. The front-end JavaScript engine was used to support the front-end framework with the option of running in the browser. Agile teaming lead to the discovery of the potential difficulty in modifying and rapidly producing the software using the JavaScript/ReactJS technology. Even though this technology is still valid and could provide the same end results, an exploration was made into the use of Unity [8]. Unity is a cross-platform game engine produced by Unity Technologies which is generally used for 2D and 3D game development and simulations. During the exploration of Unity, a faithful reproduction to roughly 80% of the advanced display demonstrated on the Command Center of the Future Touch Tables was developed in less than a month. Further development of new components further validated the selection of Unity by demonstrating the development capability for new features. It became apparent that this rapid and agile development tool would be a natural fit for the project. The outcome using either Unity or ReactJS is the same but the saving and added capability in development validated the switch from ReactJS to Unity.

The development effort, utilizing Unity, provides for dynamic UIs for both web-based applications in a browser (WebGL based compile of Unity) and desktop applications. The development process allows the ability to switch out new user interfaces without complex and costly rewrites. There are many advantages to this approach: UI interactions are faster and use less bandwidth, it allows for developers to specialize and have less dependency of single stack experts, it makes it easier to build automated testing and it minimizes technical debt in agile software development. Furthermore, Unity is proven as a leading gaming engine technology at the current state of the art.

2.2 Employment of the Toolkit

The UIPT tool will provide an interface where designers can work with the users to quickly produce high-fidelity UI prototypes of future naval applications. The high-fidelity UI prototypes will be presented to users in a “mock” command center environment for CONOPS exploration and validation.

The high-fidelity UI prototypes will provide the framework to present new UI components as either real code, animations (animated gifs, etc.), or images. Track data will be presented on the UI screen from a simulator (i.e. NGTS [9], JSAF [10]), but other necessary data and information will be simulated and accessed from the UIPT’s internal database and files. There will be a simple messaging capability to allow different UI prototype instances to communication and share information, alerts, and notification. The Fleet user will be able to experience the “New Application” capability as if it were a finished product. Additionally, multiple touch tables will be networked together and co-located to recreate a multi-person staff collaborating on mission objectives and execution.

3 Approach – Design and Development

The technical approach of UIPT has two main goals, first the creations of user interface designs which meet the end users’ objectives and requirements, and second the creation of the toolkit which allows that to happen. Both goals are required to meet the objective of the UIPT project. Designers and software developers work hand in hand in an Agile development process with weekly evaluations of the toolkit for both form and functionality.

3.1 User Interface Design

UIPT use a relatively straight forward design process, performing user research, developing multiple design concepts, testing with users and iterating many times. This standard design process to produce a high-fidelity mockup that was used to generate discussions addressing the future direction of Navy information system.

The original mockup research mainly borrowed from current and past NIWC Projects. Since the objective was to create a unique rapid prototyping tool for use to discover how new CONOPS and new technology furthered the Navy mission, there was no available Navy application or Fleet user group to enlist.

The following Figs. 4, 5, 6, 7 and 8 show the progress of the original mockups design to the current UPIT Unity build.

Fig. 4.
figure 4

Wireframe of base design – homepage

Full size image
Fig. 5.
figure 5

Early HUD design

Full size image
Fig. 6.
figure 6

Early design

Full size image
Fig. 7.
figure 7

Intermediate design

Full size image
Fig. 8.
figure 8

Final design

Full size image

3.2 Experimentation Plans and Metrics

UIPT will engage users early in the design process and conduct preliminary user engagements evaluating the prototype. Prior to installation and deployment, UIPT will need to work with users to determine baseline condition. During user events, users will review and comment on a selected Navy Mission domain user interface, as presented in an interactive scenario. User feedback will be reviewed and assessed by the UIPT development team and improvements will be made rapidly during the actual event if possible.

The critical technical parameters will be divided into prototype usability and target concept utility [11]. For prototype usability a combination of the following metrics will be collected during testing (Table 1).

Table 1. Usability metrics
Full size table

For target concept utility, we will focus on Situational Awareness. The final metrics have not been determined at this time, however a modified Situation Awareness Global Assessment Technique (SAGAT) [12] could be used to collect metrics during the course of testing (Table 2).

Table 2. Utility metrics
Full size table

3.3 Toolkit Development

Software development of the UIPT toolkit is based on an Agile development process which is closely coupled with designs and feedback from the designers of the toolkit. An iterative process is performed to design, develop, test and evaluate the UIPT toolkit. Unity currently serves as the software platform upon which the UIPT toolkit is developed.

Unity used as a development tool gives UIPT developers the ability to create interface elements in both 2D and 3D. The engine offers a scripting/coding API in C# which is used within the Unity editor and other code editors such as Visual Studio. A developer can utilize assets and prefabs, a Unity term for a module, along with scripting/code insertion capability in order to add functionality to the application itself or to the assets and prefabs. The engine has support for the following graphics APIs: Direct3D on Windows, OpenGL on Linux, WebGL on the web. Although Unity is designated as a 2D and 3D game engine, it’s set of tools allows the development of advanced user interfaces which provide rich 2D menus and user interface components as well as 3D based visualizations. This permits the overlay of UI components on top of a map-based visualization of virtual real-world space. Many of the Unity 2D features support the generation of rich user interface elements that can be intermixed with the rich 3D display elements. This unique combination provides flexibility in the types of interfaces which can be built. The move from JavaScript to Unity was validated in the faithful recreation of the VIP based demonstrations on the touch tables in the Command Center of the Future. Unity’s strong base of third-party development assets and prefabs reduces development costs and saves time. These features of Unity set it as the tool of choice for UIPT based development, but the primary factor was the ability to produce components of equal quality at a much faster rate than using the JavaScript platform.

UIPT requires no network connections for normal operation under the current configuration and development effort, however, there are various prefab add-ons as well as integrated networking capability which becomes essential when driving the user interface with simulated data. All data and processes are installed on the Touch Table workstation and can run with or without a network connection in its current configuration. A simulation generator is currently being developed to drive the system. Initially the simulation generator will be self-contained in the Unity application while a more scalable and flexible approach to a simulation generator is being developed. Unity provides capabilities to move and position entities through built in structures and coding. An example is the use of waypoints which can be placed within the scene and then the use of coding permits a given asset to be moved along a chain of waypoints in a manner prescribed by the functionality added in the code. The use of finite state machines, also part of Unity, is also being used in connection with the built-in simulation generator. The simulation generator permits the creation of scenarios to drive the Unity based user interface and maintains the split stack approach. UIPT provides the ability to switch out new user interfaces without complex and costly rewrites. Typically, all software can be thought of as a stack consisting of many layers. The frontend layer consists of the user interface (UI), while backend layers usually consists of a server, an application, and a database. This split in the software stack development is an architecture pattern that’s been developing for several years. Splitting up the backend into several layers for very large complex applications is a common practice, but the new split is frontend to backend. Multiple scenarios can be generated in order to fully explore the domain for the design and functionality created through the rapid prototype toolkit. In this manner the simulation generator can be repurposed for each domain or application that uses the toolkit for UI rapid prototyping.

To date the Unity based development of the UI rapid prototyping tool has made significant progress beyond the initial task of replicating the preliminary JavaScript implementation. Development work will continue with developing common elements of the advanced UI that can be shared across domain implementations as well as the ability to use the toolkit for rapid prototyping. Rapid prototyping concepts consist of allowing the interface to be modified in certain ways as well as providing instantiated options to a given design. This allows the collaborators to relocate and resize components, repurpose components, view components in various optional configurations, and have the ability to change styles, colors or other objects. The development process allows for collaborative feedback to be given upon which the development team prototypes the desired changes. Another collaborative session is then held to further refine the prototype development. An illustration of a work in progress using the rapid prototype UI toolkit is illustrated below in Fig. 9.

Fig. 9.
figure 9

Example of unity based UIPT screen

Full size image

4 Conclusions and Future Work

The UIPT project has been in development for just over a year. A portion of that was dedicated to switching the development platform from JavaScript to Unity. The transition to Unity occurred during the last half of the time spent on the project. Not only was the initial JavaScript prototype faithfully implemented in Unity, but its use has been extended to a specific project domain. Several other project domains have since been added to the list of efforts that are porting or creating their project interface via the UIPT toolkit and the advanced display design used in the baseline user interface of UIPT.

The toolkit is still in the process of developing tools and additional views that allows an enhanced end user, researcher, engineer, designer, UCD practitioner, and software developer to collaborate on new user interfaces and CONOPs for a given domain. The value of this new interface has been validated by several domains that have started to incorporate the UIPT designs even before the UIPT Toolkit is completed.

The following describes areas of the UIPT project that require further examination:

  1. 1.

    Complete the elements of the UIPT Toolkit to allow viewing multiple options of a given display. Improve and continue the development of being able to modify the UI for such things as position and size of components or widgets, colors, styles, fonts as well as new components or widgets.

  2. 2.

    Complete baseline user interface along with weekly improvements and bug fixes. An Agile process is in place that lets all team members work together to accomplish this goal.

  3. 3.

    Apply the rapid prototyping of UIs to several new domains. This will validate the extensibility of the designs and the toolkit. One domain is currently being worked with two to three others ready to be worked immediately.

  4. 4.

    Evaluate the effectiveness of the UIPT Toolkit via user sessions from the perspective of the UCD practitioner, the software developer, the designer, and the end user.

  5. 5.

    Continue the iterative process of design and development improvements to the Toolkit to explore new areas not covered or yet to be discovered.