Keywords

1 Introduction

Information visualization is used in various applications, such as searching, monitoring, and analyzing data. The demand to use information visualization increases according to the increase in data size. To utilize information visualization effectively, developers have to design appropriate visualization methods for target systems.

The number of appropriate visualization methods for a certain purpose or data is limited. Then, the effectiveness of a visualization method is difficult to determine before visualization results are observed, which requires implementation of visualization methods. Therefore, developers are required to have many trial-and-error attempts involving repeated implementation of these methods. As a countermeasure, visualization systems are often developed according to the following processes: developers implement many visualization methods on a prototype environment, and appropriate methods are implemented into a target program. However, we believe that the implementation costs would prevent the utilization of visualization.

This study aims to reduce the implementation costs of visualization methods to promote their utilization. We developed a visual tool to support implementation of visualization methods, because use of visual tools is one of good solutions to reduce the implementation costs [1, 2]. We supported prototyping by building a development environment called Iv Studio (Fig. 1) with a data flow visual language (DVL) for information visualization [3]. However, we need to enhance the following for improving the practical applicability of Iv Studio.

  1. (i)

    Embeddability of visualization methods into programs that require visualization features.

  2. (ii)

    Extensibility of the DVL parts.

We assumed that visualization features are added to existing programs. To enable the addition of visualization features to existing programs, (i) is important. Then, various visualization methods are required in practical use, which makes (ii) also important. Our objective is to enable developers to embed easily the developed methods by Iv Studio into their programs even if parts of the DVL are extended.

We consider various data formats, description languages, and Graphics APIs because developers are assumed to add visualization features to existing programs. Developing runtime libraries that support all environments is ideal, but it is not realistic. Thus, we designed an embeddable visualization execution environment that includes the following features:

  1. 1.

    Independence of data formats and Graphics APIs used in target programs.

  2. 2.

    Executing visualization methods on the embeddable virtual machine of the script language.

  3. 3.

    Exporting developed visualization methods as source codes.

  4. 4.

    Low-cost implementation of interface functions.

This study provides an implementation guide for development environments of visualization methods not only for Iv Studio but also for other environments.

Fig. 1.
figure 1

Screen shot of Iv Studio.

Full size image

2 Target Environments

Visualization programs require features that include loading data and showing images. Additionally, receiving inputs from viewers is necessary to support interactive visualization. We aim to execute visualization methods developed by Iv Studio in the following environments:

  • Programming language

    • C, Java, JavaScript, C#

  • Data format

    • CSV, Spreadsheet, SQL, Original formats, on memory, etc.

  • Graphics API

    • GDI, Processing, Java2D, Canvas, OpenGL, DirectX, etc.

  • GUI system (GUI toolkit)

    • WIN32 API, Cocoa, X, Swing, Qt, etc.

3 Design of the Execution Environment

We want to support various environments considering Graphics APIs and GUI Systems. Developing runtime libraries that support all environments is ideal, but it is not realistic. Therefore, we designed an execution environment according to the following procedure:

  1. 1.

    Separating dependent parts from independent parts.

  2. 2.

    Choosing an execution environment of independent parts and adding an export feature to Iv Studio.

  3. 3.

    Designing interface functions between dependent and independent parts such that the implementation costs are reduced.

3.1 Separating Dependent Parts from Independent Parts

We investigated the support range of execution environments by considering the dependent/independent parts of visualization execution systems. Data formats, Graphics APIs, and GUI systems were considered. Visualization systems are formed similar to the information visualization reference model [4]. Therefore, based on the model, we considered parts that depend on Data formats, Graphics APIs, and GUI systems (Fig. 2).

Fig. 2.
figure 2

Separation of dependent parts from independent parts.

Full size image

Raw Data are just Data format; therefore, Data Transformations depend on Data formats. Rendering is the process of showing Views on displays, which is dependent on Graphics APIs. Considering interactive visualization, the process of receiving inputs from viewers depends on the GUI systems. Conversely, Visual Mappings and View Transformations are independent from the target programs because they are pure logical operations.

Based on the above consideration, we decided that Data Transformations, Rendering, and the process of receiving inputs from viewers are executed by the target programs. Then, Visual Mappings and View Transformations are executed by our designed execution environments. Figure 3 illustrates the execution image, which is realized as follows.

Fig. 3.
figure 3

Execution image of visualization.

Full size image

  • Choosing an execution environment.

  • Developing a feature to export visualization methods to the execution environment.

  • Designing interface functions that receive Data Tables and inputs from viewers and send graphics parameters to Rendering.

3.2 Choice of Execution Environment

The requirements of execution environments of Visual Mappings and View Transformations are as follows:

  1. R1.

    Target programs can call the processes easily.

  2. R2.

    Functions implemented in the target programs can be registered into execution environments easily.

  3. R3.

    Execution environments are capable of general-purpose computing.

R1 and R2 are requirements for easy embedding of execution environments into target programs. R3 is a requirement for extensibility of the parts of the DVL. In this study, we aimed to extend the parts of the DVL by general-purpose programming languages, which would require general-purpose computing.

We considered three strategies to satisfy these requirements as follows.

[Use of virtual machines]

In this strategy, a runtime executed on JavaVM or .NET Frameworks is provided, and data and graphics parameters are communicated to target programs by sockets or interprocess communication. By exporting visualization methods from Iv Studio as a dynamic link library that can be linked to the runtime, the visualization methods can be executed, as if the visualization methods were embedded into target programs. However, this strategy is not enough to R1 and R2 because developers have to implement booting the runtime process and communicating to the runtime. Additionally, communication to web pages (the program in JavaScript) is difficult to implement.

[Exporting source codes in C]

In this strategy, source codes in C are exported from Iv Studio. This strategy reduces implementation costs of booting and communicating. Programs written in C can use the source codes directly. Programs written in Java or C# can use the source codes with native interfaces, such as Java Native Interface. Programs written in JavaScript can use the source codes with EmscriptenFootnote 1. However, this strategy also remains difficult for R1 and R2.

[Use of embeddable script languages]

In this strategy, embeddable script languages of which execution environments are provided as software libraries are used. Visualization methods can be used from various programs by exporting source codes in a script language from Iv Studio and executing the source codes. Embeddable script languages are well designed to communicate to target programs; therefore, calling and registering functions are easier than the strategy of exporting source codes in C.

Based on the above consideration, we adopted the use of embeddable script languages. There are some embeddable script languages, such as LuaFootnote 2, SquirrelFootnote 3, AngelScriptFootnote 4, and XtalFootnote 5. We adopted Lua because of its implementations in C, Java, .NET, and JavaScript. Accordingly, Lua was also adopted to the description language of the parts of the DVL.

3.3 Exporting Feature from Iv Studio

We developed a feature that exports source codes in Lua from Iv Studio to execute visualization methods on LuaVM, which is the execution environment of Lua. The processes of the parts of the DVL were described in Lua, and we enabled Iv Studio to generate source codes to call the processes according to a dataflow diagram. Then, we enabled Iv Studio to export a visualization method as a single Lua script file by integrating generated codes and codes of the parts of the DVL. Figure 4 illustrates the procedure of exporting source codes.

Fig. 4.
figure 4

Procedure in exporting source codes.

Full size image

Iv Studio exports only source codes for Visual Mappings. For other processes, such as View Transformations and receiving inputs from viewers, a support library in Lua was provided.

Developers can execute visualization methods developed by Iv Studio on target programs by linking Lua libraries to the programs and executing exported script files and the support library. Additionally, in our environment, even if target programs are running, developers can update the visualization methods, which is highly advantageous in the development of visualization programs because it is assumed that fixing visualization methods is a repetitive process.

3.4 Design of Interface Functions

For easy embedding, the implementation costs of interface functions between target programs and LuaVM must be reduced. Therefore, we reduced the number of essential functions requiring implementation. Additionally, we have shortened the implementation of the functions.

The functions were designed as low-level commands to reduce the number of functions. The implementation costs of each function were also reduced because we designed the functions such that they could be implemented using only the commands of most programming languages and Graphics APIs. Additionally, the use of arrays for arguments and return values was avoided because sending or receiving arrays required complex descriptions.

Table 1 shows the specifications of the functions to send Data Tables to LuaVM. The functions are called from LuaVM. Practically, the functions for reading files or HTTP connections are provided, such as ivs_OpenFile, ivs_ReadLine, and ivs_Close. In Table 1, names that express their operations are shown for explanation. Sending Data Tables to LuaVM is executed by implementing the functions in programming languages used in target programs and registering them to LuaVM. The implementation of the functions is simple. For example, in C, developers can implement reading CSV or TSV files by calling fopen, fread, and fclose in ivs_OpenFile, ivs_ReadLine, and ivs_Close. Developers can implement not only reading files but also connections to SQL databases, by executing SQL queries in ivs_OpenFile and returning one record in ivs_ReadLine.

Table 1. Functions for sending Data Tables.
Full size table

Table 2 shows the specifications of the essential draw functions that developers must implement and register. The upper four functions are for setting colors or font sizes. The lower five functions are for drawing or painting figures. There is a design choice where the upper four functions are integrated into the lower five functions. A function to draw lines is DrawPath only. Therefore, if DrawPath receives the thickness and the color of lines, SetPenColor and SetPenWidth are not necessary. However, to reduce the implementation costs of the optional draw functions, which are described below, we chose a design where functions for setting parameters are separated from the functions for drawing.

Table 2. Essential draw functions.
Full size table

We provided the optional draw functions that developers do not always need to implement and register. Table 3 shows the specifications of these functions. The optional draw functions are higher-level functions than the functions in Table 2, but they can be also implemented by using only the commands of most Graphics APIs. If the optional draw functions are not registered, the support library executes them by breaking their operations down to the essential draw functions. However, this execution will cause deterioration in execution performance. Therefore, we provided choices that allow developers to select priority either implementation costs or execution performance.

Table 3. Optional draw functions.
Full size table

Table 4 shows the functions for interactive visualization. Developers call the functions from target programs. All functions receive the position of a pointer, the operation amount of the mouse wheel, and the information of down buttons. Then, they return True if some operations are done in the functions. Developers can execute interactive visualization in a target program by calling it when input events occur. This design of the functions can be used in touch panel systems.

Table 4. Functions of receiving inputs from viewers.
Full size table

4 Performance Evaluation

Visualization is executed on LuaVM in our environment. Although Lua has relatively higher performance among script languages, the performance is lower than native codes. To confirm whether our environment has enough performance to treat visualization, we conducted evaluations of the execution performance.

First, we evaluated the performance of about four visualizations shown in Fig. 5. Figure 5(a) is a bar chart, Fig. 5(b) is a bar chart of sorted data, Fig. 5(c) is a scatter plot, and Fig. 5(d) is a node-link diagram using a force-directed algorithm [5]. The size of data is shown in Table 5. We evaluated the performance on C(C++) and JavaScript. On C(C++), LuaJITFootnote 6 was used. LuaJIT is a library that has Just-In-Time Compiler for Lua. On JavaScript, lua.vm.jsFootnote 7, a library that is made by compiling an original Lua library with Emscripten, was used. JavaScript was run on Electron 1.4.1Footnote 8. A PC with Windows 10 and Intel Core i7-6700K was used. We measured the execution time of Visual Mappings exported from Iv Studio.

Fig. 5.
figure 5

Visualization used in the first evaluation.

Full size image

Table 5 shows the results of the first evaluation. The time unit used is millisecond, and each result is an average of 100 executions. Note that (d) shows the time of executing a single iteration in the layout algorithm.

If visualization is executed in more than 30 fps (frames per second), we can conclude that the environment has enough performance for interactive visualization. However, this evaluation does not include the time for rendering. The execution of Rendering often takes longer time than Visual Mappings empirically. Considering this finding, we concluded that our environment had enough performance when the measured time is less than 10 ms.

As a result, on C(C++), all visualizations were executed in less than 3 ms. Our environment fast treated (d), which visualized large data. We believe that this is the effect of the optimization of JIT compiler in LuaJIT. On JavaScript, (a), (b), and (c) were executed with enough performance, while (d) took a long time. We need to optimize the exported source codes from Iv Studio.

Table 5. Performance evaluation (time unit: millisecond).
Full size table

Second, we conducted an evaluation on the number of records that could be treated by our environment. We measured the execution time of the visualization of two-dimensional data with scatter plots. LuaJIT and lua.vm.js were also used. The size of data records increased by 500 from 500 to 50000, and we measured an average time of 100 executions in each the number of records.

Figure 6 illustrates the results of the second evaluation. In this figure, the X-axis illustrates the number of records, and Y-axis illustrates the average time of 100 executions. Figure 6(a) and (b) shows the same data, but their scales are different for explanation. On lua.vm.js, our environment crashed because of lack of memory, and we could not measure the performance. Therefore, only the results until 22500 records are shown.

For the result of LuaJIT, suppose that in interactive visualization, we focus on the number of records that visualized about 10 ms. The time taken to visualize 9000 records was 10.11 ms. Suppose that in static visualization, our environment can be used for visualizing large data because 50000 records were visualized with 53.50 ms. Recent information visualization treats more than a million records and several tens of dimensions of Raw Data. This visualization often reduces the data size by using filtering, dimension reduction, or clustering. However, the reduction is performed before Visual Mappings, then our environment receives the result of the reduction. Therefore, we conclude that our environment has practical and enough performance.

For the result of lua.vm.js, suppose that in interactive visualization, the visualization of 1000 records took 12.99 ms. If interactive visualization is used on web pages, it is difficult to visualize more than 1000 records. However, the commonly used scenes of visualization on web pages are presentations. In this visualization, small size data, for example, several tens of records are often visualized. We conclude that our environment has enough performance for commonly used scenes on web pages. Suppose other implementations exist in static visualization. For example, to avoid JavaScript on clients, an SVG of a visualization result is generated on a server. Our environment has high flexibility for implementations; therefore, we conclude that executing large data in JavaScript is not needed always.

Fig. 6.
figure 6

Relationship between performance and data size.

Full size image

5 Use Cases

In this section, two systems are shown using Iv Studio and our environment. Although each system was developed by the first author, it had been developed before the addition of visualization features. Therefore, the use cases shown in this section are examples that have visualization features added to existing systems.

5.1 The Web Site

The first use case is a management web page of digital contents managed by the first author. The first author was motivated to add a feature to observe the overview of the downloads and time trends. Then, using Iv Studio, a feature visualizing the overview of trends was added (Fig. 7).

Fig. 7.
figure 7

A web page added a visualization feature. (Color figure online)

Full size image

First, a feature that exported lists of downloads and contents as CSV from SQL database was added into a program on the server side. Next, a visualization of the CSV was developed by Iv Studio. In the visualization, a ChronoView [6] was shown in the left side to show the overview of the download time, and a matrix to show contents’ names and bars representing the number of downloads was shown on the right side. Then, when marks were selected by viewers, the ChronoView and the matrix were connected by blue lines, and blue circles on the circumference of the ChronoView and marks of the ChronoView were also connected by gray lines. After the development of the visualization, source codes in Lua were exported. Finally, the codes were executed on the web page.

The first author had knowledge on JavaScript and Canvas, but no development experience with them. Therefore, we conclude that implementing this visualization on web pages using only JavaScript took more time than using Iv Studio.

5.2 Test Program of the Developing Device

The second use case is a test program for a pose capture device developed by the first author. The device is a doll equipped with multiple 3D accelerometers and 3D magnetic sensors. The program reads acceleration and magnetism data from the device, computes poses, and renders a posed 3D character model. While in development, errors of computing poses from magnetism were found. To analyze the errors, a visualization feature was added (Fig. 8).

Fig. 8.
figure 8

Device and the test program.

Full size image

This program was written in C++, and it was built on the original GUI toolkit with the original Graphics APIs. By implementing the draw functions with the GUI toolkit, the visualization feature was added in about 30 min.

In the visualization, three scatter plots show the values of magnetism on the X-Y, Y-Z, and X-Z planes, and a chart at the bottom right shows values of magnetism seen from the vertical direction. In these scatter plots, the vertical directions obtained from 3D accelerometers are also shown by orange lines. After the design of the visualization, the source codes in Lua were exported. Then, the codes were executed in the test program. To compare a result of the visualization with a pose, the visualization was semi-transparently shown overlapping the 3D character model. The semi-transparent display was realized by drawing the result of the visualization with the same Graphics APIs as the target program.

While developing the visualization, the transparency and color of the charts were tuned frequently as the results of the visualization were confirmed on the test program. Our environment allows the developer to tune the parameters while executing the program, resulting in efficient tuning of the parameters.

6 Related Work

Textual programing languages are widely used to develop programs not only for visualization. There are some toolkits to support the development of visualization programs with textual programing languages, such as Prefuse [7], Protovis [8], and D3 [9]. Alternatively, some researchers provided design patterns [10, 11] to support building visualization programs. The methodology to build visualization system and data structure for visualization was proposed.

There are other information-visualization development tools, e.g. Flina [12], Snap-Together Visualization [13], Lyra [1], iVis Designer [2], and GeoVISTA Studio [14]. Visualization methods developed with some of these tools can be used from external programs. For example, visualization methods developed with Lyra or iVis Designer can be used on web pages. GeoVISTA Studio can export Java components.

The major difference from the above research is that we focus on execution environments for embedding visualization methods into existing programs. Toolkits can be used only if they support the environments of the target programs. In other words, the use scene of toolkits is limited. The design patterns show us important guides. However, there is a difference that we focused on more practical problem, for example, the choice of execution environments considering programing languages and Graphics APIs. Although some existing visual tools provide execution environments for external programs, they have not achieved the level defined in Sect. 2, such as independence from Graphics APIs and GUI systems.

7 Conclusion

In this study, we designed an execution environment that could be embedded into various program environments easily. Then, the use scenes of visual tools were expanded. We separated the independent parts from visualization systems, adopted the embeddable script language to execution environments of independent parts, and designed interface functions such that implementation costs of embedding were reduced. Then, an exporting feature of source codes in the embeddable script language was added to Iv Studio, which is a visual tool for information visualization. We showed that our environment had enough performance for practical use. Additionally, using use cases, our environment could be used in the development of visualization systems with low costs.

The design of the execution environment is an implementation guide for visual tools for visualization development. Visual tools are one of good solutions to support the development of visualization systems. We conclude that the design reinforces the practicability of the visual tools for development of visualization.