Keywords

1 Introduction

Audiovisual production, which in early days was mostly investigated by film and television studies, now becomes broader involving aspects of psychology, pedagogy, neuroscience and especially computing [1]. The term audiovisual, according to [2] refers to works that mobilizes at once, sounds and images. This general definition does not, in turn, include the existence of more complex products or experiences. In this context, we present the compound term audiovisual system, understood by us as a merging of any elements of software, hardware and content that compose an artifact in order to provide solutions for entertainment, health, art, education, and many other subjects.

In parallel to this evolution, studies on Universal Access and Universal Design have been expanding their scope for services, technologies and educational processes [3]. This revealed the opportunity to survey fields previously mentioned and seek eventual interconnections. Although universal design discussion has an interdisciplinarity that involves guidelines for the fields of architecture, product design and software, we do not find a systematization of such principles that contributes for the construction of audiovisual systems focused on education.

This paper presents a systematic literature review (SLR), which objective is to research Human Computer Interaction (HCI) theories and methods main usage and applications related to audiovisual systems developments and possible contributions to the Universal Design for Learning (UDL).

Present work is structured as follows: section two presents general concepts about UDL; section three details systematic review and its methods; section four categorizes research results into perspectives of problems and solutions; section five shows analyses and discussion results; the 6th and last section bestows this paper’s conclusion.

2 Background

Universal Design for Learning proposed by [4] splits its framework into three networks that toils on different features belonging to strategies used in content production regarding the act of teaching. Each UDL network has its own specific goal, which considers apprentices individual capacities and their different previous experiences, promoting multiple ways of engagement, representation, action and expression for learners.

The Affective Network is branched into grids, and its objective is to make learning purposeful and motivating for the apprentice. This network is known as the “WHY” of learning and is sorted into three categories or lines of treatment for teaching content called: access, build and internalize. The access line is concerned in providing options to recruit interest, which can be done by optimizing individual choice and autonomy, optimizing content relevance, value and authenticity and minimizing threats and distractions. The build line suggests options to sustain learners efforts and persistence, vary demands and resources to optimize challenges, foster collaboration, community and increase mastery-oriented feedbacks. The internalize line recommends options for self-regulation with topics to promote expectations and confidence by optimizing motivation, facilitating coping skills, strategies and developing self-evaluation and reflection.

Recognition and Strategic Networks lines are also divided into tables with the same access, build and internalize grids, working respectively with aspects of “WHAT” and “HOW” of learning. Their goals are focused on learners who are resourceful and purposeful (Recognition Network), strategists and more objective (Strategic Network). In the access line of the Recognition Network, the Perception grid suggests topics of perception, offering means to customize the display of information and alternatives for audible and visual information. The build line proposes options for the grid of Language and Symbols in topics that recommends treatment on vocabulary and symbol clarity, clear syntax structure, text decoding support, mathematical notations and symbols, the promotion of understanding between languages and demonstrations through different media. The internalize line indicates needs in the Comprehension grid to activate and supply background knowledge, highlight patterns, critical features, great ideas and relationships.

In the Strategic Network, the access line suggests Physical Action, with topics including the variation of methods for response and navigation and the optimization of access to tools and assistive technologies. In the building line we have the Expression and Communication grid with topics suggesting the use of multiple media for communication, multiple tools for construction, composition and construction of fluences with graduated levels of support for performance practices. The internalize line offers a grid that indicate needs for options of Executive Functions in topics of: appropriate goal-setting guidance; support planning and strategic development; information and resource management facilitation; monitoring progress capacity enhancement.

3 Methods

This review includes articles from three different databases and classifies the results from problems and solutions proposed by the analysed papers. For this SLR, we adopted the PICO (Population, Intervention, Comparison and Outcomes) protocol for initial organizing. Population comprises primary studies on HCI theories and methods. Intervention indicates theories or reports on use, development and evaluation of audiovisual systems. For Comparison and Outcomes it was sought to see which studies were directly involved with UDL principles or had potential contribution to the development of audiovisual systems aimed at the universal design for the learning field. Based on these topics, the following questions were raised: what is discussed about audiovisual systems at HCI, what are the main technologies and methods used for systems development, and what practices or recommendations converge with UDL principles.

3.1 Search Protocol

The string used for the article gathering at Springer, ACM Digital Library and Scopus was “(‘Audiovisual System’ OR ‘audiovisual’ OR multimedia OR hypermedia) AND (‘Human Computer Interaction’ OR ‘User experience’) AND ‘Design Method’”. These bases were chosen because they are the most relevant for the topics of computing, multimedia and interdisciplinarity. In addition to the search string, the following filters were added: articles published exclusively in English language, complete articles published in periodicals or conferences, articles referring to human computer interaction sub discipline or group of areas of computation and applied social sciences and temporal filter with articles published between 2010 and 2018.

3.2 Inclusion and Exclusion Criteria

All articles were reviewed considering inclusion (I) and exclusion (E) criteria. Studies were included in the systematic review by meeting the following conditions: I1, only primary study; I2, available for full access; I3, include the elements of the string in title, abstract or keywords; I4, and deal objectively with theories or methods of development and /or use of audiovisual systems. Regarding exclusion conditions (E), the studies were rejected when: denying criteria I1, I2 and I4; or if the paper has been duplicated, thus being considered only the most recent publication.

3.3 Studies Selection and Classification

Applying this search protocol to mentioned databases, we obtained 336 articles for review (Table 1).

Table 1. Application of search criteria.
Full size table

At the end of the review, 31 articles followed the conditions proposed by the scope of this research. The results were: two articles at ACM database, two at Scopus and 27 at Springer. Only one article was unavailable for access.

The approved articles were classified by the context of the research problems and solutions proposed using principles of the grounded theory, that is, the codification of data through hierarchical categorization, establishment of relation and creation of central categories. Initially the 31 articles were organized by affinities in order to find patterns between the themes. After this first organization general groups were created. The coding occurred in two cycles, one for the research problems (P) and the other for the proposed solutions (S).

4 Results

In this section, we present the results of the systematic review.

4.1 Problems

Research questions were classified in this review from three large groups. The first group is related to interfaces possibilities and includes new affordances (P1) and optimization and redesign (P2) categories. The second group contains discussions about theoretical bases and individual behaviors from scenario inspection (P3) and scientific gaps and theoretical limitations (P4) categories. The third and final group refers to diversified use of audiovisual, multimedia or multimodal systems, which includes categories of artifact development (P5) and special needs (P6).

User Interfaces

Studies [5,6,7,8,9,10,11,12,13] within this group range from investigations on creation of minimalistic graphic interfaces to complex multimodal experiments that establish new modes of interaction between individual-system and individual-individual.

New affordances (P1): issues related to new modes of systems interaction and perception are in such themes as the poetic exploration of location-based narratives [5]; variables that stimulate immersion in panoramic media [6]; user data snatching to generate new requirements and symbols for interaction [7]; spatial and visual interaction dynamics that integrates visitors in the same enjoyment experience [8].

Optimization and redesign (P2): the work on improvement or reconfiguration of projects is an issue addressed in studies in contexts of language aspects optimization, layout and narrative to facilitate the awareness of children about sexual violence [9]; navigation problems in systems with complex and extensive information generates cognitive overload [10]; interaction with Head Mounted Displays [11]; content production in virtual reality [12]; information consumption while watching television [13].

Theoretical and Behavioural Bases

Search problems of the second category includes articles [14,15,16,17,18,19,20,21,22,23]. Studies range from theoretical researches to surveys about consumption habits, perception and reception of technology, and conceptual discussions on demands for new routines and standardization processes.

Scenario inspection (P3): the understanding of scenarios comprehend studies from themes such as interface development for children from 9 to 11 years old using electronic games [14]; how do children perceive and interact with interfaces visual elements [15]; generation of informative content on treatments for cancer on toddlers [16]; the unfolding of gender demography in the scenario of development and use of electronic games [17]; the teaching of affordance concept within interaction design [18].

Scientific gaps and theoretical limitations (P4): discussions and comparisons on methodological practices are present in methods to evaluate and support emotion and affectivity aspects in interface design [19]; absence of application development guides in the context of body interactions [20]; lack of foundation to describe and foster the development of complex audiovisual systems [21], teaching skills and abilities with focus on children with Autism Spectrum Disorder [22] and procedural content generation in games [23];

Systems Application

This category include studies [24,25,26,27,28,29,30,31,32,33,34,35]. The research questions of these studies incorporate the development of solutions for diverse contexts and targets.

Artifacts development (P5): studies indicated the interest in artifacts production in contexts such as content teaching on marine animal species using augmented reality [24]; content adaptation interface for engaging students in learning [25]; simulation of classroom situations to improve teacher education and decision making [26]; audiovisual production to compose exhibition environments [27]; game for Brazilian Sign Language learning [28]; engagement in informal learning activities in the museum [29]; multisensory environment for reinforcement in offline stores [31];

Special needs (P6): issues related to the development of artifacts for individuals or contexts are framed in topics of autonomy of people with intellectual disabilities [32]; development of interfaces for the elderly in hybrid TV systems [33]; Degradation of spatial visualization capacity by elderly people [34]; Limitations in rehabilitation therapies for stroke patients [35];

4.2 Solutions

All 31 articles approved in the review were also classified according to the proposed solution. The result of this process defined three general groups: theories and methods, design and evaluation of interactive systems and case studies.

Theories and Methods

The first group is related to a proposition of notes, guidelines or theoretical reviews on production, use, and analysis of audiovisual systems. This set is divided into the categories of new modes of interaction (S1), development tools (S2) and methods and frameworks (S3).

New modes of interaction (S1): the proposition of new interfaces or affordances is present in topics such as the insertion of audio to establish interaction feedback in spatial and 3D information systems [10]; in the creation of an interactive gestural language that allows individuals to communicate with character on a second screen while accompanying a main narrative on a television [14];

Education in HCI (S2): The central idea of this category is the discussion about work processes for designers and developers. There is only one work related to this theme. The study proposes teaching the concept of “affordance” through the contextualization of the limitations and comorbidities of individuals with autism spectrum disorder [18];

Methods and frameworks (S3): the absence of design guides, implementation and conceptual bases capable of describing certain processes motivated the studies of this category to investigate topics such as the creation of a framework for the development and evaluation of poetics in narratives based on location [5]; pervasive media framework and augmented reality as a focus on integrating experience into multiple spaces [30]; framework that integrates aspects of media and learning particularities of autism for the development of audiovisual systems focused on teaching skills and abilities [22]; method to analyze emotions and motivations to design affective user interfaces [19]; method of interaction integrates multimodal and affective elements to use Virtual Reality systems [11]; method to design multimedia interfaces for m-learning [25]; model of creation and analysis of complex audiovisual systems that considers multiple levels of identity, motivation, content and experience from the convergence of concepts of human computer interaction with studies of media and audience [21]; a method that works on personalizing difficulty levels of games as a way to expand the potential of m-learning applications [35]; architecture that assists the development of multimodal interfaces for TV use by the elderly [33]; agile methods for applications development in the context of corporal interactions [20];

Design and Evaluation of Interactive Systems

The second group of researches consists of the following articles [6, 8, 13, 16, 17, 23, 24, 26,27,28,29, 31, 32, 34]. This set is divided into the categories of educational tools (S4), assistive technology (S5) and experimental artifacts (S6).

Educational tools (S4): issues related to development of artifacts for educational processes or activities is the focus of this category. The studies cover topics such as the development of an educational game that teaches entrepreneurship principles geared towards female players demographics [17]; the production of an interactive audiovisual system to broaden the engagement of museum visitors [27]; development of a multimedia and interactive environment designed to help young patients understand cancer-related medical treatments from language and age-appropriate representations of these individuals [16]; development of an augmented reality educational system with the ARToolkit library [24]; development of a dialog-based game where teachers can simulate teaching and learning situations in classroom [26]; development of a guessing game for listening users learn sign languages [28];

Assistive technology (S5): the development of artifacts for individuals in specific contexts is the focus of this category involving two studies: the creation of an augmented reality spatial visualization training system to reduce or delay the loss of the ability of elderly in perception of space surrounding them [34]; development of a collaborative communication and localization system for people with intellectual disabilities and their caregivers on daily commutes [32].

Experimental artifacts (S6): this category focuses on unconventional systems, interfaces and solutions. Studies that integrate item (S6) presents themes such as the development of immersion from three-dimensional audio applied in omnidirectional movies [6]; prototype of an android application that detects the program being watched on TV and provides real-time information in a contextualized way [13]; tool that assists the generation of procedural content for game design [19]; development of a visit support system that integrates augmented reality with the social content and geolocation of the public [8]; audiovisual system integrating the experience in museums from the virtual abstraction of the operation of the objects on display [29]; multisensory experiment for store environments [31].

Case Studies

The last group of solutions (S7) involves studies [7, 9, 12, 15]. These articles have in common the development of studies on the understanding of the use, reception and impact of the proposed artifacts. The works that integrate this group present the following themes: production and verification of an experiment that involves aspects of presence, disorientation, sense of control, pleasantness, exploration and vertigo for the experience of immersion with virtual reality [12]; study of usability and engagement for contents generated by the collection and recommendation of multimedia data in social media [7]; comparative study on design improvements [9]; organization of a participatory design process with children to raise requirements as they interact with aspects of color, layout and visual perception [15].

5 Analysis and Discussion

In this section the results obtained in the systematic review will be discussed and related to the theoretical framework of this research.

5.1 General Analysis

Studies identified within this review vary from methodological expositions to diverse levels of complexity and support technical experiments. It is possible to identify the use of technologies such as head-mounted displays [11, 12]; motion and odor sensors [19]; cybergrasp data glove [31]; computer tables [35]; Eye-Tracking, Set-top box [33]; smartphones [12,13,14, 32] and computers [24, 34, 35] in contexts as virtual and augmented reality [6, 11, 12, 30, 34]; web application [7, 15, 26]; embedded systems [13, 33], digital games [17, 28] and wearable and tangible interfaces [18].

The classification proposed by this review between research problems and solutions allows an insight into how researchers have directed the themes of their studies. Analyzing Fig. 1 we can see the predominance of the category of problems in the item development of artifacts (P5) and the strength of the category of Theories and Methods (S3). This scenario reflects a typical approach of HCI to be concerned as much with the particularities of the artifacts development in certain themes or scenarios as the organization of knowledge, processes and standards in theories and methods to assist designers and developers.

Fig. 1.
figure 1

Relationship between categories

Full size image

The research question with the greatest recurrence among the studies is the production of artifacts (P5). In response to this theme, the studies propose two methods and frameworks to increase engagement on young people (S3); four systems aimed at teaching marine animals, history, sign language and improvement of teacher didactics (S4); and two experiments on the development of multisensory artifacts for exposure environments (S6).

Yet the solution with greater recurrence is the item of methods and frameworks (S3). The proposition of processes, recommendations or development models is related to five of the six research questions in this review. Solution S3 is presented in the questions, use of poetics in location based narratives of the new affordance category (P1); improvements in the experience of using Head Mounted Displays of the Optimization and Redesign category (P2); theoretical limitations to describe and assist the development of complex audiovisual systems, absence of guides for the development of applications in the context of corporal interactions, use of emotion in the design of interfaces and absence of approaches to help the use of audio and video technologies for children autistic patients of the Scientific Gaps and Theoretical Limitations category (P4); interfaces for increased engagement in museums and learning from the Development of artifacts category (P5); therapies in patients suffering from stroke and television interfaces for the elderly special needs category (P6).

In relation to studies temporal occurrence, two works were published in 2018, eight in 2017, two in 2016, five in 2015, five in 2014, four in 2013, one in 2012 and four in 2011. Regarding its publications, studies are divided in four periodical papers and 27 conference papers, published as book chapters or as a unity at conference proceedings. International Human Computer Interaction event went out as the highlight between publishing databases, containing 12 papers out of 31. These 12 articles are divided between event’s main track and affiliated conferences: Design, User Experience, and Usability (DUXU) and Universal Access in Human Computer Interaction (UAHCI). The other articles were published in different places, with publication recurrence only at International Human Computer Interaction, which emphasizes its noticeable relevance as a publishing environment for this field.

5.2 Theories and Methods Used

Studies reviewed different methodological and conceptual approaches. Analyzing the articles, discussions were recognized between principles of cognition [11, 18, 21, 22, 25, 31, 34]; semiotics [10], use of emotions [10, 11, 19, 35]; affordance [5, 17, 18, 22] and ergonomics [20] of Human Computer Interaction.

User-centered design (UCD) has been identified as the major method to assist in the development of design and software solutions. Fifteen of the 31 articles declare this approach in the section of methodological procedures, however, other articles use data collection, evaluation and systematization of UCD requirements in their processes, which also constitutes a use, even if partial [9, 10, 12, 14, 16, 17, 19,20,21, 28,29,30, 32, 33, 35]. Another approach reported in the studies is the use of participatory design, sometimes associated to the UCD or other concepts, as a way of approaching individuals in the development of the proposed solutions [5, 8, 15, 18, 23, 32]. Besides these two central topics, there are references in the studies to the concepts of color, layout and visual structures from the principles and methods of User Interface Design. There is also the use of the task analysis technique to understand the scenario of the proposed artifacts [10, 17, 20, 25, 34, 35].

Evaluating the use and proposition of methods by the chronology of publication of the studies, we can notice the predominance of UCD and participatory design in the years 2010 to 2013. From 2014 it is possible to identify other methodological and conceptual perspectives among the studies. For example, in the year 2015 Universal Instructional Design is applied associated with the UCD [35]. In 2016, UCD is used in conjunction with agile methods of project management to build a game focused on the particularities of young girls [17]. As early as 2017 the UCD appears related to the concepts of multimodal systems design and affective computing [11]. In the same year the study [19] works on the expansion of user-centered design for analysis of scenarios and affective contexts and the study [21] propose a new theoretical methodological model that integrates general principles of HCI to media studies, denominated Audiovisual Design (AD). Finally, in 2018 we have the study [22] that starts from AD to generate a sort of adaptation of the method to the context of educational products aimed at autistic children.

Synthesizing theoretical foundations of the studies and their propositions, a table was assembled with methods and frameworks that present a potential contribution to the development of audiovisual systems to support learning (Table 2). Although the studies of category (S3) have proposed several models or frameworks, only the researches [19, 21, 22] are configured as potential theoretical-methodological approaches, since they are not restricted to guides or exclusively technological recommendations. Contributions from other studies will be elaborated in Sect. 5.4 of this review.

Table 2. Methods to artifacts development.
Full size table

The first table item is user-centered design. UCD is a widely consolidated method in software development studies. Its use in parallel to Universal Design can occur in a variety of ways, for example, in the construction and validation of personas and scenarios from the learning requirements and common points between users that have turned into design and software solutions. Although the UCD does not directly provide educational solutions, the method offers a set of practical tools to assist in the development of any kind of software. These instruments, together with UDL guidelines, can enable artifacts aimed at educational purposes.

The second item is the method proposed by study [19]. The work adds the traditional perspective of user-centered design (UCD) theories of motivation and emotion. Incorporating a taxonomy of 22 emotions to the UCD the study propose a method to initially help the construction of interaction scenarios solely. As in the previous item, there are no particularities of the educational context incorporated in the method. However, the increase of the emotional aspect expands the potential of the UCD as a method to support the creation of audiovisual systems, since the creator of the system will have more information to design the content according to the audience.

The third table item is the theoretical methodological model named Audiovisual Design (AD) proposed by [21]. The study incorporates central discussions on Human Computer Interaction as utility and quality of use issues, identity and motivation of media studies. The model defines four possible levels of interaction and states that the alternation between them occurs through four design lines: content, identity, motivation, and experience. Audiovisual Design is not necessarily contrary to the UCD approach; while the UCD is a method of supporting any type of software, the AD is directed specifically to the interaction project of audio and video systems. Like the previous items the AD does not mention learning in its scope. However, its understanding of interaction levels and design lines is a potential practice for the creation of education systems when combined with the UDL guidelines.

The fourth and final item of the table is a framework for the development of audiovisual systems for children with autism. Study [22] is an application of the model developed by [21]. The framework for autistic children incorporates the aspects of media and interaction of audiovisual design into the taxonomy of educational objectives. This arrangement is positioned to particularities of individuals as an instrument that comprises levels of interaction and learning in cognitive, affective and psychomotor domains. Among the methods and models, this is the only one that incorporates the learning process. Nevertheless, its scope of use is initially centered on the particularities of individuals with Autism. The systematization of these recommendations, together with the UDL guidelines, is a potential approach for the construction of educational systems for broad access.

It is not the intention of this work to point out a definitive solution to build audiovisual systems based on the UDL principles. The table is only intended to organize identified approaches and to reflect on their potential aspects and limitations. The perceived proposals vary in two perspectives of UCD and AD as general methodological proposals to the development of computational and audiovisual systems that can be applied in several contexts and two more specific approaches one for the analysis of scenarios and the other for autistic children. These approaches have been shown as potential practices that can be harnessed in other classes of problems outside of their initial proposition. Comprovations and checkings on the effectiveness of the methods may be verified on future studies with this purposes.

5.3 Contribution to UDL

This review gathered a total of 31 different works. From these studies, a total of six articles were compatible with the UDL principles: the Affectivity (I), Recognition (II) and Strategic (III) networks. From the total, twenty-seven articles presented some development potential for audiovisual systems focused on education.

Studies [14, 15, 17, 19, 22, 28] presented direct relationship with UDL principles, since they work on issues such as: the varied experiences of the individual during learning; the different ways of executing a specific activity; the best forms of content representation, considering the medium in which it is inserted and the forms of representation itself (audio, text, images, etc.) [25]; use of pictorial language to increase apprentices engagement and broaden learning necessities [14]; aspects of game development to motivate girls in entrepreneurship, taking gender features into account [18]; facilitation of sign language teaching from appropriate representations [28]; survey of requirements for the construction of interfaces that improve aspects of representation, engagement and expression for children and their learning characteristics [15]; the proposal of building interactive artifacts based on cognitive, affective and psychomotor domains related to media enjoyment and learning focused on children with Autistic Spectrum Disorder [22];

Although 24 articles from the total have no direct connection with the principles of the UDL, 20 of them showed solutions with some potential to help the development of audiovisual systems oriented to education, thus forming a total of 27 works (those 20 added to the other 7 with direct relation with UDL) that may contribute in some way to the development of these systems [5,6,7,8, 10,11,12,13,14,15,16,17, 19,20,21,22, 25,26,27,28,29,30,31,32,33,34,35]. The contributions identified are systematized in Table 3.

Table 3. Potential contribution to UDL.
Full size table

The contributions presented in Table 3 are divided into ten technological resources to enable systems and interfaces of interaction, eight conceptual recommendations and seven strategies to use systems to achieve results such as engagement and learning. The influence of these studies vary according to their impact on UDL networks. Nineteen studies contribute to the goals of the affectivity (I) and recognition network, while 18 studies contribute to the Strategic (III) network. Only four studies have direct collaborations with all UDL networks. The integration of these approaches to UDL principles is seen by this review as a potential contribution to the development of new audiovisual learning systems.

6 Conclusions

This article presented a systematic literature review with the objective of investigating main uses and applications of theories and methods from Human Computer Interaction (HCI) related to the development of audiovisual systems and their possible contributions to Universal Design for Education (UDL). HCI’s state of the art productions and methods using audiovisual systems, i.e. audio and video, multimedia, hypermedia or even multimodal systems, has revealed a scenario of products, devices and interaction methods that have evolved and become more complex over the years. While not working directly with the educational context, the various solutions present results that can benefit the development of learning support systems.

Among these 31 studies there are researches that highlight the most varied technical solutions that extend the way in which the systems perceive better the interaction of the individuals, in real time or not, in such a way as to return adequate solutions to their cognitive, motor and affective capacity. Although none of the identified studies propose a totally direct relation of their results to the context of universal design, it is possible to recognize that the lessons learned can be systematized and attributed to a context of greater benefit of individuals. For example, studies that have worked on understanding forms of content representation and interaction for the elderly, autistic, and children have pointed to the same premise of the need for direct communication that avoids mental overload (excessive use of colors, shapes and movement) and approaching more natural interaction interfaces. Another example of this relationship are the several studies that highlight the variation in the interfaces of representation and interaction as a way of guaranteeing to the public new forms identification and engagement for the execution of some activity.

Finally, it is believed that audiovisual system production can benefit by incorporating already consolidated discussions such as intelligent tutoring or smart learning, virtual learning environments and video lectures.