Keywords

1 Introduction

The decision to make an investment of resources in any context requires a thorough study of the current situation. Through an analysis of the scenario, it is possible to detect the strengths, threats, opportunities as well as any other important factors influencing that decision. When Public Universities investing in technology arises the need to know what the improvement entails and if it is possible to assume that investment will improve learning [6]. Obviously, it does not seem reasonable to assume that such an increase in technology resources will lead to an improvement in learning, if a prior study of how to measure the impact of online teaching and learning ecosystems has not been done before [8]. To measure efficiency in the use of these ecosystems; as a support to the teaching - learning strategy in the face - to - face classes of higher education, seeks to measure the total impact of this tool in its environment, this means to measure a space of value broader than economic, including economic dimensions, also social dimensions that contribute to show the true value of these tools, showing the benefits of its activity in the information society in which it operates and, at the same time, giving relevance to the needs, aspirations and expectations of its actors.

It should be emphasized that the term ecosystems in the learning environment refers to technological ecosystems: that is, ecosystems defined as the evolution of traditional information systems to support the management of knowledge in heterogeneous environments and when talking about measuring [7], as a result of the fulfillment of the objectives of a program aimed at improving the teaching-learning process by applying these tools.

In this paper we study different models to try to find out which are the relevant aspects to study to measure the impact of the investment of learning ecosystems. Thanks to the boom in the use of TIC’S in classrooms over the last few years, there is not only a growing participation of higher education institutions (HEIs) in technological issues, but organizations are more aware of their effect on the knowledge society and the impact it produces [16]. In this sense, HEIs realize that the value generated by the use of technological platforms is not only economic, financial, but also social.

The use of learning platforms generates changes in the organization that use them [9]. By building knowledge, autonomous learning, virtual group work, virtual contact with the tutor, transfer of information, broadening access to the resources and services of the platform, tools impact on the lives of users, Their capabilities, their future opportunities and their knowledge-building styles. Thus, by assessing their socio-economic impact, HEIs can reduce costs and risks and create new opportunities for action that minimize negative impacts and maximize positive ones. Equally, impact assessment can help HEIs show their interest groups the generation of (socio-economic) benefits for the environment in which they operate.

In the evaluation process, and in particular in the measurement phase, HEIs can understand the needs, aspirations, resources and incentives of the actors, allowing them to develop new knowledge that in turn will generate new products and services or improve existing ones [13].

In summary, efficient use of online learning ecosystems through a value assessment process can produce important benefits such as: improving investment efficiency, demonstrating results beyond purpose and involving all stakeholders. To evaluate the impact on the use of online learning platforms, a series of criteria to be measured should be established in order to verify the efficiency of these tools.

1.1 Analyzed Aspects

The study seeks to determine the impact of two aspects:

  • Economic aspect focused on:

    • Growth (access to global knowledge, innovation generation and knowledge transfer).

    • Return on investment (operational efficiency, level of development of stakeholders).

    • Risk management (operational, regulatory and operational risk).

  • Social aspect focused on the changes produced in the lives of users of this tool, measuring the following aspects [11]:

    • Increase of users with access to this service.

    • Increases of new types of devices to access e-learning spaces.

    • Reduction of time spent learning a topic.

    • Level of perceived knowledge transfer.

    • Level of achievement of objectives of each of the actors.

    • Number of trained, sensitized and trained beneficiaries on the use of infrastructures.

The economic and financial value is reflected in income and expenses, balance sheets and income statements, but human value is not reflected in any state of the organization. If it is necessary to describe the value generated by learning ecosystems, it is necessary to have a method to measure and reflect human value, considering all the actors that are part of the platform implementation and use environment [10].

Two main problems arise in order for these platforms to be evaluated and to determine their true value: the existence of a large number of measurement methodologies, each of which are based on dissimilar assumptions and functionalities, focusing on different types of impact; adapted to different purposes [12].

After reviewing the two models that will serve as the starting point of this work, we present the possibility of evaluating the common and specific situations of each of these models by means of heat maps, representing the information in a simple and intuitive form, without reducing the characteristics of the original information. By means of heat maps it is sought to graphically represent each of the indicators of the models, in order to facilitate the understanding of the elements and their relationships. The addition of a color code to represent the level of influence of each element allows the construction of the map so that, in addition to the structure given by these maps, it is also possible to visualize the state of each model [1].

Heat maps are powerful tools of representation that have the following main objectives:

  • Provide a systematic way of structuring the components of the models studied.

  • Evaluate the characteristics of the selected models.

  • Identify relevant and common aspects of these models.

2 Methodology of Research

There are numerous and very diverse works published in the different databases. In this situation, it is difficult to obtain an objective view of a topic since many works are published with different points of view and different results. A Systematic Literature Review (SLR) [4] helps to obtain objective information on a research topic by identifying, critically assessing and integrating the results of all relevant studies in that subject. An SRL has several objectives. Among them we can highlight: existing research advances, identification of relationships as well as formulations of general concepts. Therefore, an SLR [14] is in itself a research work and serves much more than to obtain a mere compilation of published works on a topic. In the initial phase of application of the systematic literature review the following questions have been raised:

  • RQ1: What methodologies have been proposed to evaluate the impact of e-learning?

  • RQ2: What methodologies quantify impact considering return on investment, expectations and welfare?

The answers to these questions mark the starting point of the research and, in a certain way, conditions the following stages in our research. The queries made in the databases have been:

  • Q1: e-learning OR elearning OR blearning OR b-learning OR technology -enhanced AND ((Effectiveness, impact, evaluation) OR (metrics, “return on investment”, “return of expectation”, “return of welfare”))

In the next phase the criteria for determining the inclusion or exclusion of the work resulting from the consultations have been established. These criteria are based on the type of database, date of publication and area of knowledge. The application of these restrictions allows us to select the most interesting works for our study.

The databases handled in the review of systematic literature have been: Scopus, Web of Science (WoS) and Google Scholar. Due to the characteristics of the research, it is also interesting to consult other sources, considered of gray literature. The gray literature [3], also called unconventional literature, semi-published literature or invisible literature, it is any type of document that is not disseminated through ordinary commercial publication channels, and therefore poses problems of access.

The period of time that has been considered relevant covers from 1996 to 2016. This restriction is based on the fact that the first concepts of e-learning appeared in early 1996. As for the areas considered to restrict the search have been: education, engineering and social sciences. Although the concept of e-learning is used in many and varied areas, in which no metrics or indicators are mentioned, it is only discussed and described its use, for that reason, it is of no interest for our study. Therefore, the work will focus on the determination of relevant variables that make feasible the faithful representation of the factors included in a model that seeks to measure ROI, ROE and welfare, in order to construct an instrument to measure variables considered appropriate and define standards for testing and measurement.

Once all the documents of interest have been collected and in this phase of the investigation, we have been able to study in detail some of the relevant variables of the process.

As mentioned, the available models are diverse, so it is necessary to identify which are the most suitable. For this study several of them have been reviewed and the work is focused on two that are estimated to be oriented to assess the human and financial part (Duart and Kirkpatrick’s models) [2, 12] from which relevant indicators will be extracted. From these two models four perspectives are observed: Improvement of affective aspects, quality and efficiency of the teaching-learning process, transfer of knowledge and infrastructure [5].

Within these perspectives, influence factors have been defined for each model. In future work, the indicators will be defined in a specific way and how their measures will be carried out.

In order to determine the influencing factors of online learning ecosystems, the SROI (Social Return on Investment) impact chain concepts are applied. This concept has great importance since the value of what is created day by day goes far beyond what can be measured in monetary terms, this is, in most cases, the only type of value that is measured and quantifies. Therefore, many decisions that are made may not be as adequate, because they are based on incomplete information, regarding their true impact.

Around SROI there is a framework for measuring and quantifying a broader concept of value, an account of how change is generated by measuring social, environmental and economic outcomes using monetary terms to represent results. SROI deals with value rather than money; as a common unit [15].

A SROI analysis can take different forms. You can group the social value generated by the whole organization, or just focus on a specific aspect of your work.

As can be seen in Fig. 1, the chain of impact creation involves a set of concepts that start from the definition of the resources that contribute the ecosystems in the process of teaching - learning, definition of new activities that are carried out with contribution of learning ecosystems, to determine the measurable results and then to determine the quantifiable changes produced, this will allow to define results attributable directly to the ecosystems.

Fig. 1.
figure 1

Chain of impact creation.

Full size image

Within the idea of impact is taken into account four concepts:

  • Displacement: what percentage of the change has shifted other changes.

  • Dead Weight: reflects if the changes could have been achieved without the use of the platform.

  • Attribution: is the percentage of changes that is not attributable to the management of the organization.

  • Decrements: is the deterioration of a change over time.

  • All this will be applied to carry out the calculation of the SROI. At this stage all changes will be added and any negative impacts will be subtracted.

In addition to ROE, ROI and SROI, heat maps [1] have been used to visually define common and important indicators of the two models. The objective is to continue incorporating other models of interest to the heat map so that it reflects in an intuitive way information from different perspectives. The map thus generated will provide a global and immediate vision.

The reason for using these two-dimensional graphical representations of data is the ease of symbolizing the values of a variable with colors, its intuitive nature of the color scale in relation to temperature facilitates its comprehension. From the experience, by A. Bojko and according to the same nature the intense color is perceived as hotter than the medium intensity, and the medium intensity as hotter than the less intensity. So it is not difficult to understand that the amount of heat is proportional to the level of the represented variable. In addition the heat maps show the data directly on the variable to which they are representing. According to Bojko: “Because the data could not be closer to the elements to which they belong, it takes little mental effort to read a map of heat”.

The reason for using these maps is to evaluate the influence factors of the two models chosen, from each model have been considered common and relevant characteristics. The criterion applied in the development of the preliminary phase to the heat maps is shown in Table 1. As can be seen, each perspective is assigned a color and each influence factor an X, if the influence factor has been considered in the model and the cell is left blank if this factor has not been considered in the model. From the values assigned, the above table is obtained, with the analysis of the presence or not of the influence factors in the two models.

Table 1. Heat map of influence factors of the models studied. Affective and quality perspectives of the teaching - learning process.
Full size table

A third map is generated with the crossing of the two previous ones where it is possible to visualize more clearly, what are the influence factors that the two models have not approached. The areas painted with lower tonalities such as expectations and the affective perspective of both organization and teachers and innovation in the quality perspective are examples of what is being affirmed. From the third heat map of Figs. 2 and 3, important information has been obtained to measure the impact of ecosystems applied to higher education, through the collection of existing information and analysis of the importance and influence of each factor On the agents involved. The heat map shows this information graphically and intuitively and serves as a basis for the following analyzes.

Fig. 2.
figure 2

Heat map of influence factors of the models studied. Perspectives transfer of knowledge and infrastructure

Full size image

As can be seen in Fig. 2, Kirkpatrick’s model works in the four perspectives, specifically focused on students, not so with the other actors (organization and teachers). Neither model is oriented to measure affective aspects so factors such as motivation and interest, critical behavior, attitudes and satisfaction, and fundamentally academic prestige will be of particular interest in the study.

3 Results and Future Works

Regarding the transfer of knowledge, as can be seen in Fig. 3, the two models approach the subject in a superficial way, with respect to this perspective is a set of aspects that can be of influence and that has not been taken in counts as the interactivity between the user and the way in which the innovation of each one of the actors can be valued. On the infrastructure, in the two models there are no aspects that allow to evaluate the services that the actors receive and the availability, through the heat maps it is possible to establish the influence factors that are going to be considered as the basis for the proposal of a new model.

Fig. 3.
figure 3

Heat map of influence factors of the models studied. Perspectives transfer of knowledge and infrastructure

Full size image

Having analyzed perspectives and factors of influence between the two models, such as: improvement of aspects of affectivity, quality and efficiency in the teaching-learning processes, knowledge transfer and infrastructure, it is concluded that there are some influence factors that can be approached in greater depth and from these to define indicators that allow to estimate the real impact of online learning ecosystems within the higher education environment.

Heat maps are a technique that allows determining the factors that must be worked more strongly and that have not yet been taken into account by currently recognized models, this will allow to consolidate a robust and valid model to measure the impact of ecosystems Technological advances in higher education and the perspectives and factors with which they should be considered.

For now, two models have been chosen to perform this test, but other models will be analyzed with the same technique and the next phase will be defined where the indicators and the methodology that each will apply for their measurement will be defined.

Therefore, the model that will be implemented later, the accepted and accepted perspectives, the students and the organization as welfare and the return to expectations, and these results will be like the values for the return of the investment. In addition, this map shows that universities should improve the tools that students provide to facilitate the knowledge transfer process with simple infrastructure, testing the support and quality services that are available to all users.