1 Introduction

1.1 The Partners and Main Objectives of the Project

Ten partners make up the Consortium of the Project. La Salle Campus Barcelona (FUNITEC-Ramon Llull University, Spain), has the role of coordinating partner and there are three other main partners: FIDAE (Federazione istituti di attività educative) and ISP (Intesa SanPaolo Fromazione Scpa) in Italy, and ASSEDIL (Association Europeenne des Directeurs d’Institutions Lasalliennes) in France, plus the six other associate partners that are the vocational training centers in which the Project will be developed: 2 in Spain (La Salle Palma School, and Salesians de Sarrià), 2 in Italy (Instituto Cavanis and Suore Salesiane dei Sacri Couri) and 2 in France (Ensemble Solaire Jeanne d’Arc and Groupe Scolaire Saint Joseph La Salle).

The fundamental objectives of the Project are to:

  • Extend the use of practical, motivating, effective and international learning methodologies in the vocational training sectors in Spain, France and Italy,

  • Increase the success rate of the students through providing experiences which they find highly-motivating and which also require a certain degree of individual commitment,

  • Create a bank of educational program based in immersive worlds, projects and roles,

  • Foment international co-operation through joint projects carried out by heterogeneous work groups made up of members from different social and cultural backgrounds,

  • Pinpoint the degree to which different learning innovations can be applied to a group of students (role, online, international, etc.),

  • Verify whether these new methodologies manage to meet the objectives set out and compare these results with those of traditional learning techniques used in Vocational Training centers.

These objectives are set out in the framework of the European Union (EU) strategic priorities 2014–2020 in reference to: “The development of basic and transversal skills, the development of adapted evaluation systems, an increased and more intensive use of ICTs, more cohesion between the different evaluation systems and the promotion of the transfer of learning strategies and methodologies among the countries of the EU”.

1.2 Project Phases

The Project is divided into five stages which are in turn organized into activities:

  • Preparation: The activity program is reviewed by the main partners.

  • Content Generation: The main partners have the responsibility to design the teaching materials for the courses, a guide to the SCC teaching methodology and a guide on the evaluation methods used in the learning and professional insertion process.

  • Implantation: The teachers of the schools are trained in the method to implement both traditional and SCC methodologies in two courses: A pilot program with the Search Engine Optimization (SEO) course is carried out to compare traditional methodologies with the SCC; and the results of the learning are analyzed and the second year students are taught the Mobile Commerce (MOBCOM) course using SCC methodology in all the centers.

  • Tracking: Collection of data from students’ performance and employability.

  • Results analysis and breakdown at the end of each of these courses.

The methodology for the development of the project is based on a two-phase spiral model. Initially the learning methodologies are implemented in the traditional form in a school in each of the three countries and then the same training is implemented in a different school using the SCC methodology. It is hoped that the impact and results of the project will strengthen pedagogical capabilities in the vocational training sector in the following ways:

  • Reduction in school absence and increase motivation and commitment from the student,

  • Increased satisfaction level of the students,

  • Creation of the need to learn, from their roles with the immersive world proposed,

  • Creation of a certain degree of rivalry in the completion of ubiquitous tasks thanks to the use of various ICT systems,

  • Boost knowledge of students’ mother tongue and the establishment of a lingua franca to carry out international tasks,

  • Train the students and those already working and ensure that they are fully prepared to carry out the work expected of an officially qualified technician.

1.3 SCC Assessment

The experimental model suggested by the L4W project is based on a new SCC approach and on the training programs development methods. SCC is a methodology inherently based on objectives. Its objectives are the same ones as those pursued by students when undertaking an education or training course and actively attending a particular training program. Its objectives should coincide with those same objectives set by students when thinking about their job and career aspirations. SCC is built on activities: i.e. activities concerning the adoption of the SCC methodology must be related to the long-term goals set by students in view of their specific roles to be played in real life. A SCC should starts with the definition of what the scenario is or will be. Subsequently, within the given scenario, the training provider shall decide, on the basic elements to be developed, the specific conditions and “parallel” or “side” elements to be implemented to enhance students’ ability to play their role in their future professional career. SCC works in any complex learning environment, as long as mentors are available and willing to coach students in their learning process and future realistic roles to play. A meaningful experimental environment of such a scenario must therefore be put in place. This environment can be built on the web or at school. In both cases, teamwork and mentoring, as well as the subsequent evaluation of products resulting from these specific activities, are the key pivot around which the teaching methodology revolves. It is therefore fundamental to start from the teachers’ training, in the first place, and only later provide training to students themselves.

In order to fulfill our objectives, we need to develop a new approach that, on the one hand, takes into account the evaluation questions related to the project success, effectiveness and impact generated by the latter on the target audience; and on the other hand, it addresses the issue of evaluation of learning and tools useful for the transparency and validation of skills acquired by students through an innovative methodology aiming at developing not only vertical professional skills but also transversal soft skills and key skills.

2 Related Work

The SCC approach directly recalls the acquisition and validation dynamics of skills (such as life skills, key skills, and citizenship skills), which become a central element in the evaluation processes [1]. The issue of skills has progressively come to the limelight on the international scene, calling the attention of vocational training and education scholars, production organizations and policy makers [2]. As repeatedly stressed, being skilled implies in itself a social dimension, since someone can be recognized as competent, without necessarily acquiring skills though academic qualifications. At the same time, it implies an effective action dimension focusing on the flexibility and adaptability of competences as such on different levels, across the board.

At school, competences have been at the center of the debate for years; yet, the issue of how to assess them is underestimated, since the complexity of all the underlying issues is not directly taken into account, also considering the fact that some of these skills are prescriptive. Nowadays, we can find different references into the literature developing the “edumetrics” concept [3, 4], which has deepened the theme of competence assessment, by highlighting its specificity and discussing the translatability of psychometric criteria generally used in testing in the domain of education.

It is necessary to strengthen curricular (school) programming based on core competences, in order to facilitate the early acquisition of active citizenship competences at school. This would be desirable also in view of a wider dissemination of inquiry learning methodologies addressed to the most disadvantaged recipients who need to reposition of their skills in evidence based perception and self-assessment processes and more “manageable” cognitive processes. In this sense, intensifying the use of systems promoting the mutual recognition of European qualifications for employment, improving matching between job descriptions and skill profiles of diplomas and qualifications (European Skills/Competences, Qualifications and Occupations), can and must be regarded as a process involving all the training processes that are expressed in terms of learning outcomes. In addition, a lively debate has emerged on the link between instrument referencing systems, such as EQF (European Qualification Framework), with the national qualifications frameworks, NQF (National Qualification Framework), starting from the enhancement of already existing transparency and traceability tools, in particular, ECVET (European Credit system for Vocational Education and Training).

2.1 What to Assess and Why

Assessing training is always a difficult task, since it is closely connected to the context where (formal, non-formal and informal) training to be assessed has taken place and to the type of assessment approach that has been used. Generally, in the literature on training evaluation, two major theoretical approaches can be found: evaluation training and effectiveness training. The former is based on the evaluation of learning outcomes achieved at the end of training, in other words is based on the effectiveness of training that was provided. In the former approach, objectives, content and design of training become the object of evaluation; in the latter approach, however, the training process is examined in all its stages (pre, ongoing and post) considering the variables that might have influenced the effectiveness of training activities. Assessment supports and fosters the quality development of an education and training system because it:

  • Identifies the strengths and weaknesses of an education and training system and action,

  • Observes and analyses how resources are used,

  • Involves and empowers the stakeholders engaged in the training system and actions,

  • Ensures that a change has indeed occurred with effects on the institutional and social context,

  • Allows to identify critical issues in a primary phase using Pre and Profile tests, and using mixed methods (combining quantitative and qualitative approaches) for a better interpretation of the results [5].

When we try to incorporate new educational methods using different technologies, we need to incorporate them into teaching in a controlled manner; there are some risks that need to be controlled before one can improve not only the curriculum but also student skills and knowledge. With technology, the professor must be trained and capable of providing full-time support to students: he or she must be capable of offering a good and precise explanation of the practice and methodology, must correctly select the applications, and must provide clear final objectives. Previous studies describe “critical mistakes” in the implementation of educational technology - mistakes that can generate negative perceptions among the students and which need to be avoided [6–10]. The need of and justification for incorporating IT into the educational process are particularly relevant, and they are described in the main roles of the European Higher Education Area (EHEA), which runs the university studies of member countries, including Spain, where this project was undertaken [11].

2.2 Mixed-Methods Assessment for Pedagogical Purposes

Quantitative and qualitative approaches have historically been the main methods of scientific research. Currently, a hybrid approach to experimental methodology has emerged that takes a more holistic view of methodological problems: the mixed-methods research approach. This model is based on a pragmatic paradigm that contemplates the possibility of combining quantitative and qualitative methods to achieve complementary results.

The value of research lays not so much in the epistemology of the method but in its effectiveness [12]. On the one hand, quantitative research focuses on analyzing the degree of association between quantified variables, as promulgated by logical positivism; therefore, this method requires induction to understand the results of the investigation. Because this paradigm considers that phenomena can be reduced to empirical indicators that represent reality, quantitative methods are considered objective [13, 14]. On the other hand, qualitative research focuses on detecting and processing intentions. Unlike quantitative methods, qualitative methods require deduction to interpret results. The qualitative approach is subjective, as it is assumed that reality is multifaceted and not reducible to a universal indicator [15].

The current methods in UX do not necessarily include the end user to participate in the creative process of the product. Most of them are guides of imagination exercises to be more emphatic with the user in concrete scenarios as cognitive walkthroughs [16], or user persons [17]. On the other hand, there are also qualitative methods far from usability standards which allow obtaining subjective information from users themselves, such as contextual design [18], or diary methods [19].

To provide a quality management, which is likely to attain objectives and to meet users’ needs, it is necessary to rely on timely information on the efficiency and effectiveness of the training schedule. The main need that arises from a training project is to:

  • Check the internal consistency of the programming procedures that are implemented and to describe the gap between expectations, processes and outcomes, resulting from the procedures in use,

  • Describe the effectiveness of innovation processes that are implemented in terms of: enhancement of knowledge, expertise, skills, activities of each individual, change in attitudes and behaviors of individuals and organizations, impact of innovation on the professional, social and institutional context, and identify the transferability elements emerging from the innovation process, in order to translate them into educational policy choices.

3 Pre-test in SCC Applied to SEO and MOBCOM Courses

The SCC approach introduces an innovation element in the educational process, where it supposes an adjustment of content and models of learning units to the expected use of these tools by students in the workplace and in professional contexts. The implementation of the evaluation model is subdivided into three phases:

  • First phase: research activities aimed at identifying the SCC approach centered variables. This phase aims to define the assumptions and provides for the definition and analysis of the critical variables measured; the identification and determination of strategic processes on which the assessment process is to be based; and the data collection tool development.

  • In the second phase, tools will be administered and data will be collected within the sample made up of the teachers and learners community and in the control group (population not covered by the action).

  • In the third phase, the processing of results will take place to answer the evaluative questions posed during the model development, by means of a comparison between the built model (initial research hypothesis) and the results obtained in terms of effectiveness and impact. The final phase is intended to predict any changes in the model, compared to the preliminary setup of the training model.

The methodological approach that has been adopted aims to give an account, on the one hand, of the complexity of the research field, in terms of enlargement of training systems (formal, non-formal, informal); SCC approach consideration; key process analysis in terms of skills development and performance of the extreme variability of training behaviors by organizations. In the process of design the test, we need to identify and choice the indicators to be obtained in order to validate our methodology. The indicators can be metaphorically understood as signals, “arrows” that specify, clarify and describe the characteristics or properties of training to differentiate, to take evolution under control, to observe the direct and indirect effects caused by the project on individuals and their related institutions.

To formulate effectiveness and impact indicators and to identify areas subject to assessment it is necessary to remind, explicitly, and to explore the main features that characterize the SCC method and its building procedures, such as the development of learning units. For example, some aspects to be reviewed are:

  • The meaning of training unit of learning,

  • The reasons why the SCC approach is chosen,

  • The implications on teachers’ work,

  • The SCC/modulation relationship for students’ learning, etc.

3.1 Courses Assessment

The following Table 1 shows the basic scheme of the courses assessment:

Table 1. Course assessment. (1) For traditional and SCC groups. (2) Only to evaluate in the SCC group. (3) For local classes and international groups: one school of each country will develop the MOBCOM course in an international model, in collaboration with the selected schools of the other countries.
Full size table

Preliminary information on the students profile and their initial motivation are basic information in order to develop the methodology proposed. With the information extracted from these surveys we can detect differences across countries and educational institutions and the motivation of the students of the schools involved in the project.

3.2 Designing the Pre-test

With all the data collected we can adapt the method in function of the characteristics of the students, their needs, or for example technology difficulties of the students. The pre-test was designed to ask students about the technologies they are familiar with, possess or use (Table 2), and their motivation in front of the use of SCC methodology (Table 3). This information provides us with the level of advanced preparation using interconnected systems through different devices such as computers, mobile phones, tablets, etc. A classical mistake is assuming the presence of knowledge, use or possession of technologies required to complete a project; when this assumption is later proved wrong, the experiment fails due to the design errors in the implementation and development processes.

Table 2. Pre-test 1: Technological Student Profile. Questions 1 and 2: 5-Daily, 4-Occasionally, 3-Only at school, 2-Rarely, 1-Never. Questions 4,5,7 and 8: A-Very much, B-Somewhat, C-Slight, D-Note at all.
Full size table
Table 3. Pre-test 2: Student Motivation using SCC Methodology. Questions 1 and 2: 5-Daily, 4-Occasionally, 3-Only at school, 2-Rarely, 1-Never. Questions 4,5,7 and 8: A-Very much, B-Somewhat, C-Slight, D-Note at all.
Full size table

4 Conclusions

The paper presents the design of a specific pre-test for testing the student profile and motivation in a course where we adapt the SCC methodology. Scenario Centered Curriculum is being applied to promote the acquisition and development of international cooperation skills through the use of Information Communication Technologies (ICT) tools and systems. The L4W project is based on the development of learning strategies within the framework of Vocation Training, in order to improve learning processes and make them more applicable in the real working world while minimizing the number of student drop-outs. Learning methods are re-conceptualized through the use of immersive worlds and role and project-orientated-learning. As we have demonstrated the process of designing the assessment surveys it is critical in order to obtain the complete feedback of the student. The collected information allows us to evaluate the impact of the new educational methods proposed and the need of change any educational exercise or strategy, something that it is very easy to find according with the fact of work in three different educational sectors as are the Spanish, French and Italian schools. At the moment of the publication of this paper (2016, February), we are collecting the data and beginning the analysis of the Pre-Test data, following the schedule of the project.