Development of a Seminar Management System: Evaluation of Support Functions for Improvement of Presentation Skills

Abstract

We propose that seminar activities should be a focus of university education in the future. Under this principle, we have been developing a seminar management system and a learning management system for traditional classroom instruction and e-learning environments for university education. The main point of seminar activities is not only acquiring knowledge and skills, but also the ability to mutually assess growth with appropriate instructor support and foster student self-learning. Recognizing the importance of seminar activities in university education, we discuss seminar activities that should be the center of university education five to ten years from now to consider policies and support methods now.

1 Introduction

Since the early 2000s, different methods for providing universal access via distance education, such as the OpenCourseWare program and massively open online courses, have rapidly gained prominence, and universities worldwide have been pressed to change with the times. Although it seems that in the near future most lecture-type classes will likely be offered through distance education to off-campus locations, discussion- and participatory-type lessons continue to be performed mainly at university campuses, and even at Japanese universities still require in-person attendance.

The transformation of Japanese universities advances along with the dynamics of Japanese society. The significance of the existence of the university is about to change. As an example of this, corporate society requests that universities cultivate the Fundamental Competencies for Working Persons, which consist of three competencies (Action, Thinking, Teamwork) and 12 capacity elements (Ministry of Economy, Trade and Industry 2006). Face-to-face communication is essential in nurturing these abilities. From the above, expert knowledge education will shift to distance education, and the center of competency education will shift to university face-to-face education respectively.

Professor Keizo Nagaoka of Waseda University suggests that the core role of competency education in face-to-face education is “seminar activities” (Nagaoka and Kometani 2016). There are some peculiarities in Japanese seminar activities. In most Japanese universities, seminars are institutionalized in the curriculum. Seminars and laboratory classes taken by undergraduates at science and engineering universities emphasize training similar to that of traditional craftsmen, in which tacit knowledge, rather than intellectual knowledge, is emphasized.

Such family-like seminar activities have worked well for human-resource training of engineers from science and engineering universities and departments, especially during Japan’s period of high economic growth from the 1960s to the 1970s (McGuire 1996). They were perhaps consistent with the goal of fostering capable talent who could work in an industrialized society. However, a more rational sophistication that is compatible with the mature post-industrialization society of Japan and the present knowledge society is necessary.

Learning management systems (LMS) that facilitate the operation and management of lectures have been introduced in 74.8% of national universities in Japan (Ministry of Education, Culture, Sports, Science and Technology 2013). If we consider seminar activities as the center of university education, a similar seminar management system (SMS) is necessary. We are currently developing a SMS and using it in actual situations. In this paper, we report on the development state. First, we organize and structure seminar activities and introduce learning management functions for competency education in seminar activities. We then evaluate the developed SMS’s effectiveness at developing presentation skills, a required competency, through usage results and a questionnaire survey of the perspectives of seminar students using the developed SMS.

2 Structure of Seminar Activity

2.1 The Process of Skill Improvement

As a measure of competency training, it is effective to repeat instruction until students become accustomed to the process of goal setting, behavior, evaluation, and reviewing. Instructors are required to reflect on seminars and form educational philosophies and facilitation methods that build a better learning community. In this research, we applied triple-loop learning (e.g., Romme and Van Witteloostuijn 1999) to seminar activities for organizational learning (Fig. 1), and we developed supporting functions on the premise of this learning process. Mouri (2007) states that instructor encouragement to enhance the group’s positive entrainment in seminar activities is indispensable, and Fushikida et al. (2014) stated that instructor encouragement led to a sense of growth in generic skills (competencies) and was effective toward student satisfaction with seminar activities. The “community improvement” process shown in Fig. 1 is thus important for effective learning in seminar activities.

Fig. 1.

Improvement cycle of seminar activities

2.2 The Curriculum of Standard “Seminar Activity”

The expertise to be acquired varies among seminars, so it is difficult to realize faculty development through seminar activities (Mouri 2007). In contrast, the goals of activities and their required communication skills are common to many seminars, so support is possible. We therefore categorized representative seminar activities (Table 1). The classifications are based on the seminar activity studies of Mouri (2007) and Fushikida et al. (2014). Both affirmed the benefit of incorporating group activities, and additionally Fushikida et al. (2014) suggests the effectiveness of incorporating situational aspects, such as job hunting. Mouri (2007) affirmed the technical reading conventionally implemented in seminar activities as a means of knowledge composition through communication.

Table 1. Classification of seminar activities based on educational goals (competencies) and activity goals

To incorporate the merits of science and engineering seminars pointed out by McGuire (1996), in this research document production (text communication) is positioned as a competency, and “developing student-initiated research” as an activity goal. We have already developed functions aimed at supporting document production in seminar activities, and our SMS has been expanded to integrate the results (Kometani and Nagaoka 2015, 2016).

3 Functions for Skill Improvement in Seminar Activities

3.1 System Targets and Difficulties During Action Improvement

In previous research, many methodologies have been used to improve students’ presentation skills. However, this has not yielded sufficient research for supporting students through seminar activities on a daily basis. Therefore, our system used the action improvement cycle in Fig. 1 to specifically target support for daily improvement.

To improve presentation skills, students must objectively know their own presentation behavior, and change this behavior based on the specific needs of the presentation. Without support, it is difficult for novice presenters to improve their reflections and planning skills in the action improvement process.

3.2 Functions

To address these difficulties, we developed presentation summary functions. Figures 2, 3, 4, and 5 show the system’s user interface (UI).

Fig. 2.

Presentation summary UI (elements i, ii, iv, v)

Fig. 3.

Evaluation and comment UI (element iii)

Fig. 4.

Overall comment function (Popup UI; element vi)

Fig. 5.

Two presentations being compared for reflection (element vii)

The UI consists of 6 parts:

• i. Presentation slide-sharing function

• ii. Presentation video-sharing function

• iii. Presentation evaluation and comments function

• iv. Self-evaluation, peer evaluation, and instructor evaluation overlay radar-chart

• v. Real-time comments function

• vi. General comment-sharing function

• vii. Presentation comparison function

Figure 2 shows the UI for learning outcomes in the developed SMS (Nagaoka and Kometani 2016). Figure 2 shows the presentation “mode”, which can be combined with presentation files, videos, and colored radar charts for self and peer assessments, along with peer comments. It is designed for feedback regarding the activities listed in Table 1 by associating multiple students. In the case of a speech, the presentation file is hidden. During discussions, 360° video can be shown, and then, and hidden during document production.

4 Practice

4.1 Methodology

The students are third-year undergraduate students (7 males and 3 females), and are novice presenters enrolled in a seminar taught by the authors. Each student gave five presentations throughout the period, during 120-minute seminar classes that introduce previous research (Table 1). The feedback methods varied. Paper assessment sheets were used in the first presentations. In the second and third presentations, students used a prototype SMS that provides raw assessment data. In the fourth and fifth presentations, the functions described above were used. Before using the proposed functions, the students assessed a senior student’s presentation as an exercise. The data was obtained in 2015.

In each seminar activity, two to four students made presentations of about ten to fifteen minutes per person, and received peer assessments using the assessment items in Table 2. The assessment scores were 3 (“excellent”), 2 (“good”), or 1 (“not good”).

Table 2. List of assessment items

4.2 Change in Peer Assessments

As Table 3 shows, mean peer assessments improved after each presentation. Improvements between the third and fourth presentations are particularly notable. This may be a result of assessing presentations by fourth-year students following introduction of the new system, through which the novice presenters gained various insights.

Table 3. Changes in peer assessment scores

5 Questionnaire Survey

5.1 Overview

We developed these functions and implemented them over the 2015 and 2016 school years. The presentation theme was a literature survey for third-year students enrolled in the seminar. A total of 18 students (8 in 2015, 10 in 2016) answered the questionnaire. The question items inquired into the following:

• (1) Usefulness of the functions for preparation

• (2) Usefulness for increasing awareness of areas for improvement

• (3) Usefulness for deciding which presentation behaviors must be changed.

Each question was evaluated on a 6-point Likert scale, along with a free description section for describing the reasons for the evaluation or situation where students could use the functions effectively to improve their presentation skills. Function (vii) is planned to be offered in future development, so this is excluded from the evaluations considered in this paper.

5.2 Results

5.2.1 Usefulness of Functions for Preparation

Figure 6 shows student responses regarding usefulness of the developed system for preparation of presentations. Many students responded that the functions were useful. Figure 7 shows which functions were reported to be useful for preparation. Presentation slides were most useful for students. Student comments stated that “seeing the good points of other students’ slides were useful when making my own slides”, that “good slides provide good examples”, and that “beautiful slides made by more experienced students are useful”. During preparation, presentation slides with assessment data are effective.

Fig. 6.

Usefulness for preparation

Fig. 7.

Useful functions for preparation

5.2.2 Effects on Awareness

Figure 8 shows the usefulness for awareness. Many students reported that the functions were useful when reviewing their presentations. Figure 9 shows which functions were useful for awareness. The presentation video and radar chart functions were especially useful, with students making comments such as “I can check results of the evaluation, comments, and the presentation video, so I can clearly determine where I need to improve”, and “I can check my own presentation objectively”. The presentation video with assessment data and comments were effective for improving awareness.

Fig. 8.

Usefulness for awareness

Fig. 9.

Useful functions for awareness

5.2.3 Effect for Behavioral Change

Figure 10 shows the usefulness for preparation. Many students reported that the functions were useful for effecting behavioral changes in their presentation. Figure 11 shows which functions were useful for behavioral change. As in the results for awareness, the presentation video was most useful and the presentation slides, radar chart, and overall comments were useful as supplementary resources. Students made comments such as “I can imitate the good behavior of other students”, and “I can objectively observe the volume of my voice and my body movement, so I can check if my behaviors are improved”.

Fig. 10.

Usefulness for behavioral change

Fig. 11.

Useful functions for behavioral change

5.2.4 Overall Evaluation

Figure 12 shows overall student evaluations of the functions. All students reported that the functions were useful overall, indicating that there is a need for these functions. Therefore, the functions can be used in daily seminar activities.

Fig. 12.

Overall evaluation of usefulness for presentation skill improvement

6 Conclusion

The results of the peer assessment described in Sect. 4 and the questionnaire survey revealed that students have positive opinions of the presentation skill improvement functions of the developed SMS, showing that its functions can be introduced in actual situations. To increase the convenience of its functions, we intend to develop a data analysis and feedback method in future studies. Furthermore, functions for improvement of overall communication skills will be developed.

Undergraduate students who are new to the seminar community have self-confidence in their knowledge, because of their success at university entrance examinations. However, the entrance examination experience causes them to become rote learners who simply accept knowledge delivered by instructors. They are too afraid of making mistakes and failing. Such attitudes are not suitable for the future Japanese society, and should be changed to address the global society of the twenty-first century.

We believe that such changes in university education can be realized through seminar activities including the functions described above.