Introduction

Our research and development of an Intelligent Collaborative Learning System (ICLS), a software framework in a network environment, included a Group Leader (GL) Tutor component which provided an important platform for researchers exploring collaborative learning. In this paper, we present (a) “a rearview mirror” to explain the foundation for the system, (b) a “universal rearview mirror” to describe the significant contributions of the system, (c) a “wide angle rearview mirror” to show how the system influenced subsequent research, (d) “blind spots in the mirror” to illustrate what could now be perceived as shortcomings of the system, (e) “flipping the rearview mirror” to speculate how the ICLS or an enhanced version could be utilized in current and future computing environments, and (f) “magnification mirrors” to challenge researchers to continue to expand and enhance our work.

Using a Rearview Mirror

Our research was conducted in the late 1980’s and early 1990’s (McManus 1995; McManus and Aiken 1993, 1995, 1996). In the 1980s to 1990s, Artificial Intelligence, the new area of Groupware, and the rediscovered pedagogy of Collaborative Learning provided the foundation for wide-spread research in learning theories. Computer systems were text based and networked labs were new tools which allowed for collaboration. The World Wide Web and mobile devices were in the early stages of development and implementation. Computer and networking specifications severely limited the scope of designing and implementing our model. The system was confined to specific labs because personal computers were not affordable nor did network speed allow for distance collaboration.

Intelligent Tutoring Systems (ITS), specialized Artificial Intelligence (AI) systems, assisted the student in learning specific content. An ITS contained four essential components for single users in a stand-alone environment (Wenger 1987):

  • domain expertise

  • pedagogical expertise

  • a student model

  • a user interface

Computer-supported collaborative work (CSCW), originating with the name “Groupware,” supported multiple users working together on the same projects (Bannon and Schmidt 1991). CSCW systems were constructed of a shared database, real-time communications, and a common application program (Wexelblatt 1992). During the 1970s through 1990s, educators studied and utilized collaborative learning (CL), rather than a competitive approach. Collaborative learning, an effective pedagogy for students to work in groups on a project, benefited students as they interacted with and helped other students to learn. Students benefited most when they received help from peer students and when they gave help to others (Webb 1989). Additionally, collaborative learning helped students with their self-esteem and their interest in school. (Aronson et al. 1978). Collaborative learning was built on collaborative skills, including positive interdependence, face-to-face promotive action, individual accountability / responsibility, and group processing (Johnson and Johnson 1991). The interaction of CSCW and CL had recently led to the rise of Computer-Supported Collaborative Learning (CSCL) (Resta and Laferriere 2007). Systems such as CSILE, the Computer Supported Intentional Learning Environment, (Scardamalia and Bereiter 1991), enabled students to work and comment on each other’s work. Local area networks (LANs) were used for sharing messages and files between and among users. Accessing a shared database, stored on the Internet through a web interface was not yet feasible.

While many ITS taught content knowledge, systems to develop collaborative skills were not available. Therefore, the motivation for our research was to develop an ITS, specifically the Intelligent Collaborative Learning System (ICLS), to achieve this goal. Our preliminary work addressed the Group Leader component, which monitored computer-based collaborative problem solving. A shared multi-user extended relational database system (FoxPro 2.5 for DOS) in a client–server (Ethernet Local Area Network) provided the software platform for implementing and testing the ICLS with several groups of users (students). This environment challenged the technical agility and ease of use of the ICLS. In order to implement the multi-user aspect of students working together on a project, database records had to be locked to avoid concurrent writing to the same record. Our approach to synchronous communications (chat) was achieved by sending messages in files on the LAN; opening and closing, reading and writing these files yielded slow response time, a major drawback to the system. Other limitations, such as small memory capacity and slow processor speed also hampered the ICLS’ efficiency. Likewise, the use of procedural programming languages with their tasks and subroutines did not provide the key features of object-oriented program languages, such as data abstraction and object inheritance, useful for manipulating data.

Despite the technological challenges, the key drivers for our approach included not only utilizing, but also interweaving, the components of an ITS and a CSCL system in a GL Tutor module to teach collaborative skills in a network environment. At the time, this proved to be an innovative approach to solving the problem of teaching collaborative skills, particularly utilizing the Jigsaw method (Aronson et al. 1978). In the Jigsaw method, students were first arranged into expert groups with each group solving one problem or discrete aspect of a large project. Students discussed their work as they solved the problem, resulting in a short report or solution. The students were then rearranged into home groups, with one member from each expert group. The home groups tackled the large project, with each member contributing their expertise. It was during the expert and home discussions that the students utilized and enhanced their collaborative skills, monitored and guided by the GL Tutor.

Universal Rearview Mirror

The GL Tutor, a software process and the key contribution of our research, monitored and taught the effective use of collaborative skills. Soller et al. (2005) state that “The success of McManus and Aiken’s Group Leader began a proliferation of systems that take a finite state machine approach to modeling and advising collaborative learners” (p. 18). The significant features of the GL Tutor were to encourage students to work together, sharing expertise and communicating effectively. The GL Tutor achieved its goals of fostering a collaborative environment, monitoring the students’ participation in discussions, and promoting collaborative skills through scripts and a Collaborative Skills Network. The scripts provided the algorithms for the GL Tutor to manage the Jigsaw collaborative learning method. The goals were to:

  • enable the students to communicate (text) with each other through the network

  • foster the use of CL skills

  • allow students to assess their cooperation

Further details regarding the Jigsaw learning method and our use of it can be found in (Aronson et al. 1978 and McManus and Aiken 1995).

The Collaborative Skills Network (excerpt provided in Fig. 1) represented the skills, their subskills and associated sentence openers. Each sentence opener in a discussion was related to a particular CL skill (Communication, Trust, Leadership and Creative Conflict) or subskill (Openness for the Communication skill) as defined by Johnson and Johnson (1991). The students wrote their own sentence closer to complete their thought. A student’s use of the sentence opener “I think” indicated the openness attribute of the sending subskill of the communication skill. For example, “I think that the counter needs to be set to zero.” This enabled the GL Tutor to interpret what students were doing and what collaborative skills they were using. The GL Tutor accomplished this through tracing the discussion and the use of the collaborative skills through a finite state machine, comparing the used skill to the optimal skills defined in the network. The GL Tutor assessed the student’s mastery of the skill by comparing the number of times the skill was successfully used to the number of times it was missed. The GL Tutor generated tutoring feedback according to response types and levels of usage, according to previously generated feedback stored in the historical database (McManus 1995).

Fig. 1
figure 1

Collaborative skills network (Excerpt from McManus 1995)

Full size image

Two experiments were conducted in which undergraduate students used the ICLS to write a statistics program in the C programming language. The hypotheses were that students would:

  • enjoy using the system

  • find it useful for working in groups on projects

  • strengthen their self-esteem and interest in the subject

  • improve their effective use of collaborative skills

  • increase their content knowledge

Research questions which addressed these hypotheses included:

  • How successful is the ICLS as perceived by the students as measured by system satisfaction?

  • How likely is it that students will accept such a system as indicated by their perception of its usefulness?

  • To what extent and in what ways is the ICLS used in cooperative learning activities?

  • Does the students’ academic achievement of modular programming concepts before using the ICLS change after using the ICLS?

  • Do the students’ attitudes before using the ICLS change after using the ICLS, confirming or refuting Aronson’s studies? (Aronson et al. 1978)

The group work was conducted in the Jigsaw method of both expert and home groups through the ICLS. In both the expert and home group phases, the students worked synchronously in the ICLS. The expert group work was performed prior to the home group work.

In the first experiment, the subjects were 18 students in the first computer science course at a local institution of higher education. The subjects’ task was to write algorithms for a database project in the preliminary ICLS environment; the ICLS at that time was not fully implemented. Each expert group developed algorithms for two functions and the home groups combined them to form an entire project.

In the second experiment, the subjects were nine students, also in the first computer science course at the same institution. The subjects’ task was to write algorithms for a statistical analysis project in the full ICLS environment. The expert groups’ topics included the computation of the mean, the standard deviation, the range, and the percent of scores which are within the normal range. The home group built the complete project from the expert group algorithms.

In both experiments, the students used the sentence openers to build a discussion. For example, a student said “I think that Dan’s algorithm is right and your work isn’t totally wrong” selecting “I think” from a menu of sentence openers to indicate the collaborative skill of openness. However, some students selected a sentence opener but appended a sentence closer that was not appropriate. In one instance, a student opened with “Thank you” followed by “but what about the first algorithm?” thus mixing the subskill of acknowledgement followed by a question. Another example was the case in which a student provided an off-topic closer, such as “I think you will have a great time this summer with all of your classes.” In practice discussions, the GL Tutor assessed the student’s use of the relevant collaborative skills and subskills in a debate type of discussion. In debate discussions, the students interact with the possibility of differing views, until they agree on a conclusion. The GL Tutor traced the students’ interaction through three phases in a Finite State Automata (FSA): 1) Display (“Let’s examine the issue of”), 2). Confirm or Disconfirm (“The advantages of this idea are”), and 3) Repair until convergence (“Let’s draw a conclusion that”). Based on this information, it generated suggestions of appropriate sentence openers as a means of teaching collaborative skills. Later researchers extended the use of FSAs and the Collaborative Skills Network to include more sentence openers and even sentence closers in both structured and unstructured discussions, to enhance the tutor’s support and guidance.

The annotated screen shot from a session conducted in the experiment (Fig. 2) showed the discussion screen when the current speaker Eugene (A) was talking with Christine (B). Justin, another team member, is not involved at this time. Eugene initiated a “request” type of discussion (C), resulting in the list of sentence openers, (D) e.g., “I think,” “Do you need more information?” “Here is more information” and others being displayed. Eugene selected the “I think” sentence opener (E), indicating his use of the “Trust” skill, “Task” subskill, and “Openness” attribute, a path shown in the Collaborative Skills Network. He later completed his sentence in a text box. In the menu at the bottom, Eugene has also selected “Ask the Group Leader” for suggestions (F).

Fig. 2
figure 2

Screen shot showing interaction between two students

Full size image

The results of the experiments indicated that the students were satisfied with the system and most thought that it was useful for working collaboratively. However, they were concerned with the slowness of the system which was due to the implementation of message passing through files. The technical issues may likely have affected their responses. The students’ academic achievement improved their knowledge of modular programming. However, results showed that the students did not have a significant change in their self-esteem and interest in the subject, but this may have been due to the small sample size.

Wide Angle Rearview Mirrors

The Collaborative Skills Network proved to be a key practical contribution of this research. Later researchers not only utilized and extended the Collaborative Skills Network, but also enhanced the network as well as the usage of sentence openers.

A Collaborative Learning Conversation Skill Taxonomy (Soller 2001), was based on and advanced our Collaborative Skills Network with additional sentence openers. Soller (2001) concluded that a CL model should “dynamically analyze peer-to-peer conversation and actions, identify a group’s strengths and weaknesses, and determine which methods and strategies to apply in order to best further the group learning process.” (p. 47). A Collaborative Learning model was developed in a Java platform utilizing this extended taxonomy and with an enhanced user interface containing videos of the team members. Soller concluded that more work needed to be done with not just enabling, but supporting, user interaction. Soller noted that her research was limited by its use of structured interaction and she suggested that speech recognition and machine learning techniques could be useful to identify sentence openers.

The Intelligent Collaborative Support System (ICSS), developed by Israel (2003), extended the ICLS to a web-based platform with an updated interface and quicker response time. Additionally, the ICSS connected the sentence openers with text in the sentence closers to better analyze and facilitate discussions (Israel and Aiken 2007). The Collaborative Skills Network was extended with more sentence openers, such as “I want to” in the sending subskill of communication and “the next step should be” for working on task openers. Agents were used to couple students’ sentence openers with text from their sentence closers to more correctly classify the skill and discussion type. Students’ online discussions were more accurately monitored “to promote effective collaborative learning while accomplishing the task” (p. 8). A limitation of both the ICLS and the ICCS is that they were both scripted, menu driven systems. Allowing participants to use a more flexible format would be advantageous.

Israel noted that Soller (2001, 2004) “found that the sentence opener matched the intended attributes in only 68 % of the conversation, unless the set of sentence openers was adjusted to account for differences between face-to-face interaction and collaborative online interaction.” (Israel and Aiken 2007, p.15) Israel’s work of connecting sentence openers and closers resulted in “responses were correctly classified 100 % of the time when a keyword [in the closer] was recognized but keywords were not recognized in 30 % of responses” (p. 15).

Students’ collaborative skills become especially important in an e-learning or online learning environment. Communities of practice united students through their interactive knowledge sharing. The Collaborative Learning Conversation Skill Taxonomy (Plantamura et al. 2010), also based on our Collaborative Skill Network and Soller’s work, synthesized three main collaborative skills: active learning, conversation, and creative conflict. Their network utilized agents to monitor a web discussion forum, enabling it to identify the kind of dialog, even in an unstructured context. Their system did not tag the dialog to identify Soller’s interaction patterns; they intended to overcome this limitation by applying conversational analysis to the dialog.

Adaptive Collaborative Learning Support (ACLS) systems reacted and changed as students’ discussions were monitored. Walker et al. (2014), who classified our and Israel’s research as “guiding systems,” developed the Adaptive Peer Tutoring Assistant (APTA) for peer tutoring. Students served as peer tutors to each other while the APTA intelligent tutor provided feedback to the student tutor. Their work studied the effects of monitoring collaborative work on the actual achievement of collaborative skills and domain knowledge. Walker et al. concluded that students collaborated better and learned more when they were being monitored, due to an increased sense of accountability, one of Johnson and Johnson’s collaborative skills. Because the ACLS provided a limited adaptive support to the students, Walker et al. intended to expand their system with “long-term adaptive support which they need to collaborate more effectively.” (p. 58)

While the previous systems were built as stand-alone platforms, Calvani et al. (2006) extended the use of Moodle, an open source learning management system (LMS), with an enhanced collaboration module incorporating three functions: Forum Plus, Reflection Board, and Planner. The Forum component classified “Thinking Type” (TT) labels to communication messages. Example TTs were hypothesis, synthesis, and conclusion. The teacher could analyze the messages by type and “mark” them as significant in the collaboration and construction of knowledge. The Reflection Board tracked the construction of knowledge and logged activity in the Forum. By referencing the Forum and the Reflection Board, the Planner analyzed the users’ interaction and accordingly coached them on the appropriate use of specific TTs. Because this was an open platform system, it could be further developed and customized for specific needs and courses.

Scardamalia’s CSILE (Scardamalia and Bereiter 1991) evolved into Knowledge Forum, a product by Learning in Motion (2014). This product enabled users to form a community, sharing ideas and building a database of connected knowledge. However, tutoring or guidance of actual collaborative skills were not components of the Knowledge Forum.

Other related works also utilized the concept of sentence openers. Michaels et al. (2008) investigated deliberative discourse through “Accountable Talk,” a methodology which teachers used in the classroom. Just as in our ICLS, teachers used “Accountable Talk” techniques of conversation openers to guide student’s conversations for deliberative discourse. Over time, the students used the conversation openers on their own to stimulate and guide their conversation, creating a strong learning community.

Blind Spots in the Mirror

When our research was performed, the use of sentence openers alone (given the resources available) was determined to be the best method to analyze conversation and as a result, collaboration. Based on the work of later research projects, we recognize that the use of sentence openers alone for identifying, monitoring, and guiding collaborative skills is insufficient. For example, the connection of the sentence openers with the sentence-closers provides a better context for monitoring, guidance and assessment. Also, the exponential advance of computer (and supporting) systems in the past 20+ years allows the development of more complex and nuanced models. These models can then be implemented and tested in more realistic environments – both synchronous and asynchronous.

Improvements to the ICLS could be addressed by adding conversational agents. Tegos et al. (2014) used conversational agents in their MentorChat cloud-based system which analyzed students’ interactions by a conversation agent. The conversational agent used natural language processing to scaffold the discussion so that it would be a productive dialog enhancing the students’ interactions. One of the primary purposes of the system was to “identify efficient techniques of modeling and triggering constructive peer interactions through appropriate agent interventions” (p. 67). The system could be used with any domain, but was tested with a Computer-Assisted Language Learning (CALL) system for second language learning. A teacher also interacted with the system to configure the agent’s behavior. Just as in our study of the ICLS, the authors measured system usability and acceptance, finding that students found the system to be useful but preferred to interact with a person who was the language expert, rather than the built-in computer tutor. It should be noted that this system did not teach collaboration skills per se. However, some of the agents’ techniques for analyzing discourse efficiently could be useful to enhance an ICLS based system.

Adamson et al. (2014) also utilized conversational agents in collaborative discussions in Academically Productive Talk (APT). APT essentially is an automated response system providing positive feedback to students within a collaborative chat environment. Their experiments, in which students used the Bazaar system platform for collaborative discussions, concluded that the effectiveness of an agile system with multiple strategies depends on several factors, such as the type of the material and the skill level of each student. Review material and a higher student skill level normally yielded better results in learning. Both the MentorChat and Academically Productive Talk were examples of systems which analyzed students’ discussions and provided feedback about the discussion. Another improvement to the ICLS could be addressed by considering the use of an authoring tool to build an ITS. For example, Olsen et al. (2013) developed Cognitive Tutor Authoring Tools (CTAT). Their CTAT platform was used to build collaborative scripts which enabled dyads (pairs) of students to see each other’s behavior in their collaborative problem solving. The students discussed their work using a Skype videoconference. A similarity of the CTAT and the ICLS is that the former built a behavior graph for each student to monitor his/her behavior while our ICLS used a finite state automaton (FSA) to track the students’ use of collaborative skills. The advantage of CTAT is it created a generic script system in which pairs of students could work together, the students were aware of their peer’s knowledge, i.e., “cognitive group awareness,” and each student had his/her own tutor synchronized with the other student’s tutor. This resulted in multiple synchronized tutors running simultaneously. A limitation of this system’s supporting only two students could be addressed by supporting a larger group and also by building multiple behavior graphs.

Flipping the Rearview Mirror

Our research has the potential to be extended to future problems regarding online learning, social media and assessment. First, online courses and programs that use LMS have become more common, especially in higher education, because they offer flexibility with respect to time and location. Just as an open source LMS can be enhanced to support collaborative learning, can proprietary LMS also facilitate collaborative learning? If so, can the students be guided in its effective use? Some Massive Open Online Courses (MOOCs), which provide an unlimited number of participants with the opportunity to take an online course, use the approach of content delivery while others focus on structured discussion and yet others on peer assessment (Association for Learning Technology 2012). De Ward et al. (2011) have begun to explore the connection between MOOCs, mobile learning (mLearning), and collaborative learning through the use of Google Groups. Knox et al. (2012) noted that the challenge of MOOCs was for the teacher to provide sufficient attention to each student and suggested that structured dialog as in collaborative learning might be an appropriate monitoring pedagogy.

Second, social media, including online chats and blogs provide other means for students to collaborate. Badea and Trausan-Matu (2014) studied the interaction of student pairs using natural language processing of the HTML code. They based their work on Bakhtin’s dialogism (Clark and Holquist 1984) in which “every text, every word is echoed in many previous lines” (p. 880). They equated this dialogy to computer supported collaborative learning in which students’ discussions built on each other’s sentences. Using time-series analysis, they assessed the discussion according to the frequency of words to identify which concepts were being discussed, and which user was providing interventions. Building on this information, they provided a score for each participant. This aspect of time-series analysis could be utilized in the ICLS in its assessment of students’ collaborative skills to identify who is using the skills most often and effectively.

Third, faculty now use social media, such as Web 2.0, to enhance the synchronous and asynchronous interaction of their students, focusing on content knowledge. Some faculty use several different forms of social media for a class, such as forums, wikis, blogs, social bookmarking, and other media in which students’ constructively share information and collaboratively build knowledge. Popescu (2014) devised an integrated environment in the eMUSE platform to track students’ learning collaboration over different platforms, and to provide grading and evaluation support.

Social networking services such as Facebook and Twitter provide platforms for participants to communicate their thoughts and activities. Spyros et al. (2009), building on our use of collaborative skills, studied Facebook postings for struggle, their term for social interaction in collaborative problem solving. They used a variety of data including activity logs, email messages, file uploads and downloads, forms, analysis of struggle interactions and dialogs. They concluded that students involved in scaffolded problem solving used struggle interactions, supporting them in their problem solving.

Fourth, assessment of learning is a crucial component in education. Student learning outcomes (SLOs) are defined and evaluated, identifying areas that need continuous quality improvement (CQI). In assessing students’ achievement of collaborative skills, can we define appropriate student learning outcomes and if so, how can we best utilize them? By providing improvements, such as the appropriate use of the Trust collaborative skill in Request-type conversations, educators “close the loop” in striving to achieve the SLOs. In exploring these ideas, our Collaborative Skills Network may form the basis for an enhanced network. The Group Leader tutor model may be incorporated into a LMS to guide students in their work. In addition, the group processing module in our ICLS may be enhanced for more in-depth analysis and CQI processes.

Yoo and Kim (2014) assessed student learning of computer operating systems when they interacted in an online asynchronous discussion board (forum). In particular, they investigated what kind of participation in the discussion would predict the student’s grade for a computing project. Using predictive analysis, they parsed and computed the students’ contributions to the discussion board both quantitatively and qualitatively: quantitatively counting the number and frequency of entries, and qualitatively determining the students’ emotions or feelings in the process. They then correlated these results with the students’ understanding of the domain knowledge. The authors concluded that students who answered other students’ questions more frequently received a better grade for the team project, as did those who had a positive attitude, including “praising others.” By analyzing these results, the teacher or discussion leader knew which students needed more attention. Yoo and Kim’s research leads us to believe that predictive analysis could be used in a future version of the ICLS to determine which students the Group Leader tutor may need to focus on.

Magnification Mirrors

Our work on the Intelligent Collaborative Learning System (ICLS) has endured over the past 20+ years, from our development of the model and its implementation on what are now considered to be obsolete client–server, text-based systems to current, web-enhanced, interactive, mobile and cloud-based systems. Our research was based on the intersection of Intelligent Tutoring Systems, Computer-Supported Cooperative Work, and Collaborative Learning. A number of researchers have since incorporated various features of our ICLS, including its Group Leader Tutor and Collaborative Skills Network. Their work built upon our use of sentence openers as indicators of collaborative skills and subskills through enhancements such as:

  • using sentence closers

  • developing a collaborative skill taxonomy

  • creating an adaptive system which changes as students’ discussions are monitored

  • using Accountable Talk to guide conversations

Recent researchers have developed their systems in newer platforms, such as Moodle and a Knowledge Forum. While our system used its Group Leader Tutor, subsequent systems moved beyond this tutor to conversational agents and authoring tools.

Networks have evolved from the simple LAN environment in which our ICLS ran to highly complex Web 2.0 cloud systems which can be accessed from devices we could not even contemplate then (smartphones or wireless laptops with more processing power than the supercomputers of our generation). Current researchers are investigating how well they can study collaborative skills in MOOCs and in social media environments.

Our ICLS included an assessment module for the Group Leader Tutor to assess the students’ use of collaborative skills. Recent researchers use natural language processing to analyze conversations in online discussion boards.

As theories of pedagogy evolve and as technology changes even more quickly than Moore’s law predicts, we challenge researchers of the future to continue to build upon, improve, adapt and preserve the benefits of the ICLS, just as many other researchers have done. Big data analysis from large scale experiments, such as those in which students interact collaboratively in learning management systems or even MOOCs, may yield unforeseen and challenging insights into future enhancements of the ICLS.

As we look forward, our magnification mirror may enlarge not only the image of our ICLS, but more importantly the image of even more supportive and effective collaborative learning systems.