1 Introduction

Mobile devices can enhance online discussion forums (ODFs) by allowing learners to interact with each other anywhere and anytime. Since this enables participants to access the posts as soon as they are made, the time between the posts and their replies should decrease and the participation should increase [1]. Moreover, enabling participants to check messages easily and more frequently reduces the accumulation of unread messages, which are not relevant to them anymore [1]. Additionally, if the participants are available at the same time, a critical mass can occur and the discussion can be synchronous [2]. The benefits of a synchronous ODF are immediate feedback and motivation [1].

However, just making ODFs available on mobile devices might not yet yield all of the true benefits noted above. This is because the level of activity on the ODFs still depends on participants’ diligence to check their mobile devices to see new posts. Moreover, the difference of using a computer compared to a mobile device is that mobile device users cannot focus on the device and the screen for a long time, especially while they are travelling [3]. These reasons impede the participants from checking their devices frequently; as such, the effect of mobile ODFs on discussions could be similar to traditional ODFs.

The current research proposes a design to integrate a notification and recommender system (NARS) into ODFs on mobile devices to address the above issues. Notification is used to inform participants that there is a new activity on the forum. Recommender systems suggest posts that might be relevant to the individual participants. Specifically, the current research focuses on ODFs that allow learners to have a discussion outside a classroom. Since the discussion is an addition to classroom time, it is difficult for the learners to know when other learners are free to discuss. With NARS, participants can check the discussion on their mobile devices when they receive prompts or when they think it is necessary.

2 Information Systems Design Theory for NARS for Educational Online Discussion

This current research uses a design science research approach to develop NARS for an educational ODF. Information system design research articulates and develops a specific class of information systems [4] or artifacts [5] in terms of an Information Systems Design Theory (ISDT) [6] in order to prescriptively guide a design of other systems in that particular class. An ISDT is design knowledge that is expressed as a theory in order to make the information system research rigorous and legitimate [6]. The design theory in this current research is described using eight components of an ISDT proposed by Gregor and Jones [6]. The remaining sections of this paper describe the eight components and evaluation of the artifact in more detail.

2.1 Purpose and Scope

The current research proposes to develop and integrate NARS into an ODF on mobile devices. Notifications are expected to increase the participants’ awareness of new posts and recommendations are expected to decrease the negative effect of information overload on participants [7]. Therefore, by adding the artifact to the ODF, the effectiveness of the discussion should be improved.

2.2 Constructs

The major constructs for this research are notification, recommendation, and topic discovery. The following subsections describe each construct in more detail.

Notification. One of the benefits of an ODF is that there is no requirement to instantly reply to messages. However, long delays in responses are problematic because they tend to stifle discussion. It is possible to reduce the effect of long delays by notifying users about what is going on in the discussion [3]. Notification is a service that delivers messages to users’ devices instantly or at a specific time [8]. The purpose of notification is to help users be aware of the most recent events in the current task-oriented interaction [9]. A notification system is a lightweight display of information, which is triggered by specific events and delivered to a person with a current task-oriented concern [9].

Notification is classified as push technology, which delivers “right” messages to the right users based on predefined rules or triggers [10]. One of the advantages of push technology is that it takes less time for the users to browse for their relevant information. Moreover, users are always made aware when there is an update and they can respond immediately. The disadvantages of push technology are that it can become annoying with interruptions and requires more bandwidth to deliver [10].

Recommendation. Push systems that actively deliver information to the user without a request cause information overload [11]. That is because they potentially increase the amount of useless information that a participant must handle [12]. Recommendations can help users deal with the information overload [7]. Recommender systems suggest a set of relevant posts or threads to users based on feedback such as ratings from other users and what the users post [13]. They not only mimic a person who is knowledgeable in a topic, but also take the person’s tastes and preferences into account [14].

Recommender systems can be classified into three categories: content-based, collaborative-based, and hybrid [7]. Content-based recommender systems provide an item that is similar to the ones the user preferred in the past. Collaborative-based recommender systems provide a recommended item that people with similar tastes and preferences to the user liked in the past. Finally, a hybrid recommender system is the combination of content-based and collaborative-based [7]. Additionally, a recommender system provides a service to users based on explicit feedback (ratings from other users) and implicit feedback (what the other users post) [13].

Topics Discovery. The designed system needs a document clustering method to see whether a newly created post is similar to the posts that are known to be relevant to the participant. This kind of method discovers the semantic relationships between individual terms using statistical analysis on a whole dataset of documents [15] and groups the documents based on them. The research uses Latent Dirchlet Allocation (LDA) since the algorithm can discover the probability that a latent topic belongs to a document. Moreover, the model has been successfully used in order to discover topics in news articles and academic abstracts [16].

LDA is described as a generative probabilistic model that can be used with sets of discrete data such as a text dataset. The model is a three-level hierarchical Bayesian model, in which there are three levels: corpus, document, and word. The assumption of the model is that a document is a mixture of topics with each topic having different probabilities in each document [17, 18]. Moreover, in addition to words, the model is also valid for larger structure units such as n-grams or paragraphs [17]. LDA can cluster words into topics and a document into mixtures of topics [18].

Most of the methods for representing texts used by most of the clustering algorithms, including LDA, are derived from a “bag-of-words” model [15, 17]. In this model, an attribute is created to represent each word in the corpus and each document is assigned those attributes with their values corresponding to the number of times the word occurs in the document [15]. However, the bag-of-words model has a limitation when it is used with short text documents due to sparseness of data [15, 19]; it needs to measure similarity but short texts do not have sufficient terms to make them appear in more than one document [19]. Moreover, synonymy (different words that have the same meaning) and polysemy (a word that can have multiple meanings) make it even harder to analyze the texts [15].

Unfortunately, the length of ODF posts made on a mobile device tends to be short. This is because authoring a post on the forum might not be the participants’ primary work or when on the move participants cannot focus their attention on writing more than a few words in a message. The small size of the screen and the on-screen keyboard of the mobile device can also impede participants from creating a long post.

2.3 Principles of Form and Function

The current research adapts the generalized architecture of an adaptive educational hypermedia systems (AEHS) model [20] to describe the implementation of NARS. The architecture is chosen because it depicts the main components of the systems and their structural interconnections. AEHS addresses the same issue as recommender systems in learning [21]. The approach deals with the problem that learners with different goals and knowledge might need different information or treatment. It overcomes this problem by adapting the presented information to an individual learner with respect to his or her information in the user model [22]. The architecture of NARS for an educational ODF is shown in Fig. 1. It defines the structure of the components of NARS, which can be used as a blueprint or framework to design the artifact.

Fig. 1.
figure 1figure 1

NARS for an educational ODF architecture (adapted from karampiperis and sampson [20]).

Full size image

2.4 Artifact Mutability

Courses might be different in terms of content, requirements, ontology, etc. The design of the artifact needs be adjusted to fit the course for which it is implemented. For example, the notification rules need to be defined in the way that meets the course requirements. Recommender systems can also be designed to fit the ontology of the class. The topic discovery method can be designed to utilize the structure of the content in the forum.

2.5 Testable Propositions

Three propositions are derived from the kernel theories described in the next section. As shown in Fig. 2, notification gives the participants awareness of posts, and the awareness should improve the effectiveness of the ODF. In addition, recommendation reduces the effect of participants’ information overload [7]. It is expected that recommendations will manage participants’ feelings of being overwhelmed, and therefore increase their confidence to contribute to the forum. Overall, the effectiveness of the discussion on the forum should be improved by NARS. The propositions are as follows:

  • Proposition I: Notifications can increase participants’ awareness.

  • Proposition II: Recommendations can reduce the effects of information overload.

  • Proposition III: NARS can improve the effectiveness of learning.

Fig. 2.
figure 2figure 2

Model for the effects of NARS on ODFs on mobile devices

Full size image

2.6 Justificatory Knowledge

An ISDT should be based on natural and social science theories, which are referred to as kernel theories [5]. The research adopts awareness and information overload concepts and uses them as kernel theories to guide the development of the artifact.

Awareness. The current research suggests that the long delay in response in typical asynchronous online discussions is caused by lack of awareness of participants. For example, if they are not aware that a new comment is posted, they do not check the forum and respond to the new post. Awareness in computer-supported cooperative work (CSCW) is referred to as an understanding of the activities of others in order to provide a context for one’s own activity. This context makes sure that the individual’s actions are relevant to the group’s activities and contribute toward the group’s goals [23].

Awareness in the current research is defined as knowledge of existence of posts that are newly created on an ODF. This awareness is delivered to the learners via notification messages on mobile devices. It is expected to provide opportunities for learners to contribute to the discussion and allow the contribution to be relevant to the group. As a result, learning should be more effective.

Information Overload. Because learners simultaneously use knowledge in a shared knowledge space, it is highly likely that they will receive many active awareness messages at the same time. These messages can cause information overload for the learners [24]. Moreover, unlike traditional ODFs, those on mobile devices have almost constant access; therefore, participants can be overloaded with seemingly endless opportunities to learn and work [11]. In general, information overload refers to the concept of receiving too much information [12]. Having too much information can be confusing, can reduce the person’s ability to set priorities, and makes it harder to recall previous information [12].

Information filtering, which refers to both looking for wanted information (filtering in) and eliminating unwanted information (filtering out) [24], can help handle information overload. The goals of knowledge awareness message filtering are to filter out irrelevant messages that might disturb the learner and to prioritize the messages with respect to their significance to the learner priority [24]. Recommender systems provide filtering.

2.7 Principles of Implementation

A prototype has been developed as an instantiation of the ISDT in order to test the propositions. Fifteen graduate students were asked to evaluate the mobile app prototype for usability. Feedback from the evaluation were used to improve the prototype system so that the undesirable or missing features were corrected.

2.8 Expository Instantiation

Enhancement Approach for the Topic Discovery Method. The current research proposes an enhancement method to improve the performance of LDA on short text documents. The proposed method is intuitive and does not require a modification of the LDA clustering algorithm. This method applies to the preparation process that creates a representation of data before feeding it to the algorithm. The proposed method starts with identifying important words from a well-organized tree structure [25] of an ODF as described in Table 1. In addition, Banerjee et al. [26] find that doubling the weight of the terms in the title of a document yields better results. The current research follows this suggestion; however, it doubles the weight of all of the important words and terms in the ontology described previously. This approach can be used with any clustering methods that are based on a bag-of-words approach.

Table 1. Proposed rules to identify important words
Full size table

Prototype. There are three major components of the prototype: ODF, mobile application, and recommender engine. The ODF web application was adapted from an open source system called Simple Machines Forum (SMF).Footnote 1 One of the advantages of SMF is that various modifications developed by a community of SMF developers can be installed to the system. Secondly, the mobile application used to access the ODF was developed for the iPhone and Android platforms (Fig. 3). It is an interface that exchanges information with the web application. The current research adopted a technique called data scraping to extract data from the mobile forum web application and visualize it on the mobile application. Finally, the recommender engine selects relevant posts and threads for participants based on predefined rules. In addition, in order to find topics of posts, an LDA module in Mahout, which is an open source machine learning library from Apache, is used. In this research, Mahout is modified with respect to the way that the data is prepared according to the enhancement method proposed above.

3 Evaluation of the Enhancement Approach for the Topic Discovery Method

In order to evaluate the proposed enhancement to the topic discovery method, a reliable benchmark dataset of text documents with labeled topics was used. The dataset is from Travel Stack Exchange,Footnote 2 which is a Q&A site for traveling. In total, the dataset contains 1000 posts for topics that have accepted answers that more than 10 people voted as being useful. The criteria ensure that the discussions on the topics are highly active. In the dataset, the first post of each discussion thread contains tagging terms that describe the thread. These tagging terms are used as a benchmark or labeled topic.

Fig. 3.
figure 3figure 3

Screenshots of the application prototype

Full size image

Two versions of a topic discovery function were implemented. The first version is the baseline method that has only the LDA algorithm. The second version is the treatment method that has the LDA algorithm with the proposed enhancement method, but not all of the proposed important words identification rules are implemented because the dataset does not have some information to implement some rules. For example, the dataset does not have quotes or an ontology. Moreover, the tagging terms are not included in the rules since they are used as a benchmark.

Both versions of the method were run on the dataset 100 times. A match is made if a post is clustered into the topic that contains keywords that appear in the post’s tagging terms. The total number of matched posts for each run was recorded.

An independent samples t-test showed that here was a significant difference between the mean of the treatment method (x̄ = 492.57, s = 19.29005) and the baseline method (x̄ = 474.58, s = 19.23443); t(198) = −6.604, p < .001. This result suggests that the proposed enhancement method is better than the baseline method for classifying posts. The relatively low averages of the matched scores (less than 500 out of 1000) is possibly explained by the fact that the benchmark dataset contains only 5 tagging terms. Since the purpose of the evaluation is to see whether the proposed method increases the number of matched posts, the raw averages are not important.

4 Evaluation of the Prototype

An experiment to evaluate the propositions was conducted with 7 students enrolled in a statistics course. The experimental subjects were asked to install the mobile application on their smartphones and participate in an ODF. The experiment lasted for two weeks and was divided equally into a control phase and a treatment phase. NARS was disabled in the control phase and was enabled in the treatment phase. At the end of the experiment, survey questions were distributed to the subjects to inquire about their experiences with the notification and recommender with respect to the ODF. Since the subject size was small and the experimental period was short, it was hardly likely that information overload would happen. Therefore, the participants were asked to imagine that they were using the artifact in a large class.

The majority of the participants thought that notification would be useful because people would be noticed right away after a reply or relevant comment has been created. One participant mentioned that the notification features helped in finding posts; as a consequence, the person could post more. Another participant mentioned that the feature would help people follow the discussion, which is very difficult to do using a website. In addition, it was interesting that most participants automatically compared the notification on a smartphone to email. For example, one participant said that having many smartphone notifications, which take only a small space of a smartphone’s screen, is not as difficult to process compared to receiving them in email. Another comment suggested that students do not check their email regularly, so a smartphone notification would be helpful. Overall, the students felt that the feature would help them to continue the discussion. However, there are some concerns that too many notification messages could disturb people and they might ignore the messages potentially leading to missing some content that is relevant to them. However, an option to turn notifications on and off would prevent the students from being overwhelmed. Two participants worried about their privacy; they thought that a smartphone was for their personal use.

Participants found that the recommendation feature would help them find other discussions that may be of interest to them. This would be true especially in a big class since nobody would read all of the posts. Two participants thought that the feature would expedite the process of looking for a post to comment on or replying to a post because the participants would know where to look. One participant suggested that the feature would allow people to respond to multiple posts in a short time. However, there were some participants who still preferred to look for a relevant post by themselves because there might be a topic aside from the one recommended in which they are interested. Another participant was concerned that the feature might discourage people from reading other posts.

Overall, the participants thought that these two features would help them contribute and be more involved in the discussion, and would make the system organized and manageable.

5 Conclusion

The current research is expected to contribute to the educational industry and the research area of technology enhanced learning (TEL). For the educational industry, the ISDT can be used as a guideline to design NARS for an ODF for learning. For researchers, the current study presents an approach to improve the performance of the LDA topic discovery method. However, the approach can also be useful for any clustering methods that are based on a bag-of-words representation. The research also proposes a conceptual model to explain how NARS can improve the quality of learning for students participating in an ODF. That is, notifications delivered by NARS can increase awareness of participants, and recommendations can reduce the effect of information overload. The results of the experiment supported the propositions, although some students expressed concerns that the notifications might disturb some participants.