Keywords

1 Introduction

The use of information technologies has dramatically changed the way people learn. Online courses are now widely available to remote learners, who maybe students enrolled in a formal education program for a degree or certificate, or maybe informal learners who just want to learn something new.

In the meantime, information, including learning materials, has increased explosively on the Web that is open to the public. Learning objects [3] have been created specifically for supporting learning, and various digital libraries consolidate information resources on the Internet in supporting learning [13]. Many people now learn directly from internet sources for different learning objectives [7]. Because of the support of information technologies and the accessible information on the Internet, self-directed learning, informal learning, life-long learning, as well as formal education in higher education, can happen as elearning in a technology-rich environment, which is more convenient for learners.

This research considers search systems as an IT tool that can play an important role in supporting learning. While it is not difficult to find literature on the use of technology in general to support learning (e.g., [5, 6, 11, 14]), there has not been much research yet on learning while conducting interactive searches. In this research, we investigated into this emerging and important aspect of the learner’s use of searching systems such as web search engines. Today’s web search systems provide fast access to the vast amount of information on the Web. Naturally, searching has become a common activity in both formal classroom and in other informal learning settings. The combination of searching and learning has created the phenomena that is referred to as “learning by searching” [16]. For this research, specifically, we are interested in the relationships between online searching and learning: whether there would be any user search behaviors that could be indicative of learning, and whether or not learning during searching could be assessed by using the existing search performance measures. While learning is a task by people at a wide span of ages, the population that is considered in this study is adult learners.

2 Searching and Learning

Searching and learning behaviors have traditionally been two separate areas of studying. To lay a foundation for the current research, a discussion on learning and searching, and the relationship between the two, is necessary.

2.1 Searching is Part of Learning Process

Searching is a process of information seeking, in particular, for digital information. Because search always starts with an information need or statement, which represents the user’s intended information, searching also involves the evaluation or judgement on the search results to see if the search results are indeed related to the search objectives. A complete search process normally includes the following steps: forming the search query (or transforming the internal information need into a formal, explicit search statement); submitting the search query to the search system; evaluating the search results; if not satisfied or the information need changed, revising the search query and resubmitting it, and reevaluating the results. This information searching process has long been recognized as part of the learning process because the information retrieved is used as the input for learning. In [10] the author developed an information search process model, describing different stages during the search process that people seek information to learn the world around them. This model is part of people’s learning process. If the search system could not provide the needed information in the learning process, problems may arise which would hinder learning.

Despite the recognition of the importance of searching or information seeking for learning, searching and learning have traditionally been viewed as two separate things: searching is to collect information for learning, and learning is another process that uses the information being collected.

2.2 Learning by Searching

Searching is not just part of learning process. Searching has become more than just finding pieces of information: it shares learning activity features. People, particularly students, often employ explicit search as part of the learning process in studying of a specific topic. [12] argued that learning was a key process within the common activity of exploratory search, among three kinds of search activities: lookup (finding a fact, etc.), learning (acquisition of knowledge, etc.), and investigation (analysis, evaluation, discovery, etc.). As more primary materials go online, searching to learn is increasingly viable. Exploratory search systems are needed to support the full range of users’ search activities, especially learning and investigation, and not just lookup, which usually can be completed by one iteration of query-results. This inseparability of learning and searching is also promoted by “informed learning” [2], in which it is argued that information activities and learning are simultaneous processes.

Further evidence to support that searching is learning process is provided by [9], based on the cognitive processes involved. The cognitive processes involved in learning are summarized in the “Taxonomy Table” in [1], which includes six major categories of cognitive processes: remember, understanding, apply, analyze, evaluate and create, in the order of from simple to complex. Based on these cognitive processes, [9] classified 426 searching tasks according to cognitive process features in [1]. Seventy-two participants were asked to perform these tasks in a laboratory experiment. The results showed that information searching was a learning process with searching behavior characteristics specific to particular cognitive levels of learning. The results indicated that applying and analyzing, the middle two of the six categories, generally took the most searching effort in terms of queries per session, topics searched per session, and total time searching. The lowest two learning cognitive processes, remembering and understanding, exhibited searching characteristics similar to the highest order learning cognitive processes of evaluating and creating. Based on the findings, [9] suggested that a learning theory may be better to describe the information searching process. Such a theory, however, will need to be based on more understanding of learning during the search process.

In the present research, “learning (by searching)” means acquiring new knowledge about a topic through interactive searching activities. “Learning by searching” can be evidenced by the use of digital content as well as search engines in classrooms: students often employ searching as part of their classroom learning process when studying a specific topic. In this scenario, learning happens during the search process, not after searching.

There has been some research on learning by searching. To support learning by searching, [16] designed a system that could do automatic analysis on the search results for a particular course. This system combined searching and learning automatically, and helped address the fundamental issue of supporting learning while searching.

The factors (other than the behaviors) that may be associated with perceived learning were investigated in [17]. These factors include the user’s prior knowledge, prior search skills/experience (because the searches were on genomics documents, search skills/experience was restricted to Medline database search experience), search task characteristics such as general/specific, and the user satisfaction with the search results they found in relating to a specific search topic. The results showed that in general (without considering the task characteristics), all three factors: prior knowledge, prior search experience, and satisfaction, were significantly correlated with perceived learning.

3 Research Questions

Based on the previous work, the current work seeks to extend the scope of previous research to the understanding of how users learn by investigating the relationships between users’ learning and their search behavior and between learning and their actual search performance in terms of search outcome. Although searching and learning share common cognitive processes and are considered inseparable in this study, the learning tasks are normally not explicitly defined, but are implied in searching tasks or topics. Therefore, when investigating learning during searching process, instead of using some actual learning tasks, this study uses “perceived learning,” i.e., the user’s feelings about their knowledge gains from the searching process. Two research questions are addressed in this research:

  1. 1.

    What search behaviors are associated with perceived learning?

    Users’ behaviors on a search system have been studied for decades. These include querying behaviors, search result accessing behaviors, and so on. Behaviors which are significantly associated with perceived learning need to be identified. This identification will help understand when and how when users conduct interactive searching, they are actually learning.

  2. 2.

    Are user’s learning correlated with the search success as measured by typical search performance measures, so that learning during searching could also be indicated by using such measures?

    For learning during interactive searching process, one possible solution for learning assessment would be using the existing measures for search performance to assess learning. Search performance has traditionally been evaluated using the classic measures of Precision and Recall. These “relevance” based measures test the user’s ability to find relevant documents using the search system. They do not relate to learning directly. It is unknown if these measures would be able to be used for assessing learning by searching. [4] demonstrated the correlation between users’ knowledge, represented by a pre-knowledge score, and search performance, represented by a search score. Participants took a knowledge test before searching and were scored by the proportion of items correctly answered. The same test was applied after searching. It was found that the score after searching was positively correlated with the pre-search knowledge score. Their study [4], however, did not explicitly evaluate the difference between the search score and knowledge score. This study will compare these measures with the user’s perceived learning, hoping to find out if the measures could also be applied to assess learning in searching.

By answering the above research questions, the current research seeks to further the understanding of learning by searching, and to provide evidence for developing needed search technologies to support learning.

4 Methods

This research used the data collected and shared by a large research project on user’s information searching behavior at a major US research university. A detailed and complete description of the research design and the user experiment for the data collection can be found in [17]. In this article, we describe the resulting data and the measures that are used in this study.

4.1 Data Set

The data was collected through a laboratory user experiment in which 35 participants performed four search tasks using the standard Text Retrieval Conference (TREC) Genomics Track data, which are PubMed documents. The search topics were also adopted from the Genomics Track dataset [8]. The search topics are classified into two categories: general and specific, based on each topic’s relation to the National Library of Medicine’s controlled vocabulary, Medical Subject Headings (MeSH) tree. The participants’ search behaviors were recorded by the system. Before and after each searching task, participants were also asked to fill out the pre- and post-task questionnaires. The logged behavior data and the completed questionnaires consisted of the primary data used in this research.

Based on the original research design [17], the collected data set included the following types of data on two sets of search topics: general and specific topics:

  • Users’ search behavior data, such as the number of queries submitted to the system for a search topic, the average query length for a topic, the time spent on a topic, the documents selected from the search results pages (SERPs), and so on.

  • Users’ search performance data, mainly the number of documents judged and saved by the user as relevant for a topic, and

  • The questionnaire data, from both the pre- and post-task questionnaires, which included questions on demographic data and a question of if the user felt that new knowledge was learned through the search on a topic.

4.2 Measures

The following measures were used in the study:

  • Perceived Learning

    This was a user self-reported rating on a 7 point scale in the post-task questionnaire, to the question if new knowledge was learned from the search, from 1 for Not at all, 4 for Somewhat, and 7 for Learned a lot (Extremely).

  • Search Interaction behaviors/activities

    In total, 11 behavior variables, as listed in Table 1, were analyzed in this study.

    Table 1. Behavior variables
    Full size table

These behavior variables have been frequently used in information seeking research.

  • Search performance measures

    The classical performance measures, Precision and Recall [15], were used in the study. Precision is the number of correct search results divided by the number of all retrieved results, and Recall is the number of correct search results divided by the number of all possible relevant results in the search system.

Each participant’s performance measures are included in the dataset. These performance measures were calculated based on the participants’ evaluation of search results. In the experiment that generated the data set, Participants were asked to conduct searches on the experimental system and to find and save as many relevant documents as possible. After finishing their search activities, they evaluated all saved documents. Participants rated the relevance of the saved documents using a five point scale ranging from “not relevant” to “highly relevant” with “somewhat relevant” as the mid-point.

4.3 Data Analysis

Pearson correlation analysis and GLM/ANOVA procedures were the main statistic methods used in the study. The data analyses were performed using SPSS.

5 Results and Discussion

5.1 Search Behaviors and Perceived Learning

The results of correlation analysis are presented in Table 2.

Table 2. Correlations between perceived learning and behavior measures
Full size table

As presented in Table 2, among all the 11 behavior variables investigated, only three variables have significant correlations with perceived learning: the number of documents saved, the ratio of the number of saved documents and the number of documents opened or viewed, and the average ranking position of the documents opened.

Of the three variables, the ratio of the documents saved from all viewed is significant at p < 0.01 level. The other two are significant at the 0.05 level (n = 140). All correlations are positive. The results indicate that when more relevant documents are being saved, without viewing too many documents, it is more likely the user will report that they have learned. Since the number of documents opened alone does not have a significant correlation with perceived learning, it seems that the significance of the ratio mainly comes from the contribution of the number of documents saved. The significant correlation between the ranking position and perceived learning indicate that the lower (the larger the number) the mean rank of the opened documents in SERPs, the more likely the user will feel they have gained new knowledge.

A follow-up GLM/ANOVA analysis identified that the ratio was the only variable that had a significant effect on perceived learning (F = 10.838, p = .001). All other behavior variables do not show significant effect on perceived learning.

The results show that stronger perceived learning is associated with more document savings and lower ranking in the SERP list, which imply that more effort is needed during the search interaction process. These two factors could be indicators of a user’s learning. The finding sheds light on how the system may predict how much users gain knowledge through observable search behaviors.

No significant correlations are found between perceived learning and other user behaviors or efforts, such as the amount of time spent on the task, time spent on each page, number of pages viewed, number of queries issued, etc. Intuitively, learning might be associated with some of these behaviors. Future work will need to continue examining the relations between learning and these behaviors. If some additional important behaviors could be identified, it will help the system infer users’ learning, and adapt search accordingly for the user.

5.2 Perceived Learning vs. Performance

The correlation analysis found that perceived learning is not necessarily associated with any of the search performance measures: Precision and Recall. There are no significant correlations between the two sets of measures. The results are listed in Table 3. A follow-up GLM/ANOVA analysis found no significant effect on perceived learning from any of the performance measures.

Table 3. Correlations between perceived learning and performance measures
Full size table

One possible explanation for the result could be that a document judged relevant to the search topic does not necessarily add new knowledge to the user, i.e., the relevant document is not connected to the users’ knowledge status. “Learning” new knowledge is a goal different from finding “related” documents. A document could be “relevant” in many ways.

5.3 Effects of Topic Characteristics on Perceived Learning

The analyses described in the above sections did not consider the search task characteristics, i.e., the difference between general topics and specific topics, which intuitively is related to learning.

The data was further separated into two subsets, one for general topics and the other one for specific topics. The same statistical analyses were conducted separately for each of the two types. The number of participants included in the data set for the general topics is different from that of specific topics, due to the unbalanced number of topics in each category. The correlations between perceived learning and search behaviors are presented below first, which is followed by the results of correlation analysis on search performance.

Search Behaviors.

The results here are slightly different from the results in Sect. 5.1. Table 4 presents the correlations between the three behavior variables (that are discussed below) and perceived learning, under the two conditions separately: one for general topics and one for specific topics. All other behavioral variables that did not change from the previous analysis are excluded from the table.

Table 4. Correlations between perceived learning and behavior measures for general and specific topics separately.
Full size table

As Table 4 shows, for both general and specific topics, the average ranking on SERPs is no longer a significant factor that is associated with perceived learning. One direct consequence of slitting the data into two subsets is that the sample size in either subset is much smaller than that in the whole data set. It is possible that while checking down the search result list may be associated with perceived learning in large samples, it may not be the case for smaller samples. Given that one individual user’s data size is normally small, a document’s ranking position in SERPs may not be an important factor to consider.

Interestingly, the number of saved documents was found significantly correlated with perceived learning in the whole data set. But this further analysis found that it actually is significant only for the specific topics, not for general topics. It could be the case that specific topics are easier to learn than general topics because specific topics are relatively clearer than general ones, which normally are vaguer than specific ones.

The ratio is still significantly correlated with both general and specific topics. However, different from the test results from the whole data set where a significant effect is found with the ratio on perceived learning, a GLM/ANOVA analysis does not found significant effect of this variable. Again, it may be because the samples are not big enough to draw the results. In fact, none of the behavior variables is found to have an effect on perceived learning in either the general or specific topic case.

Performance Measures.

A GLM/ANOVA analysis was first conducted to examine if the search topic characteristic would have significant effect on participants’ perceived learning, if a given topic is general or specific. Similar to the result from the analysis on the whole data set, no significant effect is found on perceived learning from performance measures in either general or specific topic case.

While the result is not significant from the GLM/ANOVA analysis, the correlation analysis found some meaningful results that show significant correlations with perceived learning. The results are presented in Table 5, which shows that Recall score has significantly positive correlation with perceived learning for specific tasks, at p = 0.037, which for some reason the GLM/ANOVA was unable to detect. This finding is different from the result with the whole data set, where no significant correlations were found with either performance measures, despite a larger sample size of whole data set. It shows that when searching for specific topics, whether or not to find as many as possible all the relevant documents does seem to be related to whether or not the user would feel having learned new knowledge.

Table 5. Correlations between perceived learning and different types of topics
Full size table

It could be that in the case of specific topics users are able to gain more concrete ideas and, thus, are more capable of identifying the relevant documents, are thus are able to learn. In general topics, users may not be able to learn much in abstract terms on the general topic. They might also have difficulty in gaining a clear idea of how much they had learned about a general topic than from working on a specific topic.

6 Conclusion

In response to the two research questions, the research found that:

  • Perceived learning is only associated with limited types of search behaviors: the number of documents saved (as relevant). The more the saved documents, the strong feeling of having learned. The ranking position of the documents opened in SERPs can also significantly correlate with perceived learning, but only if the sample size is large: the lower ranking positions of the documents opened, the more the user would perceive learning. But when the sample size is smaller, the correlation is not strong. Realistically for an individual user, this perhaps means the ranking position is not a strong behavior factor to consider as an indicator of learning.

  • Perceived learning does not show, in general, significant correlation with search performance, measured by the classical information retrieval metric: Precision, Recall and F2 measures. However, considering the search topic characteristics and focusing on specific topics, Recall is significantly correlated, at p = 0.05 level, with perceived learning. This result suggests that for a specific search topic, a user’s learning is related to the number of relevant documents the user can find: the more relevant documents found the more the user may learn. Recall may be used as an indicator of the user’s learning, if the search topic’s specificity could be determined.

It should be admitted that the study focuses on a narrow domain: genomics. Therefore, the findings may not be appropriate to generalize to other subject areas. Similar research is need in other areas to collect empirical evidences.

The results of the study have significant implications for search-based technological support for elearning. Supporting learning has been the major goal and in the meantime a great challenge for the design of many information systems, particularly digital libraries. Such systems need to develop and incorporate new, learning supportive functions.