Keywords

1 Introduction

Adaptive Instructional Systems (AISs) are intended to help the learner to acquire knowledge and skills, to practice them, and to achieve expertise through the progression of stages that can be visualized as a learning curve, which has shown to impact learning in various task domains including procedural troubleshooting tasks, mathematics, physics problem-solving, etc. Learning curves are useful to visualize performance changes, and to evaluate adaptive instructional systems (AISs). Particularly, as a formative assessment, where a student is being taught about a concept, a fact, or a task, an adaptive instructional system should appropriately assess the student’s learning progress (or changing knowledge states), and properly inform the student of personalized and instructional contents. This is a very important but hard issue, but essential for advancing the learning experience.

As an assessment of performance changes, comparisons of two scores is a simple method. The two scores would be pre- and post-test results by comparing two learning materials, or by comparing before and after a unit of work, and by comparing scores after a period of time. This is called a summative assessment. In the meantime, the learner can be assessed during a course of learning, rather than at the end of a course, which helps us to see the skill development progress. The progress (i.e., the changing knowledge states by deliberate practice) could be visualized, and summarized as a learning curve.

This formative assessment tool can be useful in a case of military training. For example, soldiers are instructed to perform varying skill components. In general, a set of subtask skills for psychomotor tasks would consist of a physiological control skill (e.g., deep slow breathing while determining Point of Aim against a moving target during a marksmanship training). This skill set would not be not overtly identified whether it is being practiced correctly or not, which necessitates a new look toward assessment through learning curves.

1.1 The Learner and Domain Model

The domain model can be considered as a repository of knowledge and skills. That is, a domain model is a representation of knowledge for a task including domain content (scenarios, problems, or knowledge components), a learner model (including both a novice and an expert), and common misconceptions, and tactics/actions that can be taken by an intelligent tutoring system to help the learner engage in an optimized learning environment [see 2, p. 1].

The learner model specifies how the learner acquires knowledge and skills in the domain model through the stages of learning (e.g., from declarative to procedural stages). The domain and learner model can be comprised of knowledge components (e.g., declarative and procedural knowledge in ACT-R) that have been used to generalize the terms for describing pieces of cognition or knowledge including production rules, facts, principles, concepts, and schema [3]. Knowledge components can be authored manually based on a modeling framework (e.g., rule-based, or constraint-based). Domain models created by experts can be sometimes wrong, and thus, it is necessary to be evaluated by some types of the learning curve data—that is, the error rates decrease through practice [4]. One kind of the knowledge component model is rule-based cognitive models that can be computationally runnable [5]. A production rule-based model can help in thinking about what knowledge may be needed to perform a particular task, how that knowledge might be decomposed to capture what the learner would do, and how widely specific knowledge components will transfer [6].

The different domains (e.g., cognitive, domain, and social domains) can affect the way we understand the domain and learner modeling. In a cognitive domain, one of the objectives of the learner would acquire the maximum number of knowledge components specified in the domain model. In the psychomotor domain, the learner model would need a finer tuned domain model. For example, a novice golfer would acquire knowledge and skills about a putting task, and these knowledge and skills can be modeled using a rule-based system. The learner would go through cognitive processes (e.g., judge the line of the ball, etc.) before making an action (e.g., hit the ball) while he/she would control his/her breath to increase accuracy. In seconds, the task would be completed. Compared to other cognitive tasks (e.g., solving an algebra question), the golf putting task would require a finer granularity of instructions about how to successfully and fluently coordinate the cognitive, physiological, and physical processes. The knowledge components based on the learner (domain) models can shape the behavior of an adaptive instructional system. Thus, the efficiency of maintaining and updating the knowledge components, and their associated learner (domain) models would play a considerable role in the design of the grain size of adaptivity and in the behavior of the AIS. Authoring tools can be also improved by providing mechanisms to maintain and update the learner models with knowledge components, and to effectively alter aspects of the instruction as well [6].

1.2 The Adaptive Level and Assessment

Adaptive instruction can arise from an understanding of variations by individuals and by tasks/subtasks. As a formative assessment technique, learning curves can provide important insights of how to assess performance and define the adaptive levels. Educational outcomes generally reside in a certain time band of hours, months, and years. In the meantime, outcomes in a cognitive learning theory would be in the time band of seconds or milliseconds. Regarding a meaningful assessment in training, it would be necessary to consider the time band of human actions that are ranging from milliseconds to years [7, p. 122]. It is argued that there is a significant gap in an analysis of performance changes [8], and the gap can be defined by the Newell’s time scale of human action. We need an appropriate level of granularity and relevant theories [e.g., 3].

Learning curves have been used in industry as well in an attempt to investigate the prediction time (or cost) to produce a product [9]. In Cognitive Science and Education, learning curves have played a role to investigate practice effects by a certain knowledge representation in human memory systems [e.g., 10]. In both cases, learning curves generally follow a log-linear model [11, 12]. In general, the task completion time follows a power law of learning representing a speed-up effect.

One of known pitfalls of learning curves is that a larger domain model or a large student sample size is likely to exhibit a better fit than a smaller one, even if the system does not teach the students any better [13]. For example, a larger task with a large number of sample sizes with a sufficient amount of practice can still exhibit a power law of learning. But some subtasks would be learned differently and some subtasks would be learned slowly compared to others [14]. Thus, a simple analysis of learning curve in a large task seems not sufficient enough to make an instructional strategy. Furthermore, a near-term assessment by comparing learning curves would not be related to the long-term stability of learning [e.g., 15]. Thus, it would be necessary to use a probabilistic model and its parameter estimation. Particularly, Bayesian hierarchical model would be useful because it supports multilevel structures of variables to model variation explicitly. Inappropriate averaging to construct variables can sometimes remove variation, leading to inappropriate certainty of data and its handling [e.g., 16, p. 356].

Decomposing the Task to Subtasks

In Psychology, there is Reducibility Hypothesis [17]. Simply, it indicates a larger task can be meaningfully and functionally decomposed to smaller unit tasks. Thus, a subtask (in seconds) might be relevant to educational outcomes. An experimentation that investigates human attention in milliseconds provides meaningful implications toward a longer time duration of a task. The adaptive level would be related to how to decompose a task into smaller tasks. Therefore, a task can be meaningfully decomposed to a smaller unit of subtasks. A psychomotor task can be decomposed to cognitive subtasks, sensorimotor subtasks, and physiological control subtasks (like a tactical breathing subtask in a larger psychomotor task). This is useful in designing an assessment tool in an adaptive instructional system because a large task can be meaningfully decomposed and monitored.

Time Bands

To define the adaptive levels in an instructional system, it is used to consider behavior with respect to a time scale. Newell’s seminal work in Unified Theories of Cognition proposed a Time Scale of Human Action [7, 8]. Newell first mentioned the Bands: Biological, Cognitive, Rational, and Social Band. 100 ms to 100 h, even we think 10000 h of deliberate practice to be an expert.

It has inspired the efforts in the AIS community, and is seen in Anderson’s and Koedinger’s work. It has been pointed out that there is a gap between millisecond level experimentations in cognitive psychology and months/years outcomes in education [8]. Anderson made a suggestion to use the Newell’s time scale of human action. In terms of his arguments, the learner model would reside in Newell’s Cognitive Band (100 ms to 10 s), and the educational outcome lie in Social Band ranging from days to months. He argues that using cognitive modeling approach can bridge the gap in terms of Reducibility Hypothesis.

Ken Koedinger and his colleagues proposed a cognitive science based framework, called Knowledge-Learning-Instruction (KLI), in an attempt to promote high potential principles for generality [3]. He mentioned also Newell’s time band of human action. Firing one production rule is assumed to take 500 ms (0.5 s). His Cognitive Tutor uses a set of production rules as a learner model. Reading time can be modeled using a set of production rules, and can be plotted in a log-log scale, representing a power law of learning. Educational outcomes would lie in the Social Band with days to months. The granularity of adaptive instruction can go down to the Cognitive Band with a production rule in ms. It is claimed that cognitive modeling can provide a basis for bridging between events on the small scale and desired outcomes on the larger scale.

In this paper, we summarize the aspects of knowledge and skills that can be represented in a domain and learner model within a cognitive architecture. Knowledge and skills can be referred to as knowledge components that are grounded in different theories of cognition. Based on the knowledge components in the learner (domain) model, learning curves (e.g., log-linear models) is modeled and compared to summarize performance changes using the Bayesian hierarchical modeling approach. We report a case study of learning a psychomotor task (golf putting) in the context of multi-domain to test the proposed framework. Limitations and learned lessons are discussed at the end in order to improve our understanding of the learner and domain context in the development of AISs.

2 Constructs of Knowledge and Skills in a Psychomotor Task

Identifying what is learned would be useful to design instructions. Unobservable nature of the knowledge and limited scientific tools lead to a need to delve into cognitive task analysis [3]. It has been recognized that cognitive architectures have been played an important role in understanding the level of knowledge, and its learning [18]. We have taken an approach of Cognitive Science to fathom the unobservable nature of knowledge and focus on learning a psychomotor task—a golf putting task [19, 20]. Learning a psychomotor task can be simply considered as achieving a fluent coordination of cognitive, physiological, and physical systems of human body. For example, a golf putting task consists of subtask skills of: (a) cognitive skills including “judge the line of the ball”, (b) a physiological control skill including “slow breathing”, and (c) a physical skill including “hitting the ball”. These systems need to be interdependently executed to produce accurate performance. We describe our theoretical base for the construct of knowledge and skills that can be usable for an adaptive instructional system.

2.1 The Declarative to Procedural (D2P) Construct

The declarative to procedural (D2P) construct for knowledge and skills is based on the learning process implemented in ACT-R [21] with an understanding of the KRK learning and retention theory [22]. Learning theories [e.g., 12, 23,24,25] suggest a consensus understanding that a learning process consists of a number of stages from acquiring declarative knowledge to forming procedural knowledge by practice as shown in Fig. 1.

Fig. 1.
figure 1

The KRK theory of skill learning and retention [26].

Full size image

Several learning processes are implemented in the ACT-R cognitive architecture. Facts of task knowledge are encoded and stored in declarative memory. With practice, the learning process converts the acquired knowledge into a procedural form of knowledge based on both the activation mechanism of declarative chunks and production rule compilation [18]. As seen in Fig. 1, for the first stage, knowledge in declarative memory is strengthened by practice or is degraded by lack of use in terms of the activation mechanism. For the second stage, declarative knowledge is compiled into procedural knowledge, and the task knowledge is represented as a mix of declarative and procedural memory. With lack of use, declarative knowledge is forgotten, leading to missed steps and mistakes, and procedural knowledge is basically immune to decay. For the third stage, task knowledge is available in both declarative and procedural forms, but procedural knowledge predominantly produce performance.

2.2 Knowledge Components

The D2P construct is related to knowledge components (KC) in the Knowledge-Learning-Instruction framework proposed by Koedinger and his colleagues [3]. KC is defined as an acquired unit of cognitive function or structure inferred from performance on a set of related tasks. The main purpose of using KCs is to generalize terms for describing concepts, facts, cognition, and knowledge that are represented as production rules with declarative knowledge. They mentioned that many KCs representing mental process at the 10 s unit task level in Newell’s time band. But, we recognize these KCs are not directly related to a psychomotor task; they are mostly limited to cognitive tasks. Unit tasks usually last 10 s. A single golf putting task can be finished less than 10 s. Decomposing KCs in a psychomotor task would require smaller time scales to provide an improved assessment. The KLI framework focuses on the analysis of academic learning (e.g., geometry, multiplication, etc.); it appears to have a loose linkage to psychomotor domain learning (e.g., tennis, golf, archery, etc.).

Knowledge components can be characterized by Newell’s time bands, and can be also characterized by the properties of application conditions and the student responses [3]. We take such an approach to summarize the properties of application conditions and responses for a golf putting task learning domain, shown in Table 1. The relationship between application condition and response can be constant to constant, variable to constant, constant to variable, and variable to variable. For example, some KCs are applied under constant conditions when there is a single unique pattern to which the KC applies, but others are applied under variable conditions indicating there are multiple patterns that a KC applies to. In general, perceptual category learning can have a type of a variable-constant KC that is essentially category-recognition rules with many-to-one mappings. A more complex production rules can be described by a type of the variable-variable KC. In our task dome, practicing a slow breathing technique can be applied to the variable-variable KC. The conditions and responses can then be further specified by the relationships that can be either verbal or non-verbal, indicating whether it is expressed in words or not [3]. This is similar to the binary memory classification of declarative and procedural knowledge [18].

Table 1. Examples of different kinds of knowledge components in a golf putting task.
Full size table

3 The Framework

Based on the insights about the unobservable knowledge and its states, we need to fit learning functions by varying knowledge components separately so that we could identify different learning mechanisms—e.g., insights or gradual accumulation of knowledge, strategy shifts by altering the subtasks structures of the task, or subtasks trade-offs by understanding where the learner spends more time on one subtask that can lead to reduction in time (or accelerated time) to complete another subtask. It appears that the D2P construct is limited to implement models of subtasks with the assumption of consistent learning and most intelligent tutors appear to be so. But, our approach can be useful to extend the understanding of both consistent and inconsistent learning.

3.1 Bayesian Hierarchical Models

We can consider a simple hierarchical model for golfers (\( i = 1, \ldots , n) \), which can be nested by the golf handicap groups (i.e., groups by the golf handicap, which is how many strokes over par the golfer averages). The response variable \( (y_{ij} ) \) can be a distance to the target from the ball that is hit by the golfer. We assume that practice can help the golfer to reduce the distance as much as he/she can. We have a single predictor as a unit of practice trials (e.g., days, or months, but trials might be more appropriate in some cases but is less practical to compute). We assume that the error term (\( \varepsilon_{ij} \)) is normally distributed with mean zero and unknown standard deviation.

$$ y_{ij} = \alpha_{j} + \beta_{j} x_{ij} + \varepsilon_{ij} $$
(1)
$$ \alpha_{j} = \mu_{\alpha } + u_{j} $$
$$ \beta_{j} = \mu_{\beta } + v_{j} $$
$$ \varepsilon_{ij} = N\left( {0, \sigma_{y}^{2} } \right) $$

In Eq. (1), we have variation in the \( \alpha_{j} \), and \( \beta_{j} \). A correlation parameter \( \rho \) can be defined as follows.

$$ \left( {\begin{array}{*{20}c} {u_{j} } \\ {v_{j} } \\ \end{array} } \right) \sim N\left( {\left( {\begin{array}{*{20}c} 0 \\ 0 \\ \end{array} } \right), \left( {\begin{array}{*{20}c} {\sigma_{\alpha }^{2} } & {\rho \sigma_{\alpha } \sigma_{\beta } } \\ {\rho \sigma_{\alpha } \sigma_{\beta } } & {\sigma_{\beta }^{2} } \\ \end{array} } \right)} \right) $$

This model can be specified by subtasks and by golfers (nested by the handicap group). Subtasks can include knowledge components specified in cognitive, physiological, and physical properties. This linear mixed effect model is fit by using Markov Chain Monte Carlo sampling process in Bayesian inference. The proposed model explains varying intercepts for the golfers and subtasks, and varying slopes for the effect of the practice trials over time.

The model is useful to summarize learning rates by subtasks by subjects. It is emphasized that an adaptive instruction can arise from an understanding of variations. Successfully dealing with varying components in the model is important. Thus, the proposed hierarchical model based on knowledge components is useful to understand different learning curves by subtasks and by subjects as an improved assessment for adaptive instructions.

4 Discussion and Conclusions

This paper has discussed briefly theoretical foundations of knowledge and skills (e.g., D2P, ACT-R, KLI framework), tasks/subtasks decomposition, and granularity of adaptive level. As an assessment tool, the framework with Bayesian hierarchical models support an understanding of learning curves with varying slopes and interception. In this discussion, we note some lessons and future work.

4.1 Developing Fluent Knowledge and Skill Component

To help skill development, it would be necessary to consider the grain size of adaptive instructions and feedback. As mentioned earlier, a golf putting task can be decomposed into a cognitive subtask, a motor subtask, and for optimal behavior, a subtask about controlling breathing (a physiologically related subtask). Learning curves from all these subtasks would vary.

In an initial ACT-R model of golf putting [5, 19], the number of production rules is around 20. Learning in this domain involves the acquisition of such production rules. The production rule type knowledge component is useful, and this model can be run to find some sequence of productions that produces the behavior exhibited by the learner. A physio-cognitive model has been recently proposed to represent slow breathing in a psychomotor task as well [20]. The physio-cognitive model appears to provide a rich understanding of the task for adaptive instructions.

It is also necessary to consider a timing standard to look at performance—physiological control in seconds, cognitive thought process in seconds, and deliberate practice and outcomes in days. A time unit is used across the world as a standard, which helps us to communicate with each other regarding the time. Similarly, we could propose a standard regarding the granularity of adaptive instructions in an intelligent tutoring system. This would help us to better understand performance changes and their assessments. Newell’s time band [7] can be one candidate. There are several attempts that verify its usefulness for an intelligent tutoring system [3, 8]. Based on this, an adaptive instructional system would be cognitively and physiologically inspired to produce instructional adaptivity. Adaptations in the appropriate time band could provide an understanding of a finer granularity of adaptive instructions. This effort would be helpful to achieve a cognitive training and brain plasticity [e.g., 27] in AIS.

4.2 The Usefulness of the Framework: Predicting Readiness

Predicting readiness is important because soldier readiness is one of the top priorities in military training. Soldiers spend massive amounts of their time practicing knowledge and skills (e.g., shooting a static or a moving target). If the number of practice trials is sufficiently large, performance can be predicted by a regularity known as a power law of learning, where the time to complete a task decreases with practice or the number of errors decreases according to a power function [e.g., to note a few, 3, 4, 28, 29]. At the same time, if there is a period of skill disuse, soldiers and squads might not be fully ready for a military mission. A soldier’s performance would lie on a certain range in the curve shown in Fig. 1, summarizing a learning and retention theory in ACT-R.

There have been important attempts to create such predictions—to note a few, models in the KLM GOMS framework (not including learning) or in the ACT-R cognitive architecture. The KLM models predict expert performance, but they do not model learning. Models in ACT-R predict learning. It may be still not fully known if a soldier is actually ready for a combat mission and task under time stress and fatigue. Furthermore, a type of a power law of learning would provide much lesser predictability when the task is applied to multi-domains (e.g., in the wild or in a synthetic training environment). We and others suggest that psychomotor performance is interrelated with cognitive, physiological, and physical factors [e.g., 19, 30, p. 31, 31].

Predicting soldier and squad readiness is challenging, which will fundamentally play a crucial role in enhanced soldier lethality. A statistical and probabilistic model of the soldier’s changing knowledge and skill state can be useful to identify and predict performance readiness. We have reviewed attempts to monitor the changing knowledge state: one approach is to use Bayesian knowledge tracing [1, 32, 33], and the other one is to use performance factor analysis [34]. These approaches appear to be limited and unable to describe characteristics beyond the desktop environment and multi-domains. It would be necessary to extend the existing theories. Based on a preliminary physio-cognitive model in ACT-R/Φ, it is worth exploring Bayesian hierarchical model based estimates of probabilities that the learner have learned each of production rules in the physio-cognitive model until the learner has reached to the procedural stage shown in Fig. 1. This approach can increase the probability of learner and warfighter readiness.