Keywords

1 Cyber Domain in Military Operations

While cyber incidents raise increased media attention, the role of the human factor in cyber defense still lacks a comprehensive scientific framework [1]. Interdisciplinary approaches to investigate human factor in cyber defense operations include human interaction, the physical and social operating environment, decision-making processes and psychological determinants of performance in cyber officers [2, 3].

Rapid technological developments and definition of the cyber domain as a battle-field changed the cognitive demand profiles for cyber defense officers and challenged the traditional military organizational structures. Decisions on tactical levels can have large geo-strategical implications and are often highly complex, based on insufficient or unreliable information, and under time pressure.

The changed cognitive demands on cyber officers can be characterized by the requirement of enhanced cognitive agility, i.e., a parallel processing and constant monitoring of events, decisions and consequences in and between the intertwined physical and the cyber domain and their tactical and strategic dimensions. To assess, describe, and visualize the cognitive landscape in which officers operate, the Hybrid Space (HS) framework was developed [2]. The HS allows to assess performance in the cross-section of socio-technical and cyber-physical systems. This model was further expanded and tested by including metacognition, inter-individual differences, and macrocogntive (see Fiore et al. [33] for review) factors on HS performance.

1.1 Human Factors and Cyber Operations

Due to the lacking research within cyber defense, an applied paradigm to address this shortcoming is needed and from previous research, cognitive engineering is a human centered approach that addresses continuous changing complex environments [4]. This approach is supported through ecological approaches where participants are immersed in their domain, and the interaction of the person in their domain becomes the focus, unlike traditional experimentation. Such an approach supports the development of an understanding of influences of performance through an interactionist model. This is due to the cyber domain’s high information load without any objective goal or measure [5]. It is important to understand the situational factors influence on the mental workload demands involved as these new situations (the cyber domain) in itself have no value, that is no correct choice, but a person’s choice is dependent on the situational understanding and its influence on mental workload [6].

The goal of cognitive engineering is to develop an understanding of the fit of how an operator can better perform by taking more critical and complex decisions, in the system he finds himself in [5].

1.2 Cognitive Factors in Performance

The cognitive demands required for a successful cyber defense include a cognitive skill set with emphasis on cognitive flexibility, situational awareness, sustained attentional control, and motivation [7,8,9]. Being able to control one’s attention is dependent on the ability to maintain alertness, being able to orient oneself to relevant sensory input, and to make decisions based on the perceived information [10].

But factors such as anxiety can inhibit attentional control on specific tasks thus reducing processing efficiency [11]. This is caused through inhibition dysfunction, where stimuli unrelated to a task able to enter attention, and by affecting attentional shifting, where irrelevant information requires conscious processing thus diminishing the efficiency of working memory. Emotion regulation describes the process of how one is able to emotionally respond to an ongoing situational that allows for flexible decision-making strategies [12]. This process begins when an emotionally relevant situation is encountered that requires attention to and appraisal of relevant events that require a response that is dependent on previous experience as well as current psychological and physiological states. This process occurs through antecedent strategies and response modulation. For a well trained cyber defense operator, this process could begin situational selection where they approach a threat and try to identify the source through gathering information from relevant sources. Due to the difficulty of predicting emotion responses to this threatening situation, the situation could elicit emotional responses from the operator that can lead to less desirable outcomes. This would lead operators to then modify the current situation by including emotional processing strategies. This then influences their attentional deployment, where one can either focus on emotionally relevant cues (i.e. increased heart rate) that can lead to preservative cognitions (rumination, worry) that often produce incorrect decision making behaviors. But if an operator is properly trained, being able to identify a maladaptive emotional response and reappraising the situation, this leads to a reduction in physiological responses and subjective emotional responses, by reappraising the situation and thus leading to a more objective based decision making approach by understanding the impact of tactical decisions and strategic goals. If the decision taken leads to a negative outcome, for example, the network is breached and one cannot identify the source or stop the breach, an operator will try to modulate the emotional impact that they experience from the situation. This can be seen through increased sympathetic activation and a reduction in adaptive behaviors such as decreased communication and cooperation.

Metacognition refers to ‘thinking about thinking’ and includes the knowledge of one’s abilities, situational awareness, and behavioral regulation strategies [13]. Cadets with high metacognitive skills have more accurate and confident in their judgments performance within different situations. Individuals who have higher metacognitive awareness are also more accurate when describing their capabilities and identifying strategies that can improve performance. High metacognitive awareness of one’s cognitive processes involves monitoring, planning and evaluating one’s behavior in a given situation. If an operator is able to recognize the emotional impact of a breach and understands how the emotional response (e.g. rumination) affects their performance, they are better able to implement emotion-regulation strategies that would help to reappraise the situation and thus implement more objective decision making strategies.

Metacognition and emotion regulation can be viewed and intertwining processes that functions at a conscious level (awareness) that helps self-regulation, but also at a non-conscious level and co-regulates cognitions. Recent models [14, 15] of metacognition identify three distinct aspects of metacognition that also includes emotion regulation strategies that are based on declarative knowledge but also on subjective experiences based in emotional processes (intuition). These three aspects, metacognitive knowledge, experience and skills, work on three levels, social personal-awareness, and nonconscious level. While the social and personal-awareness level work through monitoring and controlling behavior (conscious level), the nonconscious level deals with emotion and cognition regulation.

1.3 Situational Factors on Performance

Due to the integration of cyber operations at each level in the military, from soldier support on the battlefield to network defense at higher command levels, cyber defense operators are dependent on team functioning. Their environment consists of several operators working in teams within a traditional military framework that responds to military leadership. Since this domain is novel to most personnel cyber operators must be able to work efficiently in these hybrid domains through proper communication and cooperation with both cyber and other military personnel. Research has identified several aspects of team functioning that can influence performance. Gutzwiller et al. [16] identified three areas that are crucial for performance: having an understanding and awareness of the network, which includes technical aspects and the behavior of the network; the world – how the physical world is affected or may be affected by events in the world including emergent threats and abnormal activities and behaviors; and finally the team, where awareness of work (completed, in-progress tasks), processes (demands, needs), and bootstrapping, being able to communicate with other inter-agencies to maintain focus. Cooke et al. [17] proposes the Interactive Team Cognition (ITC) that cognition is an activity that can be studied at the team level but only in its contexts. This has lead Salas [18] to identify several factors that influence team functioning. TO increase team performance, training, and interventions directed at improving both cognitive and affective cognitions, teamwork processes and performance outcomes helps, interventions suited at improving team processes have better effects. More specifically, improving communication and cooperation skills lead to better outcomes than task training [19]. While communication and cooperation demands within teams helps performance, increased team support or team emotional demand have been shown to decrease performance. So if a situation is able to initiate and increase emotional content and unclarity within a team, team processes focused at emotional communication and team member support will be more dominant that task communication and cooperation and thus lead to worse performance.

2 Method

The presented series of studies applies cognitive engineering and psychological approaches and attempts to provide a better understanding of determinants and limitations of cyber operator performance and discusses its implications for selection, training, and testing.

Both micro- and macrocognitive approaches were used to profile cyber officer qualities and to understand how cognitive and situational factors would influence performance. Whereas microcognitive approaches can be seen as more basic research approaches, macrocognitive approaches are applied in nature. Researchers stress that both approaches are important and need to be included in naturalistic decision making paradigms to better capture cognitive processes, both individual and situationally induced [6, 20].

2.1 Micrognition

Microcognitive approaches have the most used approaches to experimental settings. Cognitive functions are usually assessed in artificial laboratory settings under controlled situations [21]. It has the advantage of isolating individual cognitive functions and allowing for identifying outcomes of specific manipulations. Historically, micro-cognitive approaches have helped identify processes in daily functioning to decision-making strategies. Microcognitive approaches are also important in uncovering traits that might be specific to the cyber domain. For this research, microcognitive approaches are used to identify leadership constellations, decision-making strategies, and how these influence performance. For profiling of personality traits, the Big Five Inventory [22], a self-constructed self-efficacy scale, Embedded figures test [23] and trait affect measurements [24] were used. Participants were also subjected to experimental paradigms such as emotional go-nogo [25] and a modified Cognitive Reflection Test [26]. For emotion regulation strategies and nonconscious metacognition, both psychophysiological markers [27] (interoception) and cognitive aspects (rumination: response styles questionnaire [28], and the Penn state Worry Questionnaire [29]) were used. For the other metacognitive aspects, the Self-regulation Questionnaire [30] and the Metacognitive Awareness Inventory [31] was used to measure the conscious aspects.

But these approaches are limited in that they do not take into account external situational aspects that are found in natural environments that could influence behavior [20, 32].

2.2 Macrocognition

Macrocognition provides a framework to study cognitive processes as they affect real-world task performance, and is addressed as a complement, rather than a competitor, to microcognition by incorporating both individual and team processes [33] and is defined as “the internalized and externalized high level mental processes employed by teams to create new knowledge during complex, one of a kind, collaborative problem solving.” The term macrocognition emerged from a need to address the broad variety of cognitive processes in a naturalistic settings [20, 32, 34] and has gained recent focus through naturalistic decision making studies in sociotechnical systems [20, 33, 34]. Macrocognitive approaches are divided into three broad areas [21]: (1) macrocognitive modelling, where expert behaviors are compared to modelled systems, (2) macro cognition architectures, application of microcognitive functions to real world situations and, (3) team cognition, processes that arise due to several operators inter-acting in sociotechnical systems. To gather macrocognitive data, the Team Workload Questionnaire (TWLQ) was used [19]. The TWLQ was used to assess the workload demand in team tasks from two sub-scales: Team Workload Demands, which is concerned with the demands of team interactions (communication, coordination, team performance monitoring); and the Task-Team Workload Demands, which assesses the management of task and team workload demands (time share demands, team support demands, team emotion demands).

Measurements for dependent variables for naturalistic decision making paradigms (macrocognitive approaches) were developed. The Hybrid Space is mapped on a Cartesian plane and cyber operators marked their position simultaneously every hour during the third day of exercise (see Jøsok et al. [2] for description). In addition, students noted their current task at each position, to give context to further analysis. Movement in the Hybrid Space is operationalized through four constructs:

  1. 1.

    HSDT: distance traveled in the Cartesian Plane measured by Euclidian distance

  2. 2.

    HSxM: Movement along the cyber-physical domain (x-axis)

  3. 3.

    HSyM: Movement along the strategic-tactical domain (y-axis)

  4. 4.

    HSQC: Number of quadrant changes.

2.3 Data Collection and Analysis

Data for all studies was collected through self-reports and from a cyber-defense exercise where officer cadets were tested on breech detection and intrusion of a secure network.

Data was collected from mixed methods experimentation that included self-reports, expert evaluations, and naturalistic observation. Profiles were computed through psychophysiological measurements (parasympathetic activation, interoceptive accuracy) and from self-reports from personality and cognitive styles (e.g. emotion regulation, field dependence/independence, self-efficacy, metacognition) inventories. Experts in cadet training were asked to rate officer candidates for leadership abilities and these were matched to personality profiles (Five Factor Model) and emotion regulation strategies (emotional response inhibition task). For macrocognitive outcomes for predictions in decision-making and performance in cyber defense, data was collected during the Norwegian Defense Cyber Academy’s (NDCA) annual Cyber Defense Exercise (CDX): Exercise Cold Matrix. This arena facilitates the opportunity for students to train in tactics, techniques and procedures for handling various types of cyberattacks. The students work in teams, take tactical decisions in response to network intrusions, and develop counter measures. Success is presented as direct feedback to the decisions and actions taken during the exercise. Intrusions are initiated by an affiliated agency who are engaged to help the NDCA with their educational program.

3 Results

Lugo et al. [35,36,37] showed that cyber operators have different cognitive processes when compared to controls. They did not show the same associations on perseverative cognitions (rumination, worry) or emotion regulation processes. When tested for learning styles through the Embedded figures test, cyber cadet officers displayed distinct cognitive learning styles (field independent) versus age matched controls, but were similar to other non-cyber engineering students. Within the military domain, they were also significantly different than non-cyber military personnel. Participants who had higher self-efficacy responded incorrectly in decision-making strategies on more cognitive tasks when gut-feelings were involved.

External military experts rated cyber operators on leadership skills through situational observation and this was matched to personality profiles and emotion regulation strategies [38]. While previous findings show that military leadership reflects that of transformational leadership (higher extraversion, agreeableness, and openness, lower neuroticism) [39, 40], cyber operators showed that emotion regulation strategies moderated the relationship between extraversion and leadership ratings, but introverts were better rated [41]. These results partially reflect the findings of Rubin et al. [40] but theoretical aspects of why introversion better predicts leadership for cyber needs further investigation.

Metacognition was associated with better performance in cyber domain contexts [42]. Metacognition could predict overall movements in the HS except for performance on the strategic-tactical axis (y-axis). Strategic and tactical decisions are reliant on more macrocognitive approaches such as communication and cooperation between operators. Lugo et al. [38] used this to further investigate HS movements and showed that the team workload demands (communication, coordination) helped cyber operators with greater movements, while task-team workload demands (dissatisfaction, timeshare demands) inhibited movement.

4 Discussion

An understanding of cognitive, metacognitive and macrocognitive factors are needed to help develop effective and efficient cyber operators who find themselves in this domain. Current military structures may inhibit performance and therefore officers need to understand both their actions and positions, be able to identify significant others in the decision making process, and give the people responsible for decision-making the proper information that is found in the cyber domain so that decisions can be done with better certainty. Due to the fact that cyber operations understanding and research is in its infancy, the presented studies attempt to systematize an approach though the development of a conceptual framework, followed by the implementation of an action research approach by first identifying cognitive processes, and then incorporating situational variables, to give a better understanding of the qualities a cyber-operator may need to possess to successfully be able to perform in an complex and uncertain cyber domain.

Profiling of the officer cadets lead to several novel findings. Cyber engineer cadets do not display the same patterns of perseverative cognitions as normal controls do. The results from this study show that cyber domain officer cadets may differ in their rumination patterns from other comparable age groups. While normal controls showed signs of negative association between interceptive accuracy and perseverative cognitions (rumination, worry), cyber cadets did not show any of these associations. Cyber cadets did show similar cognitive styles (field independence) to that of other engineering students, but were significantly different from matched age controls and other military personnel (e.g. bomb diffusing units).

Due to the nature of the cyber domain being more oriented to more objective situations instead of containing emotional valence as found in physical domains, it was also expected, and found, that cadets who displayed higher confidence but included emotionally driven intuitions, performed worse than cadets who did not include intuition in cognitive tasks. While gut feelings can help in social situations where one must consider emotionally loaded content, the cyber domain might require more objective cognitions over emotional intuition, which could hinder the decision making process.

The cyber domain may lead to a selection process that attracts different profiles of cognitive and emotional processing. The relevance of individual differences in leadership constellations and cognitive styles may be from potentially systematic but unintended biases resulting from self-selection. Selection procedures are important to understand due to their implications for later job performance.

One important finding that arose from the investigations is the role of metacognition. Metacognitive awareness and regulation predicted performance. This is comparable to recent finding in similar domains in expert development. Metacognition is not an inborn talent, but rather a skill that is developed through training, exposure, and feedback and is a skill that distinguishes experts from novices.

Using a cognitive engineering and human factors approach, these results have led Knox et al. [43] to suggest the OLB model (“orient-locate-bridge”) for communication in a hybrid cyber physical domain, where cyber cadets are trained in technical aspects as well as psychological concepts of performance, are able to orient themselves, locate others, and bridge knowledge gaps that may interfere with decision-making. Thus helping cyber officers understand the processes and influences of their behaviors, and be able to communicate and coordinate with others to improve the decision-making process. The OLB model incorporates human factors in the cyber domain to better understand the interaction of behaviors in the cyber environment, in a scientific approach that answers the need proposed by the scientific community [1, 3, 20, 44].

All of the studies used had limitations. The studies were correlational in nature due to the lack of conceptual frameworks and validated models. The approaches used in these studies were taken from other domains, e.g. team performance, and applied psychology such as education and clinical psychology and thus might not be applicable to this domain. Further research including these approaches is necessary. Dependent variables for the naturalistic decision making paradigm where constructed for these investigations.

5 Conclusion

The proposed chapter provides an overview of a series of comprehensive empirical research on determinants of cyber defense officer performance. Cyber operators show unique profiles, and their performance can be explained by both micro and macro approaches. In summary, these findings aim to pave the way for an evidence-based approach to selection, training and evaluation of cyber defense officer performance. To achieve this, the theoretical framework of the Hybrid Space has been applied to map the cognitive location and dynamics during cyber defense operations, leadership styles, problem-solving styles and group effects complement the overview of performance within the theoretical framework.