Keywords

1 Introduction

In 2006, the Transportation Security Administration (TSA), under the Department of Homeland Security (DHS), shifted from primarily screening passengers and their baggage and started training officers to identify suspicious behaviors and/or activities at airports. The officers chosen to receive this training and execute this mission are selected from the existing pool of Transportation Security Officers (TSOs) to join the Behavior Detection and Analysis (BDA) Program. Once training is completed, these officers become certified Behavior Detection Officers (BDOs). As recently as 2013, there were over 3,000 BDOs deployed to 176 airports though today both numbers are reduced (Pistole 2013). In 2012 alone, their work resulted in 30 plane boarding denials, 79 law enforcement investigations, and 183 arrests (Pistole 2013).

BDOs observe individuals’ behavior for indications that they are experiencing stress, fear, and/or deception which could indicate a passenger is participating in illicit activity. These officers are trained to assess the environment, create a baseline of normal behavior, and then identify potential high-risk passengers based on deviations from the normal behavior baseline. BDOs, from 2006 to December 2014, were trained using a method called SPOT (Screening Passengers by Observational Techniques) – now labeled as Behavior Detection (BD), under TSA Behavior Detection & Analysis Program (BDA). SPOT was originally created for Boston’s BOS airport and later rolled out nationally (Hallowell 2011; Seidman 2011). SPOT was created using subject matter experts (SMEs) who based much of the system on similar behavior detection systems deployed in international sectors and federal entities. SPOT was tailored to American airports and has continued to evolve since it was launched. Two notable SMEs utilized in the creation and modifications of SPOT include Rafi Ron, former Israeli airports security chief, and Paul Ekman – famous for his 40 year career in interpreting the emotions coded in facial expressions (Burns 2010). Since October 2014, TSA has been piloting an updated version of the BDO protocol, referred to as the optimized BD indicator protocol, or Optimization.

The BD system provided BDOs with a large list of nearly 100 behaviors (behavioral indicators) to watch for while passengers are in the TSA checkpoint queue and other locations. BDOs observe for these indicators and when a specific threshold has been met, that individual would undergo additional screening before crossing into the sterile area of the airport (i.e., the area beyond the TSA checkpoint with access to the plane terminals). If the passenger in question passed the additional screening they would proceed, without further intervention, into the sterile area of the airport. If, however, the passenger in question failed to pass the additional screening, or excessive BDO points were assigned to that passenger, a Law Enforcement Officer (LEO) would be summoned to take over.

The piloted Optimized indicator protocol training was deployed in the field in October of 2014 (Bugler 2013). It was created to address several of the criticisms and concerns of the program (Government Accountability Office [GAO] 2013). Previously, there had been criticisms that there was no scientific evidence that SPOT worked and through the optimization effort, TSA has provided scientific support from various fields associated with deception detection, verbal and non-verbal communication, and physiological areas of academic research.

The purpose of the current paper is to discuss the methodology of conducting a job analysis on the occupation of BDOs, how the job analysis will navigate the recent BD-to-Optimized indicator protocol changes in the standard operating procedures (SOP), and how to conduct a cognitive task analysis (CTA).

2 Literature

Job analysis is the foundation of industrial-organizational (I-O) psychology, and it paves the way for many other types of work to be done (e.g., CTA, assessment centers, recruitment, selection systems). A job analysis is a structured process of identifying and measuring the elements (i.e., tasks and sub-tasks; equipment, machines, tools, technology; knowledge, skills, abilities, etc.) that compose a single job (Van De Voort and Whelan 2012). Based on how the resulting job analysis will be used the more detailed aspects can be tailored to the expected need [e.g., collecting SME job task difficulty, importance, and frequency ratings (DIF analysis) to support an HR/training department].

Several methods for conducting a job analysis exist including Threshold Traits Analysis, Ability Requirements Scales, Position Analysis Questionnaire, Critical Incidents Technique, Task Inventory/CODAP, Functional Job Analysis, and Job Elements Method, to name a few (Levine et al. 1983). Each of these has different strengths and weaknesses but job analysis experts consistently rate the Task Inventory/CODAP and the Functional Job Analysis as two of the best for the purposes of defining and describing the job and also meeting legal requirements (Levine et al. 1983). To paraphrase Levine et al., ultimately, job analysis experts overwhelmingly endorse a strategy of using a combination of multiple job analysis methods (1983, p. 346). With this in mind, for legal purposes, the Uniform Guidelines on Employee Selection Procedures (Equal Opportunity Employment Commission 1978), clearly states their preference for a job analysis method which identifies job tasks that are “observable behaviors” (p. 228).

Identifying the job tasks, sub-tasks, and other elements that compose a job with a job analysis is helpful in informing high-level organizational leadership decisions to more effectively guide their personnel’s employee lifecycles (i.e., recruitment, selection, training, promotions, teams, job duties, etc.). It can also be used to perform an even deeper analysis of the work being done, as is the case with a CTA.

A CTA is used to systematically identify the critical human worker’s cognitive activities (e.g., decision making, goal forming) related to the successful completion of job tasks. Where a quality job analysis produces a list of observable tasks and work behaviors, a CTA captures many of the internal processes and steps that are not immediately observable - without specialized cognitive measures. In many ways, “Cognitive task analysis is the extension of traditional task analysis techniques to yield information about the knowledge, thought process, and goal structures that underlie observable task performance” (Schraagen et al. 2000, p. 3). A CTA is also used to understand the cognitive demands various job tasks require (Hoffman and Militello 2012). The CTA utilizes a combination of worker observations, surveys, and in-depth SME interviews to identify and map worker’s cognitive processes (Wei and Salvendy 2004).

The recent rise in popularity of CTAs has been attributed to the growing prevalence of job specialization, the growing use of smart machines, and workplace modernization in our society (Hoffman and Militello 2012). For these reasons, CTA has been used extensively in the aerospace industry (Seamster et al. 2000), as well as with air traffic controllers (Seamster et al. 1993). Indeed CTA is a great fit for assessing the BDOs at airports as well because CTA’s strength is in supporting jobs that require cognition in the form of inference, diagnosis, judgment, and/or decision making.

In 2011, a Science and Technology (S&T) validation study of the BDO program was conducted in partnership with the American Institutes for Research (AIR). This research showed just how effective and important the job tasks of BDOs were. Their study found that across approximately 70,000 random samples of passengers submitted to secondary screening, BDOs using the BD method were nine times more likely to catch a high-threat passenger than random screenings (Pistole 2013). This type of validation study demonstrates that while cognitive load may have an effect on BDO performance, the BDO’s work is still effective in contributing as one layer of TSA security keeping passengers safe. A cognitive task analysis will demonstrate where various forms of cognition support BDO work and ultimately how to improve that work process so that BDO screening is even more effective.

3 Method

The current research is ongoing and will conduct both a job analysis and a CTA. The job analysis will assess the full scope of the BDO occupation. To do this, the researchers first acquired all possible existing sources of BDO job information including job descriptions, training materials, standard operating procedures (SOPs), other relevant HR documents and policies, and any related previous job analysis research or data.

During the document review phase, a previous job analysis covering all TSA screening positions (TSO, LTSO, STSO, STI, CCO, and BDOs) was discovered called the “OJAS” (Job Analysis of Transportation Security Officer Jobs), which was conducted by HumRRO. In this job analysis, 97 of the 416 job tasks were linked to BDOs.

A second job analysis was also found, the Human Performance Requirements Analysis (HPRA) which was conducted by Applied Research Associates (ARA). This job analysis identified a total of 3,164 TSA officer screening tasks (including Behavioral, Cognitive and Communication job tasks). Unfortunately, the BDO job fell out of the scope of that research so none of the identified screening tasks were linked to BDOs – though many likely apply based on overlap between TSO and BDO job tasks identified in the OJAS. A small group of BDO SMEs will be consulted to analyze the 3,164 tasks and to identify which tasks are relevant to the BDO job and which are not.

A third job analysis, specific to the BDO job, was conducted by American Institutes for Research (AIR) and was completed in March 2010. This job analysis identified 107 BDO job tasks, of which 95 were organized under 10 duties (Hendrickson et al. 2010). This research also identified and linked 56 knowledge, skills, abilities and other factors (KSAOs) to the related job tasks they supported. Additionally, 64 competencies from the TSA Competency Catalog were linked to the relevant job tasks they supported (Hendrickson et al. 2010).

Once the document review phase is concluded, a job analysis will be conducted to validate the currency of the job tasks being performed and to analyze differences between the former protocol and the Optimized protocol. Twelve incumbent BDO subject matter experts will be sampled from airports around the country, with the minimum qualifications of at least 1 year on the job, presently holding the BDO position, and being trained in both BD and Optimized indicator screening methods. A minimum of 6 months of job experience has been suggested as a rule of thumb (Gatewood and Field 1994; Harvey et al. 2007) though the experience necessary is expected to vary from job to job and BDO work is highly cognitive.

The national sample would be selected to increase the representativeness of the SME sample for the national population. The national sample will also help transcend regional and airport specific training differences. The SMEs would convene as a group in a conference room for 2 weeks. Each SME would be given a laptop with custom job analysis software loaded on it – created to leverage the existing BDO job tasks identified in the OJAS and HPRA.

The researcher would lead the team of SMEs through the existing job tasks and add additional job tasks as needed (i.e., tasks specific to the optimized indicator protocol). During this time the researcher would also add any subtasks below tasks that were not fully described; identify a list of KSAOs that supported all of the tasks and subtasks; and identify the equipment, machines, tools and technology (EMTTs) that supported BDO job task performance.

Once all additions and modifications to the list of job tasks had been made, the list would be uploaded to their laptops. The SMEs would then, working independently, screen each job task and related subtasks in a series of steps to (a) Verify it is accurate and current; (b) Check boxes to indicate if it applies “only to Optimized” or “only to BD” (the default would be that it applies to both); (c) Identify all of the relevant KSAOs that supported each task; (d) Identify all of the relevant EMTTs that supported each task; and finally the SMEs would rate each task based on its difficulty, importance and frequency (DIF analysis), and complexity. The difficulty and complexity ratings will serve to ensure the following cognitive task analysis sufficiently explores the highly difficult and/or complex tasks (Fig. 1).

Fig. 1.
figure 1figure 1

Sandia’s existing job analysis software, formerly used with S/L/TSOs, would be tailored to the BDO job

Full size image

Once all of the SMEs complete the task of linking and rating the job tasks, the researcher will lead the group in conducting a CTA. Prior to conducting this CTA, the researcher will coordinate with TSA to understand its specific downstream use of the CTA outputs. This will ensure that all relevant and necessary data is collected during the CTA.

The CTA will be used to map out the chronology for various sequences of job tasks and sub-tasks, identify where decisions are made, what criteria on which those decisions should be based, and in general where the cognitive work is being done (Crandall et al. 2006; Jonassen et al. 1999). To complete the CTA, the researcher would begin to create a chronological flow diagram of job tasks and subtasks in a software package (i.e., Euterpe, Hierarchical Task Analysis, Cognitive Analysis Tool, Microsaint, or MS Visio). The SMEs would guide the creation of the task chronologies adding to and modifying the design until it was complete. Disagreements in the design would be discussed until there was a group consensus (Fig. 2).

Fig. 2.
figure 2figure 2

Generic example from a Hierarchical Task Analysis

Full size image

Once the observable tasks were accurately diagrammed, the researcher would walk through each sequence with the SMEs and add all of the unseen tasks (e.g., mental processes, planning, calculations, judgments, strategies, decisions, knowledge accessed, etc.) into the sequence while noting any critical cues, potential errors, strategies used, and branching in various directions as when decisions lead to different actions (Militello and Hutton 1998).

To help the same twelve incumbent BDO subject matter experts think through specific job task processes, related critical incidents will be collected from their group during interviews and then the group will be lead through a discussion to identify the specific steps in decision making and all the variables that were considered. Once the cognitive task map is completed, the SMEs will independently rate the cognitive tasks for difficulty, importance, frequency and complexity. Complex tasks are defined as those that have many steps or considerations, while task difficulty pertains to the likelihood of doing it incorrectly or producing the wrong outcome (Fig. 3).

Fig. 3.
figure 3figure 3

Math-based example of complexity and difficulty interactions

Full size image

Once the CTA is completed, the researcher will validate the data by observing SMEs on the job for a few days and asking questions about why they did various things or made various decisions. The observed cases and answers to the questions will then be mapped to the CTA diagram to demonstrate alignment between the map and what was observed.

The final products of the CTA will be the task process map and a Cognitive Demands Table. The Cognitive Demands Table will identify all difficult cognitive tasks, why it was difficult, any common errors related to that cognitive task, and the cues and strategies the BDOs utilize to successfully complete a task. It will also include a chart comparing the identified differences in cognitive tasks between the BD and Optimized indicator protocol screening methods, allowing one to infer the relative shift in cognitive load demands.

4 Discussion

Whether using existing BD or Optimized indicator protocol, these screening methods are cognitively demanding. Conducting the job analysis and CTA using the described methods will ensure that all of the observable and unobservable tasks are not only identified but also related to one another and qualified with important information. The outputs of this important work will ensure that the TSA leadership has all of the information they need to assess and compare the SPOT and Optimized screening methods. This research could also provide valuable information to help the newly created Optimized training evolve over time to most effectively train BDOs to make the best decisions at the decisions points, and increase training time for the most difficult job tasks. Future empirical research could build on this BDO CTA by measuring BDO cognitive load in the field, in real time, while officers perform the most cognitively demanding tasks. The job analysis and CTA could also be used to create better tests and measures for selecting future BDO candidates in order to maximize the human capital and minimize BDO attrition – lowering program-wide BDO training costs.