Keywords

1 Introduction

Process modeling is a cognitively challenging activity, strongly subject to individual differences in performance [6, 12, 20, 25, 27]. Nevertheless, to the best of our knowledge we currently have no systematic, objective way of analyzing the cognitive aspects of modeling skills as it occurs in the practice of IT. Recent lab evidence shows positive correlations between working memory capacity (WMC) and process modeling quality [16], but correlations can only be taken as incentives to further explore such facilitating mechanisms.

In this study, we pose the following research question: Which variables are essential to observe in a method to analyze individual variability in cognitive skills in modeling sessions? Our literature review has revealed a critical role for abstraction, relational reasoning and executive control in modeling [27]. Compatible with [16], evidence suggests that WMC expresses itself through executive control [10, 18] and is a critical facilitator of both abstraction [3] and reasoning [13]. In this study, we validate these results firstly by comparing the findings to the opinions of experienced modeling practitioners, and secondly by checking whether the findings can indeed be made observable in real modeling sessions. Finally, based on all our results, we develop an observation method for systematic analysis of modeling skills. This method should provide more insight in the context in which WMC is expressed by describing how reasoning, abstraction and executive control manifest themselves in relation to each other and in relation to the modeling process. This may give us insights in why modeling performance is so individually variable, and allow us to train weak modelers in the specific skills strong modelers exhibit most.

We begin by reviewing our key variables. Secondly, we describe our data collection process and how we integrated the results with existing models to create an observation scheme for behavioral analysis. Finally, we discuss implications for future research.

2 Abstraction

Abstraction is one of the most difficult and most important modeling skills [12, 23, 25]. The overarching term ‘abstraction’ refers to the process of performing mental operations and simulations on a set of related objects without the objects in question being present [19]. ‘Abstraction’ as a noun encompasses the static component: mental representations. ‘Abstracting’ as a verb relates to mental operations, such as instantiation and generalization, which can be applied to any mental representation on any level of abstraction. In modeling, domain comprehension on an abstract level improves performance [15]. Additionally, abstract comprehension encourages engagement in problem solving behaviors, such as testing the consequences of model facts, which in turn improves overall model quality [6].

Most of the literature focusing on quantifying abstraction has classified abstraction into different levels, based either on mental imagery triggered by the concept [8, 22] or on observation of neural activation in response to semantic prompts and relational reasoning tests [3]. All level classifications begin with a highly concrete level, which is defined as a richly detailed mental copy of the real object. Then, there are two or three gradually more abstract levels: one or two medium levels of abstraction, which encompass generic names of objects allowing us to know what one means well enough to hold a comprehensible conversation, and a high level of abstraction which is devoid of most detail. When talking on this level, if one does not know concrete domain processes and underlying infrastructure, comprehension is impossible. [21] specifically mentions that goal and focus of abstraction levels shift as they change; each level of increased abstraction shows different details which serve to specify certain systemic functions.

3 Relational Reasoning

Relational reasoning is strongly associated with success in both modeling [6, 15] and problem solving in general [2, 25]. [4] defines relational reasoning as “the ability to consider relationships between multiple mental representations”. In essence, it comprises the operational component of abstraction: mental operations one can perform to relate or modify abstract representations. One combines experience, input from peers and existing model concepts and relations to form new representations, through a process of understanding, integration and structuring. The final result should meaningfully convey the model’s purpose. Some of the most prominently occurring reasoning processes in the literature, which we use as inspiration for our method, are making assumptions [1], drawing analogies [9], explaining [25], elaborating [2], making inferences [2], integrating [26], rephrasing [25], summarizing [11] and verifying [2].

4 Executive Control

Executive functions are a set of monitoring functions on one’s own behavior, primarily focusing on control and coordination of responses to input which might originate from the environment or from one’s own thoughts [14]. This is achieved through processes such as inhibitory control, switching, working memory updating and monitoring [18]. Executive functions lie at the heart of modeling. A modeler continuously engages in inhibition and switching as he performs such diverse tasks as deducing and testing hypotheses on how model elements interact [23, 25]. He must interpret and comprehend this information, and match his own mental representation with what other modelers are saying and writing [6]. He must be able to switch between different levels of abstraction for viewing system structures, focus his attention on different aspects of the problem in scope and regulate and monitor his selection in case of multiple simultaneous inputs [21, 27]. Moreover, modelers should not only monitor themselves, but also others as the discussion progresses [27]. At the end of the session, the modeler needs to relate the modeling goals and the users’ needs to the model created to ensure final model quality [24].

For ecologically valid, behavioral assessment of executive functions, several models exist. The main concept they share is that the different dimensions of executive functioning are all facilitated by a common underlying cognitive process, such as working memory, which allows maintenance of a goal state, and active evaluation of the current state against that goal state, to take place continuously [18]. Examples are the Behavior Rating Inventory for Executive Functions (BRIEF) [7] and educational assessment methods [5, 17]. The BRIEF focuses on both ecological and clinical assessment of behavior using the factors Behavioural Regulation (BR), Emotional Regulation (ER) and Metacognition (M), whereas the educational methods aim to assess the most important executive skills in education. A comparison of the models is shown in Table 1. It is worth noting that in [5], Sustained attention is explicitly differentiated from Working memory to draw a distinction between maintaining focus and remembering and manipulating information on the short term. Both Time management and Goal-directed persistence are scales which are not formally measured by any method in existence, because they are hard to assess within the context of a single test. Nevertheless, in real settings such as education and modeling, these are essential skills that will directly contribute to task achievement.

Table 1. A comparison of executive function models.
Full size table

5 Method

We first conducted exploratory interviews with modeling experts to verify whether their notion of essential modeling skills matched with what theory suggested to be essential modeling skills. The interviews were analyzed in a bottom-up fashion, using aspects of grounded theory. Then, we observed modeling sessions in IT industry. A small sample of these sessions was analyzed for modeling skills in the same bottom-up way as the interviews. The results from both the observations and the interviews were compared to the executive control models discussed above, from which a pilot observation scheme resulted. This scheme was tested on four independent modeling sessions until theoretical saturation was achieved, and revised to create a final observation scheme.

5.1 Expert Interviews

Five semi-structured interviews were conducted with experienced modeling facilitators, of which one was female and four were male. The sample included IT architects with different specializations: an enterprise architect, a business architect and two application architects working for a Dutch bank. Also, a business engineer working for an international IT company was included to contribute the perspective of one who was also involved with implementing solutions. An interview guide was prepared with main questions and probes about how the interviewees experience the facilitation of modeling sessions, what they consider to be critical modeling skills, different types of stakeholder responses and how they deal with them. The researcher provided scenarios, such as ‘what would you do if you notice a participant in your session who does not manage to follow along’, to stimulate the interviewees to think about what they would do or consider most important in such cases. All interviews were audio-recorded, by consent of the interviewees. Free talking was encouraged, with the researcher only probing if further information was desired, or to keep the interviewees within scope.

The interviews were analyzed directly from the audio files with Atlas.ti, following a grounded theory approach. Transcription was bypassed because the essential meaning was conveyed by the broader discussion of topics, not via formulations on word level. Firstly, open coding was applied to the interviews. No specific unit of analysis was defined, codes were assigned to fragments of speech which the researcher considered representative for the code in question. A few examples of codes assigned to utterances (translated from Dutch) can be found in Table 2. After open coding, the codes were grouped during a phase of axial coding according to the emerging categories. Resulting codes were discussed with an experienced IT professional for extra validation.

Table 2. Examples of codes assigned to utterances during the interviews.
Full size table

5.2 Observations of Modeling Sessions

We included observational research in our study because real-life modeling is subject to many influences currently still unknown to us. For both codebook construction and testing for theoretical saturation, a total of seven modeling sessions were analyzed. One session took place at a Dutch bank and included an IT architect, a program director and a program manager. No video recordings were allowed but the researcher present wrote up one elaborate report immediately after the session, describing actions done by session participants. Codes were assigned to those actions, and to described responses by other participants. For example, the reported sentence of “MV immediately began pointing out errors in the Archimate model, mostly pertaining to teams that no longer existed or had been merged” was coded as error monitoring.

The other sessions took place over the course of three months as part of a larger IT project in a Dutch organization, active in the collective sector. The sessions included a business analyst, a project leader, an architect and a change manager. Camera and audio recordings were made with all participants’ consent. A camcorder on a tripod was put up in a corner so that it would capture as much of the scene as possible, without it being too obtrusive for the participants. The researcher had been present at all sessions in a non-obtrusive manner. Before the observations, the researcher had met and talked with all participants to get to know them and get them accustomed to her presence. No interventions were done during the sessions. The observations took place at the organizations’ offices, and were typically rooms with whiteboards and brown paper sheets attached to the walls. Participants were free to make sketches and notes in this way. The final products were photographed, and resulting digital documentation was also collected.

The recorded sessions were fully transcribed. We directly coded the utterances of all participants using a grounded theory approach with Atlas.ti. A phase of open coding was followed by two cycles of axial coding. During open coding, the unit of analysis was a participant’s full turn, terminated only by an interruption or a natural reaction from another participant. Pauses between speech but still continued by the same participant were taken to belong to a single turn unless they exceeded 10 s. In addition to the emerging codes, each turn was specifically assigned a level of abstraction, to monitor the flow of abstraction levels throughout the discussion. A certain amount of bias in formulating codes resulted from the literature study on executive functions, but additionally, many new codes were formulated which did not appear as such in existing literature. An example of a coded fragment (translated from Dutch) can be found in Table 3.

Table 3. Examples of codes assigned to a discussion fragment during the observations.
Full size table

6 Results

After categorizing the emerging codes from both the interviews and the observations, we compare the categories emerging from interviews and observation to those found in the literature to construct a pilot observation scheme.

6.1 Pilot Observation Scheme

An overview of the resulting categories and codes obtained from the interviews and the observations compared to the assessment items provided by existing schemes described in the literature is provided in Table 4.

Table 4. A comparison of the categories from the interviews, observations and literature study.
Full size table

The categories in the pilot scheme can be defined as follows:

  • Goal-directedness: Any act relating to any goals of the modeling session. These can for example be modeling goals, planning goals or organizational goals.

  • Initiation: Any act relating to the start of a new task or discuss a new topic.

  • Working memory: Any act in which the modeler returns to a previously mentioned topic or repeats and manipulates previously mentioned information.

  • Shifting: Any act relating to a switch in related topics or perspectives without deviating from the main focus.

  • Monitoring: Any act relating to monitoring the progress of the session, the structure or content of the model, the way utterances are related to set goals, comprehension of other modelers, guarding discussion scope and error monitoring.

  • Reasoning: Any process of considering multiple mental representations in relation to each other.

  • Abstraction: Any act of observing processes in more detail to gain better understanding, or in less detail to gain better overview of the whole picture.

The categories of Goal-directedness, Initiation, Working memory, Shifting and Monitoring, appear in the interviews, observations and the literature. It thus seems justified to keep them as categories for the final coding scheme.

Response inhibition as measured by the items in the BRIEF or the educational models is extremely difficult to implement, as the educational models are tailored to children whose inhibitory control is still developing, and most of the BRIEF items, such as distractibility or impulsivity, are also not observable in modeling sessions. The only items which could be observed were if people broke off sentences halfway. This behavior appeared meaningless in the context of modeling, therefore we follow [17] and do not include inhibition as a separate category.

Reasoning and abstraction are both complex cognitive processes facilitated to a significant extent by executive functions, but are not considered executive functions themselves by existing measurement methods. In some studies, reasoning and abstraction are used as ways to observe executive function strength. They are both critical processes in modeling, and therefore we will include them in the coding scheme as the two main variables which will be examined in relation to the different executive functions.

Communication/People skills and Modeler characteristics were factors people only talked about in the interviews, when discussing their experiences from a generalized point of view. When observing sessions, such factors cannot be directly observed when the unit of analysis is defined as a single turn. There thus seems to be no reason to include them.

The factors Time management, Planning, Organization and Emotional control typically span an individual’s functioning across a great many tasks. Within the context of a single modeling session, these factors were also not observable, and hence we have decided to exclude them from the final coding scheme.

6.2 Final Observation Scheme

We applied the pilot observation scheme to four further modeling sessions, taken from the same project, to achieve theoretical saturation.

Firstly, we found that coding Abstraction only in terms of concrete, medium and abstract levels did not capture the essence of the variable. For example, a modeler could be talking on a highly abstract level yet not be able to formulate the type of elegant solution that would solve a modeling problem. Therefore, we further refined Abstraction into semantic and relational abstraction, as in [3]. Semantic abstraction refers to the abstraction level of the words in an utterance, relational abstraction refers to the number of relations between the concepts discussed in an utterance. In this way, it became clear that, for example, semantically abstract concepts can be used in concrete relations, and that this, if not instantiated, can easily hide poor comprehension.

Secondly, we refined Reasoning into comparative and transformative reasoning processes. Transformative processes change the presentation of information, but preserve the essence of its meaning. It includes rephrasing and summarizing, but also instantiation and generalization, keeping the Abstraction category purely for classifying the abstraction level of an utterance. Comparative processes use two or more sources of input to derive some consequence for the next step in the reasoning process. Examples are inferencing, verifying, assuming and analogy. Furthermore, we merged codes which had significant semantic overlap and were indistinct in practice, such as elaborate and explore.

Thirdly, we found that aspects of communication, such as different forms of backchanneling, were after all critical to determine a modeler’s initial reaction to a peer’s utterances. We hence added Communication as a supportive category.

Finally, we found that goal-related utterances occur either within the context of a monitoring act, such as monitoring previously set process goals, or a planning act, such as articulating future session goals. Additionally, in these sessions another participant made heavy use of other planning aspects such as organizing modeling progress and articulating future actions. Therefore, we chose to eliminate Goal-directedness as an independent category, to add a Planning category into the coding scheme and to place the goal-related acts under both Monitoring and Planning.

The final coding scheme thus consists of the following categories (a full overview can be found on ilonawilmont.nl/codingscheme): Abstraction (Semantic, Relational), Reasoning (Comparative, Transformative), Initiation, Monitoring, Planning, Shifting, Working memory, Communication.

7 Discussion and Future Research

Our results strongly suggest that some aspects of executive control, in particular Monitoring and, to some extent, Shifting, are more clearly observable in this context than the more fundamental aspects such as Inhibition, Working memory and Emotional control. This does not mean that the fundamental processes do not play an important role, it simply shows that defects in the fundamental processes may no longer be so obvious in a working context as they might have been when they were still developing in a school setting. Nevertheless, this may be indicative of appropriate inhibition given the context. In future studies, we will therefore have to examine the relation between individual measures of both fundamental and metacognitive executive functions, abstraction and reasoning, and the behavioral patterns obtained through analysis with our method.

The scope of this study was limited to making observable abstraction, reasoning and executive control processes in a real modeling setting. In future work, we will analyze this data to come to actual patterns of modeling behavior. A potential technique for this is process mining. When assigning the quotations in Atlas.ti directly to the video fragment, one also has the duration of a cognitive process, and a process mining tool could then show how long people engage in certain cognitive processes, and how they consecutively follow up on each other. Analysis should also focus on the relations between the different categories and individual codes to understand the collaborative process of modeling. Which codes co-occur most frequently? What is the effect of this cluster of behaviors on other modelers in the session? Do they follow and continue the line of reasoning, or do they apply corrections?

Finally, at the current stage of research, we are left with the issue that we have no full inter-rater reliability score for the coding scheme. This will be resolved once we have applied the coding scheme to our full dataset of observations.

8 Conclusions

We have described how we have developed an observation method to analyze critical modeling skills, in particular abstraction, reasoning and certain executive control functions in real modeling sessions. We have compared results from expert interviews, observations and the extensive literature on abstraction, reasoning and executive control to construct an observation scheme. The resulting categories from the different data sources were remarkably consistent, of which Abstraction, Reasoning and Monitoring were most prominently present. Examination of the relations between codes and consecutively occurring groups of codes promises to provide insights in how psychological processes facilitate collaborative modeling.