Demonstrating the use of a professional skills framework to support the assessment of dispositions in IT education

Abstract

The skills and competencies of IT professionals are often described using employer-led skills frameworks. They express competencies as technical knowledge and skills combined with a range of personal qualities. Employers have indicated the importance of developing such qualities for new graduates. In response, recent ACM/IEEE curricular recommendations have shifted their emphases from bodies of knowledge to the development of competencies. The IT2017 ACM/IEEE Curriculum Guidelines for Baccalaureate Degree Programs proposed a model of IT competency comprising three interrelated components: content knowledge, skills, and dispositions, where dispositions represent personal qualities desirable in the workplace. The ACM/IEEE Computing Curricula 2020 (CC2020) report enriched the IT2017 disposition concept by identifying eleven dispositions that all computing programs should include for the career preparation of their graduates. However, developing and assessing dispositions in a degree program remain challenges, often involving internships, work placements and similar student opportunities. A recent mapping of the eleven CC2020 dispositions to the responsibility characteristics of the Skills Framework for the Information Age (SFIA), a widely used professional skills framework, suggested a promising approach to addressing this challenge. Inspired by this mapping, this paper aims to help educators assess students’ achievement of CC2020 dispositions by mapping real-world experiences they have recorded in individual portfolios against the SFIA responsibility characteristics. First, the selection of SFIA to operationalize the CC2020 dispositions is validated by demonstrating that alternative frameworks pose significant challenges for any assessment approach that needs to be independent of particular technical skills. A tool is described that maps demonstration of SFIA responsibility characteristics to CC2020 dispositions, applying a simple, consistent assessment algorithm. Finally, the assessment process and outcomes are illustrated using a fictional student portfolio, constructed to reflect one author’s experience of work placement students’ achievements.

1 Introduction

The competencies gap between the academic programs’ supply of work-ready graduates and what employers demand has been amply documented (Bauer-Wolf, 2018; Hart Research Associates, 2015; Shadbolt, 2016). Factors that contribute to the gap include lack of work experience, lack of industry and business awareness, and lack of social-emotional, intrapersonal, and interpersonal characteristics (Shadbolt, 2016). The ACM/IEEE-CS IT2017 and CC2020 curricular reports (Clear et al., 2020; Sabin et al., 2017) define the concept of computing competency and use competencies to frame curricular guidelines. Competencies give academic programs and employers a common language to express what graduates should achieve by the completion of their program of study and what new hires should be ready to demonstrate in their new computing professional roles.

The IT2017 and CC2020 competency definitions distinguish three interrelated components of competency: knowledge, skills, and dispositions. While the first two have long been familiar to educators, developing dispositions in students is uncharted territory. Particularly challenging is how to teach and assess dispositions when realistic and authentic tasks and environment settings are not clearly stated. The CC2020 report identifies eleven dispositions and recommends that teachers “should instill [them] in their students” (Clear et al., 2020, p.51). The report, however, does not describe concrete contexts specific to each disposition. The description of a professional skill that students should practice and develop needs to be complemented by the description of corresponding personal and behavioral characteristics that students should manifest. To enable the teaching and assessment of skills and dispositions, their descriptions need to be contextual and operational, that is, contextualized by clearly stated tasks and environment settings. Otherwise, students cannot engage effectively in tasks that give them the opportunity to develop and demonstrate workplace competencies if such tasks are not relevant and are not clearly described.

Employer-led, professional skills frameworks, such as the Skills Framework for the Information Age (SFIA), offer operational descriptions that integrate skills with personal and behavioral characteristics and situate their practice and manifestation in workplace tasks. Two foundations are required for the assessment approach proposed in the paper. First, that among the widely used IT skills frameworks, SFIA is the most suitable choice; and second, that the dispositions that are being assessed can be mapped to the SFIA responsibility characteristics.

In this paper, we analyze and evaluate the professional skills frameworks most commonly used in IT to determine the extent to which each is appropriate for assessing CC2020 dispositions. The comparison of current candidate frameworks shows that SFIA is the most suitable choice. We then demonstrate how the SFIA Level 3 responsibility characteristics can be incorporated into an assessment tool to support indirect assessment of the CC2020 dispositions. Student reflective portfolios, typically compiled during an industrial work placement or internship, are the source of assessment data expected to evidence the manifestation of the SFIA Level 3 responsibility characteristics. Using the mapping from Bowers et al. (2022b), and the weighting algorithm suggested in Bowers and Sabin (2022), we show that this approach offers indirect assessment of the CC2020 dispositions.

The contribution of this paper is to demonstrate how a portfolio can be assessed for evidence of the SFIA level 3 responsibility characteristics and thus, indirectly, for demonstration of the CC2020 dispositions. A portfolio exemplar is used to illustrate the application of the tool. By making the tool freely available, we invite the reader both to explore its use and to share their experiences, to build a more extensive evidence base.

The next section overviews related work that motivated and informed the selection of SFIA and the use of SFIA responsibility characteristics to assess dipositions. In Section 3 we confirm that SFIA is the only realistic option amongst the range of potentially suitable professional skills frameworks to appropriately support the assessment of dispositions. Section 4 reviews the assessment method and the design of the assessment tool by which the accomplishment of CC2020 dispositions are measured indirectly, using the SFIA behavioral characteristics. Section 5 illustrates the application of the tool to the exemplar portfolio. Discussion and concluding remarks are in Sections 6 and 7.

2 Related work

2.1 Competencies in the workplace

Shadbolt was not alone in highlighting the importance of dispositions to recent graduates’ success in the workplace (Shadbolt, 2016). The survey of hiring managers and executives conducted recently by the Association of American Colleges and Universities (AACU) explored employers’ views on a new set of competencies (Finley, 2021). Referred to as “dispositions”, these competencies represent mindsets and personal qualities, such as persistence, work ethic, and curiosity for lifelong learning, that complement and enable technical competencies.

Another recent survey conducted by the management consulting firm McKinsey & Company (Dondi et al., 2021) collected data from 18,000 participants to identify the set of skills that help people succeed professionally. The total of 56 skills were organized in 13 groups, including “mental flexibility”, “developing relationships”, and “self-awareness and self-management”. The study found that respondents with higher likelihood of employment had higher proficiency in competencies such as adaptability, coping with uncertainty, synthesizing messages, and achievement orientation. Higher income and job satisfaction outcome were also correlated with competencies such as self-confidence, organizational awareness, and work-plan development. Both the AACU and McKinsey & Company survey studies signal that these competencies are not explicitly taught in higher education. They recommend that academic program curricula integrate high-impact learning practices and appropriate assessments that help students develop competencies that matter in the workplace.

2.2 Competencies in education

The IT2017 curricular report conceptualizes IT competency as the intersection of three components: content knowledge or “know-what”, skills or “know-how”, and dispositions or “know why” and “know yourself” (Sabin et al., 2017, 2018). The report’s motivation for reframing IT curricular guidelines in terms of IT competencies instead of IT content knowledge areas stems from the IT discipline’s emphasis on applied practice and experiential learning. The tight connection between knowledge and skills has a large presence in the education literature, but studies of competencies in computing are just emerging. The CC2020 report (Clear et al., 2020) played a transformational role in generalizing the IT competency model for all computing disciplines and by framing their computing curricula in terms of competencies.

ABET defines student outcomes as “what students are expected to know and be able to do by the time of graduation” as evidenced by the “knowledge, skills, and behaviors” they develop during their program of study (ABET, Inc., 2023). The interpretation of these outcomes through the lens of competencies draws attention to professional dispositions that contribute to competencies, such as effective communication in professional contexts, recognizing professional responsibilities, and making informed judgments in computing practice (Raj et al., 2022).

Competency-based education is prevalent in professional education, such as teacher preparation, legal education, accounting, medical education, and health sciences education (Boritz and Carnaghan, 2003; Frank et al., 2017; Sparrow, 2003). Computing education has also signaled increased attention to competencies in recent years. For example, two working groups hosted by the Innovation and Technology in Computer Science Education (ITiCSE) conferences developed a framework for computing competency learning (Frezza et al., 2018) and explored how to design computing competency statements for academic programs (Clear et al., 2020).

2.3 Criterion-based assessment and reflective portfolios

Highly relevant to this paper is the ITiCSE 2021 working group paper Professional Competencies in Computing Education: Pedagogies and Assessment (Raj et al., 2021), which focuses on the assessment of skills and dispositions and stresses the importance of considering context and employer-led skills frameworks. The paper features a criterion-based approach that the Institute of Coding (IoC) in the UK developed to assess student demonstration of technical skills and student reflection on their professional achievement¹. The IoC criterion-based approach promotes the use of reflective portfolios, in which students record evidence of their professional practice. As known from literature on teacher education and health education (Driessen, 2017; Jones, 2013; McKenna and Connolly, 2011; Pecheone et al., 2005), student portfolios are key to the assessment of work-based learning. The novelty of the IoC approach is the mapping between the descriptions of learning experiences recorded in the portfolio and the SFIA technical skills that students develop through those learning experiences.

Assessing student learning using SFIA descriptions of professional skills is strongly supported by the definition of competence in ISO 24773 (ISO, 2019): “Competence involves the ability to apply knowledge and skills [...] in order to achieve a successful result on an ongoing basis [...] apply[ing] sound judgement, mak[ing] correct decisions, apply[ing] the appropriate skills and knowledge and mak[ing] use of relevant professional attributes.” This definition alludes to how the IT2017 report  (Sabin et al., 2017, p. 31) defines disposition, as “personal qualities (socio-emotional skills, behaviors, attitudes) associated with success in college and career”. The CC2020 report (Clear et al., 2020) identifies and elaborates on the meaning of eleven dispositions, but the descriptions are abstracted from the behaviors required in the real world. Both reports leave it to educators to work out how to operationalize dispositions through appropriate tasks and settings.

2.4 Mapping SFIA responsibility characteristics to dispositions

To define associations between professional skills in employer-led skills frameworks and learning experiences that target professional skills remains challenging. Such associations might be too course-specific and may not transfer to other courses. Educators need to become familiar with a professional skills framework. Different educators may have different interpretations of potential associations. One way to address these obstacles is to rely on a validated mapping of the computing competencies recognized by the computing education community and an appropriate professional skills framework.

Table 1 SFIA characteristics mapped to the Adaptable disposition along with partial contributions to remaining CC2020 dispositions (from Table VI in Bowers et al., 2022b, p. 6)

The recent study of the eleven CC2020 dispositions through the lens of the 24 SFIA Level 3 responsibility characteristics (Bowers et al., 2022b) investigates the many-to-many relationships between the two frameworks and validates their mapping. Table 1 shows the mapping for a single CC2020 disposition, Adaptable, by listing the SFIA characteristics that contribute to it. The table also shows how each characteristic mapped to Adaptable also partially contributes to some of the other remaining CC2020 dispositions. Another measure of interest to the proposed assessment method (as it is described later in the paper) is the total count of dispositions that each characteristic supports. This measure is shown in the last column in Table 1. A full description of all 24 SFIA Level 3 responsibility characteristics and their mapping to the eleven CC2020 dispositions is depicted in Table VI in  (Bowers et al., 2022b, p. 6) and reproduced in this paper in Appendix A, Table 16.

2.5 Summary of related work

In summary, the related work in this section paved the way to the contributions presented in our paper by offering the following guidance. To characterize and measure practitioners’ performance on the job, employers use competencies. Although professional education has normalized the integration of competencies in their curricula and assessment approaches, computing education has only relatively recently considered competencies in developing curricular guidelines for computing academic programs. Of particular relevance to our work was the 2021 ITiCSE working group paper (Raj et al., 2021), which illustrates the assessment approach developed by the Institute of Coding (IoC) in the UK to evaluate achievement of SFIA professional skills. Three elements of the IoC assessment approach were critical to the proposed assessment method in this paper: a reflective portfolio to collect assessment data, a criterion-based approach to evaluate portfolio evidence, and SFIA’s skills descriptions that operationalize how to assess the achievement of competencies.

With regard to the dispositional component of competency, it has been beneficial that the CC2020 report defines eleven dispositions expected of computing graduates. However, the dispositions are defined abstractly, with no reference to the kinds of tasks and settings that would afford behaviors by which learners have the opportunity to manifest those dispositions. That is where the mapping between SFIA responsibility characteristics and CC2020 dispositions (Bowers et al., 2022a) was instrumental to the development of our assessment tool.

3 Skills frameworks

The choice of SFIA responsibility characteristics to operationalize dispositions for assessment purposes is critical to our approach. This raises two questions:

• 1. Are there other employer-led skills frameworks similar to SFIA that our assessment method might use?

• 2. How does SFIA compare with those frameworks?

In other words, we need to determine how appropriate SFIA is for assessing IT competencies. We address these questions in this section.

Employer-led, or “professional”, skills frameworks typically describe both technical activities and behavioral characteristics valued by employers. Whilst the scope of the technical activities may be focused, e.g., on cybersecurity, the behavioral characteristics will often be relevant to any area of IT. In addition, behavioral characteristics in professional skills frameworks are usually expressed in terms of concrete actions in contrast to the abstract concepts represented by CC2020 dispositions; it should therefore be easier to decide whether or not an individual has demonstrated particular behavioral characteristics than it is to assess the extent to which that individual has demonstrated a particular disposition.

Expressing dispositions in terms of behavioural characteristics drawn from a professional skills framework focuses the dispositions on behaviors that are valued by employers. This leverages the concept of disposition, ensuring that their realization enhances graduates’ employability. Furthermore, basing the development of dispositions on a professional skills framework, ideally in a real-world working environment, enables teachers to 1) expose students to experiences that facilitate the development of dispositions and 2) assess, summatively or formatively, students’ dispositions through their engagement in workplace activities.

3.1 Skills framework criteria for mapping dispositions

For a professional skills framework to support the mapping of the full range of CC2020 dispositions, four criteria must be satisfied.

Breadth of technical content. The framework must be relevant across the whole spectrum of IT programs. Given the breadth of the IT domain, and that IT job roles may combine skills from several areas of the domain, it follows that IT programs are likely also to vary widely in both content and focus. For a single skills framework to be useful to the majority of IT programs, it needs to describe a broad range of technical skills.

Separation of behavioral characteristics. Behaviors corresponding to the CC2020 dispositions are common across nearly all technical areas. Thus, the behavioral characteristics articulated in a professional skills framework must be separable from the descriptions of technical skills if they are to correspond to the CC2020 dispositions. Separating them makes them applicable to all skills, rather than being embedded within particular skills. It also reduces duplication in the framework and ensures consistency - effectively, normalizing the descriptions in the framework.

Levels of behavioral characteristics. Professional skills frameworks describe the behaviors needed by practitioners as they progress through their entire careers from new hires to, potentially, senior management positions. As practitioners gain more experience, their demonstration of dispositions will evolve and mature. Effectively, there will be different levels of behavior, depending on experience and responsibilities. To describe graduate professional behavior, corresponding to the demonstration of CC2020 dispositions, there needs to be a consistent description of the behaviors expected of a new graduate. There will probably be other descriptions also, for example, for assistants, team leaders, managers or strategic leaders.

Relationship to CC2020 dispositions. A rather obvious requirement is that the behaviors articulated in a professional skills framework need to provide full coverage of the CC2020 dispositions.

Hence, for a professional skills framework to be able to provide support for the development and assessment of CC2020 dispositions, the framework should meet the following minimum criteria:

• 1. There is broad coverage of a sufficient range of IT technical skills

• 2. Characterization of behaviors is separate from the descriptions of technical skills and task-specific activities

• 3. There is a grouping of behavioral characteristics appropriate for new graduates.

• 4. The range of behavioral characteristics can manifest all eleven CC2020 dispositions.

3.2 Candidate professional skills frameworks

In this section we describe the key features of the most commonly used skills frameworks in the IT sector. The extent to which each meets the criteria set out in Section 3.1 is summarized in Section 3.3, Table 3.

3.2.1 SFIA

First published in 2000, SFIA has grown to become, “the globally accepted common language for the skills and competencies for the digital world” (The SFIA Foundation, 2021c), with an ever-growing user base spanning over 180 countries. Version eight was published in September 2021. SFIA is industry and business led, and is “owned” by its global user community.

The framework is structured in seven different levels of responsibility and experience, orthogonal to the technical skills. The seven levels are: Follow (1), Assist (2), Apply (3), Enable (4), Ensure (5), Initiate (6) and Set Strategy (7). At each level, a set of “responsibility characteristics” is grouped under five broad generic attributes of autonomy, influence, complexity, knowledge and business skills (The SFIA Foundation, 2021a). Table 2 lists all 24 responsibility characteristics for SFIA Level 3, grouped by the five generic attributes.

SFIA Level 3 “Apply” is appropriate for new graduates, who should be capable of applying the knowledge and skills they have gained during their program of study in an entry-level position in the computing field. The responsibility characteristics for Level 3 have been shown to provide complete coverage of the 11 CC2020 dispositions (Bowers et al., 2022b).

Table 2 SFIA Level 3 responsibility characteristics grouped by generic attribute

3.2.2 Other relevant IT skills frameworks

In addition to SFIA, additional skills frameworks considered included e-CF (European Union), SWACOM and SWECOM (IEEE), i-CD (IPA, Japan) and NICE (US).

The European e-Competency Framework (e-CF) was developed by the European Committee for Standardization (CEN) as an alternative to SFIA. e-CF defines five levels of proficiency, characterized by increasing levels of context complexity, autonomy, influence and typical behavior (IT Professionalism Europe, 2022). However, e-CF characterizes the relevant proficiencies required for each level in the context of specific competencies (skills), rather than independently. Unfortunately, these characterizations specifically exclude behavioral skills, because e-CF seeks to capture only what is unique about ICT competencies (CEN/TC, 2019, App B S B5.1 p. 70). Furthermore, e-CF delegates responsibility for graduate-level behavioral skills to the responsibility and autonomy outcomes for a level 6 qualification in the European Qualification Framework (EQF) (CEN/TC, 2019, p 6): Manage complex technical or professional activities or projects, taking responsibility for decision-making in unpredictable work or study contexts; take responsibility for managing professional development of individuals and groups.

The Software Assurance Competency Model (SWACOM) describes technologies and processes to ensure that software systems function in the intended manner, are free from vulnerabilities, provide appropriate security capabilities, and can recover from failures. The Software Engineering Competency Model (SWECOM) (IEEE Computer Society, 2014) was developed for the Software Engineering Body of Knowledge (SWEBOK) (Bourque and Fairley, 2014; Fairley et al., 2014). Both SWECOM and SWACOM describe three elements - knowledge, skills and effectiveness (SWACOM) / ability (SWECOM). The last of these three comprises a set of attributes similar to the CC2020 dispositions. However, although each framework specifies five competency levels, only the technical skills are described at different levels, with no reference to the dispositions.

The i-Competency Dictionary (i-CD) is maintained by the Information Technology Promotion Agency (IPA) on behalf of the Japanese government. Published in 2014 (IPA, 2017, p11). i-CD comprises two multi-level dictionaries of tasks and skills, linked through a “task” $\times$ “skill” table identifying the skills required for particular tasks. The lowest levels of the two dictionaries comprise, respectively, some 2,200 assessment items and 10,000 knowledge items. There is no sense of leveling associated with tasks or skills, and nor is there any separation between technical and behavioral skills. Furthermore, there are no defined groupings of interpersonal skills relevant for all individuals across all task profiles.

The US National Initiative for Cybersecurity Education (NICE) Framework defines a “reference taxonomy” for both cybersecurity work and cybersecurity professionals, and is intended to support “an integrated ecosystem of cybersecurity education, training, and workforce development” (Petersen et al., 2020). The NICE framework describes a Competency Area as a measurable cluster of related Task, Knowledge, or Skill (TKS) statements in a particular domain (Wetzel, 2021). The draft NICE Competencies list includes some Skills identified as “professional”, or alternatively “employability” or “soft skills”, which are akin to dispositions. The authors of the NICE Framework are engaged in discussions about introducing both dispositions and levels of responsibility, which would make the framework usable to assess CC2020 dispositions.

3.3 Comparison of candidate frameworks

Table 3 summarizes the extent to which the major skills frameworks that span broad areas of IT meet the requirements identified in Section 3.1. Only SFIA currently meets all the requirements. The primary omissions for the other frameworks concern the separation between behavioral factors and technical skills. However, it is possible that future versions of the NICE cybersecurity framework may have the required separation (Petersen et al., 2020).

Table 3 Criterion-based comparison of skills frameworks

4 Indirect assessment of dispositions

4.1 SFIA characteristics as proxy for CC2020 dispositions

This study builds on prior work related to the design of an assessment method and development of an assessment tool for measuring the demonstration of dispositional competencies (Bowers et al., 2022a). The design requires that students use a reflective portfolio (Driessen, 2017) to record professional activities that demonstrate the achievement of dispositions over time, through repeated and successful practice. The use of portfolios for learning, assessment of competence, and professional development has been amply documented in the literature (e.g., Crowley and Miertschin 2004; Eliot and Turns 2011; Jones 2013; Patton and McGill 2006). The assessment tool is scalable, customizable, and accounts for the many-to-many relationships between portfolio entries and dispositions to be demonstrated. This means that a single portfolio entry may generate references in the assessment tool in support of more than one disposition, and that each disposition might collect references from multiple portfolio entries.

The SFIA-based approach of assessing dispositions affirms the following guiding principles:

• 1. Students demonstrate dispositions by successfully completing tasks in realistic and authentic work environments throughout their program of study.

• 2. Course and cross-curricular portfolios record students’ accumulated evidence of practicing and developing dispositions repeatedly and over an extended period of time.

• 3. The assessment tool’s data entry section references and associates each portfolio’s item to one or multiple dispositions whose demonstration is substantiated by that item. It also allows for multiple item references to provide evidence for scaffolding the development of each disposition.

• 4. The assessment method is supported by pedagogical approaches that ensure students have opportunities to practice and develop all dispositions for which they can collect sufficient evidence in their portfolios.

• 5. The assessment method allows for customizing the scoring criteria such that the evaluation is inclusive of students with differing personal circumstances and learning experiences.

Realistic and authentic work environments. The implication of the first guiding principle is that academic programs must integrate workplace experiences across curricula, from first year introductory courses to culminating capstone courses, and at different levels of granularity, whether in a course assignment, through a lab exercise, or a large, or interdisciplinary team project. Full-term internships and other work-based extra-curricular activities are typical sources of direct exposure to a professional environment. Another example is Prior Learning Assessment that is promoted in competency-based education and takes into account activities that occur outside of the traditional academic environment (Klein-Collins et al., 2020).

Portfolio-based assessment evidence. The second and third principles assert the key importance of a portfolio as an effective and necessary tool for recording evidence of realistic and authentic professional activities. The operational definition of competency stated in ISO (2019) draws attention to the critical aspect of sustained effort and practice over time in order to become competent in a professional field, that is: apply knowledge and skills [...] successful[ly] [...] on an ongoing basis. Portfolio items and their evaluation are appropriate means of capturing and validating the development of competencies over time.

Competency-based pedagogy. A learning environment that values and enables experiential learning and creates conditions for work-based learning outside the classroom is not possible without a competency-based pedagogical approach. Although it is not the focus of this study, the fourth principle underlines the tight connection between assessment and pedagogy in competency-based learning.

Inclusive evaluation. At the practical level of assessment tool design, the last principle highlights the importance of a customizable assessment method that takes into account the diversity of personal, academic, professional, and lived experiences of the students. Learning and professional development trajectories are unique to each individual, as well as the work-based opportunities students might have. The assessment method’s configuration parameters and scoring procedure allow assessors to customize the tool according to the specifics of the learning environment.

4.2 Overview of the assessment method and tool design

The assessment method and the use of SFIA Level 3 responsibility characteristics to evaluate the demonstration of CC2020 dispositions are described in Bowers and Sabin (2022). The method takes into account the dispositions of interest, their mapping to the SFIA characteristics, and the portfolio item references that evidence the demonstration of the SFIA characteristics. The tool that implements the assessment method is downloadable from a publicly-accessible repository².

The assessment tool performs two types of assessments: direct assessment of the demonstration of SFIA Level 3 responsibility characteristics; and indirect assessment of CC2020 dispositions using the SFIA characteristics and their mapping to CC2020 dispositions. The mapping, developed by Bowers et al. (2022a), is reproduced Table 16, in Appendix A. As described in Section 2 and further explained in Section 4.1, the motivation for using SFIA characteristics to assess the CC2020 dispositions stems from the characteristics’ practical and operational value and their validated relevance in the workplace and similar authentic work experiences. The proposed assessment method brings to bear the theoretical value of dispositions for the purpose of creating a tool that actually measures and evaluates the dispositions.

An important shared component between the two types of assessment is the data entry section of the assessment tool. The data entry section is structured as a table, in which the rows correspond to the 24 SFIA Level 3 responsibility characteristics and the columns contain portfolio item references recorded against each of the characteristics.

In the rest of this section we present the elements of the assessment method and how they apply to the implementation of the assessment tool. We describe first the direct assessment of the SFIA responsibility characteristics (Section 4.2.1). We use the elements of the direct assessment as a basis for describing the indirect assessment of the CC2020 dispositions (Section 4.2.2). Both types of assessments have a set of configuration parameters that are used in defining customizable thresholds. The assessment method has formulae for calculating assessment scores that are checked against the thresholds to determine the assessment outcome. Many elements of the indirect assessment of dispositions are analogous to elements of the direct assessment of SFIA characteristics. The many-to-many mapping between dispositions and characteristics introduces scoring calculations that are unique to the dispositions assessment.

4.2.1 Assessing SFIA responsibility characteristics

The assessment tool has three sections. The configuration section comprises the configuration parameters and configurable thresholds. In Section 4.2.2 will show how they compare between the assessment of SFIA characteristics and the indirect assessment of dispositions. References to portfolio items that evidence the demonstration of characteristics and, indirectly the demonstration of dispositions (as shown in Section 4.2.2), are recorded in the data entry section. The last section contains the calculation of partial scores for each characteristic, from which the assessment method derives scores for each disposition and overall scores for characteristics and dispositions.

Table 4 lists and defines the configuration parameters for assessing SFIA responsibility characteristics.

The total number of characteristics $N_C$ is the count of the number of rows in the data entry section of the assessment tool. Each row corresponds to a characteristic, which is labelled either “essential” or “supplementary”. Separating the SFIA characteristics in this manner produces $N_E$ and $N_S$ parameters. These parameters allow for customizing the assessment such that the essential characteristics be given special attention.

The $I_{Min}$ and $I_{Max_C}$ parameters are lower and upper bounds of the portfolio item references that are recorded and validated in the tool. The $I_{Max_C}$ may seem counter-intuitive: why should it not be appropriate to record that a student has demonstrated a particular characteristic many times more than $I_{Max_C}$? The reason is that allowing an arbitrarily large number of item references to be counted against any one characteristic could distort the assessment of the student’s achievement against the whole set of characteristics. Furthermore, setting a ceiling of $I_{Max_C}$ for each characteristic means that the instructor analysing the student’s portfolio can actually stop looking in the portfolio for evidence for any characteristic for which there are already $I_{Max_C}$ entries, thereby reducing the magnitude of the analysis task.

The last two parameters in Table 4, $E_{Prop}$ and $I_{Prop_C}$, are used to configure the thresholds shown in Table 5.

Table 4 Configuration parameters to assess SFIA characteristics

Table 5 Configurable thresholds to assess SFIA characteristics

The assessment method calculates three partial assessment scores. They count the item references corresponding to a single characteristics, $I_{ch}$, which is either an essential characteristic, $I_e$, or a supplementary characteristic, $I_s$, respectively.

Two overall scores that use $I_e$ and $I_s$ partial scores are the total number of item references recorded and validated across all the essential characteristics, $S_{IE}$, and across all characteristics, $S_{IC}$. They are checked against the thresholds $T_{IE}$ and $T_{IC}$, respectively. The final overall score $S_E$ calculates the number of essential characteristics that have been demonstrated. To demonstrate an essential characteristic, the following condition must hold: $I_{ch} \ge I_{Min}$.

If the overall scores meet or exceed the thresholds, then the assessment outcome is “Pass”.

4.2.2 Assessing the CC2020 dispositions

Table 6 lists and defines the configuration parameters for indirectly assessing the CC2020 dispositions and identifies the analogue parameter in the assessment of the SFIA characteristics.

Table 6 Configuration parameters to assess CC2020 dispositions (adapted from Table 2 in Bowers and Sabin (2022, p. 105)

The data entry section continues to serve the purpose of recording portfolio item references that are counted against each characteristics. However, additional configuration parameters are needed to configure thresholds for the demonstration of dispositions. Since there is no distinction between “essential” and “supplementary” dispositions in this version of the tool, $D_{Prop}$ corresponds to $E_{Prop}$. Thus, $N_D$ and $D_{Prop}$ are used to configure $T_D$ threshold, which sets the minimum number of dispositions to be demonstrated (see Table 7). $T_D$ and its analog $T_E$ are configured in the same way. $I_{Max_D}$ and $I_{Prop_D}$ retain the same meaning as $I_{Max_C}$ and $I_{Prop_C}$. Unique to assessing dispositions is $d_{Prop}$, defined as the minimum proportion of $I_{Max_D}$ item references required to demonstrate any disposition.

Table 7 Configurable thresholds to assess the CC2020 dispositions (adapted from Table 3 in Bowers and Sabin (2022, p. 105))

The mapping of dispositions to characteristics (Bowers et al., 2022b) introduces two mapping parameters:

• $n_{ch}$ is the number of characteristics mapped to a disposition.

• $n_d$ is the number of dispositions to which each characteristic contributes.

These mapping parameters are used to calculate two partial scores and one overall assessment score that are unique to the assessment of CC2020 dispositions. Table 8 lists the names, definitions, and formulae of these scores, along with applicable thresholds.

Table 8 Assessment scores for the CC2020 dispositions

The overall score that calculates the number of dispositions that have been demonstrated, $S_D$, is analogous to $S_E$ in the assessment of SFIA characteristics and not shown in Table 8. The two overall scores, $S_O$ and $S_D$, are checked against the thresholds $T_{O}$ and $T_D$, respectively. If both scores are not less than their respective thresholds, the evaluation outcome is “Pass”, otherwise it is “Not pass yet”.

5 Worked example

In this section, we illustrate the assessment of a fictitious portfolio (shown in full in Appendix B) by which a fictitious student, Jo Garcia, demonstrates the accomplishment of SFIA Level 3 characteristics and, indirectly, the CC2020 disposition. The portfolio comprises a series of dated entries, followed by the student’s overall reflection on their experience, and comments from the student’s supervisor. For a real portfolio, the student’s reflection and supervisor comments may be separate documents; they are included here as sections of the Appendix (B.2 and B.3).

Portfolio entries are considered for evidence of demonstration of SFIA characteristics. Each entry is quite long, and it is possible – indeed, likely – that a single portfolio entry will include evidence of more than one SFIA characteristic. In some cases, a single portfolio entry may provide evidence of two or more distinct demonstrations of a single characteristic. Item references are inserted in the data entry area to record characteristic demonstrations from the portfolio entries. Data entry for a characteristic stops when the required maximum of item references is reached - that is, whichever is the greater of $I_{Max_C}$ and $I_{Max_D}$.

On the basis of the number of item references entered against each characteristic, the assessment tool evaluates:

• The extent to which the student has demonstrated the SFIA responsibility characteristics

• The student’s demonstration of the CC2020 dispositions using the mapping in Appendix A and the weighting calculations summarized in Table 8.

Jo Garcia’s example portfolio has been designed to be short enough for inclusion in this paper. That is why the portfolio is most likely less extensive than one might expect from a “real” portfolio. Although it includes evidence of many SFIA characteristics, the number of portfolio entries demonstrating each characteristic is necessarily fewer than might be the case for a realistic portfolio. This is also accommodated by reducing some of the threshold levels in the assessment tool.

In the following subsections, we discuss first the analysis of a single portfolio entry. We then make some observations about the data entered for this entry and for the rest of the portfolio. Finally, we summarize the resulting assessments of the student’s demonstration of both SFIA responsibility characteristics and, indirectly, CC2020 dispositions.

5.1 Review of a single portfolio entry

We illustrate the analysis of the portfolio by considering the portfolio entry for 23rd September. This particular entry is quite rich, providing evidence for the demonstration of several responsibility characteristics.

The identification of demonstrated characteristics entails reading the portfolio entry, and noting which characteristics, if any, are evidenced by each sentence or paragraph. “Double counting” is permitted, as activities described in a portfolio entry will often demonstrate more than one characteristic simultaneously. A reference to the portfolio entry is then made in the data entry area of the tool (Table 10) against each characteristic for which it provides evidence.

Let us consider each paragraph in turn.

¶1: At least there are full specs - including data models! - for Whale Sports’s database. They’re drawn using a notation that’s a bit unfamiliar, but, as we were taught in the DM [Data Management] module, the basic concepts represented in all of the notations are essentially the same.

This paragraph indicates that the student Understands and effectively applies appropriate methods, tools, applications and processes. (B2) relating to data models and notations. Furthermore, the student is aware of the value of complete specifications, further demonstrating the same characteristic.

¶2: Small Fry, however, is a bit more of a problem. There’s not even a data dictionary. Nor is there any security on any of the Small Fry databases - not even the one containing what seems to be customer data! That seems a bit odd, and certainly won’t be tolerated in Whale Sports. I’m going to need to reverse-engineer the data structure from Small Fry’s three separate databases and then ensure that the imported data “fits” the existing Whale Sports security schema.

The second paragraph of this portfolio entry has the richest content. It shows the student both appreciates how own role and others support appropriate SPE working practices. (B7) and demonstrates appropriate [...] SPE knowledge in non-routine work. (B6). The student also uses discretion in identifying and responding to complex issues [...] (A3) and applies and contributes to creative thinking or finds new ways to complete tasks. (C3) for [...] a range of work, sometimes complex and non-routine [...] (C1)

¶3: I also need to understand better the ...idiosyncrasies of the two DBMS, so I’ve found some online learning material. I’m focusing first on the Small Fry system, as that implementation seems to have come with virtually no documentation - so I need to be able to work out what’s going on from the code!

Table 9 Evidence in portfolio entry (23rd Sep) supporting specific responsibility characteristics

The penultimate paragraph shows that the student takes the initiative to develop own knowledge by identifying and negotiating appropriate development opportunities. (B5), and, again, uses discretion in identifying and responding to complex issues related to own assignments. (A3)

¶4: So much for a simple job of a few SQL scripts ...

The final sentence reiterates the student’s ironic awareness of the complexity of their task. [...] a range of work, sometimes complex and non-routine [...]. (C1)

Table 10 Data entry section for Jo Garcia’s portfolio (Appendix B)

These assessments are collated in Table 9, showing the paragraph numbers (¶1-¶4) in the portfolio entry and the relevant text. It must be emphasized that such a summary is not a necessary part of the analysis of the portfolio; it is included here purely for clarity. The analysis approach is analogous to assessing an essay answer for content. As with an essay assessment, whilst collating the evidence explicitly can sometimes clarify issues with the assessment approach, all that is needed to complete the analysis is to note for which characteristics the portfolio entry provides evidence. The data entry area of the tool, Table 10, discussed in the next section, offers a succinct mechanism for this.

The portfolio entry from 23rd September provides evidence that the student has demonstrated seven SFIA responsibility characteristics. In some cases, a single entry might provide evidence of the student demonstrating some characteristics more than once. For example, the entry for 11th November provides evidence for multiple characteristics including two for which there is evidence of the characteristic being demonstrated more than once. The student has interacted with and influenced (I1) two sets of colleagues - the business managers in Whale Sports and the former employees of Small Fry; and has also overseen (I2) two colleagues - Phil and Andrea. One might argue that the second duplication, recorded in Table 10, could be a little generous, since the responsibility characteristic refers to overseeing “others”, implying “several colleagues”, but this is the lowest SFIA level of responsibility which includes supervision, and overseeing one colleague would be sufficient: two is a bonus.

In effect, the analysis approach maps each sentence of the portfolio against each of the SFIA characteristics, noting that the mapping can be quite rich, since portfolios will not, in general, be written so that each entry maps on to exactly one characteristic. Explicit sequential matching of each sentence to each of the 24 characteristics in turn would be prohibitively demanding. However, having the data entry area to act as an aide-memoire will, after a little familiarization, allow the mapping to be performed sufficiently accurately with a single pass. Only if a particular characteristic is found not to be sufficiently demonstrated should it be necessary to re-visit the entire portfolio seeking evidence for that specific characteristic.

5.2 Portfolio data entry section

The data entry section of our tool, reproduced in Table 10, shows where Jo Garcia’s portfolio contains evidence of demonstration of the SFIA characteristics. To save space in Table 10, each characteristic is identified by a summary word or phrase, to act as an aide-memoire, with a label (e.g., “A1”) referencing the full characteristic description in Table 2. The next column indicates whether the characteristic is considered [E]ssential or [S]upplementary; since this version of the tool is based on one developed for the UK Institute of Coding (IoC). The E/S distinction is based on each criterion’s importance for meeting UK qualification benchmarks. The IoC assessment and its mapping to qualification benchmarks are described in technical reports on the IoC’s public repository.

The item references area of the table contains links to specific entries in the sample portfolio in Appendix B, which provide evidence that the student has demonstrated particular characteristics, following the analysis approach illustrated in the previous section. Portfolio entries are referenced by date; evidence provided by the student in the reflection section is labeled “ref”; and evidence provided in the supervisor comments is labeled “sup”. For ease of reading, the item references are hyperlinked to the relevant portfolio entries, each of which is followed by a hyperlink to return to the data entry section.

The $I_{ch}$ column shows the count of items against each characteristic. Whether or not “Essential” characteristics have been demonstrated is shown in the following column by determining if $I_{ch} \ge I_{Min}$, where $I_{Min}$ is one item of the configuration presented in Table 13. To support subsequent calculations for the satisfaction of the overall thresholds, the final two columns ($I_e$ and $I_s$) copy $I_{ch}$ for Essential and Supplementary characteristics, respectively.

Two further observations can be made about the item references identified for Jo Garcia’s portfolio. First, the overall assessment is designed not to be a conjunctive tick-list, and thus it allows students to “pass” despite having failed to demonstrate a small number of characteristics. That is why it is not necessary that every characteristic be “demonstrated” by having more than $I_{Min}$ item references. An example is the characteristic B4 - effective application of digital skills - which has no item references entered. The count of zero item references against this characteristic does not prevent a positive overall assessment outcome. This would be true whether B4 were an “essential” or a “supplementary” characteristic.

Table 11 Configuration parameters for SFIA assessment of Jo Garcia’s portfolio (Appendix B)

A second point concerns the number of item references that can be entered and counted against each characteristic. $I_{Max_{C}}$ is set to 3 for the worked example (as discussed in Section 5.3 below). This means that not all of the four item references entered in Table 10 against characteristic B7 (Appreciates how [...] role[s] support [...] SPE working practices) will be counted: only three are counted by the tool, and $I_{ch}$ for B7 is 3.

5.3 Demonstration of SFIA responsibility characteristics

The configuration parameters for the SFIA characteristics assessment of Jo Garcia example portfolio in Appendix B are shown in Table 11. In particular, $I_{Min}$ is set to 2, which may be lower than would be appropriate for a more complete portfolio. Similarly, to keep Table 10 as simple as possible for the reader, $I_{Max_{C}}$ is set to 3. These are the minimum realistic values to demonstrate the working of the assessment tool, but they are appropriate for a portfolio with less content than might be expected of an authentic portfolio for a full placement of up to a year’s duration.

Table 12 SFIA outcome for Jo Garcia’s portfolio (Appendix B)

Table 13 Configuration parameters and thresholds for CC2020 dispositions assessment of Jo Garcia’s portfolio (Appendix B)

The proportion $E_{Prop}$, which specifies the proportion of the “Essential” SFIA characteristics that must be demonstrated in at least $I_{Min}$ item references is set to 80%, which is the usual SFIA threshold for “complete demonstration” of skills and responsibility characteristics (The SFIA Foundation, 2021b).

The final configuration parameter, $I_{Prop}$, is set at 65%, to ensure an overall breadth of evidence in the portfolio across both the “Essential” characteristics and the full set of 24 characteristics. Together, these parameters ensure that there are more than the bare minimum number of item references needed to demonstrate the threshold number of “Essential” characteristics.

Based on the data and preliminary calculations in Table 10, in Table 12 we show the calculated thresholds and scores, and the resulting outcome. Each of the three scores calculated by the tool exceeds the corresponding threshold, so the overall outcome is “Pass”.

If any or all of the scores had not met its threshold, the outcome would be, “not yet pass”, emphasizing that the tool can be used to provide formative feedback to the student, showing where they need to improve, rather than representing an “all or nothing” hurdle. Even in Jo Garcia’s case, with an overall “Pass” outcome, two “Essential” characteristics have not been demonstrated, and this should be valuable feedback to the student on where they might benefit from developing their professional experience.

5.4 Demonstration of CC2020 dispositions

As set out in Section 4.2.2, the configuration parameters and thresholds for the demonstration of CC2020 dispositions are broadly analogous to those for the SFIA assessment.

For Jo Garcia’s portfolio, the configuration parameters and resulting thresholds are summarized in Table 13. Because the score for each disposition is a normalised value rather than a simple count, $T_d$ need not be an integer value. In this case, the calculated value of $T_d$ is very close to that set for $I_{Min}$ for the SFIA assessment.

Table 14 Data area and scoring for Adaptable for Jo Garcia’s portfolio

There is no data entry area for the CC2020 dispositions area of the tool. Rather, for each of the SFIA characteristics that contributes to a disposition, a nominal entry of “1” is set against each characteristic corresponding to an item reference in the data entry area.

The data carried forward for the Adaptable disposition is shown in Table 14. The first two columns reference the contributing responsibility characteristics set out in Table 2. The number of contributing characteristics is 6, shown in brackets following the disposition name. The third column shows, for each characteristic, the number of dispositions to which that characteristic contributes, according to the mapping in Table 16.

The next four columns are generated from the data entry area in Table 10, with a “1” corresponding to each reference entered against the characteristic in the data entry area. The penultimate column shows the item reference count for each contributing characteristic, $I_{ch}$, based on which the item reference score for the characteristic $S_{ch}$ is calculated and shown in the last column. For example, the item reference score for “C3 Creative” (Applies and contributes to creative thinking or finds new ways to complete tasks) is the item reference count, 2, divided by the square root of the number of dispositions to which “C3 Creative” contributes - in this case, 4. So, the item reference score for “C3 Creative” is $2 \div \sqrt{4}$, or $2 \div 2 = 1$. The corresponding item reference score for “B5 Develops” (... takes the initiative to develop own knowledge...) is $2 \div \sqrt{6} = 0.82$, because “B5 Develops” contributes to more dispositions - (6) than does “C3 Creative” - (4).

The overall, normalised, disposition score for Adaptable, $S_d$, shown below the list of item reference scores, is normalized in a similar way, to account for the varying number of SFIA characteristics that contribute to the different dispositions. Six characteristics are mapped to Adaptable, so the overall disposition score is the sum of the item reference scores for the six contributing characteristics (5.32) divided by $\sqrt{6}$, giving 2.17. Since this value is greater than the threshold $T_d = 1.95$ (see Table 13), the disposition Adaptable has been demonstrated.

Table 15 shows the score $S_d$ for each of the eleven dispositions, together with the number of contributing characteristics and the evaluation of the disposition score against the corresponding threshold. For Jo Garcia’s portfolio, only one disposition, Inventive, has not yet been demonstrated, according to the configuration set for the assessment.

The lower part of Table 15 presents the calculated thresholds and scores, using the normalization described in Section 4. $N_D$, the number of dispositions, is included as a “score” because the tool is designed to be extensible, and this value is counted by the tool rather than being “programmed in” (see Section 4.2.2).

Both of the thresholds are easily exceeded, so it can be concluded that Jo Garcia has demonstrated the CC2020 dispositions in addition to demonstrating responsibility characteristics corresponding to SFIA Level 3.

Table 15 Scoring for Jo Garcia’s portfolio using the thresholds in Table 13

5.5 Comparison of outcomes

The two outcomes, for the demonstration of SFIA (Level 3) responsibility characteristics and the demonstration of CC2020 dispositions are based on the same input data. Although there is considerable overlap in the semantics of the configuration parameters for both assessments, the parameters can be set independently.

There is, of course, a caveat. Although the default value of $I_{Max_D}$ is $I_{Max_{Ch}}$, the two parameters can be set to have different values. The value of $I_{Max_{Ch}}$ determines when item references in the data entry area stop being counted against the responsibility characteristics – and probably, as a corollary, when an assessor would stop entering them, unless they were aware that more item references could count towards demonstration of the CC2020 dispositions. It follows that item references should be entered against each characteristic up to the limit of $MAX(I_{Max_D}, I_{Max_C})$, whether or not all references entered will be counted towards the demonstration of SFIA responsibility characteristics.

One might wish to argue that a student should be required to demonstrate (nearly) all of the CC2020 dispositions in their portfolio; this could be implemented by setting $D_{Prop}$ closer to 100%. In fact, with the data entered in Table 10, and keeping all the other configuration parameters the same, any value for $d_{Prop}$ over 90% (giving a threshold value $T_d$ of 2.7 or more) would result in the dispositions outcome being, “not pass yet”, since the five dispositions with scores less than 2.7 would not be demonstrated, so the threshold $T_D$ would not be met.

Although there is no a priori reason why success in demonstrating the SFIA responsibility characteristics should automatically imply success in demonstrating the CC2020 dispositions, or vice-versa, it was shown in Bowers et al. (2022b) that the SFIA responsibility characteristics could be used as a proxy for the CC2020 dispositions. It could therefore be unfortunate if the parameters were set so that the two assessments gave different outcomes.

6 Discussion

Having described the derivation of our assessment tool, and presented a worked example showing how the tool could be used to assess the contents of a fictitious portfolio, this section explores some issues that invite further scrutiny. We consider first the reliability and scalability of the assessment approach, and make some observations on its likely stability. After considering the normalization algorithm incorporated in the tool, we then discuss how the tool might be customized, and the potential for wider application.

6.1 Reliability and scalability

Whilst the tool may seem intricate, it is designed to support a challenging task in a reliable manner - that is, one which produces reproducible outcomes that are consistent both for different assessors and across time, and is therefore scalable for large numbers of students. Considering the 14 design elements of assessment design identified in Dawson (2017), those most crucial for ensuring reliability (consistency) both between different assessors and over time are:

• the number and type of quality levels,

• the quality definitions,

• the evaluative criteria,

• judgement complexity,

• exemplars to illustrate quality.

The evaluation criteria (the number of item references required to demonstrate a SFIA Characteristic, and how they contribute to the demonstration of Dispositions) and quality levels (“Pass”, “Not pass yet”), which specify the final outcome, are fixed in the tool once the parameter values have been set; to ensure reliability (consistency); these should clearly not be varied either for different assessors or for a given assessment over time. As far as the assessor is concerned there is just one quality definition - a portfolio entry must include evidence of demonstrating at least one SFIA characteristic.

The scope for judgement around this quality definition is limited - a portfolio entry either does or does not include evidence of any SFIA characteristics: the assessment is, in essence, criterion-based. The only scope for judgement might be how explicit the evidence must be in the portfolio entry, and how much may be inferred by the assessor. For example, if a student reports on progress of one or more colleagues whom they are supervising, does the portfolio entry need to reiterate that the student is supervising those colleagues, or is the report of the colleagues’ contribution towards one or more tasks for which the student is responsible mean that “supervision” can be inferred? Such interpretations of the rubric are best clarified by exemplars, as presented in the worked example of Section 5, and perhaps supported by guidance and/or by training for assessors; assessors following the exemplar and applying any guidance should then reach consistent judgements, leading to overall reliability for the assessment.

Furthermore, recording references in the tool to specific portfolio entries makes the outcomes auditable, allowing moderation of outcomes from distinct assessors, improving reliability further.

In addition, because of the many-to-many mapping from SFIA characteristics to dispositions, the outcome for the demonstration of each disposition is dependent on the identification of a relatively large number of portfolio entries; the normalisation applied renders that outcome correspondingly insensitive to the assessment for any given portfolio entry.

An alternative perspective on the reliability of an assessment rubric is offered by Moskal and Leydens (2000), who pose three questions for a rubric:

• 1. Are the scoring categories well defined?

• 2. Are the differences between the score categories clear? And

• 3. Would two independent [assessors] arrive at the same score for a given response based on the scoring rubric?

Given appropriate exemplars and assessor guidance, as outlined above, the answers to all three of these questions is affirmative, suggesting good reliability.

Moskal and Leydens (2000) suggest also that, if the assessment is intended to prepare students for their eventual employment, then the assessment rubric should be designed to be as authentic as possible. The assessment approach described in this paper is not only “authentic” in the narrow academic sense: it is genuinely authentic, as it assesses demonstration of SFIA responsibility characteristics, and therefore disposition, regardless of the professional skills deployed by the student during a real-world placement.

A similar approach has been used, with multiple assessors and full double-marking, on a University work-based learning module. Not only were there few issues with inconsistencies between markers, but the approach was also found to be neither more subjective nor burdensome than traditional essay-marking. This corroborates the reliability and scalability of the assessment design.

6.2 Stability

Inspired by the SFIA Foundation’s approach to assessment of professional competence, and supported by the successful deployment of a similar scheme for a work-based learning course, the assessment process presented in this paper argues that it is sufficient to know that a student has demonstrated each responsibility characteristic more than once. This requires the analysis of a student portfolio, counting how many portfolio entries provide evidence of the demonstration of each of the SFIA Level 3 responsibility characteristics. Although this may be simplistic, the tool ensures that the “counting”, and the subsequent allocation against the CC2020 dispositions, is performed in a consistent, reproducible manner. Furthermore, since each of the responsibility characteristics is self-contained, and it is likely that a single portfolio entry will include evidence for the performance of several characteristics, the analysis of each portfolio item will usually result in many item references being entered into the tool, as in the worked example in Section 5. The many-to-many mapping from SFIA responsibility characteristics to CC2020 dispositions further increases the enumeration of item references, to the point that the overall outcome is based on a large number of data points, so that the outcome itself should tend to be both robust and stable.

Future work will test this tendency by exploring the sensitivity of the overall outcomes to particular patterns in the input data, in order to validate the resulting outcomes. We plan to confirm in a future paper the pragmatic perception of stability and robustness.

6.3 Normalization

The tool relies on a normalization strategy that seeks to address the differing number of dispositions to which each SFIA characteristic contributes, and vice versa. The algorithm used represents a first approximation, and may not be ideal, as the scores for dispositions with more contributing characteristics are systematically larger than for those with fewer. However, as noted in Bowers et al. (2022b) and Bowers et al. (2022a), nor is the mapping itself perfect: further work is ongoing to refine both the mapping and the normalization of the counts of portfolio items.

Although it is unlikely that the outcome of these explorations will lead to additional SFIA characteristics being mapped to particular dispositions, the overall effort may change the imbalance between dispositions such as “Professional”, supported by 20 SFIA responsibility characteristics, and “Inventive”, supported by just two: currently, “Inventive” tends to achieve a lower individual score than any of the other disposition. This, combined with adjustments to the normalization algorithm, may make it less likely that evidence from a realistic student portfolio would tend to under-represent the achievement of dispositions such as “Inventive” and over-represent that for other dispositions supported by more responsibility characteristics.

In its current form, the tool makes no attempt to normalise the other many-to-many mapping inherent in the worked example: that between portfolio entries and the SFIA characteristics. It is assumed that any portfolio entry that is considered to demonstrate, say, three SFIA characteristics should carry the same weight, in the assessment algorithm, as three separate portfolio entries each demonstrating a single characteristic. Not only are portfolio entries likely to be completely free-form in both style and length, but there is no notion (in the real world) of a "quantum" of responsibility characteristics to be shared between the characteristics. It follows that any many-to-many mapping between portfolio entries is normalised on the basis of which characteristics have been demonstrated by each portfolio entry, and how those characteristics are mapped to the dispositions.

6.4 Customization

The tool is implemented as an Excel workbook. It is normally supplied as a “locked” version, so that only data can be entered, and the logic cannot be modified inadvertently. However, on request, an unlocked version of the tool can be supplied for those interested in exploring potential customizations.

For example, there is a distinction between the approaches to the assessment of evidence for the SFIA responsibility characteristics and for the CC2020 dispositions. For the former, a subset of the characteristics are labelled as “Essential”. reflecting the particular regulatory requirements for the accreditation standard for which the tool was developed originally. The corollary of a particular characteristic being labelled “Essential” would be a higher expectation that a student’s portfolio should contain evidence of that characteristic, although the tool will still “count” item references against “Supplementary” characteristics.

There is no similar flexibility in the tool, as presented, for the CC2020 dispositions: all are weighted equally. However, in Bowers et al. (2022a) it was suggested that there could be circumstances in which an instructor may wish to focus on a subset of the dispositions. Since the tool itself is extensible, it would be straightforward to introduce a classification of “essential” versus “supplementary” for the dispositions, and applying thresholds and calculations to the dispositions scores analogous to those for “Essential/Supplementary” SFIA responsibility characteristics. This would increase further the customizability of the tool.

6.5 Extended application

The many-to-many mapping between the SFIA responsibility characteristics and CC2020 dispositions is coded explicitly in the spreadsheet implementation of the assessment tool. Where possible, parameters for scoring and normalization within the tool are derived by counting rather than being hard-coded. For example, Table 16 shows two sets of cardinalities: in the second row, the number of SFIA characteristics contributing to each disposition; and, in the third column, the number of dispositions to which each characteristic contributes. In the tool, the first set can be counted, whereas the second must be given for each characteristic, alongside the characteristic’s description. Similarly, the numbers of dispositions, characteristics and how many of the latter are “essential” are all counted by the tool. Knowing which cardinalities need to be provided explicitly and which are derived enables modification, or even replacement, of the many-to-many mapping feasible.

For example, an opportunity for customization might arise from configuring the tool on the basis of the mapping between programme-level learning outcomes for a degree and the responsibility characteristics for SFIA level 3. Given such a mapping, the programme-level learning outcomes could be treated in a manner analogous to that used currently for the CC2020 dispositions, allowing an assessment of the extent to which the programme-level outcomes have been demonstrated. Similarly, it would also be possible to instantiate the tool based on a mapping between the programme-level learning outcomes and the CC2020 dispositions – as the tool’s algorithm reflects the mapping between two sets of qualities – currently SFIA responsibility characteristics and CC2020 dispositions – it is simply a matter of capturing whatever mapping is required in the tool’s logic.

A further application could be to invite students to use the tool to assess their own portfolios, mapping individual entries against the relevant SFIA responsibility characteristics. Whilst this could be instructive for the student, it could also be quite onerous for them to analyse a single portfolio, since the student would be unlikely to have gained sufficient familiarity with the characteristics to achieve the scalability analogous to “essay assessment” suggested in Section 5.1. Of course, developing the necessary familiarity with the SFIA responsibility characteristics could be a useful outcome, for the student, in its own right.

Currently, the assessment tool allows portfolio entries to be mapped only against the SFIA responsibility characteristics. But these characteristics will be displayed in the process of demonstrating one or more technical skills, which are also defined, in detail, in the SFIA framework. A similar approach to assessing the demonstration of technical skills was developed alongside the original assessment tool, but it is necessarily more complicated, because the SFIA framework describes 121 distinct technical skills. Work is under way to develop a complementary tool to simplify the identification and selection of appropriate SFIA skills, so that evidence contained in portfolio items can be associated with the relevant skills, leading to an assessment of the extent to which a particular skill has been demonstrated.

7 Conclusions

The assessment tool presented in this paper supports the analysis of a portfolio recording a student’s experience of real-world IT activities. The tool determines the extent to which evidence recorded in the portfolio indicates that the student has demonstrated CC2020 dispositions.

The assessment is indirect, in that the content of the student portfolio is mapped against the personal and professional qualities expressed as the responsibility characteristics for Level 3 in the SFIA skills framework The equivalence between the SFIA responsibility characteristics and the CC2020 dispositions is captured in the mapping in Table 16.

SFIA was chosen as the mediating skills framework primarily because of its unique separation between “responsibility characteristics” and (technical) “skill definitions”, discussed in Section 3. Furthermore, not only have the responsibility characteristics been shown to provide full coverage of the CC2020 dispositions, but they are also more explicit and measurable than the abstract dispositions.

The configuration parameters for the assessment tool are defined in Section 4.2, and then applied in Section 5 using a fictitious student portfolio with appropriate parameter values, to demonstrate the analysis of a portfolio, entry of the resulting portfolio item references and the resulting outcomes.

Whilst the tool may seem intricate, it is designed to support a challenging task in a scalable, consistent and reproducible manner. A similar approach has been used successfully, with multiple assessors, on a University work-based learning module, and it was found to be no more subjective or burdensome than essay-marking.

The focus is on the assessment of real-world actions and activities, mediated by a student’s contemporaneous portfolio, rather than only on a student’s subsequent (academic) reflection on their performance of those activities. This supports the recognition of the achievements by students who may be better at doing things than writing about them - an important aspect of inclusivity, particularly for neurodiverse students.

In addition, the tool requires the explicit entry of specific item references. These should be de-referenceable, albeit not necessarily hyperlinks (which are used in this paper to support the reader). This provides an important quality assurance mechanism, allowing for the subsequent review or audit both of the outcome for individual students and of the process as a whole.

Finally, the tool we have presented is both customizable and extensible, so that it can be adapted to address varying needs for differing contexts.

Appendices

Appendix A: Mapping SFIA characteristics to CC2020 dispositions

Table 16 Mapping between SFIA Level 3 responsibility characteristics and CC2020 dispositions, adapted from Table VI of Bowers et al. (2022b)

Appendix B: Selected extracts from fictional student portfolio

1.1 B.1 Jo Garcia - Portfolio (extracts)

6th September My placement employer, Whale Sports, operates a large national chain of gyms and fitness centres. In the weeks before my placement started, they had completed the purchase of a much smaller local chain, Small Fry Fitness. My main role within the IT department will be to arrange for the merging of Small Fry’s records into Whale Sport’s main customer, financial and operations databases. I’m expecting to draw on my second year data management module - although it should really be quite straightforward, as both companies use standard relational database software. So, it should just be a question of writing a few scripts to transfer the data. My initial estimate is that this should take no more than a week or two.

Back to data entry table

9th September Attended a company induction today for all the placement students and interns. Whale Sports really is a bit larger and more complex than I expected - and the takeover of Small Fry makes it even more so! One of the things that came out of the induction is that there are online learning resources that cover particular areas of the company’s operations - such as customer registration and session bookings. I’m not sure how many of them will be useful - after all, I’m only looking at the data! - but I shall bear them in mind if I have time.

Back to data entry table

13th September This is not going to be quite as straightforward as I expected. The database structures for Small Fry are completely different from those of Whale Sports. They claim to serve similar purposes, with the same sort of scope and a similar set of operations. There are just so many differences - ranging from different field names for apparently the same things to completely different sets of foreign keys. What’s more, the two databases run on different DBMSs which, even though they both use “standard SQL”, seem to do quite a lot of things - particularly scripts! - in different ways. This will just make the job even more challenging…! I’ll mention it all to my supervisor next week - and if that doesn’t help, I’m expecting my University tutor to visit later in the week.

Back to data entry table

20th September An interesting week. My supervisor just told me that I was supposed to understand databases, so why couldn’t I just get on and sort it out? Fortunately, when my tutor visited, he reminded me of all the data modelling we did in the data management(DM) module. So, I’m going to try comparing the data models for the two sets of data - and match the structures rather than the field names.

Back to data entry table

23rd September At least there are full specs - including data models! - for Whale Sports’s database. They’re drawn using a notation that’s a bit unfamiliar, but, as we were taught in the DM module, the basic concepts represented in all of the notations are essentially the same.

Small Fry, however, is a bit more of a problem. There’s not even a data dictionary. Nor is there any security on any of the Small Fry databases - not even the one containing what seems to be customer data! That seems a bit odd, and certainly won’t be tolerated in Whale Sports. I’m going to need to reverse-engineer the data structure from Small Fry’s three separate databases and then ensure that the imported data “fits” the existing Whale Sports security schema.

I also need to understand better the $\ldots$ idiosyncrasies of the two DBMS, so I’ve found some online learning material. I’m focussing first on the Small Fry system, as that implementation seems to have come with virtually no documentation - so I need to be able to work out what’s going on from the code!

So much for a simple job of a few SQL script $\ldots$

Back to data entry table           Back to portfolio entry review

30th September I’ve reverse engineered a model for one Small Fry database - customers - but it leaves several questions. For example, Whale Sports allows people to be members of more than one gym; but, in Small Fry, if one person is a member of two branches, they are treated as two completely different people. This became clear when I tried comparing the two data models - and some of the data - and found that “customer_id” in Whale Sports’s database seemed to correspond to the compound value (pn, bc) in Small Fry’s database. After talking to some of the IT guys from Small Fry - those who had stayed after the merger - I discovered that “pn” stood for “person number” and “bc” was “branch code”… I need now to list all the ambiguities and oddities in the data model for Small Fry, and spend some time with their former IT guys.

Back to data entry table

14th October My supervisor is really impressed that I’ve managed to resolve all of the differences between the two customer databases. However, she’s getting a bit worried about how long it is taking, as there is a deadline to have the merged systems running by the start of January next year. And we’ve not even started on the various applications that hang off Small Fry’s customer database. And I’m not going to make any rash estimates this time. The upshot is that she’s asked me to brief one of the developers - Phil - on the structure of Small Fry’s database, so that he can export, clean and reformat the data to put it into Whale Sport’s main database. She’s also given me someone to help - another placement student, Andrea. She wants me to brief Andrea on the approach I took for the customer database, so that she can tackle the financial database while I focus on the operations database. Her parting comment was that the operations database was likely to be the trickiest, as it is in the way they run their operations that companies seek to distinguish themselves. $\ldots$

Back to data entry table

11th November Phil has more or less completed the transfer of the customer data, and has even replicated most of Small Fry’s distinctive customer functionality - such as the loyalty scheme - within Whale Sports’s systems. The loyalty scheme was a bit of a challenge, as Whale Sports had nothing like it previously. And, of course, the loyalty scheme’s use of personal (customer) data now has to be made consistent with Whale Sports’s data protection expectations. So, once Phil and I really understood what it was doing, and what we needed to add about data security, we checked our understanding with some of the Small Fry managers that now work for Whale Sports. Then, supported by my supervisor, we presented our findings to a meeting of Business Managers; they really liked the idea of the scheme, and are planning to roll it out across Whale Sports. As an aside, I would never have followed the logic of Small Fry’s loyalty scheme if I hadn’t been able to follow the detail of the code. It really is a good job I spent time learning about their DBMS a couple of months ago! Andrea has also made good progress. Having two people working on different aspects of the problem has made it much easier to check our understanding, to validate each other’s assumptions, and check our respective models. It works better, too, when we are meeting with the ex-Small Fry staff - it seems to run much more positively with two of us talking with three or four guys who - as far as we could tell - would really have much preferred to have been left alone in their tiny company.

Back to data entry table

18th November We’ve completed formal model reviews with my supervisor for the second and third data models and the additions to the security schemas, and they have all been signed off. Andrea and I are now working with Phil to get all of the data transferred by the end of this month - so that there is plenty of time for testing.

Back to data entry table

2nd December The data transfer and merge project has been signed off as complete. My supervisor seems very happy - so much so that she has asked Andrea and me, as a team, to trawl through the data models and specs for the main Whale Sports database, to discover any anomalies, odd assumptions … or even errors. She’s also asked me to present an internal talk to the development team on how I approached the modelling task, so that there will be other people who can do the job after Andrea and I go back to our respective Universities.

Back to data entry table

1.2 B.2 Reflection on placement portfolio

Reading through my portfolio again, I remember that one of the things that kept striking me - hard - was how much of the boring detail in my university modules was actually incredibly useful. I recall that I never paid much attention to all the data modelling we did - after all, it’s so easy to prototype a database nowadays, that there doesn’t seem to be a lot of point in designing it first…. It’s just not “agile”…! And, until I encountered a “real-world” database that held personal data - for unexpected, but legitimate purposes - I had never appreciated the importance of security schemas and data protection policies. But then, in my placement, I found time and again that I was using aspects of that modelling to resolve the issues. What’s more, it was because I was able to do that my supervisor was so impressed - particularly when I went on to discover a couple of howlers in Whale Sport’s main systems!

It was also an eye-opener working with real colleagues in a team, to achieve something important. I suppose I was the team leader, but it was all very collaborative - and what really seemed to matter was that we recorded and documented our decisions, rather than that any one of us was actually “in charge”. And it was quite scary at times to realise that the work I was leading actually mattered to Whale Sports. If we hadn’t got it right, then their investment in Small Fry could just have fallen over…!

Finally, I’ve just been offered a permanent job at Whale Sports …. To lead their data migration team!

Back to data entry table

1.3 B.3 Supervisor comments

Jo came to us as a fairly “ordinary” placement student, to work in the data migration team. As they mention in their portfolio, we had just acquired a small company, Small Fry Fitness, and I asked Jo to transfer Small Fry’s data into our corporate database. Their commentary shows that he started fairly optimistic about how simple it would be. I didn’t disabuse them - I knew that there would be a few problems, but, frankly, I had no idea how challenging it would turn out to be. Jo rose to the challenge superbly. They took it upon themself to learn about the systems involved, and seems now to understand more about the Small Fry system than their colleagues who used to run their IT department! Their work was so thoroughly professional, and so well-documented that it became a benchmark within the team. They even reminded us of some of our responsibilities regarding personal data - we thought we had worked all that out, but the addition of three apparently unsecured databases from Small Fry gave us a good opportunity to revisit our internal standards.

As they comment, I asked Jo to give a seminar, with the two other members of their team, so that we would retain at least some of his knowledge after they left. So, I am very happy to confirm the content and detail of Jo’s portfolio. If anything, they sell themself a little short - this really was a difficult project. Moreover, they hardly mention how effective they became as a mentor and team leader to their immediate colleagues, Andrea and Phil.

We have this morning offered Jo a position as Data Migration Team Leader when they graduate - I really hope that they will accept it!

Back to data entry table