1 Introduction

The article offers a methodological reflection on combining qualitative and quantitative text analysis methods with document-based Social Network Analysis (SNA) to understand social networks and, more specifically, policy networks (PNs). It aims to compare two different approaches that consist in combining SNA—a quantitative-structural approach based on numerical elaborations of relations—with, respectively, qualitative text analysis (including the analysis of semi-structured interview answers)Footnote 1 and quantitative text mining (TM) and Natural Language Processing (NLP) techniques.

Besides constituting a theoretical perspective on actor relations, SNA is a series of formal procedures for examining observable relations between at least two actors and their patterns, developed at the intersection of academic disciplines (Fuhse and Mützel 2011). The combination of this latter conception of SNA with qualitative and quantitative analysis methods for answering research questions about PNs is what the present study is interested in, aware of the need to develop method-oriented research in light of specific research questions and goals (Mason, 2006). SNA research quantifies collected data, analyzes them employing measures that synthesize their structural features, and provides graphical representations of network relations that enable intuitive and cognitively easy access to the underlying network structure. However, per se, formal network analysis leads to descriptions of networks that allow for their exploration, rather than explanation (Fuhse and Mützel 2011; Yousefi Nooraie et al. 2020). For this reason, SNA is normally complemented by other methods that aim to put flesh on the sociogram bones (Crossley 2010).

In this article, the author complements SNA with qualitative and quantitative text analysis as representatives of the qualitative and quantitative methods worlds, based on their appropriateness for the posed research questions, in light of the undesirability of applying all existing qual/quant methods to a specific research object—be it for empirical investigation or methodological reflection—and of a narrower focus on research methods only (Hesse-Biber 2015). The study is undertaken in the context of a dearth of literature offering systematic comparisons of qualitative and quantitative text analysis methods for complementing SNA. In the article, the comparison is applied to the study of PNs, a research object in relation to which SNA is more often complemented by qualitative rather than quantitative analyses (Zhang et al. 2021).

PNs are defined as patterns of interaction among actors in a policy system or decision-making process, with relationships such as information exchange or political coordination (Henry 2011). This interaction normally takes shape around policy problems and/or programs (Klijn 1997). The guiding hypothesis in this study is that SNA benefits from being complemented with qualitative analysis for a thorough comprehension of PNs as complex phenomena that involve a context-sensitive understanding of premises and actors. Hence, the article aims to assess whether an SNA complemented with qualitative tools is comparatively more apt than an SNA combined with quantitative tools to tell meaningful stories about what occurs in networks (Crossley 2010; Yousefi Nooraie et al. 2020). The rationale of this hypothesis is that SNA, an approach based on mathematical formulas and automatic graphic visualization (O’Malley and Marsden, 2008), is apt to identify the structure of a network, but qualitative methods allow conducting necessary checks upon the boundaries of that quantification. The properties highlighted by SNA need to be qualitatively investigated for a thorough understanding of networks; at the same time, purely qualitative analysis might lack a comprehensive focus on network configuration. Hence, such complex study subjects as networks might benefit from an approach that takes the best from both the qualitative and quantitative worlds (Mason, 2006).

Quantitative methods are here defined as any means employed to generate or record numerical observations of the social world; by contrast, qualitative methods are those that generate/record non-numerical and prevalently discursive data (Crossley 2010). By complementing SNA with qualitative and quantitative text analysis, the article analyzes the synergies and shortcomings that characterize the interaction between different methodological approaches to studying PNs while nurturing the debate on mixed methods amidst the timeless paradigm war between qualitative and quantitative research (Alise and Teddlie 2010), which has itself contributed to giving rise to mixed-methods research (Bellotti 2016).

This method-oriented article draws upon two empirical studies.

Study 1 (S1; Sect. 3) aims at highlighting the relevance of transnational network activities carried out by the Employment Thematic Network (ETN)—established under the aegis of the European Commission—to the social public policy cycle in the participating European Union (EU) member states (MSs). The hypothesis in S1 is that the more MSs engage with their peers in transnational network activities, the more they are likely to experience policy learning and change. The concept of policy learning is widely debated. Without embarking upon a review of the possible interpretations of this concept—which falls outside the scope of this article—policy learning in this study is understood as knowledge acquisition and utilization that is functional to better goal attainment by governments (Bennett and Howlett 1992). Policy change is in turn understood as a possible consequence of this kind of learning.

Study 2 (S2; Sect. 4) sets out to identify a nexus between transnational cooperation and cultural adaption within the context of the Bologna Process, aimed at harmonizing aspects of higher education policies across Europe. The hypothesis in S2 is that participation in cooperation processes triggers progressive, reciprocal cultural adaption, which can be visible in the language employed over time in governmental reports. In a network like the Bologna Process one, according to sociological neo-institutionalism, the cooperation setting can affect actors’ identities, values, and norms (Papadopoulos 2018).

These two cases have been examined as instances of, respectively, qualitative and quantitative text analysis combined with SNA, aiming to isolate the contribution given by each methodological component and hence treating quantitative and qualitative strands as separate (Yousefi Nooraie et al. 2020). In both S1 and S2, sociogram construction is based on values derived from meeting minutes. The choice to employ meeting documentation for SNA stems from the centrality of network events for the analysis of social networks, based on the assumption that “ordinary aspects in network operation matter” (Papadopoulos 2018: 431) while noticing that, despite this, not much is made of meetings in public policy research. Pal and Spence highlight that “[w]hile social networks can be traced in various ways […] they are most evident in ‘events’ that bring together their constituent members. An event, such as a conference, a workshop or other meeting, is a focal point of network activity” (2020: 92). From a different perspective, assessing the potential and limitations of conducting document-based research is particularly important in an era of increasing resource scarcity in universities (Schuelke-Leech 2013). Document-based research is indeed a relatively inexpensive research method in terms of both financial and time resources. Moreover, enhanced transparency and facilitated access to public documents—made possible by the existence and availability of open archives and online repositories—allow scholars to easily exploit the potential of document collections.

The article proceeds as follows. Section 2 illustrates the debate on the use of SNA with multiple methods. Sections 3 and 4 present two studies where SNA is respectively combined with qualitative and quantitative text analysis methods. Section 5 compares the two studies and discusses their implications in terms of research time needed, resources employed for the study, unobtrusiveness, and effectiveness of the adopted qualitative or quantitative methods. Conclusions are drawn in Sect. 6.

2 Mixing methods with SNA

A social network can be defined as a set of actors (or nodes) tied by a set or sets of relations, where nodes can be human beings as well as aggregate social units such as organizations or governments (O'Malley and Marsden 2008). One of the approaches to studying social networks is SNA. SNA is considered a paradigm or perspective, rather than a method, which does not aspire to explain social phenomena by itself but requires assumptions about how to explain those social phenomena (Marin and Wellman 2014). Examining the usage of SNA in combination with other analytical tools and comparing these applications is hence crucial.

SNA is a quantitative-dominant approach based on mathematical formulas and automated visualizations (Hollstein 2014), which helps to formalize what might not be visible to the naked eye. However, SNA can take different nuances according to the methods integrated into it (Froehlich et al. 2020), for instance for data collection. Moreover, depending on the methods it is complemented with, this approach can be suitable for the analysis of context-sensitive phenomena such as PNs (Drew et al. 2011). In this regard, over the past 20 years, there has been increasing awareness that focusing on either qualitative or quantitative methods alone causes researchers to miss important parts of a story (Hollstein 2014). To be clear, researchers had been employing mixed-methods research for a long time before explicit attention was devoted to its methodological implications. When mixed methods were recognized as a study object, in a process that has sometimes been referred to as “reification” (Hesse-Biber 2015), even the very definition of mixed methods started to be discussed. In this regard, most of the related literature sees mixed methods as the combination of at least one quantitative tool/approach and one qualitative tool/approach (Hendren et al. 2023). Alongside, there are more inclusive approaches that encompass also combinations of methods of the same kind (e.g., qualitative-qualitative; Bellotti 2016). In this study, the definition of mixed methods is extended to the combination of at least two methods regardless of whether they belong to a qualitative or quantitative tradition, provided that these methods are combined at least at one stage of the research process (Hollstein 2014).

The stream of reflexive research that focuses on mixing methods with SNA has developed over the last 15 years and has had the merit of focusing on the advantages of combining different types of data and analytical strategies (Bernhard 2018). Quantitative network data describe relations, interactions, and structures in numbers, while qualitative network data describe network aspects in text form; likewise, quantitative methods of analysis describe frequencies, patterns, and/or causal mechanisms in quantitative terms, while qualitative methods essentially aim at understanding meaning (Hollstein 2014; Mason, 2006).

While theoretically both quantitative and qualitative methods can validly complement and lend value to SNA, most studies focus exclusively on SNA combined with qualitative methods. This trend is based on the assumed suitability of SNA that combines with qualitative tools to address research questions related not only to the formal or structural side of relationships but also to the meaning of interactions or the variability of social relationships (Froehlich et al. 2020). Recently, there has been a flourishing method-oriented debate on the value of qualitatively enriched SNA to overcome the limitations of single methods. For instance, Bellotti (2016) illustrates the strength of combining SNA visualizations with interviews. In this journal, Fuhse (2023) recently noted how most of the methodological discussion on qualitatively complemented SNA pivots around interviews. Occasionally, research has undertaken integrated analyses of both qualitative and quantitative dimensions (and methods) together with structural ones to analyze networks, although not for comparison purposes (Bidart and Cacciuttolo 2013).

Overall, the advantages and disadvantages of combining SNA with other methods have been discussed through literature reviews, case studies, and theoretical essays (Crossley 2010). Noncomparative reflections have been made, for instance, regarding the value of SNA to understand complex PNs (Drew et al. 2011) or identify the role of interest groups in PNs (Varone et al. 2017). Other studies have investigated the use of SNA to understand networks of (policy) research (rather than practice) (Zhang et al. 2021) or highlighted the value inherent in the combination between SNA and Qualitative Comparative Analysis (QCA) (Fischer 2011).

There is hence a dearth of studies that compare SNA-complementary quantitative and qualitative methods as adopted to answer research questions on PNs. In trying to fill this gap, this article connects streams that have so far barely engaged in systematic dialogue, i.e., method-oriented research and research-question-led analysis (Bernhard 2018; Hesse-Biber 2015). Thus, S1 and S2, which aim to answer social science questions in the public policy and administration domain, provide empirical material for a comparative methodological reflection between qualitative and quantitative methods that combine with SNA, utilizing qualitative and quantitative text analysis proxies.

In both single-method and mixed-methods studies, the procedure for collecting information on actors and ties based on which to construct SNA sociograms is an aspect that deserves attention. Fuhse (2023) notes that most research employs questionnaires and interviews for this research step. However, the same author also acknowledges that the data collected thanks to these tools is more prone to distortion effects and missing data from non-response, compared to process-generated, non-reactive data such as that stemming from written text that does not come from individual reports of dyadic ties. This article adopts the latter option in both S1 and S2. After collecting data from meeting documents for both studies, sociograms were designed using NetDraw, an application with powerful analytical capacity (Williams and Shepherd 2017). A typical SNA procedure develops in two main steps: first, the researcher identifies the nodes (actors) of the social network; second, s/he analyzes the ties (relations) between those nodes and the shape of the network. In this article, the analysis returned matrixes that were translated into sociograms based on the absence/presence (S1) and frequency (S2) of reciprocal relations. Then, S1 was complemented with manual coding of meeting minutes and semi-structured interviews (Leech 2002) with key network stakeholders to identify hints of causality regarding the role of networking activities in policy learning and change in MSs. Analogously, S2 has been combined with automated and semiautomated TM and NLP to find hints of reciprocal cultural adaption in the language adopted over time by the examined countries.

3 Combining SNA with qualitative methods

S1 investigates the relevance of the Employment Thematic Network’s (ETN) meeting discussions to the public policy cycle in the participating EU MSs (Howlett et al. 2016) and, more specifically, the role of the network's activities to policy learning and change in those countries. The ETN is a mutual learning PN established under the aegis of the European Commission and composed of national authorities with a role in social and labor policies, such as social (and labor) policy ministries, public employment services, trade unions, NGOs, training institutes, and private companies, from sixteen EU MSs that decided to participate. Its members held periodic meetings to share knowledge about their employment and social policies and pave the way for improvement in these areas. S1’s guiding hypothesis is that the more MSs engage with their peers in transnational network activities, the more they are likely to experience policy learning and change.

The structural SNA for S1 looks at the relationships between meeting participants within the ETN context, to obtain a picture of their engagement in the network. These relationships are drawn by examining the network’s meeting minutes (see Appendix). Different participants-nodes take on different centrality values depending on the chosen centrality measures. The qualitative data regarding the contents shared with meeting attendees have been isolated by extensively reading and coding the meeting minutes, paying attention to the country that was sharing this content and the countries attending the meeting where the content was enunciated. Thanks to this qualitative data collection, in the sociograms, the direction of the links, represented by arrows, indicates whether each country was a knowledge contributor, a knowledge receiver, or both. Table 1 shows binary values: ‘1’ stands for provided contribution to the other MSs, ‘0’ stands for zero contribution, and the double occurrence of ‘1’ in the two boxes representing the relationship between two given MSs indicates that the two countries exchanged experiences mutually, hence the twofold direction of the related arrow. Thus, the matrix (Table 1) and the sociograms based on it (Fig. 1) are founded on the intersection between co-attendance data and knowledge provision data emerging from the meeting minutes available in the ETN repository.

Table 1 Overview of knowledge sharing in ETN meetings
Full size table
Fig. 1
figure 1

Source: author’s elaboration based on Zhan et al. (2017)

Name, function, and visualization of three different centrality measures.

Full size image

When using SNA software for sociogram construction, the most connected nodes are pushed to the center and the least connected nodes to the periphery of the network by a standard spring-embedding algorithm (Williams and Shepherd 2017). Three centrality measures are relevant to S1 in light of the guiding hypothesis: degree centrality, Eigenvector centrality, and betweenness centrality (summarized in Fig. 1).

In Fig. 1, we note that Belgium is central according to all centrality measures, while the centrality of the other countries varies depending on the measure employed. To complement and make sense of the structural analysis illustrated above, a non-automated qualitative text analysis has been conducted to identify the evidence relatable to the policy cycle. This has involved the manual qualitative coding (Hendren et al. 2023) of meeting minutes and related material (attached reports and presentations) available in a dedicated European Commission repository.Footnote 2 In this second coding procedure, each meaningful text portion has been assigned to a specific policy cycle stage based on its relevance to it (Malandrino 2020). Finally, to obtain proof of occurred policy change, semi-structured interviews have been conducted with (n = 5) key informants from inside the network. This interviewing technique has been chosen according to its being “middle groundFootnote 3 (…) that can provide detail, depth, and an insider’s perspective, while at the same time allowing hypothesis testing” (Leech 2002: 665). The key informants are those who responded to an email sent to all the network participants for whom it was possible to find a valid email address. The interviews have been carried out in English, either orally (using teleconference softwareFootnote 4) or in writing, based on the interviewee’s preference. Since all interviewees were high-level officers or experts and their amount of time was limited, it was decided to opt for a short six-question format (see Appendix).

The interviews have made it possible to gain insider knowledge to test (and, eventually, disconfirm) S1’s hypothesis. They highlighted how policy learning and change occurred especially in two countries with a non-pivotal position within the network: Slovenia and Lithuania. However, while Slovenia was peripherical according to all centrality measures, a visual analysis of the sociograms shows that Lithuania scored fair values in terms of both degree centrality (high number of connections) and Eigenvector centrality (proximity to nodes with a high number of connections). The interviewing qualitative tool, together with contextual investigation enabled by document analysis, also helps to explain why these two countries—rather than more central countries such as Belgium or Italy—were loci of major policy learning and change. Their relatively recent access to the European Union made some shortcomings in employment/social policy emerge and these countries were part of the network also to benefit from their peers’ experience with more advanced practices (Scoppetta and Aparicio Jodar 2019).

4 Combining SNA with quantitative methods

S2 relies upon the distinction between groups of European countries characterized by different administrative cultures. Some European countries have a prevalently procedural administrative culture, characterized by legalism and proliferation of regulatory measures and administrative burden, while other countries are considered managerial or quasi-managerial public administrations, with a greater focus on the efficiency of their action and the use of management techniques inspired by the private sector and the New Public Management movement (European Commission et al. 2018). In line with sociological neo-institutionalism (cf. Section 1), S2 hypothesizes that participation in a transnational PN triggers progressive cultural adaption between these administrative types.

S2 examines the Bologna Process network, where European governments discussed higher education policy reforms in an attempt to overcome obstacles to create a European Higher Education Area. An SNA has been conducted focusing on key nodes represented by three procedural (France, Germany, Italy) and four managerial and quasi-managerial (Finland, Netherlands, Sweden, UK)Footnote 5 countries. France, Germany, and Italy have been chosen for the analysis due to their reciprocal cultural similarities, rooted in Weberian bureaucracy and legalistic approach (Kuhlmann 2010). Finland, the Netherlands, Sweden, and the UK are instead labeled by the relevant literature as either managerial countries or countries shifting away from a legalistic paradigm (European Commission et al. 2018; Pollitt and Bouckaert 2017).

Similarly to S1, the SNA was carried out based on meeting minutes. More specifically, in S2, SNA employs the meeting minutes of the Bologna Follow-Up Group (BFUG), i.e. the group responsible for overseeing the implementation of the reforms agreed upon by the European ministers in charge of higher education. These meetings are attended by representatives of government and higher education stakeholders including students, staff, institutions, businesses, and the European Commission. The SNA output (Fig. 2) illustrates the meeting co-attendance frequency between pairs of countries. Graphically, the weighted—rather than binary– co-attendance variable is rendered through thinner and thicker lines (O'Malley and Marsden 2008).

Fig. 2
figure 2

Sociogram based on BFUG meeting co-attendance

Full size image

Table 2 shows how the differences found between pairs are neglectable and hence do not indicate different patterns of interaction between actors, whether or not these are based on cultural affinities (Roth and Cointet 2010). Figure 2 displays a “circle” network where actors’ degree, i.e. the number of their direct connections, is constant (O’Malley and Marsden, 2008). Since SNA in S2 is conceived to be based on quantitative measurements only, a more in-depth analysis—for instance concerning the types of exchanges that occurred during the meetings—has been avoided. The trivial value differences also make it futile to undertake a correlation analysis. Luckily, the next step (TM and NLP) does not depend on the SNA output, which in this case ends up merely illustrating the network structure and proving the existence of substantial, homogeneous cooperation via the meeting co-attendance proxy. In this context, NLP and TM will aim to find out whether this cooperation gave rise to change in the report language as a proxy for administrative culture and identity.

Table 2 BFUG meetings co-attendance frequency
Full size table

This structural SNA has been combined with quantitative TM and NLP of the Bologna Process country reports, which show national progress in implementing recommended reforms. TM and NLP refer to techniques based on the use of computer technology to analyze text in order to obtain valuable information (Wu et al. 2021). The analyzed countries have been further selected for the benefit of comparability, as the procedural culture is prevalent in Europe and the countries strictly labeled as managerial are only the Netherlands and the UK (European Commission et al. 2018). Hence, for analytical purposes, they are complemented by two procedural countries (France and Italy). The four country reports are available for the same year series (2003, 2005, 2007, 2009)Footnote 6 and have been collected from the national sections of the Bologna Process website. The reports have been denoised (Wu et al. 2021) by removing the format standard text,Footnote 7 preprocessed by deleting stopwords and punctuation, and normalized by converting the remaining text to lowercase. The clean report text has been quantitatively analyzed to interpret social values and norms. To this end, S2 employs both automated and semi-automated quantitative TM and NLP techniques, namely word frequency analysis, topic modeling, cluster analysis (with computation of the Jaccard similarity index), and topic classification based on word vectors.

4.1 Word frequency analysis

Text analysis techniques based on word frequency are quite common in the social sciences (e.g. Bail et al. 2017). In this study, the words ascribable to procedural and managerial administrative culture have been selected from the top 150 words and an inductive query in NVivo has aimed to find out the general (non-diachronic) quantity of legalistic and managerial language in the two groups. The query has been carried out primarily to show that the language employed in the reports effectively mirrors the administrative culture and is hence a good proxy to assess its leaning toward one type or another. The results are shown in Tables 3 and 4. As expected, managerial words prevail in managerial countries and procedural/legalistic words prevail in procedural countries, thus confirming the aptness of language as a proxy for culture.

Table 3 Managerial and legalistic language in procedural countries
Full size table
Table 4 Managerial and legalistic language in managerial countries
Full size table

4.2 Topic modeling

To obtain more fine-grained results, the country report series for the examined years have been aggregated and inspected by conducting a topic modeling analysis in R-4.3.1 based on Latent Dirichlet Allocation (LDA)Footnote 8 which aims to discover latent topics describing a certain collection of documents (Hollibaugh, 2019; Lossio-Ventura et al. 2021). Since S2’s focus is on language as a proxy for culture rather than on the main report topic (which concerns higher education), standard English stopwords to be removed have been complemented by an ad-hoc list of words identified through repeated automated topic modeling exercises and selected to make the cultural element emerge. Working on clean text data, LDA topic modeling allows automatically grouping key terms by theme (code in Fig. 3). Figure 4 illustrates the plots referring to the diachronic evolution of themes in the aggregated procedural and managerial country reports, with their respective topic models. The graphical representation reveals a pattern that aligns with the cultural adaptation hypothesis. More specifically, in procedural countries (Fig. 4’s first section) the legal “topic” (no.1) is prevalent in the first reporting year and comparatively denser than in managerial reports (Fig. 4’s second section). In these latter reports, topic identification appears less clear-cut, although hints of managerial culture can be traced in topic no.3 with words such as “quality” (which however often appears in procedural reports, as well), “funding”, “staff”, and “number”.

Fig. 3
figure 3

R code for grouping terms by topic and displaying them on the chart

Full size image
Fig. 4
figure 4

Topic models and distribution over time in procedural and managerial reports Footnote

As automatized text preprocessing has been conducted on the (aggregated) original PDF reports, some symbols have escaped the cleaning process

Full size image

4.3 Cluster analysis

Complementarily, a cluster analysis (Fig. 5) has been carried out in NVivo and the Jaccard similarity index has been calculated for same-year pairs of report groups (procedural-managerial), to assess change in similarity over time. Document clustering is a technique used to analyze text automatically. Rather than identifying latent topics, cluster analysis consists in splitting documents into a certain number of groups based on a similarity measure (Lossio-Ventura et al. 2021). The Jaccard index has been preferred to other similarity measures, such as Salton’s cosine, due to its suitability to measure broad conceptual overlaps (Leydesdorff 2008). Again, as expected, the analysis shows some hints of cultural adaption in the language adopted by the two groups of countries over time, as similarity progressively increases (Table 5). This kind of analysis, however, does not show the direction of change: it does not explain if procedural countries adapted to managerial culture or managerial countries adapted to procedural culture. Besides, its output might just be a consequence of the very scope of the Bologna Process.

Fig. 5
figure 5

Cluster analysis of aggregated managerial and procedural country reports Footnote

2003_man…” and “2003_proc…” respectively stand for the aggregated managerial and procedural country reports for reporting year 2003. The same applies to the other years in the series

Full size image
Table 5 Change in similarity between procedural and managerial countries’ report language over time
Full size table

To address this, a complementary diachronic query has been carried out to find percentage trends of managerial and procedural language over time in the two groups of documents. Table 6 and Fig. 6 show that both procedural and managerial countries might have reinforced their styles as proxied by language: managerial and procedural language increased, overall, respectively for managerial and procedural countries. However, this could be considered a hasty conclusion: the trends are not steady over time and this result could be judged as random if not corroborated by further evidence.

Table 6 Language change in managerial and procedural countries (% values)
Full size table
Fig. 6
figure 6

Change in language in procedural and managerial countries (% values)

Full size image

4.4 Topic classification

It might be argued that the inefficiency of some of the above techniques is due to their fully automated nature. To address this criticism, a topic classification based on word vectors has been conducted to analyze the data. Unlike topic modeling, topic classification is a supervised text analysis method that requires labeled training data to classify documents (Osnabrügge et al. 2023). Here, training data have been labeled starting from the definitions of managerial and procedural culture (Table 7).

Table 7 Topic relevance in managerial and procedural reports
Full size table

The pre-trained topic classification reveals an interesting result in terms of slight reciprocal alignment of (procedural) language and procedural countries’ resistance to the adoption of managerial language (Fig. 7).

Fig. 7
figure 7

Topic classification over time

Full size image

Overall, however, the employed quantitative TM and NLP techniques provide contrasting hints that seem to either corroborate or reject the S2 hypothesis. Hence, they alone can barely represent valid evidence to test it.

5 Systematic comparison and discussion

Table 8 shows a comparative assessment of the two empirical studies examined in terms of time needed for research, human and research resources employed, unobtrusiveness, and effectiveness toward the goal of telling meaningful stories about the examined networks (Crossley 2010; Yousefi Nooraie et al. 2020), thus evaluating the effectiveness of the employed methods also in light of their respective costs.

Table 8 A systematic comparison of qualitative S1 and quantitative S2 in terms of time, resources, unobtrusiveness, and effectiveness
Full size table

The time dimension includes all the preparatory operations needed for each step. For instance, interviews in S1 include the time needed to draft the questions, write reach-out emails and interact with interviewees to find a suitable time for the interview, as well as the time spent for the interview. In this regard, it must be noted that the total amount of time needed for interviews can be considerable. This data collection method is hence considered to be particularly appropriate with small/medium scale networks (Zhang et al. 2021). Likewise, the time computed for the TM and NLP techniques employed in S2 includes the time needed to preprocess the reports as detailed in the previous section. Overall, S1 has been almost twice more time-consuming than S2, despite the same amount of time allocated to minutes-based SNA. S1 was slightly more demanding in terms of human resources, although the 5 key informants were recruited on a voluntary basis and did not represent a cost in financial terms. However, if we look at skills, we note how the combination of SNA and TM/NLP techniques requires programming competencies, which are currently not as widespread in the social sciences (Bail 2023). In terms of time and human resources, both studies are fairly sustainable, with the same researcher carrying out the research during an overall limited amount of time (with an 8-h working day, even the higher amount—113 h—corresponds to just around 14 working days). This was possible thanks to the prevalently document-based nature of both studies. We can however conclude, provided that the involved researchers are equipped with the necessary skills at the start of the research process, that qualitative research can be slightly more time-consuming, especially when relying on non-automated (as is usually the case) data collection and analytical techniques (Hermans and Thissen, 2009).

A different assessment concerns (other) research resources (than time), which include documental material, software, and storage/RAM space. Here we can note that the operations needed for automated analyses that are at the basis of quantitative methods—such as text cleaning and preparation—can be avoided in (manual) qualitative research. This applies to studies involving a low-medium number of documents, as computer-assisted qualitative procedures can be applied to higher numbers of documents. From a different perspective, the public availability of internal documents has been an important asset for the study. As mentioned, the meeting minute source was chosen primarily because of its aptness to provide first-hand evidence of what happens in the meetings (Papadopoulos 2018). One significant aspect of both studies is that they leverage a kind of source—meeting minutes—that are as a rule easily accessible for all researchers, at least when concerning public organizations and networks. Under the transnationality umbrella, for instance, the European Commission made available meeting minutes and documentation relating to nine thematic networks, of which one (the ETN) was analyzed in S1. Likewise, the national reports examined in S2 were all made publicly available by the Bologna Process Secretariat. From a different perspective, designing studies that are reasonably economical from a resource deployment point of view allows researchers to meet concerns related to the increasingly scarce resources in universities (Schuelke-Leech 2013). However, difficulties may arise with document-based SNA. First, this might occur when studying networks whose documentation is not publicly available—e.g., networks made of private actors or networks where the topics dealt with are particularly “touchy” (Papadopoulos 2018). Second, we cannot assume that the information we collect from available documents will be useful for generating a meaningful sociogram, as shown in S2. Third, complications can arise due to repository arrangement issues regarding the material to be examined. Duplicates are not unusual in such contexts, while vice versa some documents are not immediately available once one lands on the network repository webpage, so the data collection process must be accurate in order not to miss relevant information. These difficulties stem from the fact that, notwithstanding the access and transparency obligations that public organizations must comply with, documents such as meeting minutes are not conceived for the main purpose of being read by the general public. They are, after all, internal documents whose primary aim is not to fulfill informative functions targeted to the outside of the network, but rather to ensure the smooth internal functioning of a cooperation forum. Despite these difficulties, documental analysis, whether qualitative or quantitative, can help optimize the use of resources provided that document access is not prohibitive and that the use of qualitative methods can be employed at least at one stage of the analysis—for instance in the qualitative coding phase preliminary to SNA as done in S1.

An important point to be considered concerns unobtrusiveness. Unobtrusiveness in research can be defined as a kind of interaction with information sources that is socially accepted and occurs in a non-distracting way (cf. Webb et al. 1999). This key feature of social science research has been ensured in both studies. In principle, both quantitative and qualitative text analysis can be unobtrusive. As for interviews, while they imply a necessarily higher degree of perceived obtrusiveness than the analysis of pre-existing documents, advancement in social science research has produced several precautions that can be taken to make interviews as unobtrusive as possible. In S1, no reference was made to the personal role of the key stakeholder in knowledge exchange or other network activities, interview questions were framed in a very formal, academic way through reference to the policy cycle phases—rather than through reference to the adoption of policies by specific actors that the interviewees might know personally—and the anonymity policy of the study was made well clear to the five key stakeholders from the very beginning of the interviewing process. On the other hand, these steps can be seen as part of a bargain that the researcher makes with interviewees to obtain the information they need, which can at times undermine data transparency. Overall, interviews are more obtrusive than pure documental analysis—whether qualitative or quantitative; however, there are golden rules that can limit this drawback of non-documental methods.

A central point to be examined is the effectiveness of the employed methods to tell eloquent stories about the examined networks in light of set research hypotheses. The interest in effectiveness stems directly from a pragmatist approach that seeks the best methods to solve given problems and answer given research questions (Mele and Belardinelli 2019). The freedom to engage in qualitative reading, interviewing, and coding produces important results in this regard. In S1, it allowed an understanding of the relevance of network activities to the public policy cycle, tracing the direction of exchanges and finding hints of causation in policy learning and change. This does not mean that qualitative research allows coming to definitive conclusions regarding causation, but accessing insider knowledge means leveraging already-processed mechanisms that would be hard to trace through quantitative text analysis methods (Hermans and Thissen, 2009). Hence, despite the increasing sophistication of existing formal quantitative and structural analyses, it can be worth relying on qualitative tools and methods to complement SNA in order to grasp the context and functioning of networks (Bellotti 2016; Mason, 2006). The analysis of structures and connections that constitutes the objective of formal SNA needs indeed to be complemented with an active interest for meaning (Fuhse and Mützel 2011), which in S1 is achieved through the application of qualitative methods. Substantial qualitative methodological complements are likely to add nuance and depth to quantitative studies when complex social phenomena are being examined (Hendren et al. 2023), thus helping to answer relevant “how” and “why” questions (Mason, 2006) and avoiding reductionist interpretations of those phenomena (Gilad 2019). This applies to all social science disciplines that deal with multifaceted social dynamics regardless of whether the research object is ascribable to the public policy and administration domain.

There is hence a need to reflect on the research goal that mixed-method SNA-qualitative research designs can help to achieve. Extending Peshkin’s (1993) classification to mixed-methods designs, we know that research can aim at description, interpretation, verification, or evaluation. Both S1 and S2 fall into the interpretation, verification, and evaluation categories, which are respectively about developing explanations, testing the credibility of claims, and providing “an examination of policies, practices, and innovations—how they are implemented, their impact, and what the process has entailed” (Hendren et al. 2023: 470). Hence, they do not have a merely descriptive goal, which would be about exploring a certain field or object of study. Towards the set research goals, qualitative methods have proven to better enable the researcher to get “an ‘understanding’ (in the Weberian sense) of the meaning embodied in networks, and the processes of creating, sustaining, and modifying this meaning” (Fuhse and Mützel 2011: 1078). Since in many social science studies, including the present one, hypothesis testing is precisely functional to achieve a better comprehension of mechanisms and dynamics, all in all, qualitative research has proven to boost explanatory power (Mason, 2006). Overall, in this article, quantitative NLP and TM techniques—both automated and semi-automated ones—have demonstrated their usefulness to guide the researcher through the general patterns of text. This makes them perfect complements for qualitative analyses. However, quantitative text analysis is less suitable as a standalone method if we want to understand networks. Unlike quantitative TM and NLP, qualitative text analysis presents the merit of understanding the deep meaning behind numbers. Let us take the term “funding” that appears in the topic modeling of managerial reports: we can assume this term indicates a managerial concern for resources, which would confirm the embeddedness of the related type of culture in the examined countries’ public administration. However, this might remain just a guess without deeper knowledge of the network and country context, which can be obtained via qualitative research. A strong qualitative component in mixed-method studies that also involve quantitative methods is indeed likely to gain new perspectives, clarify the context in which the examined phenomena take place, and all in all provide a deeper and more nuanced understanding of those phenomena (Hendren et al. 2023). In disciplines that take complex phenomena as investigation objects, such as public policy and administration, qualitative research takes on this essential function thanks to its information-rich nature as well as the central role of the researcher in considering the existing contextual framework and assigning meaning to social actors’ experiences (Hesse-Biber, 2015; Mason, 2006). In mixed-methods studies, this function is performed provided that the qualitative component’s weight is adequately balanced with the quantitative component’s weight, i.e., that none of them play a residual role in a study’s design (Hendren et al. 2023).

Qualitative research can occur in the form of qualitative data gathering (e.g., interviewing) and qualitative data analysis (e.g., qualitative coding). At both these stages, it can complement SNA to find out which mechanisms are operating in a given network. Crossley (2010) distinguishes between five mechanisms to explain why certain actors are more likely to connect: homophily (actors with similar traits), propinquity (actors who are geographically proximate to each other), transitivity (actors who share a common alter), preferential attachment (preference for connection with certain actors), focus (actors with an interest in common). Qualitative analysis in S1 highlighted a further mechanism, which could be named purpose or function: in our empirical case, actors with more advanced employment and social policies take on the function of practice disseminators for the benefit of actors with less advanced policies.

From a different perspective, it can be noted that some steps of SNA, such as the selection of centrality measures and the translation of numbers into words, also have a qualitative taste. While the application of mathematical/statistical techniques and the graphical presentation of results make SNA a substantially quantitative approach to network study, “subjectivity is an integral aspect of network analysis, manifested in the selection of measures to describe connection patterns and actors’ positions (…), in the visualization of social structure in graphs, and in translating numbers into words” (Yousefi Nooraie et al. 2020: 110). The same applies to TM and NLP techniques, as manifested, e.g., in the inductive selection of relevant words to be detected or in the addition of stopwords to make the themes of interest emerge. From this perspective, even when applying quantitative methods, the qualitative element that each of these techniques involves allows us to get closer to understanding our research object. Analogously, enriching a quantitative-structural approach such as SNA not only through qualitative SNA operations but also through additional qualitative analyses may allow researchers to think “outside the box”, creatively and multi-dimensionally (Mason, 2006) and hence to better answer research questions about complex social phenomena.

Overall, the picture is that of two sustainable studies from the point of view of resources. However, if we look at the results, S1 and S2 greatly differ in terms of effectiveness. By employing qualitative methods in combination with SNA, S2 achieves the goal of putting flesh upon the bones of aseptic measures (Crossley 2010). This achievement is precluded in S2, which only relies on automated and semiautomated quantitative text analysis processes. It may be argued that more advanced quantitative methods, even beyond text analysis techniques, could perhaps allow a better comprehension of the phenomena under study. On one side, it is a matter of scope: this study specifically focuses on text analysis, as it would be undesirable to apply and review all possible qualitative and quantitative methods. On the other side, large-n reviewing research convincingly shows how more sophisticated methods do not necessarily imply more explanatory power (Low-Décarie et al. 2014). All in all, quantitative methods appear to respond less promptly to the interpretative function requested of a method that proposes to complement an already-quantitative, structural approach such as SNA.

Essentially, methods must be understood and evaluated within the context of the function they perform in relation to the research object—in this case, PNs. Relationships, including in networks, cannot be reduced to the presence/absence (or higher/lower frequency) of ties between actors. These are important aspects but, to understand networks, it is necessary to look at two essential elements: the qualitative features of the relationships and their context. Paraphrasing Crossley (2010), SNA is about abstracting and isolating form from content, which can be analytically useful, but qualitative research is needed to look at the content that must necessarily be taken into account if we aim to tell a meaningful story about our network. If, on the contrary, we combine SNA with other quantitative methods, we continue abstracting forms—perhaps from different perspectives, as exemplified by similarity analysis in S2—but we might still be missing focus on content.

6 Conclusions

This article offers a comparative assessment of qualitative and quantitative text analysis methods that complement document-based SNA to achieve a better comprehension of PNs. Its contribution is threefold. First, it systematically compares combinations of SNA with qualitative and quantitative methods representatives, thus complementing the existing noncomparative literature on mixed-methods SNA. Second, it employs SNA with network visualization based on text that does not depend on actors’ input, which is only rarely employed for sociogram construction (Fuhse 2023). Third, it connects method-oriented comparative research to research-question-led analysis, thus avoiding a narrow focus on methods only, as recommended in the existing literature (Hesse-Biber 2015). A limitation of this study is that it does not allow for a systematic comparison of all possible qualitative and quantitative methods, since it focuses in particular on text analysis. Hence, it is essential to call for more research that compares different qualitative and quantitative approaches and methods to answer research questions related to social networks in specific disciplines and domains. Moreover, further method-oriented research should be devoted to investigating ways to integrate quantitative and qualitative methods (not only with SNA) and the results of their application, to achieve meaningful combinations of findings obtained through different research processes.