1 Introduction

Since the beginning of the current century, the acronym PISA (Programme for International Student Assessment) seems to pervade the multiple contexts where the education systems and their governing are subjected to debate. The presence of this Organization for Economic Cooperation and Development’s (OECD) international large-scale assessment in national spaces has diverse manifestations, such as supporting “analyses and rationales” for the discussion of specific issues, or being used as a “source” for secondary studies or as “learning opportunity” for the development of accountability policies (Lawn and Grek 2012). So far, PISA became a central element of a universe of knowledge, which, paraphrasing Lindblad and Popkewitz (2004: xx–xxi), ensures that expert-based education policies can lead each nation into the so-called knowledge society. The chapter addresses this umbilical relation between governing and expert knowledge by examining the meanings and processes that sustain PISA contemporary status of indispensable resource for the imagination and scrutiny of educational issues and policies.

Based on the revision of a previous article on the organizational and cognitive dimensions of PISA (Carvalho 2012), and on the posterior works on the uses of PISA in national and supranational contexts (Carvalho and Costa 2015, 2016; Carvalho et al. 2017) and on the intensification and sophistication of PISA’s association with the policy processes (Carvalho 2014, 2016), the chapter focuses on the fabrications, that is the fictions and the making (Popkewitz 2000a, b) that support the projection of PISA as a central element for thinking-acting education policies.

In fact, the chapter retrieves two pillars of the approach to PISA followed in my previous texts: (1) the representation of PISA as a knowledge-policy tool; (2) the understanding of the status achieved by PISA as the result of the making of a proper ecology. Thus, the first section of the chapter clarifies the notion of PISA as a tool that combines assessment techniques with a set of representations about education and a philosophy for the governance of education, and discusses its power effects. The second section focuses on the making of a PISA’s ecology as a process that entwines cognitive and social practices related with the construction and rise of the subject of inquiry (the competences of literacy) and, simultaneously, building public confidence and dependence on PISA. The chapter concludes with a proposal of new challenges for researching the trajectories of PISA in national contexts.

2 PISA as a Knowledge-Policy Tool

Along the last 15 years, OECD has reiterated the idea of PISA as a response based on specialized knowledge to meet the need expressed by national governments for useful and credible data on their performances (OECD 2001, 2007, 2014). This overt policy-oriented nature was recently retaken in Beyond PISA 2015: A longer-term strategy of PISA, a document that equates the future of the program:

It focuses on providing data and analysis that can help guide decisions on education policy. By linking data on students’ learning outcomes with data on key factors that shape learning in and out of school, PISA highlights differences in performance patterns and identifies features common to high-performing students, schools and education systems. (OECD n.d.: 1)

The OECD’s self-portrayed quality monitoring tool is observed differently along this chapter: PISA is rather observed as a device that embraces and conveys different ways of imagining (and doing) education, schooling, and social research, and—simultaneously—plays a part in the coordination of education policies and public action.

2.1 Fabricating Education Systems and their Steering

Drawing on the concept of “public policy instrument” as put forward by Lascoumes and Le Galès (2007: 4–6), PISA is approached as a tool that organizes social relations between administrative and administered subjects according to specific interpretations of the social world it addresses and based on a specific concept about the ways it should be oriented, coordinated, and controlled. From their point of view, each instrument is a combination of technical components and social components, that is, values, interpretations, and concepts about the social realities it describes. Regarding PISA, one might say that its sophisticated techniques are driven by a specific problematization of the role of education in contemporary times, displaying a particular way of challenging the national policies. Concomitantly, PISA bears principles and practices for policy processes.

On the one hand, PISA operates over several core categories of schooling (to a certain extent rewriting the educational model of contemporary societies), redefining students as lifelong learners, redefining teaching–learning relationships and settings, and redefining school knowledge.Footnote 1

To clarify this statement, one should look to the innovative focus that PISA claims to bring to assessments: “rather than examine mastery of specific school curricula, PISA looks at students’ ability to apply knowledge and skills in key subject areas and to analyze, reason and communicate effectively as they examine, interpret and solve problems” (PISA Website). This definition allows OECD to move away from the conventional self-reflection of national school systems based on their own categories and “outputs” (with assessments relying on tests and examinations based on national curriculum goals and content) and toward the territory of “outcomes,” thus directly connecting the contexts, practices, and results of teaching/learning with the so-called demands of the school system environment. Parallel with this differentiation comes the redefinition of the appropriate school knowledge for the so-called knowledge society: the notion of competence enacts a “utilitarian perspective” on knowledge as it takes practical usefulness in solving everyday problems as the main criterion for the assessment of school knowledge (Mangez 2008: 102–104). Therefore, it promotes a restructuring of curricular composition in trans/cross-disciplinary terms. These shifts go hand in hand with calls for change in teaching and learning structures (for example, from “hierarchical” to “organic” models or from the sequentially transmitted “bodies” of knowledge to the construction of learning by means of students’ cognitive connection with what they already know).

On the other hand, PISA also sustains particular cultural dicta about policy makers in contemporary times.

It redefines them as “problem-solvers” and “policy-learners,” which are to be decision-makers guided by searching for competitive advantages, measuring the outcomes of the school system, identifying weakness, and adopting solutions based on what works in other systems. That is to say, policy-makers that learn about competitors so as to progress more quickly. Moreover, PISA is nurtured by and nurtures several dicta on governing processes: the primacy of the rational and evidence-based model for the coordination and control of actions in the education sector, contrasting with ideological and/or opinion-based coordination; the free acquiescence of decision makers to be involved and to support, both materially (financial and other resources) and symbolically (with belief and praise) mutual surveillance as an expected and effective practice; and the systematic assessment of student literacy performances as a useful and trustworthy resource for the steering of educational systems.

2.2 Fabricating Transnational Governance

Returning to Lascoumes and Le Galès perspective (2007: 3), each policy instrument (a) “constitutes a condensed form of knowledge about social control and ways of exercising it,” and (b) each instrument produces its own effects “which structure public policy according to their own logic.” In order to discuss the presence of these features in PISA, we need to consider the OECDs’ history, resources, ideas, and choices.

First, PISA has a course that is inseparable from the OECD trajectory in the transnational governance of education. Generated in the context of the OECD’s project on the International Indicators of Educational Systems (see Morgan 2011), PISA implements an education agenda marked since the 1990s by the idea of monitoring quality, and involved in a continuous manufacturing of problems and solutions for the so-called knowledge economy (Rinne et al. 2004). It is also the most fruitful example of the OECD’s “comparative turn” (Martens 2007) and of its “infrastructural and epistemic” governance that, as Sellar and Lingard (2013: 13–14) pointed out, generates a “self-perpetuating dynamic” in which the OECD “both prescribes education policy approaches and assesses the performance of national education systems in these terms.” This is one of the particularities of the OECD intervention: a focus on “surveillance of performances” and “assessment of policies,” aiming to impact in national policies as a “creator, purveyor and legitimator of ideas” (Mahon and McBride 2008: 7–15).

Second, PISA operates through the power of guilt and hope. On the one hand, it operates through the culpability and the responsibility that it conveys to national spaces, because, together, school systems’ positioning (in a competitive space) and numbers (systems’ performances) bring “naming, blaming, and shaming” to the national policy spheres and actors. On the other hand, PISA operates also providing optimism for the possibility of reform based on evidences, and creates confidence in national policy actors as effective reformers. Moreover, PISA is an actant that brings the comfort of criticizing or legitimizing policy problems and solutions with the blessing of a putative universal, independent, expert knowledge.

2.3 Aggregation Effect

What happens when PISA frames, data, and analysis circulate (almost) worldwide? For the last 15 years, supplementary visions have been added to PISA along its travels. Thus, new knowledges, new policies, and politics have been gathered to it.

The credibility and sense of usefulness achieved by PISA is traceable in the variety of sociopolitical mobilizations of PISA already identified by a quite extensive literature on the reception of PISA in European contexts (see Carvalho and Costa 2016): there is an assignation of multiple purposes to PISA, namely of legitimation (i.e., legitimating reforms, specific policies, assessment instruments), information (as a complementary or as a compensating/substitutive source for the steering of education systems), and idealization (supporting the construction of diverse educational ideals, projections, or narratives, about education and educational reform).

Therefore, PISA objects/texts are ubiquitously present in national contexts, by the hands of different users (politicians, and other players involved in public educational debates, national experts, and researchers prolifically using it for secondary analysis). However, they are reinterpreted and made acceptable and efficient for each sociocognitive context. Thus, they are subject to diverse selections, either regarding the information displayed in the reports or the policy domains addressed by PISA recommendations. The summary of a recent review of the research on PISA effects stresses these same trends: on the one hand, “PISA has a strong influence on a variety of national reforms (…) however this influence strongly depends on domestic policy contexts” (Pons 2017: 131).

In sum, divergent uses and effects (regarding specific political choices or solutions, or to interpretations and uses of PISA products by other social actors) coexist with a convergence toward the tool. This phenomenon supports the constitution of PISA as a taken-for-granted source for public policy actors. As discussed before (Carvalho 2012), this is neither a paradox nor an inconsistency of the tool, but an effect of its power: the signal of the proficiency of a tool that keeps actors and agencies (that operate in different social worlds, and at regional, national, and supranational spaces) bound by/to PISA’s multiple activities and products. In other words, PISA is effective because of the—convergent and divergent—engagement and participation of multiple actors in its own production, dissemination, use, and consumption. Thus it performs an aggregation effect (Lascoumes & Simard, 2011): controversies, disputes regarding the analysis of data, and competing thoughts on solutions for “education problems”—the imagination and/or the scrutiny of educational systems, policies, and practices—are recurrently made by a dependence/commitment to PISA.

To understand how this effect is achieved is important to connect the dimension of the ideas, frames, expectations, and prescriptions generated by PISA to the dimension of the organizational processes that structures the relations of interdependence between the actors involved with PISA, and that concomitantly put into circulation and legitimizes specific ways of understanding what education “is” and how it should be governed. This implies taking into account the OCDE intervention, according to its institutionalized modus operandi: idealizing, aggregating actors, supervising interdependencies (Marcussen 2004) but also intense and varied diffusion of informational products generated to diverse audiences. Both issues are addressed in the following section.

2.4 PISA as the Making of an Ecology

While accepting that trust in the OECD as an “expert organization” (Noaksson and Jacobsson 2003) precedes and promotes PISA’s public judgment as a valid and useful tool, not forgetting that the success of PISA capitalizes from a cultural and political environment that disseminates the conceptions of global economic competitiveness and the knowledge economy (Broadfoot 2000), the success of this knowledge-policy tool lies also in the making of effective connections between heterogeneous actors: public and private research centers, individual experts and researchers, OECD professionals, policy-makers, high-level civil servants and technicians from multiple countries, media … in short, in making its own ecology.

2.4.1 Fabricating the Ecology

To grasp the success of such enterprise, it is enough to consider the growing number of countries involved and the extent of the geopolitical coverage achieved by PISA: in 2000, 43 countries took part in it (13 of which are not members of the OECD); in 2012, there were 65 participant countries (31 of which are not members of the OECD); in 2015, the date of the last assessment, 72 countries, from all continents and covering well-diverse cultures, economies, and political regimes. This broadening of participant countries/regions means also the enlargement of collective actors involved in PISA’s international and national/regional steering and management activities. The notion of the PISA “expansions” developed by Sellar and Lingard (2014) captures more comprehensively this success, including the following trends: “widening the scope of the assessment to measure a broader set of skills and competencies; increasing the scale of the assessment to cover more countries, systems and schools; and enhancing the explanatory power of the assessment for policy-makers and educators” (p. 924).

I relate these developments with the making of PISA’s ecology (Carvalho 2012). Adapting Everett Hughes’ notion of “ecology of institutions,” as quoted by Star and Griesemer (1989), this means the choices taken within the Program about its material, informational, and human sources, and also the actions carried out to establish continued and lasting exchanges with selected actors. In other words, the making of this ecology rests on a collection of practices that keeps PISA alive and expanding in a field populated by other agencies which also export educational monitoring devices and are involved in the making of usable knowledge-policy instruments for national or regional territories. In the following pages, I turn to two examples of the practices that support the effective connection to PISA by interested (individual and collective) actors: assembling and coordinating heterogeneous actors and knowledges; reaching and creating interest in heterogeneous actors not directly involved in PISA formal structures.

2.4.2 Assembling and Coordinating People and Knowledges

PISA is not merely a triennial survey and report. Together with the inquiry activities (design, trial, application, data-analysis) relevant face-to-face exchange activities take place (in meetings, workshops, seminars, etc.). Likewise, multiple publications—apart from the survey’s main reports—are generated and have a worldwide flow. Thus, is reasonable to observe PISA as a system of activities where communication and organization happen. All these activities involve a great variety of social worlds and multiple kinds of knowledge, interests, and perspectives. Thus, the accomplishment of PISA depends on bringing together—and ensuring the cooperation of—heterogeneous actors around a flow of activities, and on having them share the perception of PISA being a respectable provider of useful data/information/knowledge.

When asked about the role played in this process, the OECD Secretariat—the structure formally responsible for the management of the daily activities of PISA—presents itself as a catalyst for interaction between experts and politicians (see Carvalho and Costa 2009; Carvalho 2012). However, before becoming a “facilitator of relationships,” OECD began its own work as a provider of ideas, promoting the framework of literacy and reframing the old school system problem—the one of preparing the young generation for the future—in a much broader concern: the issue of the international competitiveness.

The OECD has generated its own initiative a specific framework—the competences of literacy—which became attributable to its own agency, as one OECD executive states:

We did a book—DeSeCo [Definition and Selection of Competencies]—it’s a definition and selection of competences … That’s the source of PISA. It is a project we had over 5 years where we had anthropologists, psychologists, labor economists and all these people telling us what is competence, what does it mean to do well in life, what is a successful life and they come up with different answers (…) it was a really great inspiration because if we had only used educators we’d have quite the common denominator of national curriculum. (Interview with an OECD executive, 2008, cit. in Carvalho and Costa 2009: 75)Footnote 2

Furthermore, the competences of literacy, generated within the OECD indicators project (Indicators of Education Systems—INES), build on the experiences (and reflections on the experiences) of many of the promoters of previous large-scale assessments: the OECD did not only reuse the methodological knowledge previously developed but also “recruited” actors previously linked to comparative international studies (Morgan 2011). Overall, according to Martens and Wolf (2009: 99), the conceptualization of PISA took 5 years and involved 300 “scientists” from all over the world.

The mobilization of diverse knowledges and experts has continued after PISA take-off. Along the last two decades, the development of PISA knowledge, disciplined by the literacy framework but also by assumptions, concepts, and methods from the psychometric world, entails contributions generated by experts from very different knowledge communities (Carvalho 2012): experts related to PISA core domains (mathematics, reading, and science) and other literacy domains invented along the PISA course (e.g., financial literacy) the “hard” knowledge of statistics, psychometrics, and compared assessment; knowledge of social psychology in relation to the study of attitudes; and streams of knowledge coming from policy evaluation and analysis, and from the school effectiveness tradition. Together OECD’s professionals and external experts collectively validate the data/information/knowledge they process. Finally, as the starting point of the studies carried out under PISA is partly defined by the representatives of the OECD member countries and the non-member countries associated with PISA, so the results of the technical work are open for their appreciation. In short, the knowledge produced and disseminated within PISA rests on consensus among OECD staff, research consortia, experts, and national representatives about what counts as usable and disclosable knowledge.

Along these processes, “the catalyst” has to fulfill positively the varied informational interests of those who gather and must, as well, keep them performing appropriately to what is expected from an organization that struggles for the status of a “truth teller” (Noaksson and Jacobsson 2003). Throughout the several sequences of tasks that make PISA (from building/reviewing each cycle framework to the delivery of publications), the “catalyst” also ensures that the interactions follow the common values and rules expected in a social space created by an expert organization, like consensus building. One good illustration is the management of PISA meetings.

From an examination of the narratives of national representatives interviewed in KNOWandPOL research about the dynamics of the PISA Governing Body meetings, a few themes come to the front (see Carvalho and Costa 2009): the OECD Secretariat has a leading role; national representatives perceive diverse types of participation, ranging from the convergent to the divergent type and from the active to the “reserved” type; the meetings are spaces where different—often conflictual—visions of education are shown, and where compromises and consensus are established, between participants with unequal resources. These unequal resources relate with the nearness–distance to the specific technical knowledge of PISA, the mastery of the English language, or the status of each country in the political-economic hierarchy of the OECD. What seems crucial though is the capacity to mobilize or to contest technical argument, because this one is perceived as having a central role in the achievement of political consensus. To many of our interviewees, the political building of consensus seems to be dependent on—and subordinate to—technical expertise on comparative assessments.

2.5 Multiplying the Interested Actors

PISA reaches well beyond the actors of its formal structures. It reaches and involves media agencies and national politicians, who selectively discuss the results, as well as national and/or regional governing and/or administrative structures that use PISA knowledge for creating their own assessment devices, and even researchers from various fields who use PISA data in order to build secondary analyses.

The connection that PISA has achieved with the media, and the importance given to it by the OECD, was already depict as a “media strategy,” involving the management of the media coverage of PISA’s triennial data release and the production of media-oriented county notes (Lingard 2016). But PISA establishes effective associations with other actors by diverse means, through a process that along the current decade has become more intense and sophisticated. One is the widening of the multiple public and private actors that the OECD puts in interaction, in order to construct meaning, articulate and diffuse new rules based on the use of PISA data—what has already been labeled as a “social matrix of interrelated governing actors” by Sotiria Grek (2010: 401) in the context of her analysis about the associations between the OECD and the European Commission. Another modality of creating these social matrixes are the “meditative” activities—borrowing this analytical category from Jacobsson (2006)—developed outside the PISA word, like the publication and exchange activities on teachers and teaching developed since 2011 in the context of the International Summits on the Teaching Profession (Robertson 2012), co-organized with Education International and national authorities. A different mean is the increase in surveys on literacy that has been enlarging the subjects and the objects of the survey: Programme for the International Assessment of Adult Competencies (PIACC) (see, e.g., Grek 2010, 2014); Assessment of Higher Education Learning Outcomes—AHELO (see, e.g., Shahjahan and Torres 2013; Shahjahan et al. 2015); PISA for Development (see, e.g., Addey 2016); PISA for schools (see, e.g., Rutkowski 2015, Lewis et al. 2016; Lewis 2017a, b).Footnote 3

Finally, a third mean is the widening of knowledge-related materials generated and disseminated to diverse audiences. In 2008, the diversity of the materials generated within PISA was already remarkable (Carvalho and Costa 2009): diverse sorts of reports (main, thematic, extensive, national, and technical); databases; documents with assessment basics, written for teachers, parents, and pupils. This variety of informational products were already explicitly generated to target populations, with diverse interests and skills. Moreover, opening up to a multiplicity of possible uses, whether in order to reproduce, to re-contextualize the data/information, or even in order to produce knowledge from it. Presently, this picture presents a few important changes: the objects become more sophisticatedly elaborated and their variety is amplified, thus fostering the intensification of possible uses. Three types of materials illustrate this move: materials exhorting policy emulation and policy learning; materials that provide their readers “short-cuts to knowledge”; and digital platforms supporting “do it your-self” (DIY) practices.

One of the new deliverables is the “in-depth” report suggestively entitled Strong Performers and Successful Reformers prepared by “task forces” of experts and members of national education bodies, involving meetings with national “stakeholders.” Reforms based on local adaptation of best practices are the expected outcome from these documents. A different type of deliverable for a quick access is developed since 2011, like the monthly briefs PISA in focus that display four to five pages of explicitly policy-oriented texts on a specific PISA theme, from student performances and attitudes toward school and learning to family background, from classroom environment to education policy. The DIY products, like the Interactive Data Selection and the Multi-dimensional data request, support different types of relationships between PISA and the users, by allowing these to select and compare data of school- and student-level variables. Connected to the Education GPS, these products allow the users to access data provided by PISA, TALIS, and Education at a Glance, as well as to prepare country reports, using texts and sophisticated charts provided by the tool, and to compare the countries’ performances. Set to activate a quasi-autonomous relation with PISA data, these technologies support the OECD (new) intervention as a “center of visualization” (Williamson 2016).Footnote 4

In sum, the making of PISA ecology rests on the condition of being consumed, shared, or learned by its audiences, as credible and manageable.

3 New Challenges for Researching the Trajectories of a Knowledge-Policy Instrument

Along the text, drawing on my previous works on PISA, I have discussed the twofold influence that PISA exerts as knowledge-policy tool, as well as the practices that sustain its regulatory action (in terms of ideas and within the realm of organized action). From what is written in the previous pages, it is possible to take PISA as an analyzer of the use of expert knowledge in regulatory processes. My present interest in PISA follows this direction, by focusing the dynamics of appropriation inherent to the trajectory of any policy instrument (Lascoumes & Simard, 2011). This opens the possibility of observing new fabrications of PISA and new fabrications made with—or based on—PISA. This means to observe and analyze what is occurring when its objects (texts, data, databases) circulate through national, regional, and local public action settings and are used by different social groups, according to their beliefs, interests, and knowledge repertoires, and under specific sociocultural circumstances.

In fact, previous studies on PISA reception and effects show that all the knowledge in circulation is transformable and transformed by the context and the actors who receive and eventually diffuse it to new audiences; but we cannot escape to question if this knowledge does not transform the very actors and contexts that transform it. It is, in fact, a relationship that needs to be further studied by the research that has been devoted to the PISA reception and effects in the governing of education. Two issues emerge as central: (1) the role of actors that, at a national scale, operate between PISA objects and policy actors; (2) the appropriation of PISA-labeled objects at diverse educational contexts—policy, state bureaucracy, schools.

  1. 1.

    One of the most open issues in the literature on PISA effects concerns the understanding of the varied uses of PISA by its multiple audiences. Several factors have been put forward to that purpose: structural, socioeconomic, or cultural aspects; national policy dynamics; interventions of the actors that transform the results into pressures on the educational agenda (see Carvalho and Costa 2016). In this last factor stands out the agency from those who intermediate between PISA knowledge and policy as “brokers” or as “entrepreneurs” (Van Zanten 2009), and the resonance of the media interpretations. In the Portuguese context, two aspects deserve special attention (Carvalho et al. 2017; Viseu and Carvalho 2018): the recent emergence of “intermediary actors” (Nay and Smith 2002) who intend to transform the PISA results into knowledge for national policy, after a long period of non-existence of this intention (and of their respective performers); the continued increase in coverage given to PISA by the national media, which is even becoming more specialized in a few newspapers. The first aspect concerns the transformation of PISA’s results into knowledge for national policies by national collective actors that activate diverse mechanisms of “translation” (Callon 1986) and generate compositions of technical and political arguments—argumentaires (Pons 2012)—to explain PISA results to the politicians and largely to “the public.” The second aspect concerns the identification of the ways in which the press mobilizes PISA and portrays the Portuguese performance and, mostly, how do journalists access PISA expert knowledge (what processes and which actors are involved in their behind-the-scenes work) and what are the rationales behind PISA’s retranslation to the public (see Lingard 2016).Footnote 5

  2. 2.

    Studies on the reception of PISA in Portugal conclude that PISA/OECD’s credibility has been used to certify interventions in policy processes, and also that the emergence of an improvement narrative, aligned with OECD visions, overrides the effects on the rationalization of decision-making in policy processes (Afonso and Costa 2009; Carvalho et al. 2017). These readings go along with other analyses that show knowledge-based learning is barely present when policy-makers turn to PISA data to argue about their systems (Pons 2012) or that the use of PISA involves more often externalization than learning (Lingard 2016). From our study, we also consider the existence of a gradation on how PISA has been actually used by policy-makers, between a source that is consulted and a source that is personally studied; but very little is known about how and what politicians actually learned from such use. Thus, the mechanism of “policy learning” (Freeman 2007) remains an open question. It is especially important to contrast (a) the mechanisms of knowledge declared by national policy actors and (b) the use of knowledge by politicians as imagined by the OECD, and the “systems of reason” (Popkewitz 2000b) in play in the fabrication of the policy-learner. Finally, it is important to focus on the use of PISA evaluative and statistical knowledge by the educational administration high-level officials and technicians, but also principals and teachers involved in the management, implementation, and administering of the tests in Portugal. This inquiry is needed in order to know how (and if) the appropriation of PISAs’ assessment knowledge happens within the educational system, specifically in the PISA National Project Management body and in the schools voluntarily involved in each PISA cycle; as well as how (and if) such appropriations relate to changes in their evaluative frameworks, scripts, and procedures.

In sum, all these lines of inquiry focus on the ideas, processes, and actors that link the expert knowledge disseminated by PISA to the contexts of policy-making, administration, school organization, and public debate on education. The accomplishment of these studies may contribute to a deeper understanding on how is PISA naturalized, and how it shapes and, simultaneously, is shaped through such multiple uses.