Abstract
Since the beginning of the current century, the acronym PISA (Programme for International Student Assessment) seems to pervade the multiple contexts where the education systems and their governing are subjected to debate. The presence of this Organization for Economic Cooperation and Development’s (OECD) international large-scale assessment in national spaces has diverse manifestations, such as supporting “analyses and rationales” for the discussion of specific issues, or being used as a “source” for secondary studies or as “learning opportunity” for the development of accountability policies (Lawn and Grek 2012). So far, PISA became a central element of a universe of knowledge, which, paraphrasing Lindblad and Popkewitz (2004: xx–xxi), ensures that expert-based education policies can lead each nation into the so-called knowledge society. The chapter addresses this umbilical relation between governing and expert knowledge by examining the meanings and processes that sustain PISA contemporary status of indispensable resource for the imagination and scrutiny of educational issues and policies.
Based on the revision of a previous article on the organizational and cognitive dimensions of PISA (Carvalho. Eur Educ Res J 11(2):172–188, 2012), and on the posterior works on the uses of PISA in national and supranational contexts (Carvalho and Costa. Discour Stud Cult Polit Educ 36(5):638–646, 2015; Carvalho and Costa. A European politics of education? Routledge, London, 53–72, 2016; Carvalho et al. Eur J Educ 52(2):154–166, 2017) and on the intensification and sophistication of PISA’s association with the policy processes (Carvalho. Organized—and organizing—mutual surveillance. Plenary presentation at the Seminar “Education governance and international assessments”, ESRC series on ‘The Potentials, Politics and Practices of International Assessments. University of Edinburgh, 2014; Carvalho. Educação & Sociedade 37(136): 669–683, 2016), the chapter focuses on the fabrications, that is the fictions and the making (Popkewitz. Educational knowledge. State University of New York Press, Albany, 3–27, 2000a, Popkewitz. Educational Researcher 29(1):17–29, 2000b) that support the projection of PISA as a central element for thinking-acting education policies.
You have full access to this open access chapter, Download chapter PDF
1 Introduction
Since the beginning of the current century, the acronym PISA (Programme for International Student Assessment) seems to pervade the multiple contexts where the education systems and their governing are subjected to debate. The presence of this Organization for Economic Cooperation and Development’s (OECD) international large-scale assessment in national spaces has diverse manifestations, such as supporting “analyses and rationales” for the discussion of specific issues, or being used as a “source” for secondary studies or as “learning opportunity” for the development of accountability policies (Lawn and Grek 2012). So far, PISA became a central element of a universe of knowledge, which, paraphrasing Lindblad and Popkewitz (2004: xx–xxi), ensures that expert-based education policies can lead each nation into the so-called knowledge society. The chapter addresses this umbilical relation between governing and expert knowledge by examining the meanings and processes that sustain PISA contemporary status of indispensable resource for the imagination and scrutiny of educational issues and policies.
Based on the revision of a previous article on the organizational and cognitive dimensions of PISA (Carvalho 2012), and on the posterior works on the uses of PISA in national and supranational contexts (Carvalho and Costa 2015, 2016; Carvalho et al. 2017) and on the intensification and sophistication of PISA’s association with the policy processes (Carvalho 2014, 2016), the chapter focuses on the fabrications, that is the fictions and the making (Popkewitz 2000a, b) that support the projection of PISA as a central element for thinking-acting education policies.
In fact, the chapter retrieves two pillars of the approach to PISA followed in my previous texts: (1) the representation of PISA as a knowledge-policy tool; (2) the understanding of the status achieved by PISA as the result of the making of a proper ecology. Thus, the first section of the chapter clarifies the notion of PISA as a tool that combines assessment techniques with a set of representations about education and a philosophy for the governance of education, and discusses its power effects. The second section focuses on the making of a PISA’s ecology as a process that entwines cognitive and social practices related with the construction and rise of the subject of inquiry (the competences of literacy) and, simultaneously, building public confidence and dependence on PISA. The chapter concludes with a proposal of new challenges for researching the trajectories of PISA in national contexts.
2 PISA as a Knowledge-Policy Tool
Along the last 15 years, OECD has reiterated the idea of PISA as a response based on specialized knowledge to meet the need expressed by national governments for useful and credible data on their performances (OECD 2001, 2007, 2014). This overt policy-oriented nature was recently retaken in Beyond PISA 2015: A longer-term strategy of PISA, a document that equates the future of the program:
It focuses on providing data and analysis that can help guide decisions on education policy. By linking data on students’ learning outcomes with data on key factors that shape learning in and out of school, PISA highlights differences in performance patterns and identifies features common to high-performing students, schools and education systems. (OECD n.d.: 1)
The OECD’s self-portrayed quality monitoring tool is observed differently along this chapter: PISA is rather observed as a device that embraces and conveys different ways of imagining (and doing) education, schooling, and social research, and—simultaneously—plays a part in the coordination of education policies and public action.
2.1 Fabricating Education Systems and their Steering
Drawing on the concept of “public policy instrument” as put forward by Lascoumes and Le Galès (2007: 4–6), PISA is approached as a tool that organizes social relations between administrative and administered subjects according to specific interpretations of the social world it addresses and based on a specific concept about the ways it should be oriented, coordinated, and controlled. From their point of view, each instrument is a combination of technical components and social components, that is, values, interpretations, and concepts about the social realities it describes. Regarding PISA, one might say that its sophisticated techniques are driven by a specific problematization of the role of education in contemporary times, displaying a particular way of challenging the national policies. Concomitantly, PISA bears principles and practices for policy processes.
On the one hand, PISA operates over several core categories of schooling (to a certain extent rewriting the educational model of contemporary societies), redefining students as lifelong learners, redefining teaching–learning relationships and settings, and redefining school knowledge.Footnote 1
To clarify this statement, one should look to the innovative focus that PISA claims to bring to assessments: “rather than examine mastery of specific school curricula, PISA looks at students’ ability to apply knowledge and skills in key subject areas and to analyze, reason and communicate effectively as they examine, interpret and solve problems” (PISA Website). This definition allows OECD to move away from the conventional self-reflection of national school systems based on their own categories and “outputs” (with assessments relying on tests and examinations based on national curriculum goals and content) and toward the territory of “outcomes,” thus directly connecting the contexts, practices, and results of teaching/learning with the so-called demands of the school system environment. Parallel with this differentiation comes the redefinition of the appropriate school knowledge for the so-called knowledge society: the notion of competence enacts a “utilitarian perspective” on knowledge as it takes practical usefulness in solving everyday problems as the main criterion for the assessment of school knowledge (Mangez 2008: 102–104). Therefore, it promotes a restructuring of curricular composition in trans/cross-disciplinary terms. These shifts go hand in hand with calls for change in teaching and learning structures (for example, from “hierarchical” to “organic” models or from the sequentially transmitted “bodies” of knowledge to the construction of learning by means of students’ cognitive connection with what they already know).
On the other hand, PISA also sustains particular cultural dicta about policy makers in contemporary times.
It redefines them as “problem-solvers” and “policy-learners,” which are to be decision-makers guided by searching for competitive advantages, measuring the outcomes of the school system, identifying weakness, and adopting solutions based on what works in other systems. That is to say, policy-makers that learn about competitors so as to progress more quickly. Moreover, PISA is nurtured by and nurtures several dicta on governing processes: the primacy of the rational and evidence-based model for the coordination and control of actions in the education sector, contrasting with ideological and/or opinion-based coordination; the free acquiescence of decision makers to be involved and to support, both materially (financial and other resources) and symbolically (with belief and praise) mutual surveillance as an expected and effective practice; and the systematic assessment of student literacy performances as a useful and trustworthy resource for the steering of educational systems.
2.2 Fabricating Transnational Governance
Returning to Lascoumes and Le Galès perspective (2007: 3), each policy instrument (a) “constitutes a condensed form of knowledge about social control and ways of exercising it,” and (b) each instrument produces its own effects “which structure public policy according to their own logic.” In order to discuss the presence of these features in PISA, we need to consider the OECDs’ history, resources, ideas, and choices.
First, PISA has a course that is inseparable from the OECD trajectory in the transnational governance of education. Generated in the context of the OECD’s project on the International Indicators of Educational Systems (see Morgan 2011), PISA implements an education agenda marked since the 1990s by the idea of monitoring quality, and involved in a continuous manufacturing of problems and solutions for the so-called knowledge economy (Rinne et al. 2004). It is also the most fruitful example of the OECD’s “comparative turn” (Martens 2007) and of its “infrastructural and epistemic” governance that, as Sellar and Lingard (2013: 13–14) pointed out, generates a “self-perpetuating dynamic” in which the OECD “both prescribes education policy approaches and assesses the performance of national education systems in these terms.” This is one of the particularities of the OECD intervention: a focus on “surveillance of performances” and “assessment of policies,” aiming to impact in national policies as a “creator, purveyor and legitimator of ideas” (Mahon and McBride 2008: 7–15).
Second, PISA operates through the power of guilt and hope. On the one hand, it operates through the culpability and the responsibility that it conveys to national spaces, because, together, school systems’ positioning (in a competitive space) and numbers (systems’ performances) bring “naming, blaming, and shaming” to the national policy spheres and actors. On the other hand, PISA operates also providing optimism for the possibility of reform based on evidences, and creates confidence in national policy actors as effective reformers. Moreover, PISA is an actant that brings the comfort of criticizing or legitimizing policy problems and solutions with the blessing of a putative universal, independent, expert knowledge.
2.3 Aggregation Effect
What happens when PISA frames, data, and analysis circulate (almost) worldwide? For the last 15 years, supplementary visions have been added to PISA along its travels. Thus, new knowledges, new policies, and politics have been gathered to it.
The credibility and sense of usefulness achieved by PISA is traceable in the variety of sociopolitical mobilizations of PISA already identified by a quite extensive literature on the reception of PISA in European contexts (see Carvalho and Costa 2016): there is an assignation of multiple purposes to PISA, namely of legitimation (i.e., legitimating reforms, specific policies, assessment instruments), information (as a complementary or as a compensating/substitutive source for the steering of education systems), and idealization (supporting the construction of diverse educational ideals, projections, or narratives, about education and educational reform).
Therefore, PISA objects/texts are ubiquitously present in national contexts, by the hands of different users (politicians, and other players involved in public educational debates, national experts, and researchers prolifically using it for secondary analysis). However, they are reinterpreted and made acceptable and efficient for each sociocognitive context. Thus, they are subject to diverse selections, either regarding the information displayed in the reports or the policy domains addressed by PISA recommendations. The summary of a recent review of the research on PISA effects stresses these same trends: on the one hand, “PISA has a strong influence on a variety of national reforms (…) however this influence strongly depends on domestic policy contexts” (Pons 2017: 131).
In sum, divergent uses and effects (regarding specific political choices or solutions, or to interpretations and uses of PISA products by other social actors) coexist with a convergence toward the tool. This phenomenon supports the constitution of PISA as a taken-for-granted source for public policy actors. As discussed before (Carvalho 2012), this is neither a paradox nor an inconsistency of the tool, but an effect of its power: the signal of the proficiency of a tool that keeps actors and agencies (that operate in different social worlds, and at regional, national, and supranational spaces) bound by/to PISA’s multiple activities and products. In other words, PISA is effective because of the—convergent and divergent—engagement and participation of multiple actors in its own production, dissemination, use, and consumption. Thus it performs an aggregation effect (Lascoumes & Simard, 2011): controversies, disputes regarding the analysis of data, and competing thoughts on solutions for “education problems”—the imagination and/or the scrutiny of educational systems, policies, and practices—are recurrently made by a dependence/commitment to PISA.
To understand how this effect is achieved is important to connect the dimension of the ideas, frames, expectations, and prescriptions generated by PISA to the dimension of the organizational processes that structures the relations of interdependence between the actors involved with PISA, and that concomitantly put into circulation and legitimizes specific ways of understanding what education “is” and how it should be governed. This implies taking into account the OCDE intervention, according to its institutionalized modus operandi: idealizing, aggregating actors, supervising interdependencies (Marcussen 2004) but also intense and varied diffusion of informational products generated to diverse audiences. Both issues are addressed in the following section.
2.4 PISA as the Making of an Ecology
While accepting that trust in the OECD as an “expert organization” (Noaksson and Jacobsson 2003) precedes and promotes PISA’s public judgment as a valid and useful tool, not forgetting that the success of PISA capitalizes from a cultural and political environment that disseminates the conceptions of global economic competitiveness and the knowledge economy (Broadfoot 2000), the success of this knowledge-policy tool lies also in the making of effective connections between heterogeneous actors: public and private research centers, individual experts and researchers, OECD professionals, policy-makers, high-level civil servants and technicians from multiple countries, media … in short, in making its own ecology.
2.4.1 Fabricating the Ecology
To grasp the success of such enterprise, it is enough to consider the growing number of countries involved and the extent of the geopolitical coverage achieved by PISA: in 2000, 43 countries took part in it (13 of which are not members of the OECD); in 2012, there were 65 participant countries (31 of which are not members of the OECD); in 2015, the date of the last assessment, 72 countries, from all continents and covering well-diverse cultures, economies, and political regimes. This broadening of participant countries/regions means also the enlargement of collective actors involved in PISA’s international and national/regional steering and management activities. The notion of the PISA “expansions” developed by Sellar and Lingard (2014) captures more comprehensively this success, including the following trends: “widening the scope of the assessment to measure a broader set of skills and competencies; increasing the scale of the assessment to cover more countries, systems and schools; and enhancing the explanatory power of the assessment for policy-makers and educators” (p. 924).
I relate these developments with the making of PISA’s ecology (Carvalho 2012). Adapting Everett Hughes’ notion of “ecology of institutions,” as quoted by Star and Griesemer (1989), this means the choices taken within the Program about its material, informational, and human sources, and also the actions carried out to establish continued and lasting exchanges with selected actors. In other words, the making of this ecology rests on a collection of practices that keeps PISA alive and expanding in a field populated by other agencies which also export educational monitoring devices and are involved in the making of usable knowledge-policy instruments for national or regional territories. In the following pages, I turn to two examples of the practices that support the effective connection to PISA by interested (individual and collective) actors: assembling and coordinating heterogeneous actors and knowledges; reaching and creating interest in heterogeneous actors not directly involved in PISA formal structures.
2.4.2 Assembling and Coordinating People and Knowledges
PISA is not merely a triennial survey and report. Together with the inquiry activities (design, trial, application, data-analysis) relevant face-to-face exchange activities take place (in meetings, workshops, seminars, etc.). Likewise, multiple publications—apart from the survey’s main reports—are generated and have a worldwide flow. Thus, is reasonable to observe PISA as a system of activities where communication and organization happen. All these activities involve a great variety of social worlds and multiple kinds of knowledge, interests, and perspectives. Thus, the accomplishment of PISA depends on bringing together—and ensuring the cooperation of—heterogeneous actors around a flow of activities, and on having them share the perception of PISA being a respectable provider of useful data/information/knowledge.
When asked about the role played in this process, the OECD Secretariat—the structure formally responsible for the management of the daily activities of PISA—presents itself as a catalyst for interaction between experts and politicians (see Carvalho and Costa 2009; Carvalho 2012). However, before becoming a “facilitator of relationships,” OECD began its own work as a provider of ideas, promoting the framework of literacy and reframing the old school system problem—the one of preparing the young generation for the future—in a much broader concern: the issue of the international competitiveness.
The OECD has generated its own initiative a specific framework—the competences of literacy—which became attributable to its own agency, as one OECD executive states:
We did a book—DeSeCo [Definition and Selection of Competencies]—it’s a definition and selection of competences … That’s the source of PISA. It is a project we had over 5 years where we had anthropologists, psychologists, labor economists and all these people telling us what is competence, what does it mean to do well in life, what is a successful life and they come up with different answers (…) it was a really great inspiration because if we had only used educators we’d have quite the common denominator of national curriculum. (Interview with an OECD executive, 2008, cit. in Carvalho and Costa 2009: 75)Footnote 2
Furthermore, the competences of literacy, generated within the OECD indicators project (Indicators of Education Systems—INES), build on the experiences (and reflections on the experiences) of many of the promoters of previous large-scale assessments: the OECD did not only reuse the methodological knowledge previously developed but also “recruited” actors previously linked to comparative international studies (Morgan 2011). Overall, according to Martens and Wolf (2009: 99), the conceptualization of PISA took 5 years and involved 300 “scientists” from all over the world.
The mobilization of diverse knowledges and experts has continued after PISA take-off. Along the last two decades, the development of PISA knowledge, disciplined by the literacy framework but also by assumptions, concepts, and methods from the psychometric world, entails contributions generated by experts from very different knowledge communities (Carvalho 2012): experts related to PISA core domains (mathematics, reading, and science) and other literacy domains invented along the PISA course (e.g., financial literacy) the “hard” knowledge of statistics, psychometrics, and compared assessment; knowledge of social psychology in relation to the study of attitudes; and streams of knowledge coming from policy evaluation and analysis, and from the school effectiveness tradition. Together OECD’s professionals and external experts collectively validate the data/information/knowledge they process. Finally, as the starting point of the studies carried out under PISA is partly defined by the representatives of the OECD member countries and the non-member countries associated with PISA, so the results of the technical work are open for their appreciation. In short, the knowledge produced and disseminated within PISA rests on consensus among OECD staff, research consortia, experts, and national representatives about what counts as usable and disclosable knowledge.
Along these processes, “the catalyst” has to fulfill positively the varied informational interests of those who gather and must, as well, keep them performing appropriately to what is expected from an organization that struggles for the status of a “truth teller” (Noaksson and Jacobsson 2003). Throughout the several sequences of tasks that make PISA (from building/reviewing each cycle framework to the delivery of publications), the “catalyst” also ensures that the interactions follow the common values and rules expected in a social space created by an expert organization, like consensus building. One good illustration is the management of PISA meetings.
From an examination of the narratives of national representatives interviewed in KNOWandPOL research about the dynamics of the PISA Governing Body meetings, a few themes come to the front (see Carvalho and Costa 2009): the OECD Secretariat has a leading role; national representatives perceive diverse types of participation, ranging from the convergent to the divergent type and from the active to the “reserved” type; the meetings are spaces where different—often conflictual—visions of education are shown, and where compromises and consensus are established, between participants with unequal resources. These unequal resources relate with the nearness–distance to the specific technical knowledge of PISA, the mastery of the English language, or the status of each country in the political-economic hierarchy of the OECD. What seems crucial though is the capacity to mobilize or to contest technical argument, because this one is perceived as having a central role in the achievement of political consensus. To many of our interviewees, the political building of consensus seems to be dependent on—and subordinate to—technical expertise on comparative assessments.
2.5 Multiplying the Interested Actors
PISA reaches well beyond the actors of its formal structures. It reaches and involves media agencies and national politicians, who selectively discuss the results, as well as national and/or regional governing and/or administrative structures that use PISA knowledge for creating their own assessment devices, and even researchers from various fields who use PISA data in order to build secondary analyses.
The connection that PISA has achieved with the media, and the importance given to it by the OECD, was already depict as a “media strategy,” involving the management of the media coverage of PISA’s triennial data release and the production of media-oriented county notes (Lingard 2016). But PISA establishes effective associations with other actors by diverse means, through a process that along the current decade has become more intense and sophisticated. One is the widening of the multiple public and private actors that the OECD puts in interaction, in order to construct meaning, articulate and diffuse new rules based on the use of PISA data—what has already been labeled as a “social matrix of interrelated governing actors” by Sotiria Grek (2010: 401) in the context of her analysis about the associations between the OECD and the European Commission. Another modality of creating these social matrixes are the “meditative” activities—borrowing this analytical category from Jacobsson (2006)—developed outside the PISA word, like the publication and exchange activities on teachers and teaching developed since 2011 in the context of the International Summits on the Teaching Profession (Robertson 2012), co-organized with Education International and national authorities. A different mean is the increase in surveys on literacy that has been enlarging the subjects and the objects of the survey: Programme for the International Assessment of Adult Competencies (PIACC) (see, e.g., Grek 2010, 2014); Assessment of Higher Education Learning Outcomes—AHELO (see, e.g., Shahjahan and Torres 2013; Shahjahan et al. 2015); PISA for Development (see, e.g., Addey 2016); PISA for schools (see, e.g., Rutkowski 2015, Lewis et al. 2016; Lewis 2017a, b).Footnote 3
Finally, a third mean is the widening of knowledge-related materials generated and disseminated to diverse audiences. In 2008, the diversity of the materials generated within PISA was already remarkable (Carvalho and Costa 2009): diverse sorts of reports (main, thematic, extensive, national, and technical); databases; documents with assessment basics, written for teachers, parents, and pupils. This variety of informational products were already explicitly generated to target populations, with diverse interests and skills. Moreover, opening up to a multiplicity of possible uses, whether in order to reproduce, to re-contextualize the data/information, or even in order to produce knowledge from it. Presently, this picture presents a few important changes: the objects become more sophisticatedly elaborated and their variety is amplified, thus fostering the intensification of possible uses. Three types of materials illustrate this move: materials exhorting policy emulation and policy learning; materials that provide their readers “short-cuts to knowledge”; and digital platforms supporting “do it your-self” (DIY) practices.
One of the new deliverables is the “in-depth” report suggestively entitled Strong Performers and Successful Reformers prepared by “task forces” of experts and members of national education bodies, involving meetings with national “stakeholders.” Reforms based on local adaptation of best practices are the expected outcome from these documents. A different type of deliverable for a quick access is developed since 2011, like the monthly briefs PISA in focus that display four to five pages of explicitly policy-oriented texts on a specific PISA theme, from student performances and attitudes toward school and learning to family background, from classroom environment to education policy. The DIY products, like the Interactive Data Selection and the Multi-dimensional data request, support different types of relationships between PISA and the users, by allowing these to select and compare data of school- and student-level variables. Connected to the Education GPS, these products allow the users to access data provided by PISA, TALIS, and Education at a Glance, as well as to prepare country reports, using texts and sophisticated charts provided by the tool, and to compare the countries’ performances. Set to activate a quasi-autonomous relation with PISA data, these technologies support the OECD (new) intervention as a “center of visualization” (Williamson 2016).Footnote 4
In sum, the making of PISA ecology rests on the condition of being consumed, shared, or learned by its audiences, as credible and manageable.
3 New Challenges for Researching the Trajectories of a Knowledge-Policy Instrument
Along the text, drawing on my previous works on PISA, I have discussed the twofold influence that PISA exerts as knowledge-policy tool, as well as the practices that sustain its regulatory action (in terms of ideas and within the realm of organized action). From what is written in the previous pages, it is possible to take PISA as an analyzer of the use of expert knowledge in regulatory processes. My present interest in PISA follows this direction, by focusing the dynamics of appropriation inherent to the trajectory of any policy instrument (Lascoumes & Simard, 2011). This opens the possibility of observing new fabrications of PISA and new fabrications made with—or based on—PISA. This means to observe and analyze what is occurring when its objects (texts, data, databases) circulate through national, regional, and local public action settings and are used by different social groups, according to their beliefs, interests, and knowledge repertoires, and under specific sociocultural circumstances.
In fact, previous studies on PISA reception and effects show that all the knowledge in circulation is transformable and transformed by the context and the actors who receive and eventually diffuse it to new audiences; but we cannot escape to question if this knowledge does not transform the very actors and contexts that transform it. It is, in fact, a relationship that needs to be further studied by the research that has been devoted to the PISA reception and effects in the governing of education. Two issues emerge as central: (1) the role of actors that, at a national scale, operate between PISA objects and policy actors; (2) the appropriation of PISA-labeled objects at diverse educational contexts—policy, state bureaucracy, schools.
-
1.
One of the most open issues in the literature on PISA effects concerns the understanding of the varied uses of PISA by its multiple audiences. Several factors have been put forward to that purpose: structural, socioeconomic, or cultural aspects; national policy dynamics; interventions of the actors that transform the results into pressures on the educational agenda (see Carvalho and Costa 2016). In this last factor stands out the agency from those who intermediate between PISA knowledge and policy as “brokers” or as “entrepreneurs” (Van Zanten 2009), and the resonance of the media interpretations. In the Portuguese context, two aspects deserve special attention (Carvalho et al. 2017; Viseu and Carvalho 2018): the recent emergence of “intermediary actors” (Nay and Smith 2002) who intend to transform the PISA results into knowledge for national policy, after a long period of non-existence of this intention (and of their respective performers); the continued increase in coverage given to PISA by the national media, which is even becoming more specialized in a few newspapers. The first aspect concerns the transformation of PISA’s results into knowledge for national policies by national collective actors that activate diverse mechanisms of “translation” (Callon 1986) and generate compositions of technical and political arguments—argumentaires (Pons 2012)—to explain PISA results to the politicians and largely to “the public.” The second aspect concerns the identification of the ways in which the press mobilizes PISA and portrays the Portuguese performance and, mostly, how do journalists access PISA expert knowledge (what processes and which actors are involved in their behind-the-scenes work) and what are the rationales behind PISA’s retranslation to the public (see Lingard 2016).Footnote 5
-
2.
Studies on the reception of PISA in Portugal conclude that PISA/OECD’s credibility has been used to certify interventions in policy processes, and also that the emergence of an improvement narrative, aligned with OECD visions, overrides the effects on the rationalization of decision-making in policy processes (Afonso and Costa 2009; Carvalho et al. 2017). These readings go along with other analyses that show knowledge-based learning is barely present when policy-makers turn to PISA data to argue about their systems (Pons 2012) or that the use of PISA involves more often externalization than learning (Lingard 2016). From our study, we also consider the existence of a gradation on how PISA has been actually used by policy-makers, between a source that is consulted and a source that is personally studied; but very little is known about how and what politicians actually learned from such use. Thus, the mechanism of “policy learning” (Freeman 2007) remains an open question. It is especially important to contrast (a) the mechanisms of knowledge declared by national policy actors and (b) the use of knowledge by politicians as imagined by the OECD, and the “systems of reason” (Popkewitz 2000b) in play in the fabrication of the policy-learner. Finally, it is important to focus on the use of PISA evaluative and statistical knowledge by the educational administration high-level officials and technicians, but also principals and teachers involved in the management, implementation, and administering of the tests in Portugal. This inquiry is needed in order to know how (and if) the appropriation of PISAs’ assessment knowledge happens within the educational system, specifically in the PISA National Project Management body and in the schools voluntarily involved in each PISA cycle; as well as how (and if) such appropriations relate to changes in their evaluative frameworks, scripts, and procedures.
In sum, all these lines of inquiry focus on the ideas, processes, and actors that link the expert knowledge disseminated by PISA to the contexts of policy-making, administration, school organization, and public debate on education. The accomplishment of these studies may contribute to a deeper understanding on how is PISA naturalized, and how it shapes and, simultaneously, is shaped through such multiple uses.
Notes
- 1.
These visions echoed in diverse international organizations and have been developed in many other spaces, from policy analysis texts (see, for example, Weeres and Kerchner 1996) to reports produced by specialists nominated in the mid-1990s by the (then) Commissioner for Education, as quoted and analyzed by Lawn (2003: 331): “The future of Europe has to be constructed by several shifts: from objective to constructed knowledge; from an industrial to a learning society; from instruction to personal learning; from formal educational institutions towards new organizational structures for learning (yet to be determined)” (European Commission 1997: 7).
- 2.
For a detailed analysis on the role of DeSeco in the development of PISA and other surveys, as well as on the role of DeSeco for the establishing of networks of researchers involved in the making of PISA and studies on ‘competences, see Grek (2013).
- 3.
For a recent review on the trajectory of OECD’s surveys, see Morgan and Volante (2016).
- 4.
The newest PISA product is the “pisa4u”—The Online Programme for School Improvement—oriented toward teachers, administrators, policy-makers, and parents. It intends to provide their users with “an environment for ideation and co-creation; this programme fills the need of schools and educators to connect and collaborate across silos and regions.” http://www.prnewswire.com/news-releases/pisa4u%2D%2D-the-online-programme-for-school-improvement-launched-by-oecd-and-candena-604974616.html.
- 5.
On this second aspect, I follow a proposal from my colleague Benedita Melo (IE-ULisboa), made in the course of the collective preparation of a new research project on ‘PISA and knowledge mobilization’ in public policies in Portugal.
References
Addey, C. (2016). O PISA para o Desenvolvimento e o sacrifício de dados com relevância política. Educação & Sociedade, 37(136), 685–706.
Afonso, N., & Costa, E. (2009). A influência do PISA na decisão política em Portugal. Sísifo–Revista de Ciências da Educação, 15, 53–64.
Broadfoot, P. (2000). Comparative education for the 21st century: Retrospect and prospect. Comparative Education, 36(3), 357–371.
Callon, M. (1986). Some elements of a sociology of translation. In J. Law (Ed.), Power, action, and belief (pp. 196–233). London: Routledge & Kegan Paul.
Carvalho, L. M. (2012). The fabrications and travels of a knowledge-policy instrument. European Educational Research Journal, 11(2), 172–188.
Carvalho, L. M. (2014). Organized—and organizing—mutual surveillance. Plenary presentation at the Seminar “Education governance and international assessments”, ESRC series on ‘The Potentials, Politics and Practices of International Assessments (University of Edinburgh, December). Unpublished paper.
Carvalho, L. M. (2016). Intensificação e sofisticação dos processos da regulação transnacional em educação: o caso do Programa Internacional de Avaliação de Estudantes. Educação & Sociedade, 37(136), 669–683.
Carvalho, L. M., & Costa, E. (2009). Production of OECD’s ‘Programme for International Student Assessment’: final report. Project KNOWandPOL, WP 11, March, http://www.knowandpol.eu/IMG/pdf/o31.pisa.fabrication.pdf.
Carvalho, L. M., & Costa, E. (2015). Seeing education with one’s own’ eyes and through PISA lenses. Discourse: Studies in the Cultural Politics of Education, 36(5), 638–646.
Carvalho, L. M., & Costa, E. (2016). The praise of mutual-surveillance in Europe. In R. Normand & J.-L. Derouet (Eds.), A European politics of education? (pp. 53–72). London: Routledge.
Carvalho, L. M., Costa, E., & Gonçalves, C. (2017). Fifteen years looking at the mirror: On the presence of PISA in the education policy processes (Portugal, 2000-2016). European Journal of Education, 52(2), 154–166.
Freeman, R. (2007). Epistemological bricolage: How practitioners make sense of learning. Administration and Society, 39(4), 476–496.
Grek, S. (2010). International organisations and the shared construction of policy ‘problems’. European Educational Research Journal, 9(3), 396–406.
Grek, S. (2013). Expert moves: International comparative testing and the rise of expertocracy. Journal of Education Policy, 28(5), 695–709.
Grek, S. (2014). OECD as a site of co-production: European education governance and the new politics of ‘policy mobilization’. Critical Policy Studies, 8(3), 266–281.
Jacobsson, B. (2006). Regulated regulators. In M.-L. Djelic & K. Sahlin-Anderson (Eds.), Transnational governance (pp. 205–224). Cambridge: Cambridge University Press.
Lascoumes, P., & Le Galès, P. (2007). Introduction: Understanding public policy through its instruments—From the nature of instruments to the sociology of public policy instrumentation. Governance, 20(1), 1–21.
Lawn, M. (2003). The ‘usefulness’ of learning: The struggle over governance, meaning and the education space. Discourse: Studies in the Cultural Politics of Education, 24(3), 325–336.
Lawn, M., & Grek, S. (2012). Europeanizing education: Governing a new policy space. Oxford: Symposium Books.
Lewis, S. (2017a). Governing schooling through ‘what works’: The OECD’s PISA for schools. Journal of Education Policy, 32(3), 281–302.
Lewis, S. (2017b). Policy, philanthropy and profit: The OECD’s PISA for schools and new modes of heterarchical educational governance. Comparative Education, 53, 518. https://doi.org/10.1080/03050068.2017.1327246.
Lewis, S., Sellar, S., & Lingard, B. (2016). PISA for schools: Topological rationality and new spaces of the OECD’s global educational governance. Comparative Education Review, 60(1), 27–57.
Lindblad, S., & Popkewitz, T. S. (2004). Educational restructuring: (re)thinking the problematic of reform. In S. Lindblad & T. S. Popkewitz (Eds.), Educational restructuring: Perspectives on traveling policies. Greenwich, CT: Information Age.
Lingard, B. (2016). Rationales for and reception of the OECD’s PISA. Educação & Sociedade, 37(136), 609–627.
Mahon, R., & McBride, S. (2008). Introduction. In R. Mahon & S. McBride (Eds.), The OECD and transnational governance (pp. 3–22). Vancouver: University of British Columbia Press.
Mangez, E. (2008). Knowledge economy, knowledge policy and knowledge regimes. In B. Delvaux & E. Mangez, Towards a Sociology of the Knowledge–Policy Relation (pp. 98–118). Project KNOWandPOL, Literature Review Integrative Report. Retrieved from http://www.knowandpol.eu/IMG/pdf/literature_sythesis_final_version_english.pdf.
Marcussen, M. (2004). The Organization for Economic Cooperation and Development as ideational artist and arbitrator. In B. Reinalda & B. Verbeek (Eds.), Decision making within international Organisations (pp. 90–105). London: Routledge.
Martens, K. (2007). How to become an influential actor: The “comparative turn” in OECD education policy. In K. Martens, A. Rusconi, & K. Lutz (Eds.), Transformations of the state and global governance (pp. 40–56). London: Routledge.
Morgan, C. (2011). Constructing the OECD programme for international student assessment. In M. Pereya, H.-G. Kotthoff, & R. Cowen (Eds.), PISA under examination (pp. 47–60). Rotterdam: Sense Publishers.
Morgan, C., & Volante, L. (2016). Policy futures in education a review of the OECD’s international education surveys: Governance, human capital discourses, and policy debates. Policy Futures in Education, 14(6), 775–792.
Nay, O., & Smith, A. (2002). Les intermédiaries en politique: mediation et jeux d’instituitions. In O. Nay & A. Smith (Eds.), Le Gouvernement du Compromise: courtiers et generalistes dans l’action politique (pp. 1–21). Paris: Economica.
Noaksson, N., & Jacobsson, K. (2003). The production of ideas and expert knowledge in OECD. Stockholm: Stockholm Centre for Organizational Research.
OECD. (2001). Knowledge and skills for life. Paris: OECD Publishing.
OECD. (2007). PISA 2006: Sciences competencies for tomorrows world. Paris: OECD Publishing.
OECD. (2014). PISA 2012 results: What students know and can do. Volume I, revised edition. Paris: OECD Publishing.
OECD. (n.d.). Beyond PISA 2015: A longer-term strategy of PISA. Retrieved from http://www.oecd.org/pisa/pisaproducts/Longer-term-strategy-of-PISA.pdf.
Pons, X. (2012). Going beyond the ‘PISA shock’ discourse: An analysis of the cognitive reception of PISA in six European countries, 2001–2008. European Educational Research Journal, 11(2), 206–226.
Pons, X. (2017). Fifteen years of research on PISA effects: A critical review. European Journal of Education, 52(2), 131–144.
Popkewitz, T. S. (2000a). Globalization/regionalization, knowledge, and the educational practices. In T. S. Popkewitz (Ed.), Educational knowledge (pp. 3–27). Albany: State University of New York Press.
Popkewitz, T. S. (2000b). The denial of change in educational change: Systems of ideas in the construction of national policy and evaluation. Educational Researcher, 29(1), 17–29.
Rinne, R., Kallo, J., & Hokka, S. (2004). Too eager to comply? OECD education policies and the Finnish response. European Educational Research Journal, 3(2), 454–486.
Robertson, S. (2012). Placing’ teachers in global governance agendas. Comparative Education Review, 56(3), 584–607.
Rutkowski, D. (2015). The OECD and the local: PISA-based test for schools in the USA. Discourse: Studies in the Cultural Politics of Education, 36(5), 683–699.
Sellar, S., & Lingard, B. (2013). The OECD and global governance in education. Journal of Education Policy, 28(5), 710–725.
Sellar, S., & Lingard, B. (2014). The OECD and the expansion of PISA: New global modes of governance in education. British Educational Research Journal, 40(6), 917–936.
Shahjahan, R., & Torres, L. (2013). A ‘global eye’ for teaching and learning in higher education: A critical policy analysis of the OECD’s AHELO study. Policy Futures in Education, 11(5), 606–620.
Shahjahan, R., Morgan, C., & Nguyen, D. (2015). ‘Will I learn what I want to learn?’ Usable representations, ‘students’ and OECD assessment production. Discourse: Studies in the Cultural Politics of Education, 36(5), 700–711.
Star, S. L., & Griesemer, J. R. (1989). Institutional ecology, ‘translations’ and boundary objects. Social Studies of Science, 19(3), 387–420.
Van Zanten, A. (2009). From critical intellectuals to “idea brokers?” Traditions and new development of links between research and policy-making in education in France. Nordisk Pedagogik, 29(1), 53–60.
Viseu, S., & Carvalho, L. M. (2018). Think tanks, policy networks and education governance: The emergence of new intra-national spaces of policy in Portugal. Educ. Policy Analysis Archives, 26, 108. https://doi.org/10.14507/epaa.26.3664.
Weeres, J., & Kerchner, C. (1996). This time It’s serious: Post-industrialism and the coming change in education. In R. L. Crowson, W. L. Boyd, & H. B. Mawhinney (Eds.), The politics of education and the new institutionalism (pp. 135–151). London: Falmer Press.
Williamson, B. (2016). Digital education governance: Data visualization, predictive analytics, and ‘real-time’ policy instruments. Journal of Education Policy, 31(2), 123–141.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
Copyright information
© 2020 The Author(s)
About this chapter
Cite this chapter
Carvalho, L.M. (2020). Revisiting the Fabrications of PISA. In: Fan, G., Popkewitz, T.S. (eds) Handbook of Education Policy Studies. Springer, Singapore. https://doi.org/10.1007/978-981-13-8343-4_14
Download citation
DOI: https://doi.org/10.1007/978-981-13-8343-4_14
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-13-8342-7
Online ISBN: 978-981-13-8343-4
eBook Packages: EducationEducation (R0)