Introduction

Evaluation and valuation of symbolic goods and practices have been increasingly studied by cultural sociologists (Beljean et al. 2015; Lamont 2012). While sociological research in this area has been largely focused on field analysis—e.g., in film (e.g. Cattani et al. 2014; Kersten and Verboord 2014), literature (e.g. De Nooy 1999; Janssen 1997), or music (Dowd et al. 2002)—the most recent studies have been engaged in exploring standards and practices of evaluation using an inductive approach. This new work in cultural sociology includes, for example, ethnographic research that concerns how artistic tastes are formed and performed in social interactions (Benzecry 2011; Hennion 2001) or how shared aesthetic judgments tie people together (Wohl 2015), and interview-based analysis of critics’ legitimization strategies (Chong 2013). These studies meaningfully contribute to our understanding of how experts and amateurs interact with works of art and how these interactions help them place themselves within larger social groups. In addition, researchers studying book clubs have shown that groups develop their own local evaluation standards to discuss works of art (Long 2003) and that interpersonal influence modifies evaluative orientations of group members (Childress and Friedkin 2012). Nevertheless, we still know considerably little about what evaluative criteria and concepts of artistic value are operative in group evaluations, whether members support different evaluative standards and, if so, how they define those differences. Nylander (2014) and Loots (2019) offer useful analyses of evaluation criteria used by expert panels—in music and theater, respectively—but they do not study differences between panelists, or how those differences provide opportunities for group identification.

Furthermore, researchers have studied how artistic judgments are made in group interactions (Childress and Friedkin 2012; Long 2003; Wohl 2015), but their work focuses on voluntary community settings (book clubs, etc.) where participants are not necessarily oriented toward reaching an agreement. At the same time, much remains to be known about artistic panel deliberations, where collective judgments are shaped by the constraints of the agreement-making process. Given that panel reviewing plays an increasing role in subsidizing art (Loots 2019), it is important to understand how evaluative panels work and how experts arrive at collective decisions.

We intend to fill these gaps by studying panel evaluations of theater performances. Our paper draws on interviews conducted with panelists that collectively decide upon the allocation of public funds for theater productions in Poland. We focus on two aspects of panel decision-making that play a crucial role in how evaluations are made: (1) concepts of artistic value that members of expert panels apply when evaluating art productions—namely, we investigate what properties of theater performances indicate artistic worth to theater experts; (2) interpersonal rules of deliberation, that is, norms and patterns of behavior that are seen as good or appropriate and allow panelists to attach legitimacy to the decision-making process and its outcomes.

Different Concepts of Artistic Value

One of the central concerns of researchers studying evaluations is how experts attach value to evaluated objects. Substantial effort has been made to define what criteria are used in assessment-making (Hug et al. 2013; Ochsner et al. 2013), or how particular aspects of quality (e.g., originality) are understood by evaluators (Dirk 1999; Guetzkow et al. 2004). When evaluations are made in group settings, researchers explore shared understandings of quality and worth between panel members (Lamont 2009; Huutoniemi 2012). In panel evaluations, criteria of judgment are rarely explicitly discussed or described in official guidelines but members develop tacit knowledge about the group’s assessment standards as deliberations proceed (Huutoniemi 2012; Lamont 2009; Langfeldt 2004).

Researchers have repeatedly shown that people sitting on the same panel may have different ideas of what constitutes an excellent achievement. For example, in scientific panels, reviewing members bring to the table their preferences and beliefs rooted in different epistemic cultures (Knorr-Cetina 1999). Differences in epistemological and evaluative standards pose substantial challenges to the reviewing process. The challenges in question are most evident in interdisciplinary settings (Huutoniemi 2012; Laudel 2006), but distinctions occur not only between but also within and across disciplinary fields (Abbott 2001; Huutoniemi 2012).

In the field of art, the criteria of quality judgment have been studied most extensively by philosophers. Becker (1982) believes that aestheticians (or philosophers of art) provide theories and classification systems (criteria for judgment) that critics apply when making and justifying their evaluations. At the same time, as members of an art world—a network of people who cooperatively participate in art-making—aestheticians and critics develop value systems, which are permanently revised with regard to what artists actually produce. The concepts of artistic value, that is, different ideas concerning what is a more or less worthy artistic achievement, are part of interdependent practices specific to an art world and influenced by different factors such as the type of training received by value-creators, the allocation of resources, the modes of art distribution, etc. Depending on the organization of an art world, people involved in different artistic networks or cohesive subgroups within the same network (Wasserman and Faust 1994) may consider artistic worth in different ways. The mutuality of ties is reflected in shared artistic judgments (De Nooy 1999), but at the same time, shared judgments enhance group cohesion by confirming or denying feelings of group belonging (Wohl 2015).

In our study, we investigate different concepts of artistic value that members of expert panels apply when evaluating theater productions. We are interested in what experts appreciate and care about when they critically assess works of art. Since evaluations are produced in interactions, we focus in particular on the divergences between different members’ concepts of value by analyzing the ways in which panelists draw boundaries between different “factions” within the group. Although we believe that different value concepts reflect members’ belonging to different subgroups within an art world, we do not study ties and networks beyond the panel meeting but center on how members perceive group boundaries in situ. Moreover, we do not study all concepts of value that members hold and communicate but only the most dominant concepts that determined how members perceived their in-group differences and what made them feel different from some people and similar to others.

Interpersonal Rules of Deliberation

Another critical issue in group evaluations is the process of consensus formation. A large number of studies in social psychology have carefully examined decision-making in face-to-face interactions, focusing on different forms of group influence on individual judgment (e.g., Asch 1956; Cialdini and Goldstein 2004; Deutsch and Gerard 1955; Wood 2000). Studies in this field have shown that decision-making is perceived by individual members as fair when groups are characterized by a collaborative orientation and high quality of interpersonal treatment (Schei and Rognes 2005; Tyler and Blader 2003). At the same time, perceived fairness of deliberations enhances members’ commitment to the common goal and turns their orientation toward producing agreement. It also leads panelists to believe in the legitimacy of the process and, in consequence, attach legitimacy to the outcome of their debates (Lamont 2009). Individuals presume that certain norms and procedures are legitimate when social actors in their surroundings endorse behaviors that follow these norms (Johnson et al. 2006). Legitimacy exists when patterns of action are collectively seen as desirable and appropriate, even though individual members may not always personally agree with them (Zelditch 2001).

Evaluation panels develop shared rules of behavior in the deliberation process (Lamont 2009). Members do not create them from scratch; rather, these rules stem from shared assumptions that are elements of a broader culture. At the same time, the patterns of interaction are not fully formed before the group meeting but arise when groups coordinate themselves (Eliasoph and Lichterman 2003). The existence and prevalence of shared rules of deliberation have a tremendous effect on consensus formation. Members’ faith in the legitimacy of the decision-making affects their individual behaviors and facilitates agreement (Lamont 2009). It may, therefore, play a non-trivial role in artistic panels, where consensus is hard to reach given the ephemeral nature of artistic quality and divergent ideas of what constitutes excellence and worth. In this study, we examine shared norms, beliefs, and patterns of behavior that lead panelists to believe that the decision-making process, and its outcome, are legitimate. This is not to say that we focus on how panelists deceive themselves or invent false pretexts to justify their actions after the fact; we are interested in what norms of behavior are genuinely followed to protect the legitimacy of the process. We do not adopt any abstract measures to evaluate whether the process was “truly” legitimate but instead focus on panelists’ own assessments of the decision-making. Our approach is based on the assumption that legitimacy is an intersubjective phenomenon and that people draw their own perception of rules and norms as valid from the fact that they observably govern the behavior of actors in their social reality (Johnson et al. 2006).

In the remainder of the paper, we present and discuss the results of our study. We start with a description of the Polish theater scene to explain the context of our research, which is followed by the presentation of data and methods. Next, we focus on the empirical data: Firstly, we analyze the concepts of value that panelists apply in their evaluations. To provide a comprehensive summary of our findings, we develop a framework (matrix) of values that guide the process of artistic selections. Secondly, we examine three types of interpersonal rules of decision-making: substantial, procedural, and contextual. At the end, we discuss implications of our research.

The Context of the Study: Polish Theater Scene

Our study focuses on the Polish theater scene, which is predominantly based on publicly funded performing arts institutions. The transition from a centrally planned economy into a market economy in 1989 has changed the social and political context for Polish art institutions; for example, political interference in the artistic content has been considered unacceptable. However, given the lack of coherent reforms at the time (Lewandowska 2018a), today’s Polish theater system is still based on organizational structures established in the Communist era (Płoski 2018). The system in question includes 121 institutions owned by the state (Instytut Teatralny 2018), which are supported mainly through public sources in the form of fixed annual lump sum payments that constitute, on average, 78% of their budgets (Trzeciak 2011). The most common organizational model, a “repertory theater,” is as a company…

(…) with a strong artistic leader shaping a recognizable aesthetic profile, working with a fixed ensemble of actors, as well as administrative and technical staff, in their own building with one or more stages, with their own workshops and production facilities, and counting on a regular, diverse audience—the idea [that] dominates the theater landscape of German-speaking countries, and much of Central, Eastern and Northern Europe. (Klaić 2012, 37)

The administrative power over artistic institutions is redistributed between independent local authorities. Since the decentralization process that took place in the 1990s and led to the almost complete withdrawal of theaters from the centralized control of the state (Murzyn-Kupisz 2010), the vast majority of theaters have been managed by local self-governments at the level of municipalities and provinces. The decentralization of power has direct implications for the mission of public theaters, which are expected to focus on their local environments and address the needs of diverse local audiences. Especially companies located in small towns and isolated rural areas with limited cultural offers are deemed to fulfill both artistic and welfare goals—as opposed to similar venues in metropolises like Warsaw or Krakow, which compete with many other art producers and therefore can afford more specialized and sophisticated artistic programs.

Having said that, the central political agency—the Ministry of Culture and National Heritage—is a powerful player in the system of Polish theater funding. The subsidies from local administration are generally considered insufficient and theaters compete for the Ministry’s resources through different funding programs. While the supervision of local authorities is generally confined to routine administrative tasks and aesthetic beliefs of administrative leaders have little impact on the allocation of public subsidies (Lewandowska 2018b), the Ministry exercises a more “proactive” cultural policy, especially in recent years, marked by the dominance of the right-wing parties (for example, the Ministry has withheld support from several art initiatives that were offensive to the Ministry’s constituencies, e.g., Catholic associations). The Ministry’s interferences have provoked numerous public debates and vigorous responses from many Polish theater artists whose work has become more involved in social and political issues.

At the same time, the risk of the Ministry’s political interference is mitigated by the so-called, arm’s length principle (Madden 2009; Mangset 2009), that is to say, the delegation of decision-making regarding the allocation of funds to panels of independent experts specialized in a given field. Panel experts are recruited from considerably tight professional networks; members of those networks usually know one another in person, or by reputation, and the subgroupsFootnote 1 that are formed within such networks are often connected to each other.

In this paper, we study panel settings established to allocate public funding to theater productions. This context is crucial for evaluations, as it contributes to the panelists’ sense of responsibility, their efforts to achieve consensus, and the urge to protect the legitimacy of their judgments.

Data and Methods

The study is based on in-depth interviews with panelists serving repeatedly on funding panels at the National Contemporary Polish Drama Staging Competition (NCPDSC). Organized since 1994, NCPDSC is a program funded by the Ministry of Culture and National Heritage (MCNH) with the aim to promote theater productions based on contemporary Polish playwriting. Performances that enter the competition are typically produced by publicly-funded repertory theaters and form part of their repertoires.

During a six-month period, panel members travel to theater venues and watch performances qualified for the competition (approximately 100 a year). Each production must be seen by at least four panelists, out of seven panel members. Members are asked to permanently assign preliminary scores (0, 20, 30, 40, or 50) in an online sheet. The scores indicate the proportion of refunds on the total cost of the production; for example, if a production receives 50 points, the producing theater receives a grant that amounts to 50% of the total cost of the winning production. After the six-month traveling period, panelists meet for a two- or three-day session to discuss performances and make decisions in terms of grant allocation.

Among seven panel members, two are recommended by the editors of two major Polish theater magazines, three are selected by the national organization managing the competition, and the remaining two are permanent members, including the chair. The role of the chair is to explain to the panelists the rules of the competition before they start watching performances, monitor if all the productions are covered by four members, and finally, to orchestrate panel deliberations. The competition guidelines are very general and panelists are not instructed in terms of specific evaluation criteria.

Interviews were conducted with 30 panelistsFootnote 2 (designated in the paper as P1–P30). There were 17 women and 13 men in the sample. The majority were active theater critics who make a living through different theater-related activities: teaching in academia, working in public theaters as administrative or artistic staff, and organizing theater events. Moreover, seven of our respondents did not self-identify as critics but preferred to describe themselves as “practitioners” (professional actors, curators, or playwrights). Therefore, although our sample was dominated by professional reviewers, the bulk of our respondents combine writing reviews with practice-based work in the theater sector, such as curating performing arts projects, which requires both an ability to create artistic meaning and expertise to select quality artwork (Krzys Acord 2010).

Interviews were conducted face-to-face, as we believed that in-person contact would help us build trust and our respondents would be less likely to censor themselves when talking about issues such as tensions or power imbalances between panel members. All respondents were guaranteed anonymity. Interviews were semi-structured and typically lasted 55–90 minutes, during which time we asked panelists to describe what they liked about their favorite performances, what they generally appreciate when evaluating a theatrical piece, and what kinds of group interactions happened during panel deliberations. Moreover, the reviewers were asked to describe in what way they perceived themselves as different from some panel members and similar to others, in terms of evaluative standards. This interviewing technique, known as “boundary work,” was found effective by other scholars (e.g. Guetzkow et al. 2004; Lamont and Swidler 2014) in gathering information that is hard to obtain through direct interrogation.

All interviews were tape-recorded and transcribed. Interviews were carried out in Polish and transcribed as such before being translated into English. Transcripts were first analyzed one-by-one, using an inductive approach and focusing on detecting and coding recurring themes. Secondly, we systematized our data by entering short extracts from transcripts into analytic matrices (Miles et al. 2014). A conceptually clustered matrix (Miles et al. 2014, 173–178) was used to bring together and compare different variables, concepts, and themes, such as criteria that people with different expertise (e.g., actors, critics, curators) used to assess performances and customary rules of group reviewing, such as influence and power dynamics, or acceptable and impermissible strategies of persuasion.

Concepts of Artistic Value

In this section, we focus on the concepts of value that panelists applied in their evaluations. We deal with what our respondents told us when asked about their own evaluation criteria, as well as about their sense of how they stood with respect to other panelists, in terms of what they generally valued. We found Gielen’s (2005) distinction between content and context logic very useful for structuring our analysis. Gielen asserts that the process of artistic selections can be based on a content or context logic, the former being focused on the work of art itself (its structure, meaning, expressive qualities, etc.), and the latter referring to the artwork’s creator and the institutional context. Below, we present our data according to this distinction and we add a third element –a matrix that combines both logics and illustrates different concepts of value that prevail in the context of artistic selections.

Content Logic of Art Evaluations: Aesthetics and Meaning

Reviewers asked to describe what that they value in performances, namely, what makes them rank a performance highly, offered responses such as: “It’s mostly about how it is done, how it is performed, how it is directed, how it is produced and put together” (P8), “[H]ow was the story developed? How was it written?” (P14), and “The set design is important, it can enhance or destroy the performance. Pace and timing are important, too” (P25). When asked to describe what she appreciated in her favorite performances, a critic said that “it was a beautiful performance. The set design, the acting… it all worked together so well, I was completely immersed in the atmosphere” (P29). Reviewers enjoy performances that are well done in a formal sense, which means that they focus on the aesthetic qualities of work. Some interviewees linked aesthetic perfection with technical fluency of artists. The latter was especially important for actors sitting on the panel; we were told that actors were particularly concerned with technical details and that they favored “professionalism,” that is to say, when acting, directing, or scenography was done according to academic standards.

At the same time, reviewers champion performances they describe as “wise,” which means that they perceive them as conveying non-trivial meanings and manifesting significant human concerns. One of the reviewers believed that good performances are thought-provoking and “say something important about the world in which we live” (P4). Another one, asked about her evaluative criteria, said she cares about “the content, above all else. What a performance is about, how it presents the world… Is it a mindful interpretation of reality? Does it say anything new about the world—something that people normally don’t think or talk about?”(P10).

During our interviews reviewers frequently shifted between criteria that focused on the aesthetic qualities and the meaning of the performance, showing that these two properties of an artwork always work in conjunction. However, when asked in what way their personal criteria were different from those of other panel members, interviewees pointed out that their group consisted of reviewers who were more oriented toward the work’s aesthetic quality and perfection, and those seeking primarily the meaning and sense. A panelist recalled that while she was particularly concerned about why the director had moved the action of the play to a different place and time, her colleague was more interested in how the music and lights were done. “It was like we used criteria from two different orders” (P15).

The distinction between aesthetic and interpretative responses to art has been thoroughly discussed by philosophersFootnote 3 but has received little attention in sociological studies. However, the empirical findings suggest that this is a meaningful construct that may help make sense of the artistic selections of critics. In addition, finding the origins of this distinction might allow us to learn more about the organization of the art world. Asked where these perceptive differences come from, a reviewer explained that it reflects different educational backgrounds of evaluators. In Poland, two major academic institutions that educate critics (the Jagiellonian University in Krakow [UJ] and the Theatre Academy in Warsaw [AT]) cultivate different theatrological traditions: while at the multidisciplinary university (UJ) the dominant orientation centers on contextual interpretation of a performance, critics who learn their craft in an acting school (AT) and study in proximity to future actors and directors, are more inclined toward an “aesthetic perception and appreciation of technical skills” (P10). At the same time, the attitude of actors toward performances—the fact that they persistently described their favorite projects as “professional” in opposition to “amateur” work, which they quite openly disregarded—may also stem from their professional affiliations. The commitment to professionalism may have been nurtured in actors through the training programs at art colleges, given that the institutions of higher education typically socialize their students to obey what constitutes professional behavior in a given field (Martimianakis et al. 2009). However, professional actors may also tend to draw boundaries between themselves and “amateurs” to secure their position in the occupational field (Gieryn 1983) and act as gatekeepers who use accumulated symbolic capital to defend their status of dominance from potential newcomers (Bourdieu 1993).

In summary, reviewers interrogated directly about “evaluation criteria” focused almost exclusively on the content logic, which means that they referred to the properties of the work of art itself. However, two different value dimensions appeared during interviews. First, the aesthetic dimension is concerned with the attention to structure and form of the performance and detection of its internal consistency and expressive qualities. Aesthetic evaluation also involves finding out how the performance works, in other words, what formal arrangements bring about a certain aesthetic effect. Second, the interpretative dimension is occupied with intellectual work that involves deciphering thematic messages and identification of how these hidden meanings reflect upon or interact with the world.

Context Logic of Art Evaluations: Conservative and Progressive Approach

When we asked our respondents to describe other panelists in their group, in terms of their understanding of artistic worth, interviewees referred to the distinction between, in their words, “conservative” and “progressive” members. We were told that the panel included people who “invest in the masters,” which means renowned artists, well-established in the art world, and those who supported “young artists who question current hierarchies” and “challenge the existing order” (P9). The conservative/progressive descriptors were used primarily by the panelists who considered themselves to be located on the “progressive” side. Conservative members were more often labelled as such than self-identified themselves with this ideological position.

Respondents who considered themselves “progressive” praised artistic risk-taking, courage, boundary-pushing, and gave high grades to spectacles they perceived as “radical,” which means that they “challenged the aspects of reality [that are] commonly taken for granted” (P10). One of the “progressive” reviewers confessed: “I am always taken by works that bother me, make me feel inconvenient” (P15). At the same time, she claimed to despise “mediocre shows produced for mass audiences [which] add absolutely nothing to what already exists.” Somewhat like members of scientific panels who appreciate candidates who engage in risky, ambitious research projects (Guetzkow et al. 2004), progressive members have respect for artists who are bold, “ask challenging questions,” or “put [their] artistic careers and reputation on the line” (P13). Progressive members value “[the] use of novel, experimental means,” “artists who have the desire to expand their horizons,” and penalize those who do nothing but “produce just another performance” (P15). A panelist recalls a disappointing spectacle:

[It] told a story of Julia Pastrana, a nineteenth-century slave circus performer, whose face, due to a genetic condition, was covered with thick hair. It was supposed to be a post-colonial, feminist performance, but it completely missed the target. The set was rich, full of visual attractions, and they showed all the circus curiosities quite literally. They did not solve the problem of how to tell about objectification of the otherness without voyeurism. In the culminating moment, Pastrana was to offend the audience, expose our expectations and fantasies, but instead, she was howling like a dog or a wolf in despair. I understood this symbolic gesture, but for me, it was neither abusing nor embarrassing, not even causing any cognitive dissonance... The show ended somehow without any bold conclusion, without crossing any line. (P11)

Conservative members were recognized as such because they expressed more distance toward artistic experiments than their more progressive colleagues. One of the panelists who was labeled “conservative” by other interviewees said:

You must be intellectually strict [as a critic] because many artists today believe that theater is all about deconstruction… that deconstructing reality is enough [to make a good performance], that it will justify all the other things… I find it unacceptable. (P19)

By “all the other things” a reviewer means technical flaws and aesthetic imperfections in a performance. Conservative members believe that only experienced artists, who already possess advanced knowledge of different styles, can afford experimentation and risk-taking. A progressive critic believes that the conservative part of the group failed to appreciate “new [art] tendencies,” which were “regarded as artistic ‘excesses’ or ‘ridiculous things’” (P5). Progressive reviewers believed that they had to persistently defend productions that did not fit into the established artistic standards. The conservative label was attached to those who were “unable to recognize value” in a ground-breaking work, those who “didn’t get it” (P4) because their judgments were too deeply embedded in the same old conventions and value hierarchies.

However, conservative members did not see themselves as enemies of artistic novelty and innovations, and this is probably why they were hesitant to explicitly call themselves “conservatives.” Rather, it was their attitude toward “radicality” that marked the distinction between them and their “progressive” peers. “Radicality” means that a performance transcends the frames of pure and autonomous art by making references to contemporary social and political problems, making political statements, or offering specific social agendas. Conservatives believe, however, that art is autonomous from the mundane aspects of social life, and that the central concern of a good performance is to discuss the essentials of human existence and reveal the universal aspects of human nature. In the Polish context, a conservative member would enthusiastically welcome a masterful staging of classic works (e.g. Shakespeare, Chekhov, Beckett), while a progressive member would endorse performances based on contemporary (or modern adaptations of classic) work that comment on social and political changes in the current political climate marked by the dominance of right-wing parties.

Political scientists have shown that people attach different meanings to the conservative-liberal spectrum; for example, while some citizens perceive it as primarily about economics, others understand it in terms of social issues (Feldman and Johnston 2014). The progressive/conservative self-identification seems to work in a similar fashion; when asked what they value in performances, some progressive respondents emphasized their attention to “originality of form” (“[an artist] created his own unique language” [P25]), while others focused on the political purpose of theater: enacting social and personal change by challenging taken-for-granted norms and boundaries, including the division between the passive audience and the “sacred” space of the stage. At the same time, when describing performances, respondents did not distinguish between formal experiments and their emancipatory purposes. This is how one of the “progressive” respondents talks about one of her favorite “radical” (in her words) performances:

It was a seemingly unattractive spectacle, very static, even melancholic. Every few minutes the light went out, symbolizing the idle days that passed. This monotony built tension in viewers, caused discomfort and rejection reactions. At the end, there was a participatory stage: the actors handed out baseball bats and invited viewers on to the stage where they could bash the bats into punching bags. It wasn’t until I entered the stage that I realized what strength it is, what emotions are released then, what sense of empowerment. (P10)

In addition, progressivist and conservatists seemed to have different, rather than opposite, ideological perspectives. For example, progressive panelists disregarded performances they found “homophobic,” “racist,” or “sexist” and supported performances that conveyed “feminist” messages. They highlighted the fact that other members of the group did not pay attention to these aspects or took them for granted, arguing that “it is a fair representation of a particular reality” (P10).

The division between progressive and conservative members follows the context logic, where the artworks are evaluated with reference to their creators and their engagement with the external world. Progressive panelists tend to view the originality of a performance as an indication of the creator’s integrity, artistic courage, and risk-taking. Conservative panelists also value artistic innovation, but one that sits within the established frameworks of professionalism and craftsmanship; therefore, the biographies of artists and their artistic experience may be used as a factor in decision-making. Moreover, the contexts that artists draw upon when making their works are of importance here. While progressive members expect artists to involve themselves in debates about the issues that trouble modern societies through their creative activities, conservative panelists believe that the role of artists is to address more universal and everlasting human concerns.

Value Matrix

The two logics examined above are not exclusive; our evidence shows that, for example, both conservative and progressive members made references to aesthetic- and meaning-oriented judgments. What stems from our results can be conceptualized as a matrix of value concepts that prevail in the evaluation settings (see Table 1).

Table 1 Matrix of artistic value concepts
Full size table

The matrix demonstrates that both conservative and progressive members use concepts of value pertaining either to the design appreciation (aesthetic orientation) or to the meaning of an artwork. However, while the conservative members concerned with the design elements favor artwork created for the sake of aesthetic perfection by renowned, skillful artists, their progressive colleagues applaud artistic experiments and the use of novel means to expand perceptual boundaries.

Analogically, both conservative and progressive members can be oriented toward meaning. Members who consider themselves progressive invite performances that rhetorically advance socio-political claims and regard moral indignation as a justifiable response to art. The conservative part of the panel, on the other hand, seeks universal messages about the human condition and is much less likely to contemplate performances in terms of ulterior interests, such as social agendas and political content.

Our matrix illustrates divergencies and overlaps between different concepts of what constitutes a valuable artistic achievement. The four concepts of value that stem from our analysis (1. Aesthetic-oriented—conservative; 2. Meaning-oriented—conservative; 3. Aesthetic-oriented—progressive; 4. Meaning-oriented—progressive) reflect the most dominant criteria of distinction that reviewers used to describe panel members. These four concepts are not mutually exclusive; for example, meaning is always, in one way or another, considered in relation to form, both among conservative members who support the classic principle of decorum, the fit between matter and form, and among progressive members who value formal experiments for their ability to foster emancipatory ideas. Therefore, we propose to think of our matrix not in terms of four separated value regimes where particular groups are associated with a single concept of value, but rather as different hierarchies of artistic value or “orders of worth”Footnote 4 (Boltanski and Thévenot 2006) that are operative within a theater network and applied by panelists in a particular group context, and on a case-by-case basis. Panelists’ involvement in different subgroups in the art world (e.g., actors vs. critics) influences their critical approaches and has an impact on the importance they attach to particular properties of theater performances. Nevertheless, their position in the panel is relational and depends on the group composition (where she stands with respect to other panelists), as well as on the specificity of work under consideration (e.g., some panelists that are “progressive” on other occasions, may not accept a particularly radical political work and therefore be positioned as “conservatives” by those who approve the performance in question).

Interpersonal Rules of Deliberation

Given that the task of reviewing panels is to produce common judgment, panelists must find ways to navigate different concepts of value and divergent evaluative criteria. Although conflicts and battles are unavoidable, settlements are finally reached and there are a variety of mechanisms and intersubjective rules involved in this process. The mechanisms in question are informal, taken-for-granted norms and beliefs in what is considered fair, unbiased, or well-grounded evaluation, that Lamont (2009) calls “customary rules.”

There is no doubt that consensus is a desirable outcome of panel deliberations, and throughout all 24 editions of the NCPDSC, panelists were almost always successful in reaching unanimous consent. Respondents pointed out different motivations behind striving for a consensual verdict. Among them was the panelists’ feeling that since they act on behalf of the Ministry of Culture, they must show integrity and consistency in judgments. Panelists also cared about protecting the interest of public theaters (“If we don’t award first prize, we will send a message that theaters are not worth funding” [P28]) and the public image of the panel itself.

Despite the fact that reaching a consensual decision is a shared goal of panel members, the process of producing consensus is far from simple and unambiguous. In this section, we explore informal rules that govern the behaviors of panelists and allow them to reach an agreement despite the evaluative differences described above. We found that rules of deliberation operate on three different levels. In what follows, we describe and give examples of each: Firstly, the substantive rules of deliberation are content-related and concern arguments that are regarded as a valid justification of artistic merit. Secondly, procedural rules deal with interactional and interpersonal aspects of procedures that shape cooperation within the group. Thirdly, the contextual rules of deliberation are linked to the social context of works being evaluated and to the panelists’ role as public decision-makers.

Substantive Rules of Deliberation: The Imperative to Justify

Art critics employ a variety of customary rules in order to legitimize their judgment (Chong 2013); for example, critics interrogate their artistic experiences by turning their immediate, spontaneous reactions into an object of scrutiny. The strategy in question is relevant not only to the individual judgment but is also part of group reviewing, where self-interrogation is required from a reviewer by her peers, who expect her to translate her emotions into a reasoned argument. According to our respondents, discussing art in terms of personal impressions is considered inadequate by the group, and panelists believe that artistic judgment must be justified by making reference to the “objective” properties of a performance. This rule applies as much to the analysis of the work’s structure and matter (e.g., acting, music, scenography, movement) as to the reading of its meanings and authorial intentions, which are expected to be “sound” and based on evidence: “One of the panelists explained very scientifically, scene after scene, what the director wanted to tell us by casting females in male roles” (P10). Unlike amateurs (Long 2003), experts are not allowed to make interpretations with reference to their personal experiences (for example, how the main character’s choices helped them understand their own life trajectories); instead, their job is to discover “what this work is really about” (P9).

Panel members undertake collective efforts to control one another and safeguard the intellectual and analytic character of deliberations. It was reported that:

When you really like a performance, it is really easy to let yourself go into ecstasies and start telling others: “It was so beautiful, so perfect!” But other members immediately discipline you by saying something like: “Okay, but what exactly is so great about this piece?” (P13)

Reviewers punish members who show too much excitement by ridiculing their spontaneous reactions. A panelist recalled that his “enthusiasm was killed by sarcastic comments” (P4) of colleagues who found his affective narrative unbecoming of a professional critic. At the same time, he claimed to behave the same way toward others, shaming a colleague who “tried to convince us by saying how deeply touching a performance was.” Negative emotions, such as anger or disgust, are usually not corrected, as they are not perceived to be at odds with the “critical thinking” that experts are expected to demonstrate. Also, some panelists believed that anger was a justified reaction, for example, to performances that perpetuated harmful stereotypes.

In addition, reviewers frequently emphasized that participants motivated others to explain their point of view in a clear and plainspoken manner. For example, when one of the members described a performance as “affective,” she was immediately asked to explain what she had in mind. Therefore, reviewers are expected to convince one another with the force of reason, and need not be fooled by emotions or rhetorical strategies.

What critics describe here confirms what Hanrahan (2013) discovers in her study of music critics and what Chong (2013) finds about literary reviewing: that art criticism is an evidence-based analysis, “a survey” (Dewey 1934, 320). By providing evidence, critics allow their justifications to be tested by peers. In their work on justification, Boltanski and Thévenot (2006) argue that submitting judgments and justifications of worth into tests is a necessary prerequisite for reaching an agreement. The test should examine the factual nature of what is being evaluated; it cannot rely on mere rhetoric or be reduced to a theoretical debate. In our study, the panelists’ imperative to justify underlies the possibility of achieving a consensus about the relative worth of performances.

However, as participants draw upon different concepts of value and use multiple logics of justification, disagreements can be particularly difficult to handle. One way to assuage critical tensions is to find compromise on possible common features of different “orders of worth” (Boltanski and Thévenot 2006). In our case, compromises were reached by giving highest rank to performances that were “acceptable” by all, or almost all members of the group, and at the same time, by eliminating the most polemical, “extreme” options that were highly consistent with some concepts of value but inconsistent with others (for example, an agitational street performance would be consistent only with the meaning-oriented—progressive concept of value). Our respondents confirmed this finding by telling us that the final selection consisted of performances that “did not bother anyone,” were “lukewarm,” and “compromising.” Previous research has also found that the methods of rating based on consensus tend to support uncontroversial, moderate projects while the most extreme options are left out as they are unacceptable for some members of the group (Langfeldt 2001). Although this might be considered a flaw in the evaluation of art, where taking risks and pushing boundaries is a significant part of the creative process, expert panels continuously rely on this decision-making formula and hold it in high esteem.

Procedural Rules of Deliberation: Democratic Decision-Making

Almost without exception, our respondents considered panel deliberations democratic. Panelists believed that everyone had equal opportunity to express themselves and each member’s voice was given full consideration. Democratic conduct of group deliberations was perceived indicative of a well-working panel and healthy relationships between its members. In addition, panelists attached great value to deliberative decision-making and drew satisfaction from having a chance to cooperate, discuss performances in depth, and reach a common understanding of the issues at hand. Panelists believed that they “learned a lot” when serving as experts on the panels, especially in terms of perspective-taking and giving careful consideration to other people’s opinions. Moreover, deliberations were perceived as more valuable than other forms of decision-making. The chair of NCPDSC asserted:

We do what we can to avoid voting. We always talk a lot and try to painstakingly convince one another. It is often a long and difficult process but it pays off. We don’t want people to walk out of the meeting, we want everybody to feel that it is our verdict. We vote only when there’s no other way out. (P2)

In general, people tend to hold the “unitary,” discussion-based forms of decision-making in higher esteem than the “adversarial” forms of democracy, such as the majority rule (Gastil 1992; Rozell and Gundersen 2003) and are much more likely to act cooperatively when they are given a chance to discuss the issue at hand with one another (Meleady et al. 2013; Tyler and Blader 2003). In reviewing panels, discussion is considered more valuable than voting, however costly in terms of time and energy (Laudel 2006).

Democracy is typically considered disrupted when group members engage in informal alliance-building or enter into cliques with those who share similar views (Gastil 1993). Nevertheless, our respondents talked openly about taking part in such practices and did not perceive them as illegitimate behavior. Interviewees reported building alliances with different panelists and believed that others behaved the same way. These bonds and cliques began to develop during panelists’ interactions on trains, in hotels, or theater foyers, before and after performances. Traveling together was an opportunity to “investigate what others thought” (P14) and “explore who will play on who’s side” (P7). Alliance-building was not about setting explicit arrangements but about sensing “with whom we get along” (P15) in terms of artistic preferences. Private sympathies frequently coincided with artistic affinities and panelist felt that there was more “chemistry” between people with similar artistic values:

Staying at the same hotel, having group discussions… it brings some panelists together. You feel close to someone when you can believe them. You believe they’re right about this piece because [their judgments] make sense. Alliances are not about bribing people. Your ally is someone who understands you, who knows why you’re supporting particular artists. (P13)

Moreover, personal bonds were much stronger between young panelists than among their more experienced peers. Some young members created their own online forum and arranged schedules such that they could travel together.

Alliances were formed also during panel deliberations. One of the respondents confessed: “During a break when everybody was distracted I went to T and said: ‘Listen, this performance was really great, we must persist… you said you liked it too!’” (P17). However, respondents do not think that aligning with their colleagues is an obstacle to panel democracy, because they find these coalitions weak and unstable—panelists align themselves with different reviewers at different times. “There is no loyalty in these alliances” (P13) and people often change their position, regardless of their personal and artistic affinities. Individual strategies, such as horse-trading or making strategic choices, are regarded as more efficient. Panelists take for granted the fact that “a panel discussion is like trading” (P18) and make use of different tactics to achieve their goal. One of the interviewees recalled:

I knew that some people wouldn’t let it [his favorite performance] get through. So I opted for two performances: one that I liked, and another one that was even more avant-garde and “unacceptable.” As a gesture of goodwill, I accepted the rejection [of the latter] and they had to accept my performance to give me something in return. (P4)

Horse-trading and strategic voting are also perceived as ordinary, natural behaviors that are unavoidable in panel settings, especially at the later stage of deliberations when there are few resources left. Panelists realize that their ranking of performances is relational, and that very often they need to choose between artworks whose artistic quality is more or less comparable. Strategic thinking is, therefore, part of a realistic approach to panel decision-making and shows that the reviewers’ understanding of group democracy is filtered by their view of how democratic deliberations best serve their ultimate goal: forming a consensus.

While the tactics described above were considered permissible, respondents expressed some concerns about the unequal influence that different panel members had on the group. Panelists differed in terms of age, gender, personality, as well as expertise and years of experience. Social psychologists have shown that such characteristics are related to authority and influence in group dynamics.Footnote 5 Most of these factors were not brought up by our respondents, possibly because their effect on deliberations was seen as unproblematic or “natural.” For example, younger panelists did not think that the more experienced members had greater influence than they did, in spite of the fact that they repeatedly described senior peers as those who possessed “knowledge that we [the young peers] didn’t have” (P8), who offered “valuable [industry-related] insights” (P20), which helped the group make more fair-minded decisions. It is therefore possible that the panelists took this influence hierarchy as given, which is likely to happen when people’s referential beliefs about the higher status group members lead to a voluntary deference to the most influential actors (Ridgeway and Berger 1986).

In contrast, the influence of gender was explicitly highlighted by a few interviewees. One of the female panelists was irritated by the fact that her arguments were not taken seriously enough, especially when she addressed the “misogynist” aspects of performances. Another one felt that deliberations were marked by a “paternalistic tone” of male senior members and wished future panels would be “chaired by women” (P27). It was also pointed out that the feminist standpoint was disregarded and that women had to “argue and fight hard” (P15) to be listened to.

However, even the respondents concerned about the gender effect did not describe the panels as undemocratic, perhaps because, after all, “everyone was given a chance to speak” (P15). Reviewers tended to perceive group democracy in terms of whether they had a voice, and regardless of the impact of their opinions on the actual decisions made. Similar to other studies (see Tyler and Blader 2003 for a review), in our panels the opportunity for reviewers to make their voices heard had interpersonal worth that was not necessarily linked to people’s actual influence on the outcome of the decision-making process. Panelists who complained about the inequality of influence tended to rationalize those situations later on (“maybe I wasn’t persistent enough” [P15], “I didn’t try to push it any harder because the discussion couldn’t go on forever” [P29]). Therefore, it seems like panelists hold a pragmatic approach toward the deliberation procedures. They realize that they need to reach an agreement about a significant number of productions and allocate public resources within a short span of time. This practical concern influences their understanding of what is good or desirable behavior and allows them to deal with the inconsistencies between the meanings they give to their collective and individual actions.

Contextual Rules of Deliberation: Social Contextualization

Given that subjectivity of artistic judgment is commonly accepted, members of artistic panels do not worry about individual subjectivities in the evaluation (Wilson 2016). A panelist is not troubled by the fact that: “It is hardly ever possible to find a fully valid and objective answer why we have chosen these particular spectacles” (P17). However, members make efforts to legitimize their choices by stepping out beyond their idiosyncratic tastes, not so much by applying some universalistic standards, but by judging different types of art through different lenses. Somewhat like scientists who bear in mind the fact that epistemological and methodological standards vary between disciplines (Lamont 2009), our panelists realize that criteria of quality change and, for example, a Broadway-like musical has different qualities than an existential theater based on the works of August Strindberg. Therefore, panelists believe that each performance should be judged only with reference to the works that fall into the same category:

It is impossible to compare a low-cost independent production with that of a national theater, or an entertaining comedy with the works of Krzysztof Warlikowski.Footnote 6 A performance should be assessed within its own framework and with respect to what the authors wanted to achieve. (P2)

A panelist confessed that she was not particularly fond of puppet theater, but she understood that, while acting as a panelist, “you need to set your personal criteria aside” (P28). Another one recalled that the panel chair advised panelists to “be open to different forms of theater art” and “get prejudices out of [their] heads” (P14), trying to encourage members to appreciate the idiosyncrasy of the Polish theater art, instead of focusing on big names and what is currently in vogue.

Many panelists told us that members took into account the social and institutional conditions in which productions were made and performed. Polish public theaters are located in different parts of the country, from metropolises to small provincial towns and isolated rural areas. Performances that form part of their repertoires are made for different kinds of audiences and in response to different community needs. A reviewer believes that: “A theater performance is a social situation, it is always addressed to a particular audience” and “it works best when watched in its natural environment” (P1). Since performances were watched on their own turf, their local impact was given special consideration in the evaluation process. One of the panelists recalled evaluating a production in a small-town theater: “Perhaps from our perspective it wasn’t a masterpiece, but we knew that [local] people talked about it, that it was a big deal to them” (P24).

Furthermore, our interviewees mentioned the fact that their assessment criteria change over time and that each performance is evaluated with reference to those seen before. A few reviewers pointed out that “the more shows you see, the less strict you become” because with each new performance “your expectations get lower and lower” (P25). Shrum (1996, 147), who called this phenomenon among theater critics “standard sinking,” recognized that the volume of experience may have a powerful impact on the evaluation process. Finally, reviewers are aware that their own mental and physical condition may alter their assessments. One of the panelists remembers assessing one of the performances harshly due to the fact that he was tired and hungry after spending long hours on a train. When he realized a few days later that his judgment had fallen prey to his condition, he upgraded the score. Therefore, panelists seem to realize that artistic quality is evaluated not according to some general, consistent standards, but within the universe of a particular set of performances under consideration.

Contextualization helps panelists resolve disagreements because it provides arguments that bear upon factual, or less contestable (than artistic quality) features of works under evaluation, such as social and economic situation of institutions and local communities. Panelists evaluate performances against the background of their social context also because they are committed to their roles as public decision-makers. When artistic selections involve distribution of public resources, evaluation criteria are not confined to artistic quality, but experts infer the principles of social welfare and justice (Gielen 2005). Considering the welfare-related issues helps to legitimize artistic choices of panelists and, in turn, the allocation of resources powerfully contributes to the artistic legitimation of public theaters (Baumann 2007).

Conclusions

Panelists engaged in reviewing theater performances claim to focus on one thing: artistic value. However, reviewers apply different concepts of value in their evaluations and attach different meanings to artistic excellence. The matrix of values that stems from our analysis helps understand how art experts make artistic selections, namely, that evaluations are based on a combination of two different logics: content logic, concerning two fundamental properties of an artwork (aesthetics, meaning), and context logic, involving the application of two different approaches (progressive and conservative) to the artistic form and meaning. By integrating concepts that have been studied much more often by philosophers of art than sociologists (distinction between aesthetics and meaning) with constructs that are well known in the field of social studies (the ideological divide between liberals and conservatives), our model provides a framework for understanding what critics pay attention to and what criteria they use in their evaluations.

In addition, our data suggests that experts self-identify themselves, and identify others, with respect to how they interact with the work of art: whether they are more absorbed by aesthetic perfection or a masterful presentation, whether they react negatively to gender stereotypes, or to overtly political messages. Panelists define their artistic inclinations in relation to those of their peers, which are communicated during group discussions. Their encounters with the specific features of an artwork—its aesthetic qualities, the messages it conveys, or the way it engages with the external world—allow them to recognize commonalities and distinctions within the group and help them place themselves within certain categories of reviewers and outside others. It was known before that shared aesthetic judgments influence people’s feelings of belonging (Wohl 2015), but our analysis shows that not only judgments (whether they like something or not) but also how people arrive at them (what criteria they use and what concepts of value they apply) provide social actors with opportunities to confirm group identification.

In this paper, we also study rules of deliberation and discover that they work on three different levels. While all three types are oriented toward facilitating justified agreement, they do so in different ways. First, substantive rules of deliberation are followed by reviewers to ensure the legitimacy of the artistic judgment. The fact that panelists must convince one another of the value of a performance, and that they believe in the power of the better argument, fundamentally contributes to the perceived legitimacy of the process. Panel deliberations are seen as a more credible way of making artistic judgments than individual evaluations that are not defended publicly and leave more room for personal arbitrariness. Reviewers convince one another by justifying their judgments, and justifications must rely on the objective properties of an artwork so that they can be tested by others in interaction. The imperative to justify, which exists not only on the individual but also on the collective level as the group uses different strategies (e.g. probing, ridiculing) to protect the “rationality” of group deliberations, allows the panel to consider the agreement as valid. At the same time, as panels consist of different kinds of reviewers, and different logics of justification come into play, reviewers resolve tensions by selecting productions that are acceptable by all participants, that is, performances that are defensible in terms of different concepts of value. Panelists are oriented primarily toward reaching a consensus, but consensual decision-making might penalize unsafe and risky options (Langfeldt 2001) that enhance creative progress in the artistic world.

The second type of deliberation rules, procedural rules, are the norms of interpersonal behavior that are perceived as desirable and correct. They include, first and foremost, the fact that reviewers have equal opportunity to speak and that people’s arguments are received with careful consideration. Given that quality of treatment has a powerful impact on people’s desire to engage and act cooperatively (Tyler and Blader 2003), following the procedural rules maintains panelists’ motivation to strive for consensus, even when the group faces particularly strong tensions. Moreover, procedural rules play a legitimacy-enhancing role, as individuals bestow most legitimacy upon arrangements that allow personal involvement in the decision-making process (Esaiasson et al. 2012). Although some of the practices mentioned by the reviewers might be seen as a disruption of democratic consensus-making, such as alliance-building, horse-trading, or strategic voting, panelists consider them as compatible with democratic deliberations because they are seen as either unobtrusive or unavoidable in group decision-making. Like in scientific panel decision-making (Lamont 2009), reviewers seem to hold a pragmatic understanding of democracy and evaluate the fairness of the process through the lenses of their practical concerns.

Third, contextual rules are linked to the panelists’ role as public decision-makers. Reviewers believe that the task of allocating public funds requires considering not only artistic merit of a performance (although artistic quality is still prioritized) but also its local impact—the role it plays in the lives of local community members. This conviction pushes reviewers to go beyond their idiosyncratic tastes and attachment to high-quality artistic content, and evaluate productions on their own terms, that is, within their social and institutional contexts. Panelists are committed to aesthetic quality, but they also realize that different theaters produce for diverse types of audiences and face different institutional constraints; therefore applying consistent standards across productions would not do justice to the diversity of the Polish theater scene. Social contextualization provides more “objective” grounds for collective judgments, thus supporting the legitimacy of reviewers’ decisions.

Panelists strive for consensus, and their motivation to find a consensual verdict shapes the process of judgment-making. However, panelists’ commitment to agreement-formation comes from the institutional and social context of panel decision-making rather than the feeling that a discussion about art must lead to a conclusion. Researchers have shown that when artistic evaluations are made during informal community gatherings, participants do not feel compelled to find an agreement, and enjoy the open-endedness of group discussions (Childress and Friedkin 2012; Long 2003). Without strong external constraints, deliberations about art tend to be inconclusive (Merriman 2017). At the same time, consensus-building is an essential part of many decision-making practices in the art world, such as auditions (Nylander 2014), publishing (Merriman 2017), or public funding (Loots 2019). Our case study demonstrates that when art evaluations require a final outcome, they share many characteristics with consensus-oriented practices in other fields; for example, we have found many similarities between artistic and scientific decision-making bodies. Therefore, it seems that the purpose of evaluation is crucial, and that our findings can be translated to other areas of social life where consensus-building is highly valued.

Our research suggests that groups may compromise a lot of things to reach a consensual decision. Finding an agreement requires settling for more acceptable, but less impressive options, and thinking strategically rather than idealistically about promoting artistic values. Panelists frequently need to set aside their artistic preferences and adjust their expectations to particular cultural environments. Arriving at a conclusion often takes priority over respecting some aspects of group democracy, such as equality of influence among panel members. Although sociologists have already touched upon some of these topics, including panelists’ deference in the interest of consensus (Huutoniemi 2012; Lamont 2009), more research is needed to explore what else is at stake when group members strive for consensus.