Planning and Conducting Mixed Methods Studies in Mathematics Educational Research

Abstract

In this chapter, central ideas of Mixed Methods Research are presented in which qualitative and quantitative research methods are combined or integrated. In addition to the explanation of common Mixed Methods terminology, the chapter provides an overview of the most important aspects that must be reflected in the planning and conduction of a mixed-methodological research project. On the basis of considerations on the nature of the research object and specific conditions of mathematics education research, methodological aspects of the research question, research design and data analysis are described. The chapter concludes with considerations on the challenges of Mixed Methods, as well as recommendations on the step-by-step approach to a Mixed Methods Research project.

1 Introduction

The number of studies using both qualitative and quantitative methods or combining qualitative and quantitative data has increased significantly in recent years in mathematics education research. As shown by meta-analyses by Hart et al. (2009) and Ross and Onwuegbuzie (2012), researchers accounted for both qualitative and quantitative methods in no less than 29 and 31% of mathematics education articles published in international journals between 1995 and 2005 and between 2002 and 2006 respectively. More and more researchers nowadays situate their studies in the Mixed Methods Research (MMR) methodology, and the use of multi-methodological approaches to data analyses is increasingly taking place against a methodological background that has been described more broadly and in more detail in recent years. Mixed Methods, the “third research paradigm”, as described by Johnson and Onwuegbuzie (2004, p. 22), appears to overcome the one-sided methodological thinking of both qualitative and quantitative research paradigms, despite continuing discussions about the incompatibility of qualitative and quantitative methods (Howe 1988). Therefore, Mixed Methods studies currently enjoy great popularity in the mathematics education research community. The pragmatic and innovative combination of different research perspectives promises additional insight, which might not be accessible with a single methodological research approach.

For a long time, the Mixed Methods discussion focused on the development of new, different research designs (Teddlie and Tashakkori 2009; Creswell and Plano Clark 2018; Schoonenboom and Johnson 2017) and thus on how and when qualitative and quantitative methods are ‘mixed’ in concrete research designs. Several extensive handbooks on Mixed Methods have been written (Creswell 2003, Tashakkori and Teddlie 1998; Creswell and Plano Clark 2018; Hesse-Biber and Johnson 2015; Kuckarts 2014) and numerous methodological and theoretical research articles have been published in the major journals of the MMR community—among them the Journal for Mixed Methods Research (JMMR). Meanwhile, a large number of different design types or typologies for Mixed Methods Research designs exist, so that some researchers already speak of a “design overload” (Kuckarts 2014). Exaggeratedly, researchers can just choose a Mixed Methods design suitable for their purposes and get an appropriate methodological justification and guidance for the research process immediately. This offer sounds tempting, but falls short with regard to the subject-specific methodological reflection on one’s own research approach. Not to be misunderstood: choosing a suitable Mixed Methods research design is still a crucial and important issue in the research process. But this should not be the first consideration, when one is dealing with the question of the structure of a planned research project, because, as Burton (2002) notes,

in the majority of articles in journals and books, a description is provided on ‘how’ the research was done but rarely is an analysis given of ‘why’ and more particular out of all the methods that could have been used, what influenced the researcher to choose to do the research in the manner described to do it. (p. 1)

The decisive factor when choosing a Mixed Methods research approach is, first of all, the question ‘why mixed methods at all?’

Especially for an early career researcher in mathematics education, this can be a very difficult question to answer, because it is linked to the object of research and the exact research question, which may not yet be established at the beginning of the work. Nonetheless, choosing a Mixed Methods research approach should not be based on general trends, because one thing is often overlooked in the decision: researchers not only have to be methodologically well-versed in qualitative methods, but also well-versed in quantitative methods. That is not all—in addition, good methodological knowledge in the field of Mixed Methods methodology is required. The time resources that early career researchers must devote to study all these methodologies should therefore be well considered, especially if they have been funded only for a few years. However, once a researcher decides on a MMR approach and can realize it in the framework of a study, the promise of extended knowledge gain is certainly fulfilled—even if this knowledge gain may consist in the fact that the results yielded with different methods contradict each other or are not compatible at all.

This chapter provides an overview on central questions and fundamental aspects for carrying out a multi-methodological research project in mathematics education research. There is now a broad base of literature in the field of MMR, especially developed for researchers who are at the beginning of a research process (e.g., Schoonenboom and Johnson 2017). This literature also describes the key steps in the research process and how a MMR project can be carried out. This chapter cannot and should not replace the necessary consultation of this methodological literature, but it does contribute important subject-specific reflections and justifications from mathematics education research to this topic. Therefore, the chapter is an introduction to the methodological background of MMR and refers to further literature. The terminology, methodological justifications and the central features of MMR are provided. Subsequently, central steps in the implementation of a MMR project are described.

2 Methodological Background of Mixed Methods Research

2.1 What Is Mixed Methods Research?

The discussion about Mixed Methods has gained momentum since around the 1980s and is being led more and more by methodologically reflective literature. Mixed Methods draws back on the idea of triangulation as “the combination of methodologies in the study of the same phenomenon” (Denzin 1978, p. 291). Denzin distinguished between within-methods triangulation, which refers to the use of multiple quantitative or multiple qualitative approaches, which is nowadays referred to as “Multimethod Research” (Creswell 2016, p. 216ff.), and between-methods triangulation, which involves the use of both quantitative and qualitative approaches. From the latter understanding, the current conceptual understanding of MMR has finally emerged. For a more detailed overview on the historical development of the Mixed Methods movement, I recommend reading the descriptions by Johnson et al. (2007) and Maxwell (2016). However, within the Mixed Methods Research community, the term Mixed Methods is not used as uniformly as it might appear, even though more and more consensus is emerging in some areas (Mertens et al. 2016).

In their influential article Toward a Definition of Mixed Methods Research, Johnson et al. (2007) list no fewer than 19 distinct definitions of Mixed Methods, most of them understanding Mixed Methods as a combination of qualitative and quantitative research. Johnson et al. (2007) summarized the situation in a general definition:

Mixed methods research is the type of research in which a researcher or team of researchers combines elements of qualitative and quantitative research approaches (e.g., use of qualitative and quantitative viewpoints, data collection, analysis, inference techniques) for the broad purposes of breadth and depth of understanding and corroboration. […] A mixed methods study would involve mixing within a single study; a mixed method program would involve mixing within a program of research and the mixing might occur across a closely related set of studies. (p. 123)

Although this definition is cited by many authors as describing MMR, it is ultimately only one of many different definitions. The methodological discourse within the MMR community indicates that there is disagreement, especially with regard to the definition of Mixed Methods. A too high rigidity might take the risk of being exclusive from a definitional perspective (Mertens et al. 2016), and so, for example, studies that combine hermeneutic research methods with empirical methods would be excluded from this definition. Since the discussion about Mixed Methods is based in the social sciences, the definition also unconsciously reflects the social-scientific understanding of methods. Therefore, we should ask whether and to what extent this definition is applicable to the field and to the methodology of mathematics education research (see Sect. 6.3).

2.2 What Kind of Research Questions Does Mixed Methods Research Require?

First of all, the main feature of each MMR approach is that a Mixed Methods study essentially consists of two or more research “components” (Schoonenboom and Johnson 2017) or “strands” (Teddlie and Tashakkori 2009) from different methodological perspectives (in most cases qualitative and quantitative). Secondly, in the context of a MMR approach, the components must be related to each other; in the terminology of MMR, this means combining components with each other if the goal of the research is to mutually complement findings of each. Alternatively, it means integrating the components if the goal is the mutual validation of the research results. This combination or integration is always a function of the research question and the purpose of the study. Both components should provide insights into a superordinate (integrated) research question, which should therefore be formulated in such a way that the components can also focus on corresponding sub-questions and provide insights that can be related to the overarching question, that is, they can be integrated.

Here is a combination example from mathematical education research: In a research project, a researcher developed and carried out a teacher professional development (PD) module for the promotion of certain skills. The researcher is now interested in the results of the training. An integrated research question could therefore be “How does this intervention within mathematics teacher PD affect the attitudes and competences of teachers?” This question could be addressed quantitatively if the number of teachers that participated in the PD module is adequate. For this purpose, for example, the researcher may use a questionnaire on specific attitudes or self-efficacy before and after the module, if he or she believes that attending the module has contributed to change, such as the willingness to apply the content in forthcoming lessons. The quantitative hypothesis on the overarching question is thus: “Attending the PD module on X leads to a significant increase in positive attitudes to X and the willingness to use X in the classroom”. At the same time, the question could also be addressed qualitatively, because the researcher is interested in what other effects the attendance to the PD module had, especially effects that might not have been anticipated. A qualitative partial research question can then be, for example: “Which aspects of the PD module were particularly helpful for teachers to learn X? What other impact(s) did the course have?” These questions could be addressed with in-depth interviews with some participating teachers, focusing on qualitative evaluation methods such as content analysis. When combining both findings, a broader spectrum of the PD module’s impact is considered and the results complement each other. Quantitative findings may be enhanced by results from case analyzes (for example, to find out what might be the reason why some teachers, despite attending the PD module, do not show attitude changes).

How the concrete combination or integration of the results of the different components can be realized in order to answer the overarching research question of a study must be decided on a case-by-case basis and is part of the responsibility of the researcher. The challenge here is to gain additional value from the integration of the results with regard to the overarching question, namely, “produce a whole through integration that is greater than the sum of the individual qualitative and quantitative parts” (Fetters and Freshwater 2015, p. 116; see also Bryman 2007). In the MMR community, the formulation “1 + 1 = 3” is used to symbolize the challenges of generating meta-inferences from integration, meaning qualitative + quantitative = more than the individual components (Fetters and Freshwater 2015), which is, however, difficult to endure for mathematicians. Experience has shown that this step has the greatest challenges in the research process, and because of the lack of generalizability, there is also little literature on the concrete procedure. Empirical meta-studies on the quality standards of Mixed Method studies also show that integration is often absent, and that in lower-quality Mixed Method studies, qualitative and quantitative research questions are answered rather independently (Bryman 2007), without any relation between the results produced.

2.3 What Is the Purpose of Doing MMR? And Why Should I Choose This Methodological Approach?

As Burton (2002) points out, the purpose of the study should be clarified depending on the research question of the study. The central question of why one chooses a MMR approach is at the forefront of the decision for this research approach. It should be weighed to what extent a mixed-methodical approach has advantages over a mono-methodically oriented approach. That means, a researcher should initially reflect on the specific strengths and weaknesses of qualitative research approaches and quantitative research approaches, and to what extent a MMR approach can make use of the strengths and compensate for the weaknesses. Johnson and Christensen (2017) characterize the quantitative and qualitative research paradigms briefly as follows:

First, the quantitative research approach primarily follows the confirmatory scientific method because its focus is on hypothesis testing and theory testing. Quantitative researchers consider it to be of primary importance to state one’s hypotheses and then test those hypotheses with empirical data to see if they are supported. On the other hand, qualitative research primarily follows the exploratory scientific method […]. Qualitative research is used to describe what is seen locally and sometimes to come up with or generate new hypotheses and theories. Qualitative research is used when little is known about a topic or phenomenon and when one wants to discover or learn more about it. It is commonly used to understand people’s experiences and to express their perspectives. (p. 33)

Mixed Methods researchers argue that it is important for the research process to use both exploratory and confirmatory methods. Johnson and Onwuegbuzie (2004) describe Mixed Methods in their much-cited article as the third paradigm between quantitative and qualitative research, building on the philosophy of pragmatism. The idea behind this approach is that the combination or integration of quantitative and qualitative methods or even paradigms can compensate for the weaknesses and build on the strengths of the respective research approaches (Johnson and Christensen, 2017, p. 51). Some of these specific strengths and weaknesses can be described as follows (Johnson and Onwuegbuzie 2004, p. 19):

Strengths of quantitative research:

• Testing and validating already existing theories and hypotheses

• Generalizing results using appropriate samples

• The influence of confounding variables can be controlled in experimental settings

• Providing easy-to-handle (numerical) data

• Data analysis is relatively less time-consuming (often software-supported)

• Relative independence of the results from the researcher

• Overall high credibility for decision makers

• Useful in large sample analyzes.

Weaknesses of quantitative research:

• Research categories or theories often are not culturally sensitive

• Results depend on the theoretical assumptions and are not supported by a curriculum

• Occurring phenomena are often hidden due to the hypothesis-testing procedure (confirmation bias)

• There is discussion about a replicability crisis in quantitative research (Open Science Collaboration 2015).

On the other hand, strengths and weaknesses of qualitative research can be described as follows (for instance, see Johnson and Onwuegbuzie 2004, p. 20):

Strengths of qualitative research:

• The collected data carry a subjective context of meaning of the examined persons

• Qualitative research is suitable for looking at small case numbers in depth

• Useful for describing complex phenomena or dynamic processes

• Creating a contextual context of the results, cultural sensitivity

• Generation of theories in the context of Grounded Theory

• Case studies to illustrate results.

Weaknesses of qualitative research:

• Data analysis is often very time consuming

• Often generalizability of the results is not possible

• It is difficult to verify/falsify hypotheses

• Results may be idiosyncratically affected by the researcher

• There are ongoing discussions about quality criteria.

The mutual reproach of weaknesses to the significant other methodological approach was the subject of the so-called “Paradigm Wars” (Gage 1989) within the social sciences but also in educational science research until the 1980s (Kelle and Buchholtz 2015). Since the 1990s, however, the Mixed Methods movement has increasingly and pragmatically set itself the goal of overcoming struggles between purist representatives of both research paradigms. In doing so, MMR methodology is understood as an integrative approach in which the strengths of one research method can be used to balance the weaknesses of the other research method. Johnson and Onwuegbuzie (2004) describe the strengths as well as the weaknesses of the approach as follows (p. 21):

Strengths of Mixed Methods Research:

• Simultaneous generation and testing of theory

• Possibility of answering a broader extent of research questions

• Establishing a wealth of research designs with specific strengths and weaknesses

• Provision of validation strategies through convergent research results

• Generating insights that go beyond the use of single research methods

• Added value of additional knowledge for theory and practice.

Weaknesses of Mixed Methods Research:

• Single researchers can struggle to carry out both qualitative and quantitative research at the same time; it may require a research team

• The researcher has to be firm and confident in applying multiple research methods.

If one is aware of the specific strengths (and the weaknesses) of the MMR approach, the central characteristic of methodological reflection remains the purpose or the legitimation of the study (Schoonenboom and Johnson 2017). While the overarching goal of a MMR approach is always to deepen and strengthen the study’s conclusions, depending on the research question, specific reasons may be given for why it makes sense to combine or integrate different methods. Here the MMR community has already worked out methodological justifications. Among the best known is the classification by Greene et al. (1989), which is based on an analysis of Mixed Methods studies. They identify five purposes or reasons why researchers mix quantitative and qualitative methods (p. 259):

• 1. Triangulation, which means seeking for convergence, corroboration and correspondence of results from different methods, often used for validity purposes.

• 2. Complementarity, which means seeking for elaboration, enhancement, illustration or clarification of the results from one method with the results from the other method. Here the aim of the mixing is to get a more holistic understanding of the research object.

• 3. Development, which means seeking to use the results from one method to help develop or inform the other method, where development is broadly construed to include sampling and implementation, as well as measurement decisions.

• 4. Initiation, which means seeking the discovery of paradox, and contradictory findings, new perspectives of frameworks or the recasting of questions or results from one method with questions or results from the other method.

• 5. Expansion, which means seeking to extend the breadth and range of inquiry by using different methods for different inquiry components. This for example is particularly important in follow-up studies.

Meanwhile, a variety of other classifications of purposes for Mixed Methods Research exists (e.g., Bryman 2006). However, the key point in finding the purpose for using MMR methodology is to define the research question and then carefully consider what the purposes for mixing are. As Schoonenboom and Johnson (2017) state:

One can use mixed methods to examine different aspects of a single research question, or one can use separate but related qualitative and quantitative research questions. In all cases, the mixing of methods, methodologies, and/or paradigms will help answer the research questions and make improvements over a more basic study design. Fuller and richer information will be obtained in the mixed methods study. (pp. 111–112)

3 Special Features of MMR in Mathematics Education

In addition to the methodological justifications for MMR approaches, mathematics education research also provides justifications that can be given on the basis of disciplinary prerequisites. On the one hand, epistemological orientations within mathematics education research play a role, but on the other hand, there is also the complexity of mathematics didactic research objects. The research objects are in the field of tension of various disciplines such as mathematics, psychology, sociology, philosophy and educational science. They focus on different actors in different educational levels, and stand in a systemic relationship between theory and practice. Since no comprehensive discussion of the nature of mathematics educational research objects can be given here—at this point, reference should be made to the detailed considerations in Ernest (1998), Schoenfeld (2016), Steinbring (1998) or Wittmann (1995)—we are content with the following broad understanding of mathematics education research objects for the purpose of methodological reflection. Mathematics educational research objects in the broadest sense refer to the teaching and learning of mathematics in social and institutionalized educational contexts, the development, structuring and implementation of teaching materials as well as the reflection and selection of educational goals.

Starting from different research questions, we can distinguish various methodological approaches in mathematics education research, which according to the idea of multiple research paradigms (Johnson 2015) reflect a variety of different traditions of mathematics education research. For example, Bishop (1992) distinguishes three distinct traditions in research on mathematics education. Firstly, there is a pedagogue or educational tradition, whose aim is to bring about direct improvements in practice, while experiment and observation are key concepts of research. Methodologically, this research tradition is located in the field of observation and improvement of teaching activities, whereby as an example, experience from expert teachers will be generalized or participatory research methods will be applied. Bishop further distinguishes the tradition of the empirical scientist, whose goal is to analyze the practice based on empirical data and to generate explanatory models. Especially in this tradition, Bishop sees a strong methodological fixation that understands mathematics education as an analytical science and uses theory to explain scientific evidence. This approach resonates with classical qualitative and quantitative empirical research methods that are used. As a third research tradition, Bishop cites the scholastic-philosopher tradition, which sees mathematics education research as a rigorously argued theoretical reflection process. The epistemological aim from this perspective is to establish a theoretically argued position on theory-driven research questions and curricular development—strongly based on mathematical insights and logical rigor. Steinbring (2011) describes this as a research paradigm in which “the scientific elaboration of mathematical knowledge is the central and crucial means practiced for steering and optimizing mathematical instruction, learning and understanding processes” (p. 46). Research methods within this tradition—such as subject-matter didactical analysis (Sträßer 2013) or the French didactical engineering (Artigue 1988), aim at making relevant mathematical structures accessible to the learner’s mental development and adapting them to the requirements of teaching and learning without distorting mathematical standards (see, e.g., Kirsch 1977). Griesel (1974, p. 118) points to the logical mathematical rigor of the research approach: “The research methods of this area are identical to those of mathematics, so that outsiders have sometimes gained the impression that, here, mathematics (particularly elementary mathematics) and not mathematics education is being conducted” (as translated by Steinbring 2011, p. 45).

The argument for MMR, that the combination or integration of different research methods leads to a combination of strengths or compensation for weaknesses, can be transferred to the subject area of mathematics education research, but requires an extended understanding of research methods within the MMR methodology. The understanding of research methods in this case must be extended also to theoretically oriented subject-specific research methods. Depending on the research question, the extended understanding of methods also enables the combination or integration of empirical-social scientific research methods (quantitative/qualitative/both) and non-empirical subject-specific research methods. Within the social sciences, the term Mixed Methods is relatively limited to the exclusive combination of qualitative and quantitative design elements (Johnson et al. 2007; Johnson and Onwuegbuzie 2004; Cresswell 2016; Baur et al. 2017). However, there are also representatives within the methodological discussion of Mixed Methods who regard the rigid separation of qualitative and quantitative methods as outdated and argue pragmatically for Mixed Methods (Bazeley 2018), Mixed Methodologies (Szostak 2015; Christ 2010; Huysmans and de Bruyn 2013) or Merged Methods (Gobo 2016) as an entanglement of different methodologies of any origin. By way of example, such integration of subject-didactic and empirical methods takes place within the framework of mathematics educational design science (Prediger et al. 2015; Prediger and Zwetschler 2013; Nührenbörger et al. 2016). The research process is designed as a cyclical process based on the mathematical specification and structuring of learning content, design development (substantial learning environments), and subsequent empirically researched design experiments (Nührenbörger et al. 2016). The subject-specific methodology aims at a local theory formation based on the contextual research results (Bakker and van Eerde 2015, p. 437), which in turn can serve to further specify and structure learning content. For the sake of simplicity, the following sections deal explicitly with the combination and integration of qualitative and quantitative research methods, but in principle, the considerations can also be applied to the field of specific mathematics education research methods.

In addition, research objects in mathematics education are characterized by a high level of complexity, which is due to the interdisciplinary nature of the discipline (Wittmann 1995; Bartolini-Bussi and Bazzini 2003). Furthermore, the objects studied are characterized by different intra- or interpersonal relationships on micro (individual), meso (e.g., group, class, school) and macro level (e.g., country level or educational system). Not least, research objects are temporally involved in the dynamics of action, but also in educational and social change processes (Schoenfeld 2016; Steinbring 1998). MMR is particularly suited to the complexity of research objects in mathematics education. Taking up the complexity, it is crucial that the different research methods are either directed toward different aspects of the research object, with the aim of being able to make as far-reaching statements as possible, or that the different research methods analyze the same research object from different perspectives. The aim is then the mutual validation of the findings (Erzberger and Kelle 2003). The principle of triangulation as “taking different perspectives” (Flick et al. 2012) is particularly relevant in social science contexts and enables a more comprehensive understanding of complex research objects (Creswell and Plano Clark 2018). Further reasons for MMR in mathematics education are as follows (see Buchholtz, forthcoming):

• Researchers can focus more on the interdisciplinary nature of the research objects studied through different subject-specific methodological approaches.

• Mixed Methods Research offers the opportunity for local subject theory (Herbst and Chazan 2017; Erzberger and Kelle 2003).

• The methodological focus of different levels of education (individual and collective) helps to maximize the scope of research findings. The orientation towards both individual and collective statements then corresponds to the normative orientation of the research discipline (Schoonenboom and Johnson 2017).

• By combining hermeneutic methods with empirical methods, the uptake and updating of the understanding of historical and social processes can be targeted from a contemporary perspective.

• MMR is effective for examining and validating education, action and change processes.

Overall, by combining or integrating different research methods, a broader picture of the complexity of the research objects can be gained, since the analysis takes up different perspectives.

4 Choosing a Research Design

If the rationale for the choice of a MMR project is clear, then the question arises as to how the qualitative and quantitative methods are to be related to each other in the specific case. How the researcher can compose and arrange the components in the research process is then a question of the research design. According to Morse and Niehaus (2009) and Schoonenboom and Johnson (2017), we can identify three crucial aspects that influence the design of a Mixed Methods study, which I describe shortly. First, the “theoretical drive” (Morse and Niehaus 2009) of a research question indicates whether answering a research question is more qualitatively driven or quantitatively driven, or equally qualitatively and quantitatively driven. This theoretical drive can therefore determine which component of the research is dominant in the research process or if the two components are equally dominant.

Second, the timing of the components refers to how both components are dependent on each other in the context of the research process (Morse and Niehaus 2009). Usually, the simultaneity or dependency of the components can be described in two ways: whether the research design is concurrent (sometimes referred to as parallel) or sequential (Creswell and Plano Clark 2018; Morse and Niehaus 2009). In sequential research designs, the two components are conducted at different times and one component can build on the results of the other component. In concurrent research designs, both components are conducted simultaneously. Schoonenboom and Johnson (2017) explain:

In a commonly used mixed methods notation system (Morse 1991), the components are indicated as qual and quan (or QUAL and QUAN to emphasize primacy), respectively, for qualitative and quantitative research. As discussed […], plus (+) signs refer to concurrent implementation of components […] and arrows (→) refer to sequential implementation […] of components. Note that each research tradition receives an equal number of letters (four) in its abbreviation for equity. (pp. 108–109)

This means that for example a sequential MMR design where a dominant qualitative component builds up on the results of an earlier (not so dominant) quantitative component would be labeled as “quan → QUAL”, and so on. In the handbook of Creswell and Plano Clark (2018) three core-designs are described, that frame the basis of most (even more complex) MMR designs and reflect certain purposes of mixing. The sequential exploratory design (QUAL → QUAN), seeks to generalize findings of a qualitative study:

Building from the exploratory results, the researcher conducts a development phase by designing a quantitative feature based on the qualitative results. This feature may be the generation of new variables, the design of an instrument, the development of activities for an intervention, or a digital product, such as an app or website. Finally, in the third phase the investigator quantitatively tests the new feature. The researcher then interprets how the quantitative results build on the initial qualitative results or how the quantitative results provide a clear understanding because they are grounded in the initial qualitative perspectives of participants. (p. 65)

For example, a researcher could collect qualitative interview data about teachers’ perceptions on mathematical modelling and the conditions for and contexts in which they use modelling in the classroom. Taking the resulting categories as variables, the researcher could develop a quantitative survey instrument and then use it to assess the overall prevalence of these variables for a large number of teachers or for teachers in different countries. Secondly, the sequential explanatory design (QUAN → QUAL) follows a quantitative study with a qualitative study in order to explain or expand the results of the quantitative study. For example, a researcher could collect and analyze quantitative survey data from students to identify significant predictors of mathematics anxiety. Finding a surprising association between mathematics anxiety and high mathematical achievement, the researcher could conduct qualitative focus group interviews with high achieving students to explain this rather unexpected result (e.g., finding that these students feel high achievement pressure). Third, the convergent (or concurrent) design (QUAN + QUAL) seeks for the mutual complementation or validation of results (Cresswell and Plano Clark 2018):

The basic idea is to compare the two results with the intent of obtaining a more complete understanding of a problem, to validate one set of findings with the other, or to determine if participants respond in a similar way if they check quantitative predetermined scales and if they are asked open-ended qualitative questions. The two databases are essentially combined. (p. 64)

An example of a comparison approach to the convergent design would be if a researcher videotapes mathematics teachers during teaching and subsequently conducts reflective interviews on the teaching methods used in the lesson. The researcher can analyze the video data quantitatively with a rating instrument or an observation protocol for instructional quality (this analysis could also be qualitative, depending on the methodical focus of the instrument) and can evaluate the interviews qualitatively. Subsequently the two sets of results can be used to assess in what ways the objective results about instructional quality and the teachers’ views converge or diverge. Meanwhile, there is much literature on MMR designs (Cresswell and Plano Clark 2018; Teddlie and Tashakkori 2009; Greene 2007), including so-called “emergent designs” (Morse and Niehaus 2009; Brevik, forthcoming). In these designs, the different research methods are mixed situationally during the whole research process (for example, if it is not foreseeable at the beginning of the research project that additional methodological components will be required). There is a danger that because of the “design overload” (Kuckarts 2014) one can easily lose track of the various design types, especially as different authors often use different names for similar research designs. It is also criticized that design descriptions are often too rigid to describe variable and complex designs (Guest 2013). One way out here is not to understand the multitude of design typologies as binding templates for one’s own research project, but as a framework for orientation in finding a research design.

The third aspect concerning the research design is the “point of interface” (Morse and Niehaus 2009) or the “point of integration” (Schoonenboom and Johnson 2017), which refers to the stage at which the researcher combines or integrates the results from both components and the actual mixing occurs. In the case of concurrent research designs, this can happen throughout all stages of the research process (Schoonenboom and Johnson 2017), but the common point at which the mixing takes place is during the analysis of the data gathered by both components or when the data is interpreted. More details about how and when mixing takes place can be found in the writings of Bazeley (2018) or Onwuegbuzie and Teddlie (2003). In case of sequential research designs it is often the case that one component informs the following component before it is conducted, which is a form of interface or integration too. Nevertheless, a Mixed Methods study is more than the sum of its parts (Bryman 2007), so even in sequential designs, an integration should take place when analyzing or interpreting the data or at least when the research results are written up.

5 Mixed Data Analysis: Integrating Qualitative and Quantitative Findings—Joint Displays

Once data are collected in the research process, the data analysis process begins. In the classical case, qualitatively collected data such as, for example, open-ended interviews or questionnaire responses, observations and field notes or journals are evaluated with the help of qualitative research methods. Quantitative data such as measurements based on standardized tests, rating scales or self-reports are preferably evaluated using quantitative research methods. On the one hand, this can happen in parallel, and then qualitative and quantitative results are interrelated in the interpretation of the findings (triangulation of results) in order to conclude meta-inferences. This integration requires reducing, transforming, comparing, or correlating the results (Caracelli and Greene 1993; Bazeley 2012; Onwuegbuzie and Teddlie 2003; Onwuegbuzie and Hitchcock 2015). The guiding question of this integration is: “To what extent do the quantitative and qualitative results converge or diverge?” The results should be studied according to similarities and differences, or to what extent the results of one research component can be explained using the results of the other component. Whether the different methods were used with the proviso of investigating the same research object (e.g., for validation purposes) or different aspects of the research object (for a comprehensive picture), the results of the integration may vary (Kelle and Buchholtz 2015):

This differentiation is not a mere play of words: only methods which refer to the same phenomena can yield results which may be used for mutual validation of methods. Different results would indicate validity problems here; but if separate aspects of the investigated phenomenon or even separate phenomena were examined with different methods we would expect different (but certainly not contradictory) results. […] [E]ach of the following four outcomes can arise (cf. Erzberger and Prein 1997 […]):

• 1. qualitative and quantitative results converge,

• 2. qualitative and quantitative results relate to different objects or phenomena, but are complementary to each other and thus can be used to supplement each other,

• 3. qualitative and quantitative results are divergent or contradictory,

• 4. qualitative and quantitative results refer to unrelated phenomena.

This makes clear that both types of triangulation are applicable and can make sense within a mixed methods design: triangulation as validation may lead to convergent qualitative and quantitative findings or it may result in divergent findings which point to validity problems; triangulation as investigating different aspects of the research subject may yield complementary results (if applied successfully) or it may render unrelated results (if this triangulation strategy fails). (p. 332ff.)

In a sequential analysis in the explanatory sequential design, on the other hand, the integration mostly happens when the results of the first analysis influence the second analysis with regard to certain emphases. First of all, findings must be identified in the quantitative results that require further explanation and to which the results of the qualitative study can be referred. These can be results on content-related topics, but also particularly interesting or extreme cases that will be selected as a specific sample in the following work. In the sequential analysis of the exploratory sequential design, a selection of qualitative results must be made, which are aiming for generalization. Based on these results, a contextually appropriate feature (e.g., a new instrument) is developed which is piloted and tested on a larger sample. Already in the data analysis of one research component (qual. or quan.), however, it is possible to work multi-methodically. For example, it is possible to quantify qualitative data through the use of quantitative methods or qualitize quantitative data through the application of qualitative research methods (Tashakkori and Teddlie 1998).

Schoonenboom and Johnson (2017) recommend the use of a so-called “joint display” (Guetterman et al. 2015; Cresswell and Plano Clark 2018) to facilitate the process of integration. Following Johnson and Christensen (2017), a joint display “is a matrix juxtaposing qualitative and quantitative results for cases, research questions, variables, outcomes, times, locations, or any other dimension of interest” (p. 593). The table representation condenses the results of the mixed analysis (but cannot replace it). It should be consistent with the research design or research questions and make clear where the integration of each component occurred. In any case, the joint display should therefore contain information about the results of the individual data analyses and provide the results of the integration of the qualitative and quantitative results (see Fig. 6.1).

Fig. 6.1

Joint display

The possibilities of designing a joint display are manifold, as described by Johnson and Christensen (2017, p. 595), including for example that rows equal cases, and columns equal time-ordered outcomes (time 1, time 2, time 3, time 4). Rows could also equal qualitative themes, while columns equal relevant quantitative statistical results such as on a 5-point scale (themes-by-statistics display). Cells could also include quotes and frequency counts (Creswell 2015).

6 Methodological Challenges for MMR

The choice of a MMR approach should be well considered and based on requirements of the specific research question. Throughout the research process, MMR designs may present methodological difficulties and challenges on which researchers should constantly reflect.

Frequently, a comprehensive methodological justification for Mixed Methods is already missing when determining the object of the research and the research questions (for example, if different data sources in an existing research project simply arise). In this case, there is no theoretical anchoring of the research question, and a selection of methods often takes place prior to the research question. With regard to the purpose of the study, it is then not even clear whether the combination or integration of different methods is intended to achieve mutual validation or complementarity of the results. Further difficulties may also arise in the lack of coordination between the selected research method and the empirical field.

During the data collection and the data analysis, it is imperative for a MMR study to integrate the results of the different components of the study. This integration and the interpretation of the integrated results must add value to the individual interpretation of qualitative and quantitative findings. There is a danger here that a pure parallel performance of the research methods with subsequent interpretations will be undertaken. Even in sequential designs, integration is an integral part of the research process, which must be done explicitly. The mixed data analysis does not always lead to complementary or convergent findings. In the case of a targeted combination of methods with complementary perspectives on the research object, it can happen that results can be completely incoherent and that the qualitative and quantitative results cannot be correlated at all. In the method integration with the aim of mutual validation of research results, however, it is not ensured that both methods also cover the same research object, a mandatory prerequisite for mutual validation. Results can converge here, but also diverge or even contradict each other. Often it becomes clear in the course of the research process that qualitative and quantitative components cover different aspects so that only complementary results can be achieved. Especially for early career researchers, MMR projects pose great challenges in terms of resource and time management, which is why, when working in an unsecured position or on scholarships, a reduced choice of method should be considered. The central question is: ‘Is there a recognizable added value of Mixed Methods compared to a reduced choice of method?’

In presenting the results, the complexity of MMR designs can lead to difficulties in accurately identifying qualitative and quantitative proportions of studies. Care should also be taken to ensure that the research question and the complete implementation of the research project are coordinated and that this is also clear from the description of the study. Bryman (2008) generally observes a “mismatch between the rationale for the combined use of qt and ql research and how it is used in practice” (p. 94). If the process of integration is described, it is aggravating that there is no clear terminology of “mixing”, so that the integration processes must be described as precisely as possible. Here, the recent editions of the handbooks with their descriptions of integration processes now provide support (Cresswell and Plano Clark 2018). In the end, the presentation of methodology and results of Mixed Methods studies will be of great importance. Finally, qualitative methodological procedures and qualitative results, quantitative methodological procedures and quantitative results as well as the procedure of integration and its results will have to be described. As a result, when one or more research papers are written instead of a monograph, space problems often arise because scientific journals set limits on the length of articles. A way out may be to either publish qualitative and quantitative results in separate articles or to submit the research article to a Mixed Methods journal for specifically methodologically-based studies, as these generally offer more generous space constraints.

7 Summary: How to Conduct a Mixed Methods Study

This chapter provides insight into the most important aspects to consider when deciding on a MMR approach. Reference was made to the crucial aspects of planning a Mixed Methods study in mathematics education as well as to the common terminology of the Mixed Methods community. To summarize, here is a step-by-step presentation of the sequence of important questions that should be answered in the planning and implementation of MMR studies (adapted from Schreier 2015).

• Step 1: What is the research object?

  • Does the research object have various constituent aspects, such as actors at different levels (e.g. teachers and students, or individual and institutional conditions) or different disciplinary characteristics?

  • Does the research object have a theoretical and/or practical orientation? What forms of complexity does the research object exhibit?

  • What is the relationship between the object of research and mathematics?

  • What are the special features of the research (e.g., videos, conceptual training programs, teaching materials)?

  • Does the investigation of the research object require a constructive or analytical orientation, or both?

  • Which aspects of the research object should be investigated?

• Step 2: Why mixed methods? Identifying the purpose of the study

  • Should the study bring together or compare different perspectives on different aspects of the research object?

  • Should the study validate different perspectives on certain aspects of the research object?

  • Should existing research results be extended?

  • Should statements about individual cases as well as general statements be made?

  • Should the research object be explored qualitatively and the results reviewed for their generalizability? Should an object-related theory be created and tested?

  • Should a questionnaire study be prepared and validated? Should a hypothesis be tested using qualitative data? Should causal factors be identified to better understand the underlying mechanisms?

• Step 3: What is the exact research question?

  • Which integrated research question can be formulated about the research object?

  • What could be a qualitative sub-question of the integrated research question?

  • What could be a quantitative sub-question of the integrated research question?

  • For what aspects of the research questions are qualitative or quantitative methods needed? Which methods are suitable for answering the partial research questions?

  • At which points can an integration take place?

  • What can/should the integration look like?

• Step 4: Which research design fits the question and the purpose of the study?

  • Is a sequential explanatory design QUAL → QUAN suitable?

  • Is a sequential exploratory design QUAN → QUAL suitable?

  • Which parallel design is suitable, integration or combination QUAL + QUAN?

  • Is a complex design needed (for example, longitudinal design)? Is an intervention design needed (Cresswell 2015; Sandelowski 1996)?

  • Is the Mixed Methods design developing during the entire research process?

  • Is a design-based research design needed (with alternating cycles of constructive design development and analytical empirical validation)?

  • What is the position of the individual components in the research design?

• Step 5: How is the research design specified?

  • How many components does the design have? How many components are mixed?

  • What is the priority of the different components?

  • How do the components relate to each other?

  • What is the theoretical orientation of the research design? Are local theories used (for example, to study particular mathematical content) or is the research design completely theoretically driven (for example, by feminist theories or theories of intercultural learning)?

  • At what point in the design does integration happen? Is it in the interpretation of results or across the entire research process?

• Step 6: How should the mixing happen?

  • Are qualitative data and/or quantitative data generated in the study?

  • Are the data collected and analzyed independently or is integration already taking place at the level of data collection and analysis?

  • Are qualitative data quantified? Are quantitative data qualified?

  • Which sampling strategies can be used?

  • To what extent do the methods used inform each other?

  • Which variables/cases are the focus of mixing?

There are many different aspects to consider when choosing a MMR approach, and a Mixed Methods study is certainly less easy to plan for early career researchers than a mono-methodical study. Frequently, the way in which the results of the different methods relate to each other is not yet foreseeable at the beginning of the research project. Nonetheless, a MMR project provides unique insights into the object of research that cannot be achieved in depth with the application of single methods. By validating or complementing research results, MMR provides an in-depth knowledge of research results. The methodological discussion within the Mixed Methods community continues to develop this methodology, and interestingly enough, mathematics education has proven to be a common field of application of mixed-methodology studies. In August 2018, for example, at the third International Conference on Mixed Methods (MMIRA) in Vienna, a whole section was devoted to Mixed Methods studies in mathematics education.