Background

Genetically modified (GM) crops have been cultivated around the world since 1996. In 2017, the area devoted to GM crops reached 189.8 million ha [1]. More than half of this production involved crops that are engineered to express one or more insecticidal proteins (Cry or Vip proteins) from the bacterium Bacillus thuringiensis (Bt) for controlling lepidopteran and/or coleopteran pests. Before their commercial release, GM crops undergo an environmental risk assessment (ERA) to ensure that they do not cause unacceptable detrimental effects to the environment. One area of concern addressed in this risk assessment is the potential adverse impact on non-target arthropods (NTAs) and the ecosystem services they provide, including biological pest control by predators and parasitoids [2].

Ladybird beetles (Coleoptera: Coccinellidae) are important NTAs: they are abundant predatory species in many agroecosystems  [3, 4], are valued for their biological pest control functions [5] and have been recommended as surrogate species for studies to support the ERA of Bt crops  [6,7,8]. Most ladybirds feed preferably on aphids, which do not ingest significant amounts of Bt proteins  [9]. However, they also consume other available prey and, occasionally, pollen when prey is scarce  [5], and therefore they can be exposed to Bt proteins under field conditions  [7, 10]. Some species, such as Stethorus punctillum, are specialist predators of spider mites, which are known to contain high amounts of Bt protein when feeding on Bt plants, and are thus highly exposed to Bt proteins [11, 12]. Figure 1 describes a conceptual model on how insecticidal Bt proteins could result in harm to ladybird beetles when feeding directly on plant parts or indirectly when preying on herbivores that have ingested the Bt protein.

Fig. 1
figure 1

Interactions of Bt proteins produced by genetically modified (GM) plants with (non-target) ladybird beetles including the potential pathway to environmental harm, ecosystem services and environmental protection goals. The focus of the systematic review will be on the hazard of Bt proteins on ladybird beetles

Full size image

Insecticidal Bt proteins expressed in field-grown GM crops are known to be highly specific against target pests, and the scientific evidence accumulated with the assessment of these plants over the last three decades has demonstrated their environmental safety towards NTAs  [13]. However, some recent laboratory studies reported putative toxic effects of Bt proteins on some ladybird species  [14,15,16]. Although such studies have been criticised because of methodological shortcomings or inconsistencies  [17,18,19,20], they cast doubt on the insecticidal spectrum of activity of some Bt proteins and pointed to uncertainties that may be resolved by synthesising all existing evidence on this controversial topic.

Systematic reviews are evidence synthesis approaches that can be used to support the risk assessment of GM crops  [21], especially when the available evidence shows different, or even, contradicting results. A common approach for synthesising and analysing the data collected from multiple individual studies included in a systematic review is to conduct a meta-analysis. Meta-analyses combine data from multiple studies, increasing the statistical power and overcoming the low level of replication of individual studies  [22,23,24]. Although several meta-analyses on the effects of GM insect-resistant crops on different groups of NTAs have been published to date  [25,26,27,28,29,30,31,32], none of them has focused particularly on ladybird beetles.

We intend to perform a systematic review on the effects of Bt proteins expressed in GM insect-resistant crops on ladybird beetles. Since the studies reporting putative toxic effects of Bt proteins on some ladybird species were conducted in the laboratory, the systematic review will focus on studies performed under controlled environmental conditions. The methodological quality of each relevant study will be assessed. If the number of datasets generated with comparable experimental setup is sufficient, statistical meta-analyses will be conducted.

Objective of the review

The objective of the review is to systematically search, critically appraise and synthesise all scientific literature on the effects of Bt proteins, i.e. Cry and Vip proteins, expressed in GM insect-resistant crops on ladybirds.

Review question

We plan to answer the following review question: Are ladybird beetles (Coleoptera: Coccinellidae) affected by Bt proteins expressed in GM insect-resistant crops?

Components of the review question

The review question has a PECO structure with the key elements consisting of:

  • Population All species belonging to the coleopteran family Coccinellidae, commonly known as ladybirds, ladybird beetles, or ladybugs;

  • Exposure Cry and Vip proteins (Bt proteins) expressed in GM insect-resistant crops for which there has been a regulatory approval for cultivation. This includes the Bt proteins as expressed in the GM plant and microbially-produced and purified Bt proteins (hereafter referred as to purified proteins);

  • Comparator Closely-related (non-Bt) plants for those studies using Bt plant material/tissue (e.g. leaves, pollen), or a negative control for those studies using purified Bt proteins (e.g. untreated artificial diet);

  • Outcome Effects of the Bt protein on life-history traits of the ladybird beetle (e.g. development, survival, reproduction) measured under controlled environmental conditions (i.e. laboratory and glasshouse studies).

Methods

Searching for articles

Search terms

The search will be structured to reflect the population and exposure elements of the review question because both elements can be clearly defined and translated into broad search terms. On the contrary, the comparator and outcome components of the review question will remain open and not be present in the search string to avoid narrowing the search too much and thus missing relevant articles.

To enhance the sensitivity of the search, a wide range of search terms will be used (Table 1). Search terms will cover possible synonyms, related terms, acronyms, spelling variants, lay and scientific terminology and translation issues, and specific search functions, such as truncation, wildcards, proximity operators and quotation marks for multi-word terms. The Thesaurus search tools in Biosis Citation Index (Clarivate Analytics, Philadelphia, USA), CAB Abstracts (CABI, Wallingford, UK) and Zoological Record (Clarivate Analytics) were used to ensure that terms from the controlled vocabulary are included (e.g. coccinellidae, ladybirds, ladybugs). Only English search terms will be used.

Table 1 Search terms corresponding to the population and exposure elements of the review question
Full size table

Search strings

The search strings will be composed of two parts, corresponding to the population and exposure elements of the review question, combined with the Boolean operator AND. Within each set, search terms will be linked with the Boolean operator OR (Tables 2, 3). This string will retrieve articles that contain at least one search term from each element.

Table 2 Search strings to be used in the electronic bibliographic databases hosted in the Web of Science platform
Full size table
Table 3 Search strings to be used in Agricola, AGRIS, Open Access Theses and Dissertations and Google Scholar
Full size table

The search strategy will be adapted to the different information sources, i.e. Web of Science (WoS)  [33], Agricola  [34], Open Access Theses and Dissertations  [35], and Google Scholar (https://www.scholar.google.com). The search strings that will be used in the electronic bibliographic databases hosted in Web of Science and Agricola, AGRIS, Open Access Theses and Dissertations and Google Scholar are presented in Tables 2 and 3, respectively. Searches in Google Scholar will be performed using the open source software Publish or Perish (https://harzing.com/resources/publish-or-perish). We selected Publish or Perish because it eases the process to export the results retrieved in Google Scholar and import them into EndNote. Since nested Boolean operators do not work in Publish or Perish, several searches will be performed using the all the words command. The first 200 results from each search will be considered, as suggested by Haddaway et al.  [36].

Languages

The searches are expected to retrieve articles written in English and articles written in other languages with at least a title, abstract/summary or keywords in English. Identified articles written in languages other than English will be translated as needed for further assessment of relevance and validity.

Time period

The search will be limited to those articles published from 1990 onwards. Since the commercial cultivation of GM crops started in 1996, the likelihood that relevant articles were published before 1990 is considered very low.

Estimating the comprehensiveness of the search

We performed a scoping exercise on 29 September 2017 with the aim of estimating the comprehensiveness of the search and validating the selected search string with the electronic bibliographic databases hosted in the WoS platform. For this purpose, 18 already-known relevant articles were selected: twelve articles that were listed in two reviews [20, 37] and six articles (published between 2014 and 2017) that had been previously identified by citation alert (see Additional file 1).

Individual searches were performed in five of the electronic databases that will be searched in the systematic review (i.e. WoS Core Collection, CAB Abstracts, Chinese Science Citation Database, BIOSIS Citation Index and Current Contents Connect), using the search string provided in Table 2. All retrieved lists of records were checked for the presence of the 18 relevant articles. The results of the scoping exercise are provided in the Additional file 2. WoS Core Collection and CAB Abstracts delivered the best results and were able to retrieve 16 articles each. All databases combined returned all 18 relevant articles. Therefore, the selected search string was considered valid. None of the 18 relevant articles were retrieved by the Chinese Science Citation Database. This was expected since none of them were published in journals indexed in the database. However, the Chinese Science Citation Database will still be searched for the systematic review since it might contain some other relevant articles which are not indexed in the other databases.

Electronic bibliographic databases

A broad range of relevant multi-disciplinary and subject-specific electronic bibliographic databases will be searched:

  • Web of Science (WoS) Core Collection (Clarivate Analytics). Contains most peer-reviewed scientific articles in English language. The following databases will be included: Science Citation Index Expanded, Conference Proceedings Citation Index—Science, Book Citation Index—Science, and Emerging Sources Citation Index;

  • BIOSIS Citation Index (Clarivate Analytics). Comprehensive reference database for life sciences;

  • CABI: CAB Abstracts (CABI). Comprehensive database that also includes more local and non-English articles;

  • Chinese Science Citation Database (Chinese Academy of Sciences). It provides scientific articles published in Chinese language (title and abstract in English);

  • Current Contents Connect (Clarivate Analytics). It contains peer-reviewed scientific articles in English language from different disciplines;

  • Data Citation Index (Clarivate Analytics). It contains data sets from a wide range of data repositories and connects them with the scientific literature to track data citation;

  • SciELO Citation Index (Sao Paulo Research Foundation for the cooperative publishing of open access journals on the internet). It provides access to scholarly literature in natural sciences, social sciences, arts and humanities published in leading open access journals from Latin American, Portugal, Spain and South Africa;

  • Zoological Records (Clarivate Analytics). It covers all aspects of animal research;

  • Agricola. It contains bibliographic records of materials acquired by the National Agricultural Library and cooperating institutions in agricultural and related sciences;

  • AGRIS (Food and Agriculture Organization of the United Nations). It facilitates access to articles, monographs, book chapters, and grey literature in the area of agriculture and related sciences;

  • Open Access Theses and Dissertations. Contains a collection of open access graduate theses and dissertations published around the world.

All databases, except Agricola, AGRIS, and Open Access Theses and Dissertations, are hosted in WoS.

Search engines

Google Scholar, which is a web search engine that indexes peer-reviewed journals and other scholarly literature, such as books, conference papers, theses and dissertations, and technical reports, will be searched.

Organisational websites

The websites of specialist organisations (regulatory agencies, industry organisations and civil society organisations) and web-based databases containing information on environmental effects of GMOs listed below will be searched.

Supplementary searches

Supplementary searches will be conducted to identify potential additional relevant articles not retrieved by other methods.

Citation searching

Citation searching will be conducted with those articles meeting the eligibility criteria and categorised as relevant after the screening process. All references citing each relevant article will be identified in WoS using the cited reference search tool (all eight WoS-hosted databases cited above will be considered) and screened.

Checking reference lists

The reference lists of those articles meeting the eligibility criteria and therefore categorised as relevant after the screening process will be checked.

Electronic-searching of key journals

The ten journals that yielded the highest number of records when performing the scoping exercise on 29 September 2017 will be checked electronically to identify very recent articles that have not yet been indexed in the electronic bibliographic databases. The journals are, in alphabetical order, Biological Control, Entomologia Experimentalis et Applicata, Environmental Entomology, IOBC/WPRS Bulletin, Journal of Applied Entomology, Journal of Economic Entomology, Pest Management Science, Plos One, Scientific Reports and Transgenic Research. The searches will be limited to articles published within the 12 months before the date of the search of the systematic review, and to articles published ahead of print.

Search update

A search update will be conducted if searches were performed more than 2 years prior to review completion. Search updates will be thoroughly documented and reported as described below.

Documenting and reporting the search process

The search will be fully documented and reported following the recommendations outlined in the European Food Safety Authority (EFSA) guidance on systematic review methodology and food/feed safety risk assessment  [23], the EFSA explanatory note on literature searching  [38] and the ROSES RepOrting standards for Systematic Evidence Syntheses [39]. The ROSES form is included as Additional file 3. The following details will be recorded and reported in the systematic review:

  • List of all information sources (e.g. electronic bibliographic databases, organisational websites, hand searched journals) searched;

  • The full search strategy for each information source (copied and pasted exactly as run) and details of how each information source was searched will be recorded and provided in the systematic review;

  • The date on which the search was conducted;

  • The date of the most recent update of the database that was searched;

  • The date span of the search;

  • Any limits applied to the search (e.g. article types, dates);

  • The journal name, the journal URL (internet address) or publisher; the dates, volumes and issues searched; the method of searching, e.g. scanning tables of contents for each issue, or using a search engine; the search terms used (if any) (for electronic-searching key journals);

  • The number of articles identified through each information source and the final number of articles remaining after removing duplicates (automatically and manually). This information will be part of the flow diagram that includes information on the screening and critical appraisal and synthesis steps [39];

  • Any deviations from the protocol in the search strategy, and their impact in the systematic review.

Article screening and study eligibility criteria

Screening process

The articles retrieved by the literature searches will be transferred to EndNoteX9 (Clarivate Analytics). Retrieved articles will be tagged with the database provider and stored in one single EndNote file. Then, the articles will be combined into one file that will be exported to the systematic review software Distiller SR (Evidence Partners, Ottawa, Canada). Duplicates will be eliminated automatically. Remaining duplicates, if any, will be removed manually.

The relevance of the resulting articles after deduplication will be assessed manually following a two-step approach and applying the eligibility criteria listed in Table 4. In the first step, title and abstract (or summary) will be screened, whilst in the second step, the full text of all remaining articles will be assessed. At this second step, each excluded article will be documented with the reason for exclusion. Any articles for which their relevance still remains unclear after assessing the full text will be discussed by the review team to reach a consensus decision.

Table 4 Eligibility criteria to assess the relevance of retrieved articles at title/abstract and full-text screening stage
Full size table
Screening consistency checking

Articles will be screened independently by at least two reviewers at the title, abstract, and full text level. To minimise any influence on eligibility decisions  [40], reviewers will not be allowed to assess the relevance of any article they have co-authored. Any articles considered relevant by one reviewer and non-relevant by the second one will be documented and discussed to reach a consensus decision. If no consensus is reached, the opinion of another member of the review team will be sought.

Eligibility criteria

From the articles retrieved by the different sources of scientific literature described above, only those articles fulfilling all the eligibility criteria listed in Table 4 will be considered relevant for the systematic review.

Pilot testing eligibility criteria

The validity of the eligibility criteria was pilot-tested on 6 June 2018, following the recommendations of Frampton et al.  [40]. For this purpose, a sample of 20 articles identified in the scoping exercise performed on 29 September 2017 was selected (see Additional file 4). This sample size is bigger than the size suggested by Higgins and Green  [41] (i.e. 10–12 articles). The sample included articles which were thought by one reviewer to be definitely relevant (10 articles), definitely irrelevant (seven articles), and doubtful (three articles). Articles were selected randomly but ensuring that all categories were represented. A second member of the review team assessed all articles independently. Both reviewers agreed on the relevance of all 20 articles assessed. Therefore, the defined eligibility criteria were considered valid.

Documenting and reporting article screening and eligibility criteria

The relevance assessment will be fully documented and reported following the recommendations in Frampton et al. [40], the ROSES RepOrting standards for Systematic Evidence Syntheses  [39], and the Collaboration for Environmental Evidence Guidelines and Standards for Evidence Synthesis in Environmental Management  [42].

The following details will be recorded and reported in the systematic review:

  • Summary of the finally-agreed eligibility criteria, including the instructions given to the reviewers;

  • A flow diagram showing the results of the screening: number of articles screened, number of articles excluded upon screening titles and abstracts (for both reviewers); number of articles excluded at full-text screening; number of articles used in the systematic review;

  • Number of reviewers involved; whether screening decisions were independent; expertise of the reviewers;

  • Results of the assessment of reviewer agreement, and how disagreements were resolved;

  • A list of articles that were excluded at full-text (with reasons for exclusion);

  • A list of articles which had unclear eligibility status after completion of full-text screening (with explanation why they could not be classified).

  • A list of articles that could not be obtained for full-text screening;

  • The final list of articles eligible for the systematic review;

  • Any deviations from the protocol in the eligibility criteria or the screening process, and their impact in the systematic review.

Study validity assessment

Study validity criteria and process

Relevant studies will be assessed for both external validity (the degree to which the studies are appropriate or applicable for answering the review question) and internal validity (risk of bias). The validity of each study fulfilling the eligibility criteria will be evaluated against a set of criteria. The study validity criteria are divided into three groups according to whether they pertain to: (1) test substance; (2) test organism, and (3) study design. A list of potential criteria that will be considered and the conditions that are needed to fulfil each one of them are provided in Table 5. Further details about the scientific rationale followed to underpin the selection of the proposed criteria can be retrieved from Romeis et al. [20, 43].

Table 5 Proposed criteria used to assess the validity of relevant studies
Full size table

For each study, all quality criteria will be assessed individually and categorised (e.g. fulfilled/low risk of bias; partially fulfilled/moderate risk of bias; not fulfilled/high risk of bias; not relevant; not assignable, in case that information is not provided or reported). An overall validity descriptor will be assigned to each study based on the results of the independent assessment of all criteria by two reviewers: (1) high validity/low risk of bias; (2) medium validity/medium risk of bias; (3) low validity/high risk of bias; or (4) not assignable.

In case that insufficient evidence is provided in the study to assess each of the validity criteria, corresponding authors will be contacted and asked for clarifications.

Details about the criteria assessed to determine the validity of the studies, the rationale followed to categorise each of criteria as well as the results of the assessment of each study will be provided in the systematic review. Information from the validity assessment of the eligible studies will be used in data synthesis; sensitivity analyses will be performed by comparing results with and without exclusion of studies with low and medium validity.

Consistency checking

A similar quality assurance system to that established for the article screening process will be put in place. All studies will be assessed independently by two members of the review team. Reviewers will not be allowed to assess the study validity of studies they have co-authored. Inconsistencies or uncertainties between reviewers will be discussed to reach a consensus decision. If no consensus is reached, the opinion of another member of the review team will be sought.

Documenting and reporting study quality

The rationale behind the decision for each validity criterium and the overall external and internal validity of each study will be documented in the systematic review, as well as any deviation from the protocol.

Data coding and extraction strategy

The variables that will be extracted from relevant studies and recorded in an MS-Excel spreadsheet are listed in Table 6. The extracted data will be available as an additional file of the systematic review.

Table 6 List of variables to be extracted from each relevant study
Full size table

A random sample of entered data (at least 20% of the remaining studies after relevance assessment) will be checked by a second member of the review team.

The corresponding authors of those studies in which relevant data are not properly reported might be contacted; alternatively, data figures will be scanned using the open source software Plot Digitizer (http://plotdigitizer.sourceforge.net) and means and measures of within treatment variance will be estimated.

Potential effect modifiers and reasons for heterogeneity

Several potential effect modifiers and sources of heterogeneity were identified:

  • Test substance

    • Protein type: Cry/Vip protein

    • Protein specificity: coleopteran-/lepidopteran-specific

    • Purified protein/plant-expressed protein

  • Test organism

    • Ladybird beetle species

    • Stage tested: larvae, pupae, adults

  • Study type/design

    • Study type: laboratory/glasshouse; direct feeding study/indirect (prey-mediated) feeding study; artificial diet/plant study

    • Measurement endpoint

    • Exposure duration

Those effect modifiers will be part of the data extracted from the studies and recorded in the MS Excel spreadsheet. Additional effect modifiers may be added during the course of the systematic review. The influence of the potential effect modifiers causing heterogeneity will be investigated.

Data synthesis and presentation

The data extracted from all identified studies will be synthesised in a narrative report and in different tables, including information on the ladybird species tested, type of study (e.g. direct/indirect feeding assay with plant material/purified protein in artificial diet), characterisation of the test substance, exposure to the test substance, endpoints analysed, negative and positive controls and results of the statistical analysis. The data extracted will describe the body of evidence and will also serve to identify potential knowledge/data gaps.

If a sufficient number of datasets generated with comparable experimental setup is available, statistical meta-analysis will be conducted using the effect size estimator Hedge’s d for continuous variables (e.g., developmental time, pupal/adult weight) and odds ratio and/or risk ratio for dichotomous variables (e.g., survival/mortality).

Separate analyses will be conducted for different study types, type of Bt protein, spectrum of activity of the Bt protein, measurement endpoint, etc.

Sensitivity analysis will be conducted to explore the influence of individual studies, funding sources or affiliation of investigators for the overall result and study validity (i.e., by comparing results with and without exclusion of studies with low and medium validity), etc.

Publication bias will be estimated; effect sizes will be compared for articles with different funding and author affiliation types (see sensitivity analyses described above). Systematic differences would indicate a publication bias depending on funding source of a study or authors affiliation.