2.1 Our Mission
The German Mouse Clinic (GMC) is part of the Helmholtz Zentrum München (HMGU) and located in Munich, Germany. Understanding gene function in general, and furthermore the causation, etiology, and factors for the onset of genetic diseases, is the driving force of the GMC. Established in 2001 as a high-throughput phenotyping platform for the scientific community, we have set up different phenotyping pipelines covering various organ systems and disease areas (Gailus-Durner et al. 2005, 2009; Fuchs et al. 2017; www.mouseclinic.de).
Standardized phenotyping is designated to the areas of behavior, bone and cartilage development, neurology, clinical chemistry, eye development, immunology, allergy, steroid metabolism, energy metabolism, lung function, vision and pain perception, molecular phenotyping, cardiovascular analyses, and pathology. In a comprehensive primary screen, we can analyze 700+ parameters per mouse and collect 400+ additional metadata (Maier et al. 2015). Our collaboration partners have to provide a cohort of age-matched mutant animals from both sexes and the corresponding wildtype littermates. Expectations for high-quality and scientifically valid outcomes are very high, due to widespread financial and time resources spent collectively. Therefore, study design, performance of experiments, material and equipment, as well as training of personnel have to be well coordinated, harmonized, and advanced.
2.2 Our Team and Main Stakeholders
Our GMC team consists of scientists with expertise in a specific disease area (e.g., energy metabolism or pathology), technicians performing the phenotypic analysis, animal caretakers, computer scientists, a management team, and the director. In total, we have approximately 50 team members with a third of the staff working in the animal facility barrier, which represents an additional challenge to maintaining efficient communication crucial for successful output and teamwork.
Collaboration partners who send us mouse models for phenotypic analysis are scientists or clinicians from groups at the HMGU and many different academic institutions, universities, hospitals, or industry from Germany, Europe, and other countries/continents like the United States, Australia, and Asia. Since the beginning of the GMC, we have analyzed mice from 170 collaboration partners/laboratories from 20 different countries.
The GMC is a partner in a number of consortia like the International Mouse Phenotyping Consortium (IMPC, a consortium of so-called mouse clinics all over the world, http://www.mousephenotype.org, Brown and Moore 2012; Brown et al. 2018) and INFRAFRONTIER (https://www.infrafrontier.eu; Raess et al. 2016). Together with the European mouse clinics, the GMC has developed standardized phenotyping protocols (standard operating procedures (SOPs)) in the European EUMODIC program (Hrabe de Angelis et al. 2015). These SOPs have been further developed for the use in the IMPC (https://www.mousephenotype.org/impress). Starting on the European (EUMODIC) and expanding to the international level (IMPC), we had to harmonize equipment, handling of animals, and documentation of laboratories to enable the compatibility of phenotyping data from different facilities around the globe.
2.3 Needs Concerning Quality Management and Why ISO 9001
There are logistical, experimental, and analytical challenges of systemic, large-scale mouse phenotyping to ensure high-quality phenotyping data. The mutant mice we receive for analysis are generated by different technologies (e.g., gene editing, knockout, knock-in) and on various genetic backgrounds. As a unique feature, we import age-matched cohorts of mice from other animal facilities for the phenotyping screening.
Therefore, several processes need to be quality-controlled in many different areas, including project management (request procedure, reporting), legal requirements (collaboration agreement, animal welfare, and gene technology regulations), scientific processes (scientific question, study design, choice of phenotyping pipeline, data analysis, reproducibility), capacity management (many parallel projects), and knowledge and information management (sustain and transfer expertise, internal and external communication). In order to cover this complex management situation, we decided to implement a QMS within the GMC.
Being a partner in an international consortium allows benchmarking with centers in similar environments. Due to this circumstance, our rationale for using the ISO 9001 standard was based on the following: (1) two other benchmark institutions with similar scope had already adopted ISO 9001-based QMSs, and (2) our products, consisting of research data, inference, and publications, are not regulated like data of, e.g., safety, toxicity, or pharmacokinetic studies and clinical trial data used for a new drug application to the regulatory authorities. Therefore, a process-based system seemed to be most suitable for our needs.
An ISO 9001-based QMS is very general and serves as a framework to increase quality in many aspects. In the GMC, we have implemented measures not only to improve our processes but also to directly improve instruments and increase the quality of our research results. To this end, the implementation of quality management went hand in hand with investments in information technology (IT) structure and software development.
Building a QMS and going through the ISO 9001 certification process was a project that required significant personnel effort and time (2 years in our case). Although efforts should not be underestimated, they should be viewed within the context of promoting benefits (see Sect. 2.6). To answer the question what resources are required and how one can get started, we describe hereafter how the process was carried out at the GMC as an example of implementing an ISO 9001 QMS in a German Research Center setting, which includes project management and organization, the IT infrastructure, as well as social aspects.
Project Management and Organization
After an introductory quality management training by an expert consultant for all GMC staff, we formed a project management team consisting of a quality manager (lead), the two heads of the GMC, the head of the IT group, and an affiliated project manager. An initial gap analysis provided useful data about our status quo (e.g., inventory of existing documentation). In the beginning, the main tasks were (1) development of a project plan with timelines and milestones, (2) defining “quality” in our biomedical research activities, and (3) determination of the scope of the QMS.
GMC’s current strategy and future plans were reconsidered by establishing a quality policy (highest possible standard of research quality) and corresponding common quality objectives (specific, measurable, achievable, realistic, time-based (SMART)). The relaunched version of the ISO 9001:2015 challenged us to comprehensively describe our context. The external context of the GMC includes political, economic, social, and technological factors like animal welfare regulations, funding, health and translational research for the society, and working with state-of-the-art technologies. Our internal context is represented by our technical expertise (over 15 years of experience in mouse phenotyping), knowledge and technology transfer (workshops), application and improvement of the 3 Rs (https://www.nc3rs.org.uk/), and adherence to the rules of good scientific practice (GSP) and the ARRIVE guidelines (Karp et al. 2015).
Implementation of the process approach was addressed by defining the GMC’s key processes in a process model (Maier et al. 2015) and specifying related key performance indicators (KPIs) (e.g., number of publications, trainings, reported errors, measures of internal audits). Process descriptions were installed including responsibilities, interfaces, critical elements, as well as risks and opportunities (risk-based thinking) to ensure that processes are clearly understood by everyone, particularly new employees.
It is often claimed that the ISO 9001 documentation management increases bureaucracy (Alič 2013). Therefore we did not create documents just for the sake of the ISO 9001. Instead we revised existing phenotyping protocols (SOPs) by adding essential topics (like data quality control (QC)) on the one hand and implemented missing quality-related procedure instructions on the other hand (e.g., regarding error management, data management, calibration, etc.). All documents were transferred in a user-friendly standard format and made easily retrievable (keyword-search) using commercial wiki software. A transparent documentation management system was successfully put into effect by implementing these processes.
The constant improvement of our systemic mouse phenotyping processes is continuously ensured by using established key elements of quality management such as error management including corrective and preventive actions (CAPA), an audit system, annual management reviews, and highly organized capacity and resource management structures.
IT Infrastructure and Software Tools
Although our custom-built data management solution “MausDB” (Maier et al. 2008) has supported science and logistics for many years, the decision to implement a QMS triggered a critical review of our entire IT infrastructure. As a result, we decided to adapt our IT structure according to our broad process model (Maier et al. 2015) to deliver more reproducible data.
To this end, MausDB has been restructured into process-specific modules. MausDB 2.0 is a state-of-the-art laboratory information management system (LIMS) for automated data collection and data analysis (Maier et al. 2015). Standardized R scripts for data visualization and statistics are custom-developed for every phenotyping test and routinely applied. Numerous QC steps are built into the LIMS, including validation of data completeness and data ranges (e.g., min/max). Additional modules cover planning of capacities and resources as well as animal welfare monitoring and a project database tool for the improvement of project management and project status tracking.
On a data management level, a series of SOPs regulate reproducible data handling and organization. Comprehensive data monitoring allows detection of data range shifts over time, eventually triggered by changes in methods or machinery.
On the infrastructural level, a well-defined software development process built on the Scrum methodology ensures proper IT requirements management. Thus, a continuous improvement process can be applied to our IT tools. Script-based automation of frequent tasks encompasses daily backups of data as well as software build procedures. An “IT emergency SOP” has been developed to ensure well-planned IT crisis management (e.g., in case of server failure) and provides checklists and instructions for troubleshooting.
The challenges with IT-related issues described above were primarily of two categories: resources and change management. Self-evidently, implementation of all IT improvements required years of effort. However, the resulting overall process is much more efficient and less error prone. Active change management was essential in order to convince IT and non-IT staff that changes were necessary, although these would affect daily work routines. In the end, employees have come to realize that these processes save time and produce higher data quality.
In a preclinical research environment, the members of a research group traditionally have a high level of freedom for work planning and execution of scientific projects. Often, only one scientist conducts one detailed research project, plans the next steps from day to day, and communicates progress to the group in regular meetings. Therefore, a research environment encourages a self-responsible, independent working structure, leaving room for innovative trial and error, and supports both a creative and a competitive mind.
When you plan to implement a QMS in a preclinical research environment, you want to preserve the positive aspects of open mindedness and combine them with more regulated processes. As the initiator, you might find yourself in the position where you see both the opportunities and possible restrictions like limitations for innovative and unrestricted science. We decided to limit the certification to the standard screening pipelines in the beginning and not to force every research project into the ISO framework. Still, we encountered expected resistance to the implementation, since people feared losing freedom and control of their work structure, as well as disruption with unnecessary additional bureaucracy. This is, in every case, a complex psychological situation. Therefore, the implementation of a QMS takes time, understanding, sympathy, and measures of change and expectation management. A good deal of stamina, patience, and commitment is indispensable.
Quality control and management of preclinical animal research is a topic of increasing importance since low reproducibility rates (Begley and Ellis 2012) have put the knowledge generated by basic research in question. Furthermore, low reproducibility rates have caused immense delays and increased costs of therapeutic drug development (Freedman et al. 2015). NOT implementing recommended solutions like rigorous study designs, statistics consultation, randomization and blinding of samples to reduce bias, sharing data, or transparent reporting (Kilkenny et al. 2010; Landis et al. 2012; Freedman et al. 2015, 2017) in preclinical research will keep these costs high. Practical implementation of these standards could be supported by a well-developed QMS and provide structure and ensure achievement.
However, there are many concerns about the financial expenses needed for implementing quality in research practice by establishing and maintaining a QMS. Initial minimum financial costs include gaining knowledge about the chosen standard, in our case through trainings by an external consultant on ISO 9001-based quality management and documentation organized for the whole staff. Designation of at least one person who coordinates the implementation of the QMS is essential, which brings about the issue of salary costs. We hired a quality manager for 2 years (1 FTE) and in parallel trained a project manager from our team to the standard and for becoming an auditor to take over after the first certification (0.5 FTE).
In addition, the costs for the certification body needs to be included in cost calculations. Different certification bodies perform the ISO 9001 certification with varying costs. The first 3-year period comprises audit fees for the initial certification and two annual surveillance audits. This is followed by a recertification in the fourth year. As an example, our certification body costs were as follows: ~6,500€ for the initial certification, ~3,000€ for an annual surveillance audit, and ~5,000€ for the recertification.
Since we implemented new IT solutions, we had additional financial costs. For transparent documentation management, we acquired licenses for a supporting commercial wiki software (~2,200€ per year). All other software acquirements were not directly in the context of the ISO implementation and were solely data analysis or software development related. The same is true for a permanent statistician position (1 FTE) to support study design and data analysis.
Costs for implementing quality management in research practice are often a deterrent as the advantages of saving this money might be more obvious than the disadvantages. However, “not investing in costs” with respect to quality lead to “silent” costs. An example is the nonconformity management: if errors and corresponding measures are not properly documented, reduction as well as detection and avoidance of recurring errors through a detailed analysis is hardly possible, and the positive effects of increased efficiency and reduced failure costs are left out. The financial gain of an effective QMS can hardly be calculated in a research environment; however, documented, reviewed, and continuously improved processes ensure identification of inefficiencies, an optimized resource management, avoidance of duplication of work, and improved management information reducing the general operating costs.
Why should an institution decide to improve quality in research practice by investing in an ISO 9001 QMS? We want to list a number of fundamental arguments and provide practical examples that might open a different perspective.
Building a QMS in the GMC demanded the highest efforts in the first 2 years before the certification (in 2014). However, with increasing maturation of the QMS, the process ran more efficiently due to enhanced quality awareness and a general cultural change which both led to increased quality of output (assured by monitoring the KPIs). People started to like the environment of having a QMS, and continual improvement became a habit. Over time, the benefits associated with using a QMS will offset the efforts it took to build it in the first place. Some of the most striking benefits of having an expanded QMS are listed hereafter.
These are important controlling steps as they give an annual overview of the actual state of the processes including all KPIs, the content of errors and the corresponding actions, open decisions that were supposed to be closed during the year, or specific actions that are pending. To this end, this kind of review differs from the usual reporting to funding authorities. They solely serve the quality status and reinforce focus on strategic, quality-related goals that have been identified as priorities. This is particularly useful since concentrating on important issues (e.g., increased QC issues in specific tests or applying a risk-based approach) is something that is often postponed in favor of other tasks requiring frequent or immediate attention. Here you need to deal with formal numbers and can react and adapt milestones if specific problems have not been addressed adequately. Surprisingly, this kind of review enabled us to react quickly to new developments. Since the digital assembly of the KPIs is in place, the numbers can be easily reported also during the year, and fact-based decisions can be made. To this end, contrary to the belief that a QMS causes a bureaucratic burden, the QMS actually facilitates agile project management.
Internal audits are an often underestimated element of a QMS. By performing internal audits (e.g., independent phenotyping protocol reviews, complex process audits, or audits addressing current quality problems such as reduction of bias), we ensure that standardization is guaranteed, measures for improvement are defined, and prevention of undesired effects is addressed. Internal system audits as well as the third-party audits ensure the integrity and effectiveness of the QMS.
Box 1 First Third-Party Audit
BEFORE: Being part of a third-party audit was initially mentally and emotionally demanding: just before the first certification audit, personnel (afraid of the visit by the audit team) kept calling to report minor issues and ask what to do.
AFTER: Now, members of the group are used to internal scientific method auditing and have realized that we do not run the QMS solely for the certification body but for our own benefit. Today, while presenting the systemic phenotyping methods in a third-party audit, people feel accomplished, enthusiastic, and self-confident.
Comprehensive training of personnel is time-consuming and associated with extensive documentation. However, training ensures establishment and maintenance of knowledge. New employees complete an intensive induction training including the rules of good scientific practice (GSP), 3 Rs, awareness for working in an animal research environment, QM issues, and legal regulations. Regular QM trainings build and maintain awareness of quality issues. The “not documented, not done” principle is well accepted now and supports the transparency of personnel competence assessment. In addition, we are currently building an eLearning training program in order to save time for logistics in case people missed trainings.
All processes were critically assessed for traceability. On the physical level, temporal and spatial tracking of mice and samples (blood, tissue) is an issue. We implemented a barcoding in our LIMS to register all samples. On the data level, we aim to maintain full traceability of data and metadata. This means we link file-based raw data to our LIMS and capture all metadata that may influence actual data, e.g., experimenter, equipment, timestamp, and device settings. On the process level, all transitions between sub-processes (“waiting,” “done,” “cancelled”) are logged. This enables us to monitor dozens of ongoing projects at any time with custom-built tools to identify and manage impediments.
Box 2 Traceability Versus Personalized Data Storage
BEFORE: Some 10 years ago, we had to ask a collaboration partner for a re-genotyping because of identity problems within a cohort of mice. Tail biopsies were sent, but the electronic list correlating the biopsy numbers to the corresponding mouse IDs was saved on a personal device unavailable for the team. Therefore, the results could not be matched and the data analysis was delayed for more than 4 weeks.
AFTER: Samples now carry a barcode label with the mouse ID and any lists are saved in a central project folder.
With respect to quality, reproducibility of results is of paramount importance. In addition to SOPs, which regulate how phenotyping procedures are physically carried out, we put considerable efforts in making data analysis and visualization reproducible. To this end, we seamlessly integrated R (R Core Team 2013), a free statistical computing environment and programming language with our LIMS MausDB. Upon user request in MausDB, R scripts perform customized, test-specific statistical analyses as well as data visualizations. This tool restricts user interaction to the mere selection of a data set and the respective R script ensuring that same data will always reproduce the same statistical results and the same plots.
Box 3 Taking Responsibility in Writing Up Publications
BEFORE: We always provided our collaboration partners with the raw data so that they could perform additional analyses. In the past, during drafting manuscripts, we did not verify in detail if we were able to reproduce their statistical analysis and figures.
AFTER: With the implementation of the QMS, we have formalized how a manuscript is processed. This process now includes a step in which our in-house statistician reproduces all analyses and figures using our data as well as additional data from the collaboration partner.
2.7 Lessons Learned/Outlook
Certification to ISO 9001 is not a requirement in nonregulated preclinical biomedical research and also does not define scientific standards, but it represents a reasonable strategy to improve data quality.
GMC’s QMS: A Success Story?
Our ISO 9001:2015-based QMS helps us to generate and maintain transparent and traceable data records within a broad spectrum of standardized phenotyping processes with low variability and increases collaboration partner’s trust in the analysis, interpretation, and reporting of research data. This structured approach also supports compliance with manifold regulations and promotes awareness and risk-based thinking for the institutional context as well as meeting the requirements of funders, personnel, the scientific community, and the public. However, to specifically address the quality of data output, we see the need to broaden the perspective and to reach out to other parties who perform quality assessments in preclinical research.
Although certification is rarely found in preclinical research, participating in a network of institutions in similar scientific research areas performing, e.g., annual internal ISO 9001 audits on a mutual basis is an opportunity to address common scientific quality problems and therefore a future goal. Positive examples are the Austrian biobanks (BBMRI.at) and a French network of technological research platforms (IQuaRe; https://www.ibisa.net) having built ISO 9001 cross-audit programs.
Limits of Automation
We have learned that beyond a certain level of complexity, further automation requires increasingly and disproportionately higher efforts and is therefore limited. At the GMC, automation of data analysis and visualization works well for projects adhering to our standardized workflow. Beyond that, customization of projects adds additional complexity that is not compatible with full automation. In such projects, custom data analysis still has to be performed manually.
At the GMC, information technology has supported operative processes since 2001. In that sense, “digitalization” is not just a buzzword for us, but a continuous process that aims for measurable and sustainable improvement of our work. IT solutions implemented so far mainly cover standardized processes.
“Machine learning” is another heavily used catch phrase. In the case of the auditory brainstem response test, we currently use our vast data set to develop methods for automated detection of auditory thresholds, including deep learning by neural networks. We are sure that this will provide a more reproducible method, independent from human influences. Of course, human experts will always review and QC the results. Nevertheless, setting the scene with a QMS paves the way for future investment in modern IT technologies and digitalization.
Finally, we made the experience that you need to allow flexibility and consider not including all processes and/or details in the ISO 9001 QMS. Indeed, the ISO 9001 QMS does not require to incorporate every process, and this might be also a misconception of many principal investigators (PIs) that prevents the introduction in academic settings. Real innovation is a truly inefficient and non-directed process (Tenner 2018) that needs to reside in a protected area. As soon as innovative research projects generate either new technologies or techniques, we slowly implement these into our processes and apply quality management measures step by step. Therefore, it is also important to not allow the system to “take over,” getting lost in micromanagement or obsessed by automation with new IT solutions. It is all about balancing the needs for quality and scientific freedom and keeping the expectations from all involved parties in a reasonable frame. To this end, it is necessary to allow time during the implementation process and to understand that benefits are apparent only after a longer time period.
Although efforts for implementing a QMS might be more tricky in an academic setting in a university (many PIs, high diversity of activities, rapid change of personnel) than in a mouse clinic performing highly standardized tests and procedures, the ISO 9001 standard gives a framework for introducing more quality-relevant aspects in preclinical research and helps enormously with team mindset. An ISO 9001-based QMS supports quality in manifold key process types as well as in supporting, analysis, and improvement processes like training, communication, documentation, auditing, and error management.
Nevertheless, the determination of good quality data output can only be judged by scientific peers and the respective community.