Developing a Quality Control/Quality Assurance Program

Abstract

A good quality control/quality assurance (QC/QA) program is essential to the internal and external validity of your research project. This chapter focuses on the model, rationale, and procedures for a QC/QA program for site preparation, behavioral and biological assessments, and the intervention (The NIMH Multisite HIV/STD Prevention Trial 2007). Quality control procedures are the methods used to ensure that data are collected in a standardized way and that procedures are operationalized with clarity. Quality control activities include the development and implementation of systems such as a standard protocol. Quality assurance activities address adherence to the protocol and study procedures, behavioral and biological assessments, and intervention (treatment and control conditions), and assesses whether the quality control procedures were effective. These activities can include onsite and central monitoring of data collection, implementation of interventions, and a review of a random sample of questionnaires. The chapter is organized to track the life of the research project, from start-up, through field implementation, to ensuring adherence to the study procedures, and validity of the data.

Appendices

Appendix 1

24.1.1 Sample Certification Interview

Site:

Name of Individual Being Certified:

Date of Interview:

Certifier/Interviewer Name:

• 1. Generally, describe the study.

• 2. Tell me about the population being sampled in your country (venue, micro-venue, age, gender, inclusion/exclusion criteria).

• 3. Recruiter: What are the three fundamental ethical principles (respect, beneficence, justice).

OR Give me 2–3 examples of a risk and 2–3 examples of a benefit in this study.

Assessor: When you sit down to conduct an interview, tell me what specific materials/supplies you need to have with you.

• 4. Describe the informed consent process (prior to assessment, components, signed, witnessed).

• 5. Describe the protocol for following-up with non-responders.

• 6. What incentive is being offered? What do participants have to do to receive the incentive?

• 7. Describe the computer assisted personal interviewing (CAPI) process and way in which the assessment is administered (e.g., only use probes in assessment, response cards, etc.).

OR I am a participant and you just read me a question. I tell you that I don’t understand the question. What would you do? (reread question, use probe if available, use definition of terms, ask participant to answer the best they can)

• 8. What forms must be completed for each participant (consent, contact form, STD).

• 9. Describe the site’s counseling and treatment process (e.g., administered by nurse, referred).

• 10. [Optional question at discretion of interviewer – document question]. For example: give me an example of an adverse event. Give me an example of a protocol violation.

Certification Results
Staff Member Certified	Staff Member Not Certified
Certification No:	Areas of Retraining Needed

Appendix 2

24.2.1 Sample Staff Statement of Confidentiality

(Name of your organization) assures each participant that the confidentiality of responses to this information request will be maintained by (your organization) and that no information obtained in the course of this activity will be disclosed in a manner in which the particular individual supplying the information or described in it is i dentifiable, unless such individual has consented to such disclosure, to anyone other than authorized project staff of the (name of your organization and associated offices).

Agreement

I, (printed name of staff)__________________________________, agree to provide field data collection services for the benefit of (name of your organization) in connection with the (Name of study). Further, I

• a) am aware that the research being conducted by (site-based organization) is being performed under contractual arrangement with the (U.S.-based organization);

• b) hereby accept all duties and responsibilities of performing specified data collection tasks and will do so personally in accordance with the training and guidelines provided to me. At no time will I engage the services of another person for the purpose of performing any data collection tasks for me without the prior written approval of (your organization);

• c) agree to treat as confidential all information secured during interviews or obtained in any project-related way during the period I am providing services to (your organization);

• d) agree to treat as confidential and proprietary to (your organization) any and all research instruments, materials, and documentation provided or accessed during the course of my service on this project;

• e) am aware that the instruments completed form the basis from which all the analysis will be drawn, and therefore, agree that all work for which I submit invoices (or whatever method of accounting is used to monitor work) will be of high quality and performed in compliance with all project specifications;

• f) fully agree to conduct myself at all times in a manner that will obtain the respect and confidence of all individuals from whom data will be collected and I will not betray this confidence by divulging information obtained to anyone other than authorized representatives of (your organization); and

• g) understand that my obligations under this agreement will survive the termination of any assignment with (your organization) and/or my employment by (your organization).

Employee’s SignatureDate

Appendix 3

24.3.1 Sample Site Visit Agenda for Study Site Visit by DCC

The Data Coordinating Center (DCC) will visit each site just prior to the inception of data collection for the Baseline Assessment. If feasible, these trips will be timed to allow visitors to observe both the training of the data collection staff and the initial stages of selecting, contacting, recruiting, and assessing study participants. Site visits will focus on QA/QC for procedural and operational aspects of the data collection, and for certification of the behavioral assessment field and central office staffs.

1.Agenda Items to be Accomplished During Site Visits

• Meet with Site Principal Investigator (PI) or designee at beginning of visit to review and discuss visit objectives, staff organization, schedule, protocols, logistics, supervisory process and staff performance, plan for additional training, if needed.

• Meet with data collection staff to observe, review, and supplement their training program, conduct brief interviews (individual and collective) with staff to confirm understanding of study data collection standards, clarify misconceptions (if any), and focus on points of emphasis (e.g., assessment methodology (e.g., CAPI), contact form, response rates, informed consent).

• Review staff readiness (collective and individual) to properly select, recruit, screen, transport, and assess study subjects.

• Review training and data collection procedures to ensure they meet informed consent, privacy, and confidentiality standards.

• Review sample of each data collector’s completed Assignment Control Forms and Contact Forms (practice or actual) to ensure they are accurately and promptly prepared.

• Review equipment and materials (laptop computers, paper and pencil interviewing (PAPI) version of assessment, biological specimens collection apparatus, etc.) to ensure they are in proper working order or correct version.

• Review archival records to ensure compliance with study standards, especially records that identify study participants.

• Certify data collection field and central staffs.

• Meet with Site PI at end of visit to debrief and discuss areas of emphasis.

2.Site Preparation Responsibilities

• Site will assess the need for an independent translator (should not be a member of the study assessment staff) to be available to the DCC site visitor, in order to accomplish the site visit tasks and objectives.

• Site will prepare a timeline for the site visit that will allow all required activities to be performed during the time period of the site visit.

• Site will make arrangements for the data collection field and central staff members to be available as required for meetings, discussions, etc.

• Site will make arrangements for the site visitor to observe data collection sites, if feasible.

3.Site Visit Documentation and Follow-up

• Following the site visit, the DCC site visitor will draft a site visit report that will be submitted to the client, the DCC, site PI(s), and be archived at the DCC. The report will first be submitted to the project site and DCC for review. This review will allow the DCC to clarify issues that arose during the site visit, and the report will be revised, as deemed appropriate by the site visitor.

• In addition to the site visit report, the site visitor will provide each site with a written evaluation of staff strengths and weaknesses, and a more detailed summary of discussions held with site team members about the next steps necessary to address specific issues raised during the site visit.

Appendix 4

24.4.1 Sample Site Visit Checklist for DCC

Questions	Y/N	Unable to Assess	N/A	Observations/Comments
Site Preparation, Logistics, and Monitoring
Has appropriate IRB approval been obtained from both US and country institutions and forwarded to DCC?
Is a copy of IRB approval on file at site?
Are the appropriate numbers of sites being used in the study?
Are the appropriate numbers of individuals within each site being identified and selected for interviewing?
What was done to prepare the sites for sample selection and assessment?
What is the estimated response rate? Is anything being done to improve the rate?
Are the assessment supplies being assembled properly and in advance of field visits?
Are the biospecimen supplies being assembled properly and in advance of field visits?
Were appropriate criteria (education, experience) utilized in hiring study staff for assessment?
How many recruiters, assessors, biospecimen collectors are working in a site (field)?
How are the field data collection teams organized? Is this working well?
Are new staff trained and certified? How?
Is the site coordinator conducting regular meetings with assessment field staff and properly monitoring progress and quality of work?
Is the site coordinator conducting regular meetings with biospecimen collection staff and properly monitoring progress and quality of work?
Is transportation of participants to assessment locations appropriate and on schedule?
Are the translations and language for CAPI and PAPI appropriate?
How many assessment laptops are being utilized by the site?
Is the correct version of CAPI installed on laptops?
Are completed consent forms being properly stored in ID number order and in a locked field/central office?
Are completed contact forms being properly stored in ID number order and in a locked field/central office?
Are all confidential files secured and locked?
Is confidentiality of participant data maintained by all staff? Is site utilizing a staff confidentiality agreement?
Are Protocol Violation forms being properly completed when needed?
Are Adverse Event forms being properly completed when needed?
Are Protocol Violation and Adverse Event forms being forwarded (in English) to DCC then securely filed at the site?
Are the correct versions of DMS and LMS installed in site?
Who is responsible for assigning ID’s to assessors? How often is DMS updated with assignment information?
Are data forms being keyed in a timely manner?
Are appropriate quality assurance measures being taken in keying forms (key verification)?
Are there any issues with data entry or software?
Are telecommunications and Internet connections working properly?
Are data transmissions to DCC being sent in an appropriate and timely manner?
Is the site properly utilizing DMS reports to monitor progress?
Is acknowledgement of receipt of data being transmitted to the site from the DCC in a timely manner?
Are communications between US institution and country regular and informative?
Are communications between country and DCC regular and informative?
Selection
Is the site adhering to the appropriate age range?
Is the site adhering to the appropriate inclusion/exclusion criteria?
How was the micro-site selected within each site?
How are participants selected from micro-sites?
Recruitment/Screening
Did recruiters accurately describe the study when approaching potential participants?
Are the ID labels being utilized properly on contact forms?
Are the contact forms being completed accurately and in a timely manner?
Is appropriate tracing information being obtained for follow-up in 1–2 years?
Are non-participants (e.g., refusals) being documented properly for follow-up?
Are selected participants being given appointment cards or appropriate information for appointment?
Are selected participants being administered informed consent correctly (signed) and completely?
Are incentives being offered to each participant?
Are strong efforts being made to follow-up on non-participants (minimum of three attempts)?
Assessment
How are assessments being administered?
If applicable, are the CAPI computers and software working properly?
Are PAPI copies of assessment (current version) and pens available in case of computer failure?
Are the ID labels being utilized properly on all forms (STD q’aire) and biospecimens?
Do assessors demonstrate strong professional and interpersonal skills?
Are assessors effectively establishing rapport with participants?
Do assessors accurately describe study and reiterate informed consent?
Are behavioral assessments being administered prior to STD symptoms questionnaire and HIV/STD counseling?
Are assessors properly administering CAPI, including probes?
Are response cards available and used with participants?
Are process evaluation questions being administered by assessor or separately?
Do biospecimen collectors accurately describe study?
Is site offering pre-assessment counseling and prevention education?
Are the STD Symptoms Questionnaires being administered and completed correctly?
Is site offering post-assessment counseling or appropriate referral after administering symptoms questionnaire?
Are biospecimen collectors properly disposing of all bio-hazardous material (e.g., sharps, gloves, etc.)?
Are participants being given appropriate information to receive test results?
Are participants receiving treatment or referral for treatment?
Are participant partners receiving treatment or referral?
Is site offering post-assessment counseling or appropriate referral after participant receives results?
Are proper incentives being given to participants who complete assessment?

Appendix 5

24.5.1 Sample Site Visit Procedures for Study Site Visitfor the Intervention

The data coordinating center (DCC) will conduct a site visit to each site during the course of the intervention. The site visit will be conducted by a staff member from [name of institution] who has many years of experience in implementation of [the intervention]. The site visit will focus on quality control and quality assurance for conduct of the intervention and on certification of the intervention facilitators.

Agenda

Due to travel and site logistics, it is expected that the order of the activities will vary somewhat from site to site. However, all of the activities listed should be covered during each site visit. Additionally, most of the visits will begin with a meeting between the site visitor, study investigators, and the intervention supervisor, and all will end with the debriefing activity described below.

Before the visit, the central QA site monitor will contact the site via e-mail to confirm that dates for the visit will be acceptable and to develop an agenda for the visit that includes the agenda items below.

Agenda Items

Each intervention QA site visit will include the following activities:

• An initial meeting with representatives from the site (at a minimum, the intervention supervisor) to obtain an overview of activities at the site (with attention to both successes and difficulties) and to review plans for the remainder of the site visit. At a minimum, the overview will cover the items listed in Appendix 6.

• A visit to the site offices to examine study materials, participant records (e.g., documentation of consent, contact information, confidentiality procedures), and intervention process forms (e.g., recruitment forms, attendance forms, facilitator session notes, notes on other prevention activities in the overall site and in each venue). The results of this visit will be documented in a site visit report.

• A review of video or audiotapes of mock or pilot sessions and associated feedback to document facilitator training practices. This will be documented on the Generic Intervention Checklist.

• Attend one or more training or reunion sessions in order to systematically monitor adherence to model components and study protocol. If cultural differences preclude attendance at a session, review audio or videotapes and transcripts of recent sessions conducted by as many facilitators as possible. This review will be documented on the Generic Intervention Checklist.

• Initiate informal discussions with facilitators as well as conduct a brief interview.

• Certify the intervention supervisor and facilitators based on interviews and results of the review of sessions. Document the name and Facilitator ID Number of the individuals certified in the written report.

• Visit one or more of the resource or intervention venues to see where the intervention is being implemented. This will be documented in the written report.

• Conduct a brief interview with at least ten study participants (conduct a focus group if individual interviews are not possible). Note, cultural barriers may prevent this portion of the visit.

• Debrief site investigators at the conclusion of the site visit (with written report to be provided at a later date).

Reporting

The central QA site monitor will submit the standardized forms cited above, as well as a written narrative report to:

• DCC

• Site Principle Investigators (All PIs will receive a copy of the report prior to submission to client to clarify points and check for accuracy)

• Client Program Officer

Appendix 6

24.6.1 Suggested Agenda for Quality Assurance Site Visitsfor the [name] Intervention

Day 1:

• Meet with site investigators at site office to discuss program and logistics of the intervention QA site visit

• Examine protocol and project materials accessibility, participant records, and intervention process forms

• Quickly review facilitator pilot study or mock group training videos or audiotapes

Day 2:

• Visit intervention sites/venues where the research participants are expected to interact with peers

Day 3:

• Attend one or more training or booster sessions if possible; alternately, review video or audiotapes of recent groups; review transcript of recent groups if session is in a foreign language and transcripts are available; use generic session checklist to document evaluation of session(s). Note that audiotapes, videotapes, or transcripts of sessions also should be reviewed prior to the site visit if feasible.

• Interview ten facilitators regarding project interview guide attached ( individually when possible; small group otherwise)

Day 4:

• Conduct interviews with facilitators using facilitator interview guide attached

• Certify facilitators

• Meet with site investigators at site office for debriefing

Day 5:

• Complete on-site notes; Follow-up with site investigator, as needed

• Prepare reports and complete documentation

Note: Due to travel and site logistics, tasks on Days 1–4 may vary in order. However, all of these tasks will be completed during each site visit.

Appendix 7

24.7.1 Intervention Quality Assurance Site Visit Overview

The site visit will begin by a meeting attended by the site visitor, the intervention supervisor, at least one site investigator or coordinator, and an interpreter if needed. At a minimum, the following topics will be covered during this meeting:

PROJECT OVERVIEW (Information obtained from staff service providers/facilitators)

• 1. Describe the performance sites:

• 2. Number of performance sites:

• 3. Describe the target population:

• 4. Population Size (Provide range of sizes per venue across all venues):

  • a. Total population size per performance site:

  • b. Number of facilitators to be trained:

• 5. Describe the activities undertaken to obtain community buy-in:

• 6. Number of research subjects recruited to date:

• a. How were they recruited?

• b. Did all meet criteria for recruitment?

• 7. # intervention groups currently active:

• 8. Attendance rates at sessions (# attending Session 1 and #s attending subsequent sessions/or # graduating):

• 9. Is training of intervenors/facilitators done in waves or all at once:

  • a. If training is done in waves, how have attendance rates varied across wave?

• 10. How many booster sessions been conducted?

• 11. Are booster sessions held with original intervention group or across intervention groups?

• 12. Describe how the intervention is monitored for fidelity:

• 13. Describe process of providing corrective feedback to project staff, including facilitators:

• 14. What is going well regarding intervention implementation?

• 15. Are you having any problems implementing the intervention? (Note: the site visitor can use this information, along with direct observation, to devise constructive suggestions that can improve intervention delivery.)

• 16. Review the site visit agenda to confirm dates, times, and transportation arrangements.

Appendix 8

24.8.1 Intervention Quality Assurance Visit Records Review

Intervention Materials	Yes	No	Comments:
1.Training Manual readily available	Y	N
2.Training Manual available in local languages	Y	N
3.Site Visit Protocol on file	Y	N
4.Training videos available	Y	N
[supervisor eval. of staff – 80% content covered and adequate or higher skill ratings. On-site supervisors attends or watches videotape of two sessions from each fac’s first group and one of subsequent groups, meet w/ fac w/in week and give feedback]
5.Process notes	Y	N
[summaries of group sessions and documentation of challenges]
Participant Records
6.Contact information kept	Y	N
7.Written consent forms	Y	N
8.Confidentiality Safeguards	Y	N
9.Attendance forms	Y	N
10.Parental consent forms, if minors (written)	Y	N	NA
11.Adverse events	Y	N
Comments:

Appendix 9

24.9.1 Criteria for Rating Laboratories

• Higher Standard Approval ─ Adheres to and exceeds study protocol, by demonstrating a higher performance standard according to International Organization for Standardization (ISO), Clinical Laboratory Improvement Amendments (CLIA) regulatory requirements, up to and including the “gold standard” of operations according to CAP.

• Approval ─ Adheres to study protocol; no-protocol related action items; suggestions provided for a higher performance level.

• Provisional Approval ─ Adheres to critical requirements of study protocol; protocol-related action items identified which do not affect test results; resolve action items during site visit, or resolve in writing and confirm at next site visit.

• Probational Approval ─ Does not adhere to critical requirements in study protocol; protocol-related action items identified which affect test results; resolve action items: each item in writing, written confirmation of corrective action to demonstrate consistent resolution, and on conference call and confirm at next site visit. Testing may be suspended.

• Denial or Revocation ─ Does not adhere to critical requirements in study protocol; multiple protocol-related action items identified, including those which affect test results ─ testing is suspended; resolve action items with written confirmation of corrective action and consistent improvement; testing may only be resumed after next site visit.