Keywords

1 Introduction

The number of courses and full degrees offered online, raises a considerable interest in concerns and problems associated with online education, particularly as it relates to the quality of online instruction [1]. In the case of the Community College of Baltimore County (CCBC), they started offering online courses around Fall of the year 2000, and launched CCBC Online during Fall 2018, offering not only full degrees but also certifications completely online. As of today, CCBC offers more than 20° online and certificate options, in addition to thousands of online classes, rolling admissions, and multiple starts-dates.

In this context, Quality Assurance (QA) has become a very important issue for both, educational institutions and institutional accrediting agencies. According to Universitas 21 Global approach, this QA process relates to five main areas: course content, courseware development, adjunct faculty recruitment, pedagogy and delivery [2]. The main focus of this paper is about course content, development and delivery; recruitment and pedagogy are out of the scope of the present work. In order to improve course quality, several rubrics have been developed by different institutions. These rubrics can also work well as guide when developing a course, similar to a roadmap to success.

This paper is organized as follow: Sect. 2 discusses some of quality course rubrics, Sect. 3 presents the Quality Assurance process performed at CCBC, Sect. 4 depicts the types of Quality Assurance reviews, while Sect. 5 considers some future work and provides some conclusions for this article.

2 Quality Control Rubrics

Because of the exponential growth in the offering of online education, and the aim of quality by those institutions that are offering those courses and degrees, several companies and institutions developed different rubrics to help developers in achieving good quality courses. The first institution that is going to be presented is the Online Learning Consortium (OLC) who supports higher education institutions seeking best practices for advanced quality. OLC offers a robust suite of quality scored cards providing educational institutions with the necessary criteria and benchmarking tools to ensure online learning excellence for the entire institution. The current scorecards include Administration of Online Programs, Blended Learning Programs, Quality Course Teaching and Instructional Practice, Digital Courseware Instructional Practice and the Open SUNY Course Quality Review (OSCQR) from Open SUNY. These scorecards can be used to demonstrate several quality elements not only within a particular program but also as an overall level of quality, that can be presented to the higher education accreditation institutions [3].

From all the scorecards provided by OLC, the one that focuses on the quality of the course will be discussed. OSCQR is not only a rubric, but a collaborative process to improve instructional design and accessibility of an online and blended course. The main focus of the process is continuous improvement and it is not a curse or instructor evaluation tool. Each course is reviewed using the OSCQR rubric by three different perspectives: faculty perspective (the author of the course), instructional design perspective, and an external reviewer perspective (a librarian, or another faculty member that is not familiar with the course). The team reviews the course asynchronously and provides suggestions for improvement. Those improvements are discussed, and a refresh plan is developed; once the course has been updated to the new version, then the team meet again, review the accomplishment and plan future enhancements. The rubric contains 50 quality and accessibility standards and have been categorized as ‘essential’ or ‘important’ based on research-based effective online practices. This rubric might be particularized by the higher-education institution by adding other standards [4]. It is very important to understand that OSCQR is an approach to review, refresh, and continue improve online courses, but it doesn’t help when you need to decide if the course under review has good QA to be offered online.

The next course quality rubric to be presented is the Quality Online Learning and Teaching (QOLT) Rubric developed by California State University at Chico. The rubric can be used in three different ways: as a self-evaluation tool, so a faculty member will evaluate his/her course, as a guide for developing a new online/blended course, so the rubric can be used as a roadmap, and finally as a recognition tool for exemplary online instruction [5]. The rubric is composed of six domains, and each domain is presented with different criteria. Each criterion provides rankings with clear explanations so the reviewer can see how the course meets each one of the criteria: ‘baseline’, ‘effective’, or ‘exemplary’. There are no explanations about what the requirements for a course are to be considered a good quality course. Does all the criteria for all the domains be ranked at ‘exemplary’, or only some of them?

Blackboard provides the Exemplary Course Program which recognized instructors and designers that demonstrate the applicability of best practices in the following areas: course design, interaction and collaboration, assessment, and learner support. Courses are evaluated by Blackboard clients using a common rubric. The Exemplary Course Program Rubric uses weighting values and numerical scores. A weighting value from 0.5 to 3 has been assigned to a category in each one of the areas, to indicate the relative importance of the category. Also, each category is described according to the level of mastery and assigned a numerical score: Exemplary (5–6), Accomplished (3–4), Promising (2), Incomplete (1), and Not Evident (0). Reviewers assign a numeric value to each category in each area [6]. The program does not specify a minimum numeric value needed by a course to indicate the quality of the course.

North America Council for Online Learning (NACOL) released the National Standards for Quality Online Courses (iNACOL) which provides guidelines for online course content, design, technology, assessment, and course management. These standards are available to states, districts, online programs, or other educational organizations, and are primarily geared towards K-12 online learning. Several standards are organized into five different sections: content, instructional design (how the course has been designed: learning activities and communication), student assessment, technology (tools and accessibility to those tools), and course evaluation and support. Each standard contains some considerations for the reviewers, and the corresponding rating: 0: Absent (the component is missing), 1: Unsatisfactory (needs significant improvement), 2: Somewhat satisfactory (needs targeted improvements), 3: Satisfactory (discretionary improvement needed), and 4: Very Satisfactory (no improvement needed) [7]. The rubric does not provide any suggestion on how to work with this rating to determine if a course can be considered a quality course.

Quality Matters (QM) is a nonprofit organization that supports and maintains quality assurance in online learning. The principal characteristic of the QM rubric is the concept of Alignment, which occurs when the learning objectives, assessments, instructional materials, learning activities and interactions, and course technology work together to ensure students achieve desired learning outcomes. There are a total of eight general standards: course overview and introduction, learning objectives (competencies), assessment and measurement, instructional materials, learning activities and learning interaction, course technology, learner support, and accessibility and usability [8]. These general standards are further breaking down into specific review standards, giving a total of 42. Standards are assigned different points depending on their relative importance: 3: Essential, 2: Very important, and 1: Important. The rubric also provides a series of annotations with examples to help the reviewer decide if the standard is met. For the standard to be met, the reviewer asks this question: does the course meet the standard at an 85% or better level? If the answer is yes, then the standard receives the corresponding value according to its importance. Whether or not the standard is met, the reviewer provides constructive feedback for continued improvement. Each course is reviewed by a three QM-Certified Peer Reviewers using the QM Rubric, one of them must be of the same discipline as the course being reviewed (subject matter expert), and one of them is the team chair, who is QM-Certified Master Reviewer. The team chair combined all three reviewers scoring and decide if a standard is met or not, based on the majority rule: at least two reviewers agreed on the same decision. The course can meet Quality Matter Review Expectations if all 3-points standards are met, and the review resulted in a total overall score of 85 or higher out of 100 points. There are many other rubrics available for higher-education institutions, that are not going to be presented in this work.

3 Quality Assurance Process: The Beginning

After analyzing several QA rubrics and approaches, CCBC decided to adopt Quality Matters and the decision was made because their suggested process is centered on continuous improvement. This process is designed to help institutions achieve the QA goals for online learning as they grow to address all aspects of the Online Learning Quality Pie [9] (see Fig. 1).

Fig. 1.
figure 1

Online Learning Quality Pie

Full size image

Even though quality matters focuses on course design, institutions can use the same rubric to scale the model and apply it to all the aspects of online learning specified in the previous figure. This is a progressive process and institutions go through different stages until they implement QA effectively in their institution.

3.1 Stage 1: Training

In the particular case of CCBC, some faculty and staff already have online training and have been teaching online for quite some time. Faculty were required to complete a formal training in order to be ready to teach online and to design their courses. The format and name of training has changed through the years, from Virtual Academy to Online Training Institute and Online Course Design Institution. If faculty or staff already had online training obtained at other institutions, a waiver could be obtained via a very informal process.

Based on several comments received from the Accreditation Institution in one of their reviews, CCBC decided to adopt QM as quality control in their courses. To start with this process, an Instructional Design and Online Learning (IDOL) office was created, and instructional designers were hired. Faculty that were already trained were the first ones invited to participate in the Apply the Quality Matter Rubric workshops. This was the beginning of the QM implementation. Faculty stated to be trained, courses started to be self-reviewed, and after the final review from the IDOL office, they were submitted for external certification.

Table 1 shows the number of courses by department that were externally reviewed, and the year of certification.

Table 1. List of QM certified courses.
Full size table

Without considering the ACDV course, a total of 16 courses were QM certified in two years, an average of 8 courses per year. This was a great start for this initiative.

3.2 Stage 2: Process

Because the rubric was successfully adopted for faculty and staff, the IDOL office decided that a formal structure was needed for the online learning initiative. CCBC is formed by seven schools: (1) Business, Education, Justice and Law (SBEJL), (2) Continuing Education (CE), (3) Health Professions (SOHP), (4) Liberal Arts (SOLA), (5) Mathematics and Science (SOMS), (6) Technology, Art, and Design (STAD), and (7) Wellness, Behavioral, and Social Science (WEBSS); so, each school selected a member to be design as an Online Coordinator for their school. This team meet once a month with the IDOL office to discuss all the aspect related to online learning.

At the same time, a new board was created: Distance Learning Advisory Board (DLAB) and was composed by a representative from different offices such as schools, IDOL office, the Deans of the schools, the Blackboard Department, the IT department. The mission for this board was to define distance education policies for the college. Those policies were then sent to the College Senate for approval and implementation. Among those policies are: Faculty Usage of CCBC Learning Management System Policy, Policy on Ensuring Quality in Distant Learning Courses, Learning Management System Course Menu Best Practices.

The number of courses seeking formal certification was growing as well as the cost of having those courses certified externally by Quality Matters. This forced the Online office to look for a different way to do this formal certification: Internal Certification. This new initiative required to train faculty as Peer-Reviewers, and formally organize the QA process. Several informal policies were established:

  • Courses that were Externally Certified by QM, become now Master Courses

  • Faculty teaching online was suggested to use the Master Courses

  • If faculty did not want to use the Master Course, then his/her course must go under Internal QM Review

  • Highly-enrolled courses should continue to be certified externally by QM

Table 2 depicts the courses that were Internally QM Certified, while Table 3 shows the courses that were Externally QM Certified.

Table 2. List of internally QM certified courses.
Full size table
Table 3. List of externally QM certified courses.
Full size table

As it can be appreciated through those tables, a total of 72 courses were internally QM certified, while 23 courses were externally certified, giving a final total of 95 courses.

Table 2 is not showing any courses for the year 2018, because the certification process last two semesters: during the fall semester, courses are self-reviewed by their authors, and they are submitted for review at the end of the semester. During the spring semester, the review teams are formed, the review is performed, and for those courses that do not meet the certification, a report of improvement is provided, and faculty implement the suggestions, so their courses can achieve the level needed to be certified. This process can be repeated only twice, and at the end of the third time, the decision is final, and the course Met or Not Met the qualifications needed to be QM certified. At the moment of writing of this paper, the 2018 Internal Review courses are being reviewed by the 3 peer-reviewer teams.

3.3 Stage 3: Formalizing the Process

Until August of 2018, online course development has been faculty-driven with little intervention or help from instructional designers, aside from specific course development training. As the number of online and blended courses starts to increase, a more systematic approach must be developed where faculty will continue to be the developers of the courses, with the assistance needed from the instructional designers. With this official Policy, courses will be developed with the proper online learning pedagogy, built in a timely manner, and designed following QM standards from the beginning.

Because of the launch of CCBC Online, an Online Learning office was added to control and supervise all the aspects related to online learning with its Assistant Dean and coordinate these activities with the IDOL office. CCBC identifies two categories of professional development in instructional design and online learning: course facilitation and course development.

Course Facilitation

Teaching an online or blended course requires a specific set of skills. There must be proficiency at using Blackboard as well as an understanding of pedagogy required to engage online students. Because of that, faculty that want to teach online at CCBC must complete the following training: Teaching Online Course (TOC). This course is offered usually twice a year, it is completely online, and teaches faculty how to engage with students and facilitate an already constructed course. There are no prerequisites for this training, but there is an assumption of prior knowledge of Blackboard. This training also assumes that the faculty members have a course that has already been built for them to teach.

Faculty who have expertise and prior experience teaching online from other institutions must demonstrate competency in Blackboard, online skills including facilitation, and assessment measurement & learner engagements prior to teaching online courses. Proper documentation must be sent to the Instructional Design and Online Learning office.

Figure 2 depicts this process graphically for a clear understanding of this process.

Fig. 2.
figure 2

Teaching Online Course (TOC) procedure

Full size image

Course Development

Each academic School as well as Continuing Education, have an Online Learning Coordinator (OLC) who should create a production schedule for the courses to be developed. This production schedule is approved by the Dean of the School and share with the Assistant Dean of Online Learning and the Director of IDOL. If a faculty member wants to develop a course that is not on the schedule, written authorization by the Dean and the Department Chair/Director must be presented prior to production. When a course is added into the production schedule, faculty members developing the course are also identified and an application is completed. This application is signed by the faculty member(s), OLC, and Academic Dean.

Once the application is signed, the OLC schedule a meeting with the Director of IDOL to determine the professional development needs of the faculty involved and determine the appropriate course development process.

If the faculty member has been already trained at CCBC in the past (by taking Virtual Academy (VA), Online Teaching Institute (OTI), Online Course Design Institute (OCDI)) a DOCS (Designing Online Courses) process is assigned to the course. If a faculty member already has this training from other institutions, formal documentation must be submitted to the IDOL office prior to developing a course. If it is approved, then also a DOCS process is assigned to the course.

In a DOCS process, there are some professional development requirements prior or during course development: updated APPQMR and specific Blackboard training modules. In this process, the responsibilities are divided between the faculty member and the Instructional Designer with the help of the OLC as follow:

  1. 1.

    Faculty member is able to:

    1. a.

      Write rough drafts of module level objectives;

    2. b.

      Outline the design of the course;

    3. c.

      Create Assignments/Assessments;

    4. d.

      Create content/identify instructional materials;

    5. e.

      Align content, make changes to course design;

    6. f.

      Create rubrics within Blackboard;

    7. g.

      Design overall structure.

  2. 2.

    The Instructional Designer with the help of the OLC will:

    1. a.

      Review the first module before subsequent modules are built;

    2. b.

      Monitor faculty course development process according to the plan developed in the DOCS meeting;

    3. c.

      View the course at designated points to assist overall course alignment;

    4. d.

      Assist faculty member with construction of parts of the course if the faculty member requests it;

    5. e.

      Conduct a Quality Matters review at the end of the development.

The DOCS process begins with a one or more than one meeting with all the parties involved in the process: Faculty member developing the course, OLC, Director or IDOL, Instructional Designer (ID), and Library Representative (if using OER). During this meeting the group will determine the tasks needed to build the course and assign those tasks to members of the committee. Also, the timeline for completion is set and the professional development needs of the faculty member is determined based on the tasks assigned. To ensure the quality of the course, the ID will review the first module before subsequent modules are built.

If the faculty member has not taken TOC, then the first step is to follow the procedure described above and take the course. Then, complete the following training Blackboard Rubrics, and Test Student courses; and APPQMR. According to the time constraint, two different processes can be assigned to the course: CRIRP (Course Redesign Institute Regular Program) or CRISP (Course Redesign Institute Summer Program).

This is a more extensive training and incorporates online learning pedagogy while developing an online course. The Regular Program involves two consecutive semesters: during the fall semester, online learning pedagogy is instructed, and best practices are applied to create one module for a course. At the end of this first semester, a showcase of the module is presented, and certificate is awarded. During the second semester, faculty replicates the structure of the module to the rest of the modules of the course and finalized the process by doing a QM self-review. In the case of the Summer Program, everything is compressed during the summer months of June, July, and August. At the end of this process, the course is also reviewed by an ID from the IDOL office. Figure 3 shows this process graphically.

Fig. 3.
figure 3

Course development procedure

Full size image

Independently of the process followed to developing a course, once it is finished, it will go under a Quality Assurance Review. The type of QA review that the course is going to undertake will depend on the type of course, and the professional development of the faculty member. See next section for details on these types of QA.

4 Types of Quality Assurance Reviews

The Online Learning office provides 4 different types of QA Reviews: (a) Eyes Review, (b) Second Generation Review, (c) CCBC Internal Quality Assurance Review, and (d) Quality Matters (External) Review.

In addition to these 4 types, courses that have been built using the DOCS process will undergo an Essentials Review prior going live, and then they under Internal Review.

Let start describing each one of those types.

4.1 Essentials Review

This review uses all the Essential Standards from the QM Rubric and some of the Very Important Standard, a total of 35 Standards plus 4 additional CCBC standards. This review is performed by a Subject Matter Expert and an ID or QM Certified Peer Reviewer. This process is a quick review where reviewers just mark on the form if the standards are met or not met, by checking the corresponding checkbox. No written feedback is provided in this type of review.

The output of this review could be as follow: (a) the course met the QM Essential Standards, (b) the course needs minor revisions, or (c) the course needs major revisions.

If the course needs minor revision, meaning just few changes need to be made, the faculty member is given two weeks to complete them. If a significant number of revisions need to be made and/or Standard 2 or any standard involved with alignment (2.1, 2.2, 3.1, 4.1, 5.1, 6.1) need to be revised, faculty will meet with an ID or the OLC to assist them with their revisions. Deadline for completion of the revisions will be determined at a meeting, with a maximum time of 8 weeks for completion.

4.2 Eyes Review

For this type of review, the same rubric as the Essential Review, the only difference is that this review is performed just by an ID or a Certified QM Peer Reviewer. This review is the quickest where the reviewer just marks whether the standard is met or not met by checking the respective checkbox. No written feedback is provided in this type of review.

The output of this review is the same as the previous one, and faculty should follow the procedure described above too.

This type of review is used when a course needs to be evaluated quickly, because it can be done in hours since it is reviewed by only one QM Peer Reviewer or ID.

4.3 Second Generation Review

This type of review uses the same rubric as the Essential Review, and the review is performed by the same team of reviewers. The output is exactly the same, the only difference is when this type of review is used.

If a faculty member already has a course that have been through QM (external) Review or CCBC Internal QA Review, then an assumption can be made about the other courses developed by the same faculty: they follow the same design principles and therefore the QM standards. Because CCBC wants all the courses to go under QA, this process is used to certify those courses in a quick manner and using the minimum of resources.

4.4 CCBC Internal Quality Assurance Review

If the course to be certified is the first for the faculty member, and the course is not considered to be a high-enrolled course for CCBC, then this is the QA process to follow. The only condition for a course to be reviewed using this process is that the course has been taught online at least twice before undergoing this review. The process resembles the official process that QM uses to certify courses, but it is performed totally in-house. The process starts by the faculty member doing a self-review of their course, making sure that it follows all the QM standards specify in the QM rubric.

Later, the faculty member submits some documentation of the course such as: Module objectives, objective worksheet, alignment worksheet and other information about the course. Then the Online Learning office forms a team to review the course by selecting a Subject Matter Expert, a Certified QM Peer Reviewer, and a Team Chair. The team review the course individually, and then a Final Report is generated indicating if the course met QM Standards or not. In order to be certified, all 3 points (essentials) standards must be met, and the total points must be equal or greater than 85 (out of 100). In addition, all CCBC specific standards must be met.

In this process, reviewers provide feedback on improvement of each one of the standards so faculty can perform those modifications to meet the standard. Faculty is provided with the feedback from each one of the reviewers.

If the course does not meet QM standards, the faculty have three weeks to perform the changes suggested, and the Chair of the team re-evaluates the standards marked as ‘no met’ and mark whether the changes made in the course makes the standard to be ‘met’. This process can be done twice, and after that the final decision is made: the course is QM Certified or not.

This internal process is performed once a year, usually during the fall semester, faculty perform the self-review and during spring the team is formed, and the evaluation is done. The certification is valid for five years, and CCBC provides a logo that can be added to the course.

4.5 QM (External) Review

This is the last type of review and only for high-enrolled courses, and courses that are considered Institutional Courses (used by all the faculty teaching the class as is) undergo this type of review. Before sending the course directly to Quality Matters, faculty also performed a self-review, and finally the course is reviewed by an ID from the IDOL office.

When the course is ready, then it is sent to the QM for certification. This process is entirely done by QM and CCBC have to pay a fee for the process. Because of that, just a small number of courses are sent for certification.

This certification is also valid for 5 years, and QM provides a logo that can be added to the course too.

5 Future Work and Conclusion

There is always something to improve when we are talking about QA. At this moment, the Online Learning office is working on a re-certification process, so it can also be performed in-house instead of sending the courses to QM for external re-certification.

On the other hand, OLC are reviewing all the courses that are offered online in their school and have never undergo a QA process, to determine the type of QA process for the course. This review is a long process and will take for sure several months to be completed; courses would need to be scheduled according to the resources available to perform each type of review.

Another path that the Online Learning office is researching right now, is the QA for a complete program or degree. Of course, the first step will be reading and learning how QM does the Program Review and see if that model fits CCBC or needs to be adapted to our needs.

As a conclusion, and as shown in this process, to ensure the quality of the design and development of a course, the qualification of the instructor needs to be considered first. If the instructor does not have those qualifications, training should be provided.

Second, the institution should provide a clear and very detail formal process for course design and implementation. On that way, faculty can realize that the school administrators want to ensure quality in the online instruction.

Successful online instruction is a collaborative process among instructors, administrators, staff, students, and the community at large. Courseware development industries should maintain the faculty informed about their new updates and provide them with training and technical support.