Keywords

1 Introduction

In recent years, with the vigorous development of cloud computing and Internet technology, the data show an explosive growth mode to make big data quietly come. Traditional data processing technology and tools are unable to meet the requirements of data processing in the new era, so the big data platform emerges at a historic moment. The big data platform supports a variety of computing frameworks that can serve multiple users simultaneously. However, it is an urgent problem for platform service providers that the mixed jobs of multiple users are reasonably scheduled so as to meet the requirements of resources utilization, the SLA of users and the maximum revenue.

At present, many researchers have studied the problem of job scheduling in big data platform and put forward a lot of solutions. A resource configuration optimization model based on deadline estimation for the Pig job is proposed [1]. The model eliminates the non-deterministic problem when Pig program executes jobs concurrently. However, the model does not consider the revenue of the platform. In [2, 3], researchers study the job scheduling project on basis of MapReduce framework. While taking into account deadline constraints and resource allocation, they do not consider the revenues of the platform and resource utilization. The proposed scheduling algorithms [4, 5] focus on the job deadlines. Although these algorithms are not suitable for our study, they provide a great guide. Liu et al. [6] propose a priority scheduling algorithm to divide the computing capacity of each node into a front-end virtual machine layer and a background virtual machine layer. The algorithm balances the workload of the platform, makes full use of the CPU resources, improves the execution efficiency of the job and shortens the response time of the job. However, it still does not consider the revenue of the platform. Koutsandria et al. [7] first investigate the problem of efficient resource allocation strategies for time-varying traffic, and propose a new algorithm, MinDelay, which aims at achieving the minimum service delay while taking into account the revenue of the providers.

To sum up, the above researches have been studied deeply in different constraints and different backgrounds for the job scheduling project and a series of achievements have been made. However, the methods in these achievements do not solve the problems of our study. We propose a two-stage job scheduling model based on revenues and resources in MapReduce framework and the main contributions of this paper are summarized:

  • We design a scheduling algorithm of the maximum revenue (SMR) based on the latest start time of the jobs. According to the deadlines of the jobs and the revenue rate, SMR pre-allocates the resources of the platform to the jobs and adjusts the allocation result to make the provider gain the maximum revenue.

  • According to SMR, we propose a sequence adjustment scheduling algorithm based on the maximum resource utilization of the platform (SAS). The job sequence on basis of the maximum revenue is adjusted to realize the maximum resource utilization under the premise of the maximum revenue.

The rest of the paper is organized as follows. In Sect. 2, we discuss the related work of job scheduling. Section 3 presents relevant definitions and the design of two-stage job scheduling model. The model is described detailedly in Sect. 4. In Sect. 5, the experimental results of the scheduling model are given and analyzed. Section 6 concludes the paper.

2 Related Work

Job scheduling in the big data platform is crucial to the optimization of the platform performance. In order to improve the efficiency of the job execution and optimize the performance of the platform, the researches [8,9,10] propose data placement strategy and job scheduling algorithm based on the minimum data transmission time to reduce the data transmission time and improve the efficiency of the job execution. However, the revenue is not considered in the algorithms. The key factors affecting the availability requirement of the parallel task and the guarantee of the resource availability are analyzed, and a parallel task scheduling algorithm based on usability perception is proposed [11]. Although the methods [12,13,14] optimize the performance of the platform, they do not take into account the revenues of the platform. These researches [15,16,17] optimize the performance of the platform by reasonably scheduling jobs, but the revenue and the resource utilization are not considered. In order to improve the utilization of the network and reduce the completion time of the job, a job-aware priority scheduling algorithm is proposed by monitoring the application layer [18]. The algorithm not only achieves network load balancing, but also improves the execution efficiency of the job. Kumar et al. [19] propose a scheduling algorithm based on perceptual and heterogeneous cluster to improve the resource utilization, whereas they do not consider the revenues.

In [20], job scheduling is studied from the perspective of resource allocation, and a flexible resource allocation algorithm is proposed to enable the job to be completed and consume the minimum computing resources before the deadline. The algorithm provides a fine guide for the resource scheduling. An intelligent job scheduling and workload balancing mechanism is proposed to realize the application performance with the least resources [21]. The framework does not guarantee the maximum resource utilization. The research [22] solves the issue of the maximum revenue, but there are some limitations. The method does not take into account the situation of parallel execution for multiple jobs. However, these methods do not take into account the maximum revenue and the maximum resource utilization at the same time. Therefore, this paper considers the constraints of the deadline, the maximum revenue and the maximum resource utilization and proposes a two-stage job scheduling model. The model not only meets the requirements of the deadlines, but guarantees to maximize the revenue and the resource utilization.

3 Two-Stage Job Scheduling Model

3.1 Basic Definitions

In order to facilitate the description of the job scheduling model, we give some basic definitions and formulas in this section.

Definition 1.

Total Number of Computing Resources (TR): the number of all Containers in the platform.

The big data platform that one master node and N dn workers. Each node is configured with c cores and m GB RAM. Each Container is configured with c c cores and m c GB RAM. The total number of resources of the platform:

$$ TR = min\left( {c/c_{c} ,m/m_{c} } \right) \times N_{dn} . $$
(1)

Where \( min\left( {c/c_{1} ,m/m_{1} } \right) \) is the maximum number of Containers for each node.

Definition 2.

A Set of Submitted Jobs: when the jobs are submitted by multiple users signing the SLAs with the provider to the big data platform, a set of submitted jobs is generated. It is expressed as \( J = \left\{ {j_{1} ,j_{2} ,\, \ldots ,\,j_{n} } \right\} \) , where n is the number of the submitted jobs.

We focus on the job scheduling in the isomorphic cluster, so we set identical nodes that have identical hardware configuration and performance. What is more, in this paper, we do not consider data skew so that we think the running time of every map task or reduce task is same by default. We pay attention to the execution time, required resources, the deadline and the revenue for each job. Any job j is expressed as j = (ms, rs, mt, rt, dl, rf(t)). Where ms is the number of map tasks, rs is the number of reduce tasks, mt is the average execution time of map tasks, rt is the average execution time of reduce tasks, dl is the deadline of the job and rf(t) is the revenue function of the job. The revenue function is following:

$$ rf\left( t \right) = \left\{ {\begin{array}{*{20}c} {a,} & {t \le j.dl} \\ {b,} & {t > j.dl} \\ \end{array} } \right.. $$
(2)

Where a is the revenue gained for the provider when the job is completed before the deadline and when the job is not completed on time, b is positive value as the revenue that is less than a. If b is negative value, the provider compensates the user for the loss.

Definition 3.

The Total Revenue Function: the job j i has j i .dl and j i .rf(t). The actual completion time of j i , j i .end, may be more than j i .dl or less than j i .dl. For all jobs, the total revenue function is following:

$$ R = \sum\limits_{i = 1}^{n} {j_{i} .rf\left( {j_{i} .end} \right)} . $$
(3)

3.2 Scheduling Model Design

In this section, we design the architecture of job scheduling model. In the first part of Fig. 1, the processing of job scheduling is divided into two stages. The first stage is a scheduling algorithm of pre-allocation resources based on the latest start time and maximum revenue. The second stage is a sequence adjustment scheduling algorithm based on the maximum resource utilization of the platform. In the first stage, the submitted jobs generate a set of jobs awaiting to be scheduled. Then, the jobs are scheduled firstly by pre-allocating the resources of the platform based on the latest start time and maximum revenue. The result of the first stage is that a primary sequence of scheduled jobs is got by SMR. In the second stage, the primary sequence is adjusted by SAS according to the remaining resources. The final result of the model is that generating a job execution sequence to make the provider gains the maximum revenue and improve the resource utilization. In the second part, the job scheduler launches scheduled jobs based on the optimal start time of the jobs. The resource scheduler allocates resources to launched jobs and the jobs gain resources from different nodes.

Fig. 1.
figure 1

The architecture of two-stage job scheduling model

Full size image

4 Model Implementation

4.1 SMR Algorithm

In the section, it is showed that the implementation of SMR algorithm and the definition of equations needed in the SMR algorithm.

Definition 4.

Initial Latest Start Time j i .T ols : the job is just completed at the deadline when the resources are not competed by other jobs. If the deadline of j i is j i .dl, the initial latest start time is following:

$$ j_{i} .T_{ols} = j_{i} .dl - \left( {\left\lceil {\frac{{j_{i} .ms}}{M}} \right\rceil \times J_{i} .mt + \left\lceil {\frac{{j_{i} .rs}}{M}} \right\rceil \times j_{i} .rt} \right). $$
(4)

Where M is the available resources of the platform. \( J_{i} .ms /M \) is the execution rounds of map tasks and \( J_{i} .rs /M \) is the execution rounds of reduce tasks.

The job j i is launched before \( J_{i} .T_{ols} \) to ensure the job completed before the deadline of the job when there is no resource contention. The job j i is not completed before j i .dl when j i is launched at \( J_{i} .T_{ols} \). Therefore, we should adjust the latest start time for jobs competing resources. To determine the time period for resource contention, we should pre-allocate the resources to all jobs based on the initial latest start time of the jobs. The total computing resources are counted at statistics time period. At different time period, different states may arise. There are two states, normal load state and overload state. The time period in the overload state is called the overload time period. In the overload time period, the resources of the platform could not meet the resource requirements of the jobs so as to delay the jobs eventually.

For the overload time period, we should design an optimal adjustment strategy to guarantee the maximum revenue. A feasible adjustment strategy set is defined as follows:

Definition 5.

A Set of Feasible Adjustment Strategy: there is a set of pre-scheduled jobs during the overload time period, called as Ju. A minimum proper subset is selected from Ju, \( Js\; \subseteq \;Ju \) . The initial latest start time is advanced for all jobs of Js so that the overload state turns into the normal state. Js is called as a feasible adjustment strategy. A set of feasible adjustment strategy during a certain period of time is \( CL = \left\{ {Js_{1} ,Js_{2} ,\; \ldots ,\;Js_{m} } \right\} \).

The latest start time of all jobs, in \( Js_{i} \) , is advanced to the overload state changing to the normal state. The latest start time for all jobs is changed as follows:

$$ j_{i} .T_{ls} = T_{cs} - \left( {\left\lceil {\frac{{j_{i} .ms}}{M}} \right\rceil \times J_{i} .mt + \left\lceil {\frac{{j_{i} .rs}}{M}} \right\rceil \times j_{i} .rt} \right). $$
(5)

Where \( j_{i} \, \in \,Js_{i} \), and \( T_{cs} \) is the start time of the overload time period.

During the overload time period, we should select a feasible adjustment strategy from CL. To guarantee the maximum revenue, the feasible adjustment strategies are evaluated for the revenues. Main factors of the evaluation are following two aspects, the Evaluation of the revenue (Sp) and adjustment cost.

$$ Sp = \left| {a - b} \right|. $$
(6)

Taking into account the two factors, the paper presents an evaluation function of the adjustment strategy based on the goal of the maximum revenue. The adjustment strategy of the minimum score is the optimal adjustment strategy. The evaluation function is following:

$$ Js_{i} .pf = \frac{{\sum\limits_{{m \in Js_{t} }} {j_{m} .Sp} }}{{\sum\limits_{m \in Ju} {j_{m} .Sp} }} \times lastsize. $$
(7)

Where \( \sum\nolimits_{{m \in Js_{t} }} {j_{m} .Sp} \) is the sum of the revenue valuation for all jobs in the adjustment strategy \( \sum\nolimits_{{m \in J_{u} }} {j_{m} .Sp} \) is the sum of the revenue valuation for pre-scheduled jobs during the overload time period and latesize is the ratio of the remaining resources of the current time period and the total resources of the platform.

The SMR algorithm is outlined in Algorithm 1. The initial latest start time is calculated for each job and the resources are pre-allocated to submitted jobs according the initial latest start time (lines 1–3). The resources are counted for every time period based on the result of pre-allocation, which is called as P_R (line 4). If the overload time period is existing, the initial latest start time are adjusted for all jobs during the overload time period (lines 5–21). The last overload time period is selected (line 6) and the set of pre-scheduled jobs is got during the overload time period (line 7). The adjustment strategies are evaluated by the evaluation function and the optimal adjustment strategy is selected (lines 8–13). The latest start time of the jobs in the adjustment strategy are adjusted and P_R is updated (lines 14–20). Looping through lines 6-20 until there is no overload time period in P_R.

figure a

4.2 SAS Algorithm

In the section, we should consider the pre-allocation of resources based on the SMR algorithm to ensure the jobs completed on time. To maximize the resource utilization of the platform, we evaluate the utilization of the computing resources using the rate of waste resources (W rr ). W rr is the ratio between the non-reusable resources W r after scheduling jobs and the sum of the currently used computing resources A r . The smaller the waste resource rate, the larger the resource utilization of the platform currently. W rr is following:

$$ W_{rr} = \frac{{W_{r} }}{{A_{r} }} $$
(8)

The SAS algorithm is presented in Algorithm 2. A set of jobs E j is found to be executed at T time and the required resources of E j do not conflict with pre-allocated resources (lines 2–8). If E j is not null, the job is selected that makes the waste resource rate minimize and the optimal start time of the job is T (lines 10–16). If E j is null, T is set to the start time of the next time period in P_R (line 18). Looping through lines 3–20 until every job has an optimal start time.

figure b

After the second adjustment scheduling, we adjust the initial scheduling sequence to reset the start time for each job. As the result of the second adjustment and scheduling, the resource utilization of the platform becomes larger and the actual start time and the completion time of the job are advanced. Many of the jobs discarded due to insufficient resources in the SMR algorithm have opportunity to be executed newly. Therefore, the revenue for the provider may get greater.

5 Performance Evaluation

5.1 Experiment Setup

Platform Configuration.

We experiment with the proposed algorithm in a big data platform based on MapReduce computing framework. The platform contains one master node and 20 workers that have identical configuration. The configuration information of the node is CPU 8 cores, 8 GB RAM, 1 TB hard disk, Red Hat Enterprise Linux 6.5, and Hadoop 2.7.1. Each Container is configured with 1 core and 2 GB RAM so that each node has 4 Containers and the platform has 80 Containers.

Performance Indicators.

In order to verify the effectiveness of the scheduling algorithm proposed in this paper, two - phase scheduling algorithm (TPS) is compared with FIFO and EDF Scheduler in the effect of different performances. We evaluate the algorithm using three indicators, the platform resource utilization (PRU), the job completion rate (JCR) and the total revenue (PR). The three indicators are as follows:

$$ PRU = \frac{{\sum\nolimits_{j = 1}^{k} {RE\_j_{i} } }}{TR}. $$
(9)

Where RE_j i is the occupied resources of the job i executed in the platform.

$$ JCR = \frac{n}{N}. $$
(10)

Where n is the number of the jobs completed before the deadlines and N is the number of all jobs submitted by users.

$$ PR = \sum\limits_{j = i}^{n} {a_{j} - \sum\limits_{i = 1}^{m} {b_{i} } } . $$
(11)

Where a j is the revenue when j is completed before the deadline, b i is the compensation to users when the completion time of i is more than the deadline of i and m is the number of jobs completed after the deadline (b i  < 0).

5.2 Experiment Results

In Fig. 2, the resource utilization rate is not affected by the job set size in three algorithms. The resource utilization of TPS is the highest in the three algorithms and the resource utilization of EDF is slightly lower than that of TPS. The resource utilization of FIFO is the lowest. As shown in Fig. 3, due to the limited computing capacity, the job completion rates are reduced in three algorithms when the job set size increases. Because EDF and TPS consider the deadlines and TPS also takes into account the revenues, EDF has a higher completion rate than TPS. FIFO only considers the revenues so the algorithm has the lowest completion rate.

Fig. 2.
figure 2

The effect of job set size on PRU

Full size image
Fig. 3.
figure 3

The effect of job set size on JCR

Full size image

From Fig. 4, the revenues show a tendency to increase first and then decrease when the size of the job set increases for three algorithms. However, when the number of jobs exceeds the computing capacity of the platform, the number of jobs completed on time is reduced so that the total revenue declines. From Fig. 5, it can be seen that the wastage rates of the three algorithms are roughly same and decrease with the increase of the number of computing resources. However, the resource utilization of EDF and TPS increase with the increase of the number of computing resources and the TPS rise is more than that of EDF.

Fig. 4.
figure 4

The effect of job set size on PR

Full size image
Fig. 5.
figure 5

The effect of resources on PRU

Full size image

Figures 6 and 7 show the effect of the number of computing resources on job completion rate and the total revenue. It can be seen from the figures that with the increase of the number of computing resources, the completion rates and the total revenues are increased by TPS and EDF and the revenue of TPS is much larger than that of EDF. The job completion rate and the total revenue increase with the increase of resources in FIFO when the resources of the platform are less. However, because the jobs are executed serially to waste a large amount of resources, the job completion rate and the revenue do not increase with the increase of the resources and are stabilized at a fixed value when the resources of the platform are larger than the average job input size.

Fig. 6.
figure 6

The effect of job set size on JCR

Full size image
Fig. 7.
figure 7

The effect of resources on PR

Full size image

6 Conclusion

The big data platform could serve multiple users at the same time. When users submitted jobs to the big data platform, jobs were reasonably scheduled not only to meet the requirements of users, but also improve the performance of the platform. Therefore, the two-stage job scheduling model was proposed for the jobs with the deadline constraints. In the model, the SMR algorithm calculated and adjusted the latest start time for every jobs based on the deadlines and the revenues of the jobs, which pre-allocated resources to jobs according to the result of adjustment to guarantee the jobs with the larger revenues to be completed before the deadlines. Under the premise of ensuring the maximum revenue, the SAS algorithm was developed to improve the resource utilization of the platform. Experimental results showed that the two-stage job scheduling model not only realized the maximum revenue of the provider, but improved the resource utilization of the platform. Moreover, the comprehensive performance of the platform was promoted.