1 Introduction

The concept of crowdsourcing is first coined in 2006 and simply means outsourcing certain tasks and problem formulations to an undefined (and generally large) network of people in the form of open calls [1]. With today’s development of Internet and mobile technologies, and the explosion of social media, companies are able to have a better engagement of distributed crowds of individuals for their innovation and problem-solving needs [2]. As a result, an increasing number of companies, ranging from small startups to those listed in Fortune 500, are trying to make use of crowdsourcing to access knowledge and skills that previously unavailable to them and to solve parts of business processes formerly executed in-house [3]. In this way, companies can have a more flexible workforce and higher knowledge absorptive capacity and business processes can be adapted on-demand, which results in higher level of process flexibility [4].

Crowdsourcing can be considered as an online and distributed problem-solving model [5] and suggests that engaging crowds can help companies develop solutions to a variety of business challenges. As the business challenges and tasks vary, so do the knowledge and skills that crowdsourcing participants have. Unlike simple and low-priced tasks that commonly require general skills, IT service tasks are knowledge intensive and require crowdsourcing participants with special skills and knowledge. This makes the identification of suitable participants a challenge. Matching skills to tasks often relays on sophisticated online crowdsourcing platforms to manage distributed workers and support task providers [2]. However, prior research on crowdsourcing focuses on its business models [6, 7] or brand-related and marketing alike activities [8, 9], taken the online crowdsourcing platforms and their functionalities as given [10]. The state of the art in crowdsourcing practice still lacks approaches for automated estimating participants considering their skills and knowledge [11].

In this paper, we propose a method of AHP-TOPSIS based on Grey Relation Analysis to help companies estimate and identify the best-fit participant for their IT service crowdsourcing tasks. The method is tested on a dataset of 348 completed crowdsourcing tasks in the IT service domain. A post-hoc analysis on the matching between the estimation result and the actual decision made by task providers reveals the accuracy and efficiency of this method.

2 Research Context

2.1 Forms of Crowdsourcing

The way of using crowdsourcing to abstain flexible workforce for business process execution is similar to cloud computing where computing capacity is provided on demand [11, 12]. A typical form of crowdsourcing is publishing the request for proposals through an online marketplace platform with the details of the needed service and its expected duration and (a range of) cost. Then potential participants bid on the task by submitting their proposals. Although many proposals would be received for a task, only the best-fit candidate will be selected to carry out the task. At the end, company can decide to accept and pay for the work, or refuse it if it does not fulfil the expectation. This marketplace form of crowdsourcing enables companies to access the vast potential of workforce with various backgrounds, while it has more flexibility and less risk than having a fixed outsourcing contract [13].

There are also other forms of crowdsourcing such as knowledge contributions (e.g. Wikipedia), rating (i.e. participants ‘vote’ on a given topic) and micro-task (e.g. Galaxy Zoo) where participants complete the task voluntarily. In addition, contest-based crowdsourcing is used for obtaining innovative ideas or solutions, in which all the participants make their effort and results are determined on a comparative basis and probably only the top contributor(s) would be reworded. Those forms of crowdsourcing are out of the scope of this paper, as they are less effective in providing business process flexibility.

2.2 Related Work

Estimation issue in crowdsourcing has been observed by researchers, and there are some studies for automatically estimating different submissions to find out those with sufficient quality for a task. For example, Tarasov et al. [14] proposed a dynamic estimation of worker reliability in rating-based crowdsourcing. This approach is for detecting noisy and incompetent workers by estimating their submissions, instead of estimating the workers before the task was taken into execution.

Mechanisms for estimating crowdsourcing participants for business process execution can be found in the research of BPEL4People in social networks [15], in which a ranking method based on Hyperlink-Induced Topic Search (HITS) algorithm is provided to estimate the expertise of works in a social network. In this method, a certain skill, its expected level and the importance of a task are used as input, and the ranking result presents a list of all the suitable crowdsourcing works in a social network. The underlying concept of BPEL4People is that the flexibility of traditional SOA-based business process systems can be enhanced by enabling human-based services with very the same API used by software-based Web services. In this way, tasks would be able to match to suitable workers that are registered and active on the crowdsourcing social platform [11].

However, not all crowdsourcing platforms take the form of social networks where crowdsourcing participants work with each other in a joint task and some of them would take the role of supervisor or coordinator. Instead, many crowdsourcing platforms have a form of marketplaces, where a task is completed by only one participant exclusively. In this case, many candidates will compete for the same task by submitting their proposals and the task provider has to choose the best-fit one from them. The more candidates a task has, the more difficult for the task provider to manually estimate and identify the best-fit participant, because of information overload. It is therefore important to have an estimation method to rank all the candidates for the task provider to choose. And a reliable estimation should be a necessary functionality of online marketplace crowdsourcing platforms to support task providers for quality control and solving the problem of information overload.

2.3 AHP, TOPSIS and GRA

Selecting the best-fit crowdsourcing participant from a group of candidates is a typical Multiple Attribute Decision Making (MADM) problem [16] in which decision-making is for selecting the most appropriate one from many feasible solutions. Analytical Hierarchical Process (AHP) [17] is one of the most outstanding MADM methods, which first estimates the relationship among criteria weight and then the total value of each choice based on the obtained weight [18]. The Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) is another outstanding MADM method, which is based on the concept that the best choice should have the shortest Euclidean distances from the positive ideal and the farthest from the negative ideal [16]. AHP and TOPSIS can be used in combination [19] where AHP is used to calculate the weights of the parameters and these weights are later used in TOPSIS.

The drawback of TOPSIS method is its linear variation of each alternatives, which cannot provide an accurate ranking between two alternatives that have the same distances to the ideal. This problem can be solved by Grey Relational Analysis (GRA) which is an effective method to solve decision making problems by generalizing estimates under limited samples and uncertain conditions [20]. GRA is a kind of flexible measurement of curve similarity. By using GRA, the nonlinear relationship between the sequences of each alternative can be well reflected, which can compensate the inaccuracy problem of TOPSIS method.

AHP-TOPSIS based on GRA has been proved to be useful in solving MADM problems [21]. In this study, it is employed for estimating crowdsourcing participants for given tasks.

3 Estimation Method

In this section a method of AHP-TOPSIS based on GRA is proposed for estimating crowdsourcing participants. This method has the following three phases.

3.1 Phase 1: Identifying Estimation Parameters

The estimation parameters used by an algorithm-based method should be quantitative, otherwise they cannot be calculated. In addition, the data of parameters should be easy to access, otherwise the desired automation in the ranking of participants cannot be achieved. In this study, there are two underlying assumptions. The first one is that task providers and candidate participants do not know each other in actual life. This means that a task provider makes its selection decision only based on the related candidate participants’ information that is available online. The second one is that task providers will insist on looking for the best-fit participant rather than shifting to other strategy like choosing the first acceptable candidate. This means that all related candidate participants should be involved in the consideration during the decision-making. In a typical marketplace crowdsourcing model, participants’ online information comes from either their online profiles or the proposals that they submitted to the task. Both these two sources of information are involved in the formulation of the estimation parameters in this study. Afterwards, the parameters that cannot be quantitated has to be ignored, and the parameters that reflect the same property are merged. At the end, parameters are categorized into benefit parameters (the larger the value is, the better the solution is) and cost parameters (the smaller the value is, the better the solution is).

3.2 Phase 2: Using AHP to Calculate the Weight of Parameters

In AHP, the multi-attribute weight measurement is calculated via pair-wise comparison of the relative importance of two factors. Assuming that there are N number of decision parameters, denoted as (P 1, P 2, …, P n ), its judgment matrix would be A = [a n ], in which a n represents the relative importance of P 1 and P 2. Using the row vector average normalization proposed by Satty [17], the weight of P i is calculated as:

$$ W_{i} = \frac{{\left( {\prod\limits_{j = 1}^{n} {a_{ij} } } \right)^{{\frac{1}{n}}} }}{{\sum\limits_{i = 1}^{n} {\left( {\prod\limits_{j = 1}^{n} {a_{ij} } } \right)}^{{\frac{1}{n}}} }}i,j = 1,2, \ldots ,n. $$

3.3 Phase 3: Using GRA-Based TOPSIS to Estimate Participants

In this phase the algorithm has the following ten steps.

  1. 1.

    Normalizing of Initial decision matrix \( X = \left( {x_{ij} } \right)_{m \times n} \), get the normalization matrix \( Z = \left( {z_{ij} } \right)_{m \times n} \). \( (i = 1,2, \ldots ,m;j = 1,2, \ldots n \)

For benefit parameters:

$$ Z_{ij} = \frac{{x_{ij} - \mathop {\hbox{min} }\limits_{i} x_{ij} }}{{\mathop {\hbox{max} }\limits_{i} x_{ij} - \mathop {\hbox{min} }\limits_{i} x_{ij} }} $$
(1)

For cost parameters:

$$ Z_{ij} = \frac{{\mathop { \hbox{max} }\limits_{i} x_{ij} - x_{ij} }}{{\mathop {\hbox{max} }\limits_{i} x_{ij} - \mathop {\hbox{min} }\limits_{i} x_{ij} }} $$
(2)
  1. 2.

    Calculating the weighted decision matrix \( S = \left( {s_{ij} } \right)_{m \times n} \), \( (i = 1,2, \ldots ,m;j = 1,2, \ldots n) \).

$$ S_{ij} = w_{ij} z_{ij} $$
  1. 3.

    Calculating the positive ideal solution \( S^{ + } \) and negative ideal solution \( S^{ - } \)

$$ S^{ + } = \left( {s_{1}^{ + } ,s_{2}^{ + } , \ldots ,s_{n}^{ + } } \right);S^{ - } = \left( {s_{1}^{ - } ,s_{2}^{ - } , \ldots ,s_{n}^{ - } } \right) $$

Where \( s_{j}^{ + } = \mathop {\hbox{max} }\limits_{i} s_{ij} = w_{j} ;s_{j}^{ - } = \mathop {\hbox{min} }\limits_{i} s_{ij} = 0,i = 1,2, \ldots ,m;j = 1,2, \ldots ,n. \)

  1. 4.

    Calculating the Euclidean distance between each solution and positive/negative ideal solution \( d_{i}^{ + } \), \( d_{i}^{ - } \)

$$ d_{i}^{ + } = \sqrt {\sum\limits_{j = 1}^{n} {\left( {s_{ij} - s_{j}^{ + } } \right)^{2} } } ,d_{i}^{ - } = \sqrt {\sum\limits_{j = 1}^{n} {\left( {s_{ij} - s_{j}^{ - } } \right)^{2} } } $$

where i = 1,2, ... ,m;j = 1,2, ... ,n.

  1. 5.

    Calculating the grey relation coefficient matrix of each solution and positive/negative ideal solution \( L^{ + } \), \( L^{ - } \):

$$ L^{ + } = \left( {l_{ij}^{ + } } \right)_{m \times n} ,L^{ - } = \left( {l_{ij}^{ - } } \right)_{m \times n} $$
$$ {\text{Where}}\quad l_{ij}^{ + } = \frac{{\mathop {\hbox{min } }\limits_{i} \mathop {\hbox{min } }\limits_{j} \left| {s_{j}^{ + } - s_{ij} } \right| + \theta\,\mathop {\hbox{max } }\limits_{i} \mathop {\hbox{max } }\limits_{j} \left| {s_{j}^{ + } - s_{ij} } \right|}}{{\left| {s_{j}^{ + } - s_{ij} } \right| + \theta \,\mathop {\hbox{max } }\limits_{i} \mathop {\hbox{max } }\limits_{j} \left| {s_{j}^{ + } - s_{ij} } \right|}} $$
$$ l_{ij}^{ - } = \frac{{\mathop {\hbox{min } }\limits_{i} \mathop {\hbox{min } }\limits_{j} \left| {s_{j}^{ - } - s_{ij} } \right| + \theta\,\mathop {\hbox{max } }\limits_{i} \mathop {\hbox{max } }\limits_{j} \left| {s_{j}^{ - } - s_{ij} } \right|}}{{\left| {s_{j}^{ - } - s_{ij} } \right| + \theta\,\mathop {\hbox{max } }\limits_{i} \mathop {\hbox{max } }\limits_{j} \left| {s_{j}^{ - } - s_{ij} } \right|}} $$

Where \( \theta \in \left( {0,1} \right) \), is distinguishing coefficient, Here the value of \( \theta \) is set to be 0.5.

$$ {\text{Simplifying}}\,{\text{the}}\,{\text{formulas}}:L_{ij}^{ + } = \frac{\theta }{{\left| {z_{ij} - 1} \right| + \theta }};L_{ij}^{ - } = \frac{\theta }{{\left| {z_{ij} + 1} \right| + \theta }} $$
  1. 6.

    Calculating the grey relation grade of each solution and positive/negative ideal solution \( l_{i}^{ + } \), \( l_{i}^{ - } \):

$$ l_{i}^{ + } = \frac{1}{n}\sum\limits_{j = 1}^{n} {l_{ij}^{ + } } ;l_{i}^{ - } = \frac{1}{n}\sum\limits_{j = 1}^{n} {l_{ij}^{ - } } $$
  1. 7.

    Applying nondimensionalization to \( d_{i}^{ + } ,d_{i}^{ - } ,l_{i}^{ + } \) and \( l_{i}^{ - } \)

$$ \begin{aligned} D_{i}^{ + } & = \frac{{d_{i}^{ + } }}{{\mathop {\hbox{max} }\limits_{i} d_{i}^{ + } }},D_{i}^{ - } = \frac{{d_{i}^{ - } }}{{\mathop {\hbox{max} }\limits_{i} d_{i}^{ - } }} \\ L_{i}^{ + } & = \frac{{l_{i}^{ + } }}{{\mathop {\hbox{max} }\limits_{i} l_{i}^{ + } }},L_{i}^{ - } = \frac{{l_{i}^{ - } }}{{\mathop {\hbox{max} }\limits_{i} l_{i}^{ - } }} \\ \end{aligned} $$

Where \( i = 1,2, \ldots ,m \)

  1. 8.

    Calculating the relative closeness degree:

$$ P_{i}^{ + } = \frac{{D_{i}^{ + } }}{{D_{i}^{ + } + D_{i}^{ - } }},U_{i}^{ + } = \frac{{L_{i}^{ + } }}{{L_{i}^{ + } + L_{i}^{ - } }} $$
  1. 9.

    Combining \( P_{i}^{ + } \) and \( U_{i}^{ + } \): \( Q_{i}^{ + } = v_{1} P_{i}^{ + } + v_{2} U_{i}^{ + } \)

Where \( v_{1} \) and \( v_{2} \) reflect the degree of preference of decision makers, \( v_{1} + v_{2} = 1 \), and \( v_{1} = v_{2} = 0.5 \)

  1. 10.

    Sorting solutions by the value of \( Q_{i}^{ + } \). The better \( Q_{i}^{ + } \) is, the better the solution is, and vice versa. \( { \hbox{max} }(Q_{i}^{ + } ) \) is the final decision.

4 Experiment

In order to evaluate the proposed method of AHP-TOPSIS based on GRA on its accuracy and efficiency in estimating crowdsourcing participants, an experiment was carried out on data from a popular Chinese crowdsourcing marketplace platform, Epweike (http://www.epweike.com/). This experiment used the proposed method to estimate crowdsourcing participants of certain tasks, and then the estimation result was compared with the actual selection decision made by the task provider. This comparison allows for an analysis on the accuracy of the method and the impact of the number of candidates on the actual decision-making which reflects the efficiency of this method.

4.1 Dataset

In this experiment, a dataset of 348 valid and completed tasks between 2010 and 2015 for IT services crowdsourcing is used. The content of those task includes software/mobile application development, website construction, database and system design, server maintenance, etc. In those tasks the number of openly visible proposals is more than 3. Tasks that have less than 4 proposals are ignored, because the GRA-based TOPSIS algorithm does not return a meaningful ranking result when the number of candidates is less than 4.

4.2 Approach

Phase 1: Identifying Estimation Parameters.

The following 10 parameters of crowdsourcing participants are identified for the estimation, and the data of these parameters can be accessed from the website openly (Table 1).

Table 1. The description of each parameter
Full size table

These parameters describe the information of either the participant itself or the proposal it provided. Among those parameters, P1, P2, P3, P4, P5, P9 and P10 are benefit parameters; while P6, P7 and P8 are cost parameters.

Phase 2: Using AHP to Calculate the Weight of Parameters.

By AHP, the weight of the 10 parameters is calculated:

$$ \begin{aligned} & w_{i} = ( 0. 1 4 0 9 1 1 2 2 7, 0. 1 9 4 9 1 7 1 6 8, 0. 0 5 1 3 7 7 5 0 8, 0. 0 4 1 0 4 7 8 9 2 , 0. 0 3 8 4 9 1 2 2 7, \\ &\quad\quad 0. 0 3 0 8 3 6 4 6 2 , 0. 2 5 5 8 5 7 8 3 5, 0. 2 0 9 3 3 8 2 4 4, 0. 0 1 6 2 2 3 4 0 2, 0. 0 2 0 9 9 9 0 3 5 ).\\ \end{aligned} $$

Phase 3: Using GRA-based TOPSIS to Estimate Participants.

For space reason, only the calculation of Task1 is given as an example. This task has 23 candidate participants and 18 openly available proposals. The other 5 proposals are closed and only visible to the task provider, and therefore those 5 participants are not taken into account in the experiment. The best one is identified from the rest 18 candidates by using the GRA-based TOPSIS method. Through the estimation, the final value of \( Q_{i}^{ + } \) is presented in the following Table 2.

Table 2. Value of \( Q_{i}^{ + } \) for each participant of Task 1
Full size table

By descending the order of \( Q_{i}^{ + } \), the sort of participants of Task 1 can be get.

According to the result presented in Table 3, the best candidate for Task 1 is Participant 20. The values of each of its parameters are:

$$ P_{20} = \left( { 1 7000, 100\;\% , 4, 4, 40, 5, 800, 5, 1 2 6 2 4, 1} \right) $$
Table 3. The ranking of participants of Task 1
Full size table

But the subjective choice made by the task provider is Participant 1, and the values of its parameters are:

$$ P_{1} = \left( { 9 200, 100\;\% , 4, 3, 3 9, 6, 500, 2, 2 5 8 8, 1} \right) $$

In this example, the estimation result is not matching with the actual selection.

In this experiment, Phase 3 was repeated on all the 348 tasks. Then the result of the estimation was compared with the actual selection made by the task provider to find out whether they are matchable. The latter is openly available on the crowdsourcing website.

4.3 Result Analysis

In order to evaluate the accuracy of the proposed method, the result of estimation is compared with the actual selection decision made by task providers. A metric of matching rate is used to indicate the percentages of matchable tasks. In this experiment, the overall matching rate is 88.22 % (307 out of 348 tasks). The matching rates under different number of participants are presented in the following table.

The matching rates presented in Table 4 indicate that the proposed method has a high accuracy when the number of participants is between 4 and 10. There are 248 tasks (71.26 % of the total 348 tasks) fall into this range and the matching rates are above 90 %. Specially, in the simplest situation where only 4 participants competing a task, the estimation result is 100 % matched with the manual decision-making. This means the proposed method can achieve an estimation result that is very similar with the manual decision-making, when the manual decision-making is simple and the information overload problem does not appear.

Table 4. Matching rates under different number of participants
Full size table

Technically speaking, the accuracy of this method will not be influenced by the number of participants. However, the matching rates generally declines along with the increase of participants. A possible explanation is that when the number of participants increase, it is more difficult to manually identify the best-fit participant. The task provider might have to rely on its own experience or even instinct in the decision-making for participant selection rather than an objective comparison between candidates. When the number of participants is more than 50, manually identifying the best-fit participant becomes very difficult and the actual selection result deviates very much from the estimation result, which results in matching rates below 33.33 %. The following curve demonstrate the decline of the matching rate. Under the assumption that the accuracy of manual decision-making is mainly and negatively influenced by information overload, the proposed method could solve this problem and improve the decision-making for crowdsourcing tasks with a large number of participants. To solidly prove this statement, investigation on the factors that impact manual decision-making for selecting crowdsourcing participants is desired in the future (Fig. 1).

Fig. 1.
figure 1

The matching rates decline along with the increase of participants

Full size image

5 Conclusions and Future Work

Companies want to adopt crowdsourcing to obtain on-demand IT services and enable flexible business processes. The challenge to overcome the information overload problem in estimating and identifying the best-fit crowdsourcing participant out of a large group of candidates. This paper presents a method of AHP-TOPSIS based on GRA for estimating crowdsourcing participants. The proposed method has been tested on a dataset of 348 valid and completed IT service crowdsourcing tasks from a Chinese crowdsourcing online marketplace platform. For the tasks with a small number of candidates, a matching between the estimation result and the actual selection made by task providers proofs the accuracy of the proposed method. Although the matching rate generally declines along with the increase of the number of candidates, this reflects the information overload problem in manual decision-making. The proposed method could solve this problem by providing an accurate and objective estimation which is uninfluenced by the number of participants. Employing this method allows task providers to quickly and easily identify the best-fit crowdsourcing participant. Furthermore, this method facilitates the design of a software agent for crowdsourcing platforms to rank the participants of a task automatically.

In the future work, it would be interesting to contact the task providers who had to make decision with a large number of candidates, especially those chose a participant other than the one recommended by the proposed method. Then invite them to interviews or surveys with questions such as how long it took them to make a decision on selecting the best-fit participant, what are the factors impacting their decision, and whether they think this method would help them in overcoming the information overload problem.