CrowDIY: How to Design and Adapt Collaborative Crowdsourcing Workflows Under Budget Constraints

Abstract

Workflow quality is a key determinant of crowdsourcing complex work, but finding ways to task design and plan has proved illusive. Instead, we formulate it as an optimization problem with budget constraints and fewer decision variables to set. We propose a two-staged approach CrowDIY that can not only estimate task attributes based on previous tasks but also optimize them with budget constraints in order to publish tasks more wisely in a timely manner. Several experimental studies have been conducted, and the results show compelling evidence that, under different conditions, the proposed approach can effectively reduce the workload of workflow design and plan, while avoiding commonly encountered trial-and-error in crowdsourcing workflows and leading up to successful complex outcomes.

1 Motivation and Background

The dominant infrastructure in human computation systems today is workflow, which typically splits a business process into multiple microtasks and asks distinct workers to carry them out in pre-specified steps on services like Amazon’s Mechanical Turk (MTurk), CrowdFlower and CrowdSPRING [4]. There is little doubt that crowdsourcing workflows (CWs) are powerful because they build operational knowledge into software [2], allowing people around the world to work collaboratively and contribute meaningfully.

Though CW techniques pushed the boundary of crowsourcing [8], task requesters still need to program their own workflow or intervene continuously on the execution of their manmade workflow [9]. Task requesters need to make a variety of decisions regarding the task they want to submit [12]. To understand the complexity behind practical usage, we use the example of writing short essay about Dalian – a tourist city in China. Figure 1 shows the screenshot of a crowdsourcing workflow G₁ composed of eleven tasks with indexed numbers inside circles (denoted as T₁, T₂, …, T₁₁). Tasks are of specified types: question and answer (QA), choice, merge, notification, AND- and OR-node. To design G₁ with a graphical web UI on CrowDIY (Crowdsourcing - Do It Yourselves), the requester decomposes essay writing into several steps: (1) Puts a question to crowd for suggesting aspects to describe Dalian via QA node T₁, (2) Chooses three hot aspects via majority voting (T₂), (3) Asks crowd to write about the selected aspects (later bound to culture (T₃) and architecture (T₄) and transportation (T₅) at runtime), and then asks others to read and give their rates (T₆, T₇, and T₈ respectively), (4) Combine the content via a merge node T₉, (5) Makes his own decision via choice node T₁₀, and (6) T₁₁ to notify the completion. Figure 1 also shows the execution status of G₁ that started from T₁, ran through task nodes (denoted in green), steps into T₉ for merging, and will end in T₁₁ for notification.

Fig. 1.

The motivational example workflow G₁.

Provided with task decompositions (e.g. via a map-reduce paradigm [7], a divide-and-conquer strategy [6], and a customized policy [1]), the design and plan of workflows like G₁ are still not easy because at least information such as the task type and description, time effort, time allotted, the reward for which the worker actually booked the task, and the time it took from publishing to booking should be defined for each task at the design time. We address the problem of crowdsourcing workflow optimization (CWO), and propose a two-staged approach that can not only estimate attributes and parameters but also optimize them with budget constraints, and then publish tasks more wisely in a timely manner.

2 Proposed Approach

2.1 Problem Statement

A workflow can be characterized by a directed acyclic graph (DAG) G = (T, E) where the nodes T = {T₁, T₂, …, T_n} correspond to the tasks and the edges E indicate the data dependencies between tasks.

Definition 1

(Total Cost). The total cost of a workflow G = (T, E) is the sum of all task rewards, defined as:

$$ Cost\left( G \right) = \sum\nolimits_{\forall {T_i} \in T} {r_i} $$

(1)

Besides rewards, more attributes are associated with a task in a real crowdsourcing workflow; they are type, level of difficulty, effort to complete in terms of the number of time points, time allotted, reward, latest booking time for booking, earliest publishing time and buffer time. Concisely, each task T_i is characterized by T_i = ($typ{e_i}$, $lo{d_i}$, $et{c_i}$, $t{a_i}$, ${r_i}$, $lb{t_i}$, $ep{t_i}$, $b{t_i}$).

Definition 2

(Sequential and Parallel Execution). A sequential execution of a workflow G = (T, E) is a sequence of tasks sp = [T₁, T₂, …, T_n], such that T₁ is the initial task, T_n is the final task, and for every task T_i (1 ≤ i ≤ n):

• T_i is a direct successor of one of the tasks in sp.

• T_i is not a direct successor of any of the tasks in sp.

• There is no state T_j in sp such that T_j and T_i belong to two alternative branches of the workflow.

A parallel execution of a workflow G is a set pp(G) = {sp₁, sp₂, …, sp_m} of sequential executions of G such that all the parallel branches of every AND-node in sp_j = [T₁, T₂, …, T_n] (1 ≤ j ≤ m) are executed when that AND-node is entered. Formally,

• If T_i is the initial task of one of the parallel regions of an AND-node, then, for every other parallel region C, one of the initial tasks of C belongs to the set {T₁, …, T_i-1, T_i+1, …, T_n].

The second goal of the CW research is to manage business processes in terms of time, e.g. by controlling the estimated total execution time, which means the longest sequential execution path that covers all parallel regions.

Definition 3

(Estimated Total Execution Time) Let sp = [T₁, T₂, …, T_n] $\in$ pp(G) be any sequential execution of a workflow G. The estimated total execution time of G, denoted by ETime(G), is the maximum of ETime(sp):

$$ ETime(sp) = lb{t_1} + \sum\nolimits_{i = 1}^n {t{a_i}} $$

(2)

$$ ETime(G) = \mathop {\hbox{max} }\limits_{\forall sp \in pp(G)} ETime(sp) $$

(3)

The present research makes two extensions to the available CW studies: (1) A fewer task attributes (e.g. $typ{e_i}$ and $lo{d_i}$) are mandatory while others are optional. The mandatory part are set manually while the optional part can be defined by functions that take mandatory $lo{d_i}$ and historical task data as arguments. (2) We control the execution time by minimizing the overdue risk while ensuring the deadline and the cost budget. Next we offer an overview of our approach CrowDIY before formalizing them as the CWO problem.

2.2 CWO Formulation

Throughout this paper, time-related parameters and task attributes are supposed to be characterized in terms of time points t₀ (start time), t₁, t₂, …, t_D (deadline time) such that each t_i defines a point in time that i time slices have elapsed.

Definition 4

(Overdue Risk). The overdue risk of any task T_i with respect to start time t and buffer time bt is defined as:

$$ f(lod_{i} ,t,bt) = lod{}_{i} \cdot [\alpha_{2} (t + bt)^{2} + \alpha_{1} (t + bt) + \alpha_{0} ] $$

(4)

with weights $\alpha_{0}$, $\alpha_{1}$ and $\alpha_{2}$ $\in$ [0..1].

A CWO problem is to find a solution of task attributes with minimized overdue risk while not exceeding the deadline and the cost budget. There exist two solution scenarios: static assignment, in which lbts and tas of all tasks are set while aggregating estimated execution time in design phase, and dynamic assignment, in which epts and bts are set for initial tasks to be published while aggregating the estimated execution time of tasks not yet run.

Definition 5

(Static CWO Assignment). Let G = (T, E) be a workflow under design, R_max be the budget in score points, and D_max be the deadline in time points. A static CWO assignment is to find: for each task T_i in sp = [T₁, T₂, …, T_n] $\in$ pp(G) (1 ≤ i ≤ n), the $lbt_{i}$ and the $ta_{i}$ that, minimize

$$ \sum\nolimits_{{\forall T_{i} \in sp}} {f(lod_{i} ,lbt_{i} ,ta_{i} )} $$

subject to

$$ ta_{i} \, < \,lbt_{i} - lbt_{i - 1} (i \ge 2) $$

(5)

$$ Cost({\text{G}}) \le R_{max} $$

(6)

$$ ETime({\text{G}}) \le t_{{D_{max} }} $$

(7)

Note that the real execution time of tasks may be different from what was estimated. Let T_C $\subseteq$ T be tasks that have already completed so far, and E_C = {<T₁, T₂> | $\forall$ T₁, T₂ $\in$ T_C, <T₁, T₂> $\in$ E} be edges that have already been covered. We separate G into two subgraphs: the completed part G_C = (T_C, E_C), and the part not completed $\overline{\text{G}}_{C}$ = (T−T_C, E−E_C).

Definition 6

(Dynamic CWO Assignment). Let T_C = {T₁, T₂, …, T_C} be tasks G = (T, E) of that have already completed at time point t_C, and G = G_C $\cup$ $\overline{\text{G}}_{C}$, and In($\overline{\text{G}}_{C}$) = {T_i | T_i is the initial task of any sequential execution sp $\in$ pp($\overline{\text{G}}_{C}$)}. Let R_max be the budget in terms of score points, and D_max be the deadline in terms of the number of time points. A dynamic CWO problem is to find: for each task T_s $\in$ In($\overline{\text{G}}_{C}$), and for each task T_i $\in$ sp $\in$ pp($\overline{\text{G}}_{C}$) (i ≠ s), the ept_s, the bt_s, the $lbt_{i}$, and the $ta_{i}$ that, minimize

$$ \sum\nolimits_{{\forall T_{i} \in sp}} {f(lod_{i} ,lbt_{i} ,ta_{i} )} + \sum\nolimits_{{\forall T_{s} }} {f(lod_{s} ,ept_{s} ,bt_{s} )} $$

subject to

$$ t_{C} \le ept_{s} \, < \,lbt_{s} (\forall T_{s} \in In(\overline{\text{G}}_{C} )) $$

(8)

$$ ta_{s} \, \le \,bt_{s} (\forall T_{s} \in In(\overline{\text{G}}_{C} )) $$

(9)

$$ ta_{i} \, < \,lbt_{i} - lbt_{i - 1} (i \ge 2) $$

(10)

$$ Cost({\text{G}}) + Cost(\overline{\text{G}}_{C} )\, \le \,R_{max} $$

(11)

$$ ETime(\overline{\text{G}}_{C} )\, \le \,t_{{D_{max} }} $$

(12)

2.3 Solution Algorithms

Algorithm 1 depicts the overall procedure of CrowDIY, which starts from Task, max reward R_max, max deadline D_max to perform workflow design and revise (Step 1), planning (Steps 2–4) and publishing (Steps 7–10) remained tasks to crowd workers until all tasks are finished or the Dynamic CWO has no solution.

Design(Task, R_max, D_max) means that the requester can design a CW via the Web UI in several steps: decompose complex tasks into small ones by calling divide(Task), place a choice node for selecting answers, manage task dependencies and structure (AND-node or OR-node), later combine the results into a coherent solution via merge node, and finalize with a notification node. Design can be extended recursively or revised repeatedly by Algorithm 1 (from step 1 to 6). As described by Algorithm 2, Transform(t_C, $\overline{\text{G}}_{C}$, R_max, D_max) instantiates constraints Eqs. (8)–(12) and the overdue risk function Eq. (4) for $\overline{\text{G}}_{C}$ at current time t_C.

3 Evaluation and Results

We implemented the solution method in a crowdsourcing workflow system CrowDIY in Python [5], running on the Django Web Framework with SQlite and other tools for solving the CWO problem and generating workflow. To solve the CWO problem, CrowDIY integrates Gurobi, Cplex and Choco through constraint programing in Java in order to find static and dynamic CWO assignments. We set weights of Eq. (4) with $\alpha_{0}$ = 0.25, $\alpha_{1}$ = 0.4, and $\alpha_{2}$ = 0.5.

Workflows were generated with JGraphT–a Java library of graph theory data structures and algorithms [10], and mandatory attributes such as node type and level of difficulty are generated uniformly in random. We vary the number of workflows from 1 to 500 while the number of tasks in every workflow is in [6..20]. We assumed that there were 3000 workers and task attributes were generated. For every task type, we generated other task attributes that are linearly dependent on task difficulty as we did in case studies. Also 300 workers were assigned the least time allotted to finish a task and the minimum acceptable reward, which were generated using the normal distribution based on the average reward, average allotted time and their allowable deviation parameters. So we prepared a large number of different workflows with randomizing workflow structures and diverse deadlines, and tasks in them have various allotted times and booking times.

The number (#W) of workflows ranges from 1 to 500, and each is compared with the reference case #W = 1. First, we guess the max deadline D_max for every workflow in every case. If the manmade D_max does not make sense, there is no solution to the CWO formulation of the workflow under consideration. So we can count the number of trial-and-error (#E) of CWO solving. If D_max makes sense, then we guess the max reward R_max. If the manmade R_max works, constraint solvers return the overdue risk and their execution time (#T) in seconds. In particular, #OR indicates the multiple of 329.7 or 722.1, namely the overdue risk of the reference case #W = 1. If R_max is implausible, “no solution” means that, at design time the workflow is found more likely to “fail” because it requires the deadline extension. So we compare the time extension (#X) in time points raised by failed workflows. The more time extension failed workflows require, the better solution the constraint solver can ensure. All metrics we used are reported on average for all the workflows we prepared.

The first experiment is to find the best solver for workflow plan (i.e. static CWO assignment). The comparison results were summarized in Table 1. It can be seen that as #W grows, their performance present the trend of linear growth under four metrics. Also we can see that the performance of Gurobi and Cplex are similar in #E, #OR, and #X. But Gurobi is much better than Cplex in terms of #T. So we choose Gurobi to conduct the rest experiments.

Table 1. Results from comparative constraint solvers.

The second experiment is to verify whether buffer time influences the final outcome of all workflows in the task publishing algorithm with a linear dependence $bt_{\text{s}} = x \cdot ta_{s} \,\,\left( {x \in \left\{ {0.2,\,\,0.5,\,\,1,\,\,2,\,\,3,\,\,4,\,\,5,\,\,6} \right\}} \right)$. It can be seen from Fig. 4 that the optimal results have achieved the minimum value when buffer time is almost equal to its allotted time. In case of smaller buffer time, for example $x$ = 0.2 and $x$ = 0.5 (x-axis), more tasks were not booked on time, so the reward to workers should be raised. At the same time, lack of time also increase the possibility of missing deadlines. That is why three metrics (#OR, #X and #E) have higher values. If the buffer time is larger, for instance coefficient x $\in$ [2..6], the values of three metrics are higher than the optimal results, but still much lower than the buffer time. This is because tasks can be booked earlier by workers.

Fig. 4.

Finding the most appropriate buffer time.

4 Conclusion and Future Work

The present approach eases the complexity behind collaborative crowdsourcing process, but dynamic approach to publishing cannot guarantee the time constraints because there is a lot of uncertainty in crowdsourcing, especially the anonymous people with uncertain skills and commitments. What merits future investigation includes advancing the training of Estimator and the control of workflow, and exploiting statistical sampling of people from the crowd after they contributed meaningfully in previous tasks [3, 11].