Online Crowd Learning with Heterogeneous Workers via Majority Voting

Many platforms recruit workers through crowd-sourcing to finish online tasks involving a huge amount of effort (e.g., image labeling and content moderation). These platforms aim to incentivize heterogeneous workers to exert effort finishing the tasks and truthfully report their solutions. When the verification for the workers' solutions is absent, the crowdsourcing problem is challenging and is known as information elicitation without verification (IEWV). Majority voting is a common approach to solve an IEWV problem, where a worker is rewarded based on whether his solution is consistent with the majority. However, most prior related work relies on a strong assumption that workers' solution accuracy levels are public knowledge. We relax such an assumption and propose an online learning mechanism based on majority voting, which allows the platform to learn the distribution of the workers' solution accuracy levels. In the mechanism, workers will be asked to report their private accuracy levels (which do not need to be the true values), in addition to deciding their effort levels and solution reporting strategies. The mechanism computes the workers' rewards based on their reported accuracy levels, and the workers obtain rewards if their reported solutions match the majority. We show that our mechanism induces workers to truthfully report their solution accuracy levels in the long run, in which the platform asymptotically achieves zero regret. Moreover, we show that our online mechanism converges faster when the workers are more capable of solving the tasks.