Value at risk for risk evaluation in information retrieval

In Information Retrieval (IR), evaluation metrics continuously play an important role. Recently, some risk measures have been proposed to evaluate the downside performance or the performance variance of an assumingly advanced IR method in comparison with a baseline method. In this paper, we propose a novel risk metric, by applying the Value at Risk theory (VaR, which has been widely used in financial investment) to IR risk evaluation. The proposed metric (VaR IR) is implemented in the light of typical IR effectiveness metrics (e.g. AP) and used to evaluate the participating systems submitted to Session Tracks and compared with other risk metrics. The empirical evaluation has shown that VaR IR is complementary to and can be integrated with the effectiveness metrics to provide a more comprehensive evaluation method.