Investigating the dynamic decision mechanisms of users’ Relevance judgment for information retrieval via log analysis

Measuring relevance of documents with respect to a user’s query is at the heart of information retrieval (IR), where the user’s relevance judgment criteria have been recognized as multi-dimensional. A set of relevance dimensions that are considered as critical factors in document relevance judgment have been investigated, such as topicality, novelty, and reliability. However, most existing work focuses on individual relevance dimensions, yet neglecting how different dimensions would interact with each other to influence the overall relevance judgment in real-world search scenarios. This paper aims at an initial step to fill the gap. Specifically, we divide 7 relevance dimensions in an enriched Multidimensional User Relevance Model (MURM) into three categories according to three main requirements for document relevance, i.e., document content requirement, document quality requirement and personalization requirement. We then exploit the Learning to Rank framework to conduct document ranking experiments on a query log dataset from a prominent search engine. The experimental results indicate the existence of an order effect between different dimensions, and suggest that considering different dimensions across categories in different orders for document relevance judgment could lead to distinct search results. Our findings provide valuable insights to build more intelligent and user-centric information retrieval systems, and potentially benefit other natural language processing tasks that involve decision making from multiple perspectives.