TY - GEN
T1 - Investigating the dynamic decision mechanisms of users’ Relevance judgment for information retrieval via log analysis
AU - Su, Yi
AU - Li, Jingfei
AU - Song, Dawei
AU - Zhang, Pengqing
AU - Zhang, Yazhou
N1 - Publisher Copyright:
© Springer Nature Switzerland AG 2018.
PY - 2018
Y1 - 2018
N2 - Measuring relevance of documents with respect to a user’s query is at the heart of information retrieval (IR), where the user’s relevance judgment criteria have been recognized as multi-dimensional. A set of relevance dimensions that are considered as critical factors in document relevance judgment have been investigated, such as topicality, novelty, and reliability. However, most existing work focuses on individual relevance dimensions, yet neglecting how different dimensions would interact with each other to influence the overall relevance judgment in real-world search scenarios. This paper aims at an initial step to fill the gap. Specifically, we divide 7 relevance dimensions in an enriched Multidimensional User Relevance Model (MURM) into three categories according to three main requirements for document relevance, i.e., document content requirement, document quality requirement and personalization requirement. We then exploit the Learning to Rank framework to conduct document ranking experiments on a query log dataset from a prominent search engine. The experimental results indicate the existence of an order effect between different dimensions, and suggest that considering different dimensions across categories in different orders for document relevance judgment could lead to distinct search results. Our findings provide valuable insights to build more intelligent and user-centric information retrieval systems, and potentially benefit other natural language processing tasks that involve decision making from multiple perspectives.
AB - Measuring relevance of documents with respect to a user’s query is at the heart of information retrieval (IR), where the user’s relevance judgment criteria have been recognized as multi-dimensional. A set of relevance dimensions that are considered as critical factors in document relevance judgment have been investigated, such as topicality, novelty, and reliability. However, most existing work focuses on individual relevance dimensions, yet neglecting how different dimensions would interact with each other to influence the overall relevance judgment in real-world search scenarios. This paper aims at an initial step to fill the gap. Specifically, we divide 7 relevance dimensions in an enriched Multidimensional User Relevance Model (MURM) into three categories according to three main requirements for document relevance, i.e., document content requirement, document quality requirement and personalization requirement. We then exploit the Learning to Rank framework to conduct document ranking experiments on a query log dataset from a prominent search engine. The experimental results indicate the existence of an order effect between different dimensions, and suggest that considering different dimensions across categories in different orders for document relevance judgment could lead to distinct search results. Our findings provide valuable insights to build more intelligent and user-centric information retrieval systems, and potentially benefit other natural language processing tasks that involve decision making from multiple perspectives.
KW - Information retrieval
KW - Order effect
KW - User relevance dimensions
UR - http://www.scopus.com/inward/record.url?scp=85051935156&partnerID=8YFLogxK
U2 - 10.1007/978-3-319-97304-3_74
DO - 10.1007/978-3-319-97304-3_74
M3 - Conference contribution
AN - SCOPUS:85051935156
SN - 9783319973036
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 968
EP - 979
BT - PRICAI 2018
A2 - Kang, Byeong-Ho
A2 - Geng, Xin
PB - Springer Verlag
T2 - 15th Pacific Rim International Conference on Artificial Intelligence, PRICAI 2018
Y2 - 28 August 2018 through 31 August 2018
ER -