TY - GEN
T1 - Investigating order effects in multidimensional relevance judgment using query logs
AU - Uprety, Sagar
AU - Song, Dawei
N1 - Publisher Copyright:
© 2018 Association for Computing Machinery.
PY - 2018/9/10
Y1 - 2018/9/10
N2 - There is a growing body of research which has investigated relevance judgment in IR being influenced by multiple factors or dimensions. At the same time, the Order Effects in sequential decision making have been quantitatively detected and studied in Mathematical Psychology. Combining the two phenomena, there have been some user studies carried out which investigate the Order Effects and thus incompatibility in different dimensions of relevance. In this work, we propose a methodology for carrying out such an investigation in large scale and real world data using query logs of a web search engine, and device a test to detect the presence of an irrational user behavior in relevance judgment of documents. We further validate this behavior through a Quantum Cognitive explanation of the Order and Context effects.
AB - There is a growing body of research which has investigated relevance judgment in IR being influenced by multiple factors or dimensions. At the same time, the Order Effects in sequential decision making have been quantitatively detected and studied in Mathematical Psychology. Combining the two phenomena, there have been some user studies carried out which investigate the Order Effects and thus incompatibility in different dimensions of relevance. In this work, we propose a methodology for carrying out such an investigation in large scale and real world data using query logs of a web search engine, and device a test to detect the presence of an irrational user behavior in relevance judgment of documents. We further validate this behavior through a Quantum Cognitive explanation of the Order and Context effects.
KW - Multidimensional Relevance
KW - Quantum Cognition
UR - http://www.scopus.com/inward/record.url?scp=85063527174&partnerID=8YFLogxK
U2 - 10.1145/3234944.3234972
DO - 10.1145/3234944.3234972
M3 - Conference contribution
AN - SCOPUS:85063527174
T3 - ICTIR 2018 - Proceedings of the 2018 ACM SIGIR International Conference on the Theory of Information Retrieval
SP - 191
EP - 194
BT - ICTIR 2018 - Proceedings of the 2018 ACM SIGIR International Conference on the Theory of Information Retrieval
PB - Association for Computing Machinery, Inc
T2 - 8th ACM SIGIR International Conference on the Theory of Information Retrieval, ICTIR 2018
Y2 - 14 September 2018 through 17 September 2018
ER -