How Users Interact with Generative Information Retrieval Systems: A Study of User Behavior and Search Experience

The development of LLM has facilitated the emergence of generative information retrieval (IR) systems, such as “Bing Chat”. Generative IR systems return generated text with citations rather than a list of ranked search results. User studies on IR systems are essential for understanding users’ interaction patterns, evaluating and optimizing systems, and improving search experience, particularly in the context of generative IR systems with novel conversational interfaces and responses. However, systematic investigations into user behavior and search experience on generative IR systems are notably lacking. To address this gap, we conducted a user study using Bing Chat to explore user behavior and feedback on generative IR systems. The participants were required to accomplish three types of tasks using Bing Chat. During the search process, we collected their various behavior (e.g., click, query reformulation) and explicit feedback (e.g., satisfaction, credibility, and success). Additionally, the same study was conducted on traditional IR systems Bing for comparison. Analyses of these data show that Bing Chat can reduce the user’s search effort and lead to a better search experience without any decrease in credibility compared with Bing. We believe that this work provides valuable insight into the design and evaluation of generative information retrieval systems.