Query Optimization for Hybrid Plans in Row–Column Dual Store HTAP Databases

As data volumes grow and business requirements become increasingly complex, Hybrid Transactional/Analytical Processing (HTAP) technologies, capable of handling both Online Transaction Processing (OLTP) and Online Analytical Processing (OLAP) workloads on a single platform, have gained prominence. HTAP databases typically maintain dual data storage formats and dual query engines: one row-oriented for OLTP, and another column-oriented for OLAP. Query plans, known as hybrid plans, can be segmented and pushed down to execute on these different formats. However, existing HTAP solutions still face challenges in optimizing these hybrid plans, struggling to explore the vast space of potential execution strategies effectively. To address these issues, this study introduces a learning-based query optimizer for row–column dual store HTAP database systems, which automatically generates multiple high-quality query optimizer hints (HINTs) to derive candidate plans. To balance plan generation efficiency with plan quality, a lightweight, learning-based algorithm using Monte Carlo Tree Search (MCTS) for generating hybrid access HINTs is proposed. Moreover, a Transformer-based neural network model coupled with a hybrid plan feature representation method is developed to select the candidate execution plan with the lowest predicted execution time. This work focuses on latency-oriented hybrid-plan selection for analytical queries in a row–column dual-store HTAP architecture; the current evaluation does not cover full mixed OLTP/OLAP workload scheduling, transactional interference, or concurrency control, which are left as future work. Experimental results on AlloyDB Omni, a recent row–column dual-store HTAP database, using the real-world IMDB dataset and JOB benchmark demonstrate that our system reduces execution time by 75.02% compared to the Cost-Based Optimizer (CBO) and by 62.23% compared to the state-of-the-art row-store-based learning query optimizer in this evaluated analytical-query setting.