Hybrid Energy Management Optimization Combining Offline and Online Reinforcement Learning for Connected Hybrid Electric Buses

Energy management is crucial to guarantee the long-term fuel economy for hybrid electric systems. This article proposed a novel energy management strategy (EMS) for a hybrid electric bus (HEB) based on an "offline training, online fine-tuning"methodology. The proposed strategy innovatively combines the advantages of classical online deep reinforcement learning (DRL) and novel offline DRL methods, avoiding issues including the sim2real gap and low sample efficiency. First, leveraging the existing offline suboptimal dataset, an initial strategy is pretrained by the offline algorithms and deployed into the onboard control unit (OBU) and cloud platform. Second, the initial strategy is optimized in real-time by the enhanced online algorithms during the online fine-tuning phase. Meanwhile, an action masking mechanism is proposed to avoid unreasonable actions. Finally, the effectiveness of the pretrained method and the proposed strategy is validated. The proposed strategy enhances the fuel economy by 6.51% compared with the initial strategy, achieving 95.20% performance of the dynamic programming (DP)-based EMS. This article improves energy optimization with less dependence on the simulation models and realizes the "model-free"concept in the strict sense, providing an attractive solution for applying DRL to intelligent power allocation.