Dynamic grouping evolutionary reinforcement learning algorithm for scheduling workflows in hybrid clouds

Although several Evolutionary Reinforcement Learning algorithms (ERLs) benefit from adopting Evolutionary Algorithms (EAs) into RL, most of them focus on either single-objective or linear combination-based multi-objective optimization, and have high computational cost. Due to their poor combination mechanisms, they still face sparse rewards, stability issues, and a lack of exploration. This work proposes a Dynamic Grouping Evolutionary Reinforcement Learning algorithm (D-ERL) for workflow scheduling by combining Particle Swarm Optimization (PSO) with Deep Deterministic Policy Gradient (DDPG), which maps a particle in the PSO swarm to an agent (i.e., actor or policy network) in the DDPG framework, and is structured as an iterative improvement cycle benefiting from the alternating evolution of PSO and DDPG. First, it uses two sub-populations to focus on makespan and cost, respectively such that specific regions can be extensively explored to avoid gradient confusion, and the gradient diversity can be enhanced by filling the replay buffer with diversified experiences, along with a dynamic group learning-based update strategy to reduce the time complexity. Second, a hybridized elite solution-guided evolutionary mechanism is developed by iteratively constructing a global shared elite policy set including both compromise and extreme policies, thus accelerating the entire system convergence and enabling DDPG's cross-refinement. Third, we adopt an external objective-aware complementary heuristic into action selection to adjust the actor's output probability, which enables finely grained objective balance and diversity maintenance to further reduce vanishing gradients and premature convergence.Extensive experiments are conducted to verify the effectiveness and superiority of D-ERL, and the results show that our D-ERL outperforms its four peers in diversity and policy quality, e.g., its resulting hyper-volume is at most 882.21% and at least 5.83% greater than its peers. D-ERL can enhance the generalization and robustness of the obtained Pareto optimal solution set, which can not only provide a novel intelligent solution paradigm, but also be readily applied in executing large-scale scientific applications to enable more efficient and effective cloud data centers.