Escaping Stability-Plasticity Dilemma in Online Continual Learning for Motion Forecasting via Synergetic Memory Rehearsal

Deep neural networks (DNNs) have achieved remarkable success in intelligent systems such as autonomous vehicles and robots. However, most DNN-based methods suffer from catastrophic forgetting, where the DNN may fail to maintain its performance in previously learned tasks after adapting to new data. This phenomenon gives rise to a critical tradeoff between memory stability of the DNN, that is, the ability to retain learned knowledge, and learning plasticity, that is, the capacity to acquire new information effectively. To address such stability–plasticity dilemma, this study proposes a novel CL method named synergetic memory rehearsal (SyReM). SyReM maintains a compact memory buffer to represent learned knowledge. To ensure memory stability, it employs an inequality constraint that limits increments in the average loss over the memory buffer. Synergistically, a selective memory rehearsal mechanism is designed to enhance learning plasticity by selecting samples from the memory buffer that are similar to recently observed data. This selection is based on an online-measured cosine similarity of loss gradients, ensuring targeted memory rehearsal. We validate SyReM under online CL tasks for motion forecasting. Comprehensive experiments on naturalistic driving datasets demonstrate that, compared to non-CL and CL baselines, SyReM significantly mitigates catastrophic forgetting in past scenarios while improving forecasting accuracy in new ones.