Hyper-relational interaction modeling in multi-modal trajectory prediction for intelligent connected vehicles in smart cites

Trajectory prediction of surrounding traffic participants is vital for the driving safety of Intelligent Connected Vehicles (ICVs). It has been enabled with the help of the availability of multi-sensor information collected by ICVs. For accurately predicting the future movements of traffic agents, it is crucial to subtlety model the inter-agent interaction. However, existing works focus on the correlations between agents and the map information while neglecting the importance of directly modeling the impact of map elements on inter-agent interactions, the direct modeling of which is beneficial for the representation of agent behaviors. Against this background, we propose to model the hyper-relational interaction, which incorporates map elements into the inter-agent interaction. To tackle the hyper-relational interaction, we propose a novel Hyper-relational Multi-modal Trajectory Prediction (HyperMTP) approach. Specifically, a hyper-relational driving graph is first constructed and the hyper-relational interaction is represented as the hyperedge, directly connecting to various nodes (i.e., agents and map elements). Then a structure-aware embedding initialization technique is developed to obtain unbiased initial embeddings. Afterward, hypergraph dual-attention networks are designed to capture correlations between graph elements while retaining the hyper-relational structure. Finally, a heterogeneous Transformer is devised to further capture the correlations between agents’ states and their corresponding hyper-relational interactions. Experimental results show that HyperMTP consistently outperforms the best-performing baseline with an average improvement of 4.8% across two real-world datasets. Moreover, HyperMTP also boosts the interpretability of trajectory prediction by quantifying the impact of map elements on inter-agent interactions.