Multi-agent reinforcement learning-based routing and scheduling models in time-sensitive networking for internet of vehicles communications between transportation field cabinets

Abstract

Future autonomous vehicles will interact with traffic infrastructure through roadside units (RSUs) directly connected to transportation field cabinets (TFCs). These TFCs must be interconnected to share traffic information, enabling infrastructure-to-infrastructure (I2I) communications that are reliable, synchronous and capable of transmitting vehicle data to the Internet. However, I2I communications present a complex optimization challenge. This study addresses this by proposing the design, implementation, and evaluation of an automated management model for I2I service channels based on multi-agent reinforcement learning (MARL) integrated with deep reinforcement learning (DRL). The proposed models efficiently manage the routing and scheduling of data frames between internet of vehicles (IoV) infrastructure devices through time-sensitive networking (TSN) to ensure real-time synchronous I2I communications. The solution incorporates both a routing model and a scheduling model, evaluated in a simulated shared environment where agents operate within the TSN control plane. Both models are tested for different topologies and background traffic levels. The results demonstrate that the models establish the majority of paths in the scenario, adhering to near-optimal routing and scheduling policies. Recursively, for each individual request to create a service channel, the system establishes online an optimal synchronous path between entities with a limited time budget. In total, 71% of optimal routing paths are established and 97% of optimal schedules are achieved. The approach takes into account the periodic nature of the transmitted data and its robustness through TSN networks, obtaining 99 percent of compliant service requests with flow jitter levels below 100 microseconds for different topologies and different network utility percentages. The proposed solution achieves lower execution delays compared to the iterative ILP approach. Additionally, the solution facilitates the integration of 5G networks for vehicle-to-infrastructure (V2I) communications, which is identified as an area for future exploration.

This work was supported by the Spanish Ministry of Economic Affairs and Digital Transformation and the European Union—NextGenerationEU, in the framework of the Recovery Plan, Transformation and Resilience (PRTR) (Call UNICO I+D 5G 2021, ref. number TSI-063000-2021-15–6GSMART-EZ), as well as by the Agencia Estatal de Investigación of Ministerio de Ciencia e Innovación of Spain under project PID2022-137329OB-C41/MCIN/AEI/10.13039/50110001103.

Postprint (published version)