AVDDPG – Federated reinforcement learning applied to autonomous platoon control

1.1.1. Federated reinforcement learning

There are two main areas of research in FRL currently: horizontal federated reinforcement learning (HFRL), and vertical federated reinforcement learning (VFRL). HFRL has been selected as the algorithm of choice for the purposes of this study. HFRL and VFRL differ with respect to the structure of their environments and aggregation methods. All agents in an HFRL architecture use isolated environments. It follows that each agent's action in an HFRL system has no effect on the other agents in the system. An HFRL architecture proposes the following training cycle for each agent: first, a training step is performed locally, second, environment specific parameters are uploaded to the aggregation server, and lastly, parameters are aggregated according to the aggregation method and returned to each agent in the system for another local training step. HFRL may be noted to have similarities to "Parallel RL". Parallel RL is a long studied field of RL, where agent gradients are transferred amongst each other ^[5,10,11].

Reinforcement learning is often a sequential learning process, and as such data is often non-IID with a small sample space ^[12]. HFRL provides the ability to aggregate experience while increasing the sample efficiency, thus providing more accurate and stable learning ^[13]. Some of the current works applying HFRL to a variety of applications are summarized below.

A study by Lim et al. aims to increase the performance of RL methods applied to multi-IoT device systems. RL models trained on single devices are often unable to control devices in a similar albeit slightly different environment ^[5]. Currently, multiple devices need to be trained separately using separate RL agents ^[5]. The methods proposed by Lim et al. sped up the learning process by 1.5 times for a two agent system. In a study by Nadiger et al., the challenges in the personalization of dialogue managers, smart assistants and more are explored. RL has proven to be successful in practice for personalized experiences; however, long learning times and no sharing of data limit the ability for RL to be applied at scale. Applying HFRL to atari non-playable characters in pong showed a median improvement of 17% for the personalization time ^[10]. Lastly, Liu et al. discuss RL as a promising algorithm for smart navigation systems, with the following challenges: long training times, poor generalization across environments, and storing data over long periods of time ^[14]. In order to address these problems, Liu et al. proposed the architecture 'Lifelong FRL', which can be categorized as an HFRL problem. It is found the Lifelong FRL increased the learning rate for smart navigation system when tested on robots in a cloud robotic system ^[14].

The successes of the FedAvg algorithm as a means to improve performance and training times for systems have inspired further research into how aggregation methods should be applied. The design of the aggregation method is crucial in providing performance benefits to that of the base case where FRL is not applied. The FedAvg ^[3] algorithm proposed the averaging of gradients in the aggregation method. In contrast, Liang et al. proposed using model weights in the aggregation method for AV steering control ^[15]. Thus, FRL applications can differ based upon the selection of which parameter to use in the aggregation method. A study by Zhang et al. explores applying FRL to a decentralized DRL system optimizing cellular vehicle-to-everything communication^[16]. Zhang et al. utilize model weights in the aggregation method, and describe a weighting factor dividing the sum batch size for all agents by the training batch size for a specific agent^[16]. In addition, the works of Lim et al. explore how FRL using gradient aggregation can improve convergence speed and performance on the OpenAI-gym environments CartPole-V0, MountainvehicleContinuous-V0, Pendulum-V0 and Acrobot-V1 ^[17]. Lim et al. determined that aggregating gradients using FRL creates high performing agents for each of the OpenAI-gym environments relative to models trained without FRL^[17]. In addition, Wang et al. apply FRL to heterogeneous edge caching ^[18]. Wang et al. show the effectiveness of FRL using weight aggregation to improve hit rate, reduce average delays in the network and offload traffic^[18]. Lastly, Huang et al. apply FRL using model weight aggregation to Service Function Chains in network function virtualization enabled networks^[19]. Huang et al. observe that FRL using model weight aggregation provides benefits to convergence speed, average reward and average resource consumption^[19].

Despite the differences in FRL applications within the aforementioned studies, each study maintains a similar goal: to improve the performance of each agent within the system. None of the aforementioned works explore the differences in whether gradient or model weight aggregation is favourable in performance, and many of the works apply FRL to distributed network or communications environments. It is the goal of this study to conclude whether model weight or gradient aggregation is favourable for AV platooning, as well as be one of the first (if not the first) to apply FRL to AV platooning.

1.1.2. Deep reinforcement learning applied to AV platooning

In recent years, there has been a surge in autonomous vehicle (AV) research, likely due to the technologies potential for increasing road safety, traffic throughput and fuel economy ^[6,20]. Two areas of research are often considered when delving into an AV model: supervised learning or RL ^[20]. Driving is considered a multi-agent interaction problem, and due to the large variability of road data, it can be quite challenging (or near impossible) to gather a data set variable enough to train a supervised model ^[21]. Driving data is collected from humans, which can also limit an AI's ability to that of human level ^[6]. In contrast, RL methods are known to generalize quite well ^[20]. RL approaches are model-free and a model may be inferred by the algorithm while training.

In order to improve the limitations of vehicle following models, DRL has been a steady area of research in the AV community, with many authors contributing works to DRL applied to CACC ^[8,9,22,23]. In a study by Lin et al., a DRL framework is designed to control a CACC AV platoon^[22]. The DRL framework uses the deep deterministic policy gradient (DDPG) ^[24] algorithm and is found to have near-optimal performance ^[22]. In addition, Peake et al. identify limitations in platooning with regard to the communication in platooning^[23]. Through the application of a multi-agent reinforcement learning process, i.e. a policy gradient RL and LSTM network, the performance of a platoon containing 3-5 vehicles is improved upon that of current RL applications to platooning ^[23]. Furthermore, Model Predictive Control (MPC) is the current state-of-the-art for real-time optimal control practices ^[25]. The study performed by Lin et al. applies both MPC and DRL methodologies to the AV platoon problem, observing a DRL model trained using the DDPG algorithm produces merely a 5.8% episodic cost higher than the current state-of-the-art^[25]. The works of Yan et al. propose a hybrid approach to the AV platooning problem where the platoon is modeled as a Markov Decision Process (MDP) in order to collect two rewards from the system at each time step simultaneously^[26]. This approach also incorporates jerk, the rate of change of acceleration in the calculation of the reward for each vehicle in order to ensure passenger comfort ^[26]. The hybrid strategy led to increased performance to that of the base DDPG algorithm, as the proposed framework switches between using classic CACC modeling and DDPG depending on the performance degradation of the DDPG algorithm ^[26]. In another study by Zhu et al., a DRL model is formulated and trained using DDPG to be evaluated against real world driving data. Parameters such as time to collision, headway, and jerk were considered in the DRL model's reward function^[27]. The DDPG algorithm provided favourable performance to that of the analysed human driving data, with regard to more efficient driving via reduced vehicle headways, and improved passenger comfort with lower magnitudes of jerk^[27]. As Vehicle-to-Everything (V2X) communications are envisioned to have a beneficial impact on the performance of platoon controllers, the works of Lei et al. investigates the value of V2X communications for DRL-based platoon controllers. Lei et al. emphasizes the trade-off between the gain of including exogenous information in the system state for reducing uncertainty and the performance erosion due to the curse-of-dimensionality^[28].

When formulating the AV platooning problem as a DRL model DDPG is prominently selected as the algorithm for training. DDPG's ability to handle continuous actions space and complex state's is perfect for the CACC platoon problem. However, despite the DDPG algorithm's success in literature, there are still instability challenges related to the algorithm along with a time consuming hyper-parameter tuning process to account for the minute differences in vehicle models/dynamics amongst platoons. As previously discussed, FRL provides advantages in these areas where information sharing can accelerate performance during training and improve the performance of the system as a whole. In addition, the ability to share experience across like models has been proven to allow for fast convergence of models, which further optimizes the performance of DDPG when applied to AV platoons ^[5].

2.2.1. State space

The state space formula (6) can be discretized using the forward euler method giving the system equation below

where $$ x_{i, k}=[e_{pi, k}, e_{vi, k}, a_{i, k}, a_{i-1, k}] $$ is the observation state for the MDP problem that includes the position error $$ e_{pi, k} $$, velocity error $$ e_{vi, k} $$, acceleration $$ a_{i, k} $$, and the acceleration of the predecessor vehicle $$ a_{i-1, k} $$ at time step $$ k $$. Moreover, $$ A_{Di} $$, $$ B_{Di} $$, and $$ C_{Di} $$ are given as

2.2.2. Action space

Each vehicle within a single lane platoon follows the vehicle in front of it, and as such the only action the vehicle may take to maintain a desired headway is to accelerate, or decelerate. The action for the system is defined as the control input $$ u_{i, k} $$ to the vehicle.

2.2.3. Reward function

The design of a reward in a DDPG system is critical to providing good performance within the system. In the considered driving scenario, it is logical to minimize position error, velocity error, the amount of time spent accelerating and the jerkiness of the driving motion. The proposed reward thus includes the normalized position error, $$ e_{pi, k} $$, velocity error $$ e_{vi, k} $$, control input $$ u_{i, k} $$ and lastly the jerk. The vehicle reward $$ c_{i, k} $$ is given below, where $$ a $$$$ b $$, $$ c $$ and $$ d $$ are system hyperparameters.

2.3.1. DDPG model description

The DDPG algorithm is composed of an actor, $$ \mu $$ and a critic, $$ Q $$. The actor produces actions $$ u_t \in \mathbf{U} $$ given some observation $$ x_t \in \mathbf{X} $$ and the critic makes judgements on those actions while training using the Bellman equation ^[12,24]. The actor is updated by the policy gradient ^[24]. The critic network uses its weights $$ \theta^q $$ to approximate the optimal action-value function $$ Q(x, u|\theta^q) $$^[24]. The actor network uses weights $$ \theta^\mu $$ to represent the agents' current policy $$ \mu(x|\theta^\mu) $$ for the action-value function ^[24]. The actor $$ \mu(x): \mathbf{X} \xrightarrow{} \mathbf{U} $$ maps the observation to the action. Experience replay is used to mitigate the issue of training samples not being independent and identically distributed due to their generation from sequential explorations ^[24]. Two additional models, the target actor $$ \mu' $$ and critic $$ Q' $$ are used in DDPG to stabilize the training of the actor and critic networks by updating parameters slowly based on the target update coefficient $$ \tau $$. A sufficient value of $$ \tau $$ is chosen such that stable training of $$ \mu $$ and $$ Q $$ is observed. Figure 2 provides a high level simplified overview of how the DDPG algorithm interacts with a single vehicle in a platoon.

Figure 2. High level flow diagram of the DDPG model for a general vehicle $$ V_i $$ in a platoon.

2.3.2. Inter and intra FRL

Modifications to the base DDPG algorithm are needed in order to implement Inter-FRL and Intra-FRL. In order to implement FedAvg the following modifications are required:

$$ 1. $$ An FRL server: responsible for averaging the system parameters for use in a global update

$$ 2. $$ Model weight aggregation: storing of each model's weights for use in aggregation

$$ 3. $$ Model gradient aggregation: storing of each model's gradients for use in aggregation

In order to perform FRL, it has been proven that including an update delay between global FRL updates is beneficial for performance ^[5]. In addition, turning off FRL partway through training is important to allow each agent to refine their models independently of each other such that they can perform best with respect to their environments ^[5]. Lastly, it has also been shown that global updates and local updates should not be performed in the same episode ^[15].

Two methods of aggregation are implemented in the system design, Inter-FRL (see Figure 3), and Intra-FRL (see Figure 4). The proposed system is capable of aggregating both the model weights and gradients for each model so that either type of parameter may be averaged for use in global updates. The FRL server has the responsibility of averaging the parameters (model weights or gradients) across each agent in the system.

Figure 3. Inter-FRL.

Figure 4. Intra-FRL.

The pseudo-code for the Inter/Intra-FRL algorithm is presented in Algorithm 1. The system is designed to allow the training of any number of equal length platoons. At the lowest level, a DDPG agent exists for each vehicle in each platoon. As such, a DRL model must be initialized for each vehicle in the whole system. Each DDPG agent trains separately from the others before data is uploaded to the FRL server. Federated averaging is applied at a given time delay known as the FRL update delay, while being terminated at a given episode as defined by the cutoff ratio as seen in Table 3. Currently, Algorithm 1 is synchronous, and the FRL server is also synchronous.

Algorithm 1: FRL applied to an AV platoon.

3.2.1. Performance across 4 random seeds

The performance for each of the systems is calculated by averaging the cumulative reward of each vehicle in the 2 vehicle 2 platoon system, as summarized in Table 4. For each of the 3 cases (base case, Inter-FRLGA and Inter-FRLWA), training sessions were run using 4 random seeds. In order to determine the highest performing system overall, an average and standard deviation is obtained from the result of training using the 4 random seeds. From Table 4,it is observed that both Inter-FRL scenarios using gradient and weight aggregation provide large performance increases to that of the base case.

3.2.2. Convergence properties

The cumulative reward is calculated over each training episode, and a moving average is computed over 40 episodes to generate Figure 6a-6f. It can be seen that the cumulative reward for Inter-FRLWA not only converges more rapidly than both no-FRL and Inter-FRLGA, but Inter-FRLWA also appears to have a more stable training session as indicated by the lower magnitude of the shaded area (the standard deviation across the four random seeds).

Figure 6. Average performance across 4 random seeds for a 2 platoon 2 vehicle scenario trained without FRL (Figure 6a, 6d), with Inter-FRLGA (Figure 6b, 6e), and with Inter-FRLWA (Figure 6c, 6f). The shaded areas represent the standard deviation across the 4 seeds.

3.2.3. Test results for one episode

In Figure 7a and 7b,a simulation is performed over a single training episode plotting the jerk, along with the control input $$ u_{i, k} $$, acceleration $$ a_{i, k} $$, velocity error $$ e_{vi, k} $$, and position error $$ e_{pi, k} $$ for each platoon. There are 2 platoons in the Inter-FRL scenario, and a simulation is provided for each platoon. The simulation environment is subject to initial conditions of ($$ e_{pi} = 1.0\:m $$, $$ e_{vi}=1.0\:m/s $$, $$ a_i = 0.03\:m/s^2 $$). It can be seen that each DDPG agent for both vehicles within both platoons quickly responds to the platoon leader's control input $$ u_{i, k} $$ to bring the position error, velocity error and acceleration error to 0. In addition, each DDPG agent closely approximates the Gaussian random input of the platoon leader, eliminating noise in the response to maintain smooth tracking across the episode. Finally, each DDPG agent in the platoon also minimizes the jerk effectively. These results are indicative of both a good design of the reward function (10), and also a suitable selection of parameters $$ a, b, c $$ and $$ d $$ in (10).

Figure 7. Results for a specific 60s test episode using the 2 vehicle 2 platoon environment trained using Inter-FRL with weight aggregation.

3.3.1. Performance across 4 random seeds

The performance for the platoon is calculated by averaging the cumulative reward generated by the simulation for each of the 4 random seeds and is summarized in Table 5. The results in Table 5 summarize the performance for no-FRL, Intra-FRLGA, and lastly Intra-FRLWA. It is observed that Intra-FRLWA performs most favourably, followed by no-FRL and lastly Intra-FRLGA.

3.3.2. Convergence properties

The cumulative reward is calculated over each training episode, and a moving average is computed over 40 episodes to generate Figure 8. Similar to the Inter-FRL experiments, Intra-FRLWA shows the most favourable training results. In addition, the rate of convergence increases with Intra-FRLWA over no-FRL and Intra-FRLGA. Lastly, the stability during training is also shown to be improved as the standard deviation across the four random seeds is much smaller than the other two cases (as evident in the shaded regions of Figure 8).

Figure 8. Average performance across 4 random seeds for 1 platoon 2 vehicle scenario trained without FRL (Figure 6a), Intra-FRLGA (Figure 6b), and with Intra-FRLWA (Figure 6c). The shaded areas represent the standard deviation across the 4 seeds.

3.3.3. Test results for one episode

A single simulation is performed on an episode plotting the jerk, along with the control input $$ u_{i, k} $$, acceleration $$ a_{i, k} $$, velocity error $$ e_{vi, k} $$, and position error $$ e_{pi, k} $$. Figure 9 shows the precise control of Intra-FRLWA on the environment. The environment is initialized to the same conditions as that of the Inter-FRLWA scenario ($$ e_{pi} = 1.0 m $$, $$ e_{vi}=1.0 m/s $$, $$ a_i = 0.03 m/s^2 $$), and each DDPG agent in the platoon quickly and precisely tracks the Gaussian random input $$ u_{i, k} $$ from the leader while minimizing position error, velocity error, acceleration, and jerk. Much like the Inter-FRLWA scenario, it is observed that a strong optimization of the reward function (Equation 10) has occurred. This is an indication of a good design of the reward function in addition to a good balance of parameters $$ a, b, c $$ and $$ d $$ in the reward function.

Figure 9. Results for a specific 60s test episode using the 2 vehicle 1 platoon environment trained using Intra-FRLWA.

3.5.1. Performance with varying number of vehicles

The performance for each experiment is calculated by taking the average cumulative episodic reward across each vehicle in the platoon at the end of the simulation episode. Table 7 presents the results for no-FRL and Intra-FRLWA for platoons with 3, 4, and 5 follower vehicles. Table 7 shows that Intra-FRLWA provides favourable performance in all platoon lengths. A notable example of Intra-FRLWA's success is highlighted when considering the poor performance of the 4 vehicle platoon trained with no-FRL using seed 1. The Intra-FRLWA training strategy was able to overcome the performance challenges, correcting the poor performance entirely.

3.5.2. Convergence properties

The cumulative reward is calculated over each training episode, and a moving average is computed over 40 episodes to generate Figure 10. Intra-FRLWA shows favourable training performance to that of the no-FRL scenario for all platoon lengths. In addition, the rate of convergence is increased using Intra-FRLWA versus no-FRL. Furthermore, the shaded areas corresponding to standard deviation across the seeds are reduced significantly, indicating better stability across the seeds for Intra-FRLWA than no-FRL. Last, the overall stability is improved as shown by the large noise reduction during training in Figure 10d,10e,10f when compared with no-FRL's Figure 10a,10b,10c.

Figure 10. Average performance across 4 random seeds for 3 platoons with 3, 4 and 5 followers trained without FRL (Figures 10a, 10b, 10c), and with Intra-FRLWA (Figure 10d, 10e, 10f). The shaded areas represent the standard deviation across the four seeds

3.5.3. Test results for one episode

As with all previous sections, a single simulation is performed on a 60 second episode plotting the jerk along with the control input $$ u_{i, k} $$, acceleration $$ a_{i, k} $$, velocity error $$ e_{vi, k} $$, and position error $$ e_{pi, k} $$. Figure 11 showcases the ability of Intra-FRLWA to control a 5 platoon environment precisely when compared to a platoon trained without Intra-FRLWA. The environment for Intra-FRLWA is initialized with the same values as no-FRL, just like all previous experiments: ($$ e_{pi} = 1.0 m $$, $$ e_{vi}=1.0 m/s $$, $$ a_i = 0.03 m/s^2 $$). Each DDPG agent trained with Intra-FRLWA quickly and precisely tracks the Gaussian random control input $$ u_{i, k} $$ from the leader minimizing $$ e_{pi, k} $$, $$ e_{vi, k} $$, $$ a_{i, k} $$ and jerk. In particular, the response for $$ e_{pi, k} $$ and $$ e_{vi, k} $$ in the platoon trained using Intra-FRLWA (Figure 11b) appears to respond to the platoon leader's input quicker and in a much smoother manner than that of the no-FRL scenario (Figure 11a).

Figure 11. Results for a specific 60s test episode using the 5 vehicle 1 platoon environment trained using no-FRL (Figure 11a), and with Intra-FRLWA (Figure 11b)

The large difference in performance for no-FRL versus Intra-FRL can be explained by understanding how Intra-FRLWA works. With no-FRL, each agent trains independently, and the inputs to the following vehicles are directly outputted from the predecessors. Thus, the followers farther back in the platoon take longer to train as their predecessors' outputs can be highly variable while training. As the policies of the predecessors converge, the policy of each follower can then begin to converge. This sequential convergence from predecessor to follower can be seen in Figure 10,where the convergence during training is slower for vehicles 4 and 5 than it is for 3, 2 and 1. Intra-FRLWA helps to resolve this challenge by allowing vehicles to average their model weights, thus distributing an aggregation of more mature predecessor parameters amongst the platoon.