Conflict resolution strategy based on deep reinforcement learning for air traffic management
Abstract
With the continuous increase in flight flows, the flight conflict risk in the airspace has increased. Aiming at the problem of conflict resolution in actual operation, this paper proposes a tactical conflict resolution strategy based on Deep Reinforcement Learning. The process of the controllers resolving conflicts is modelled as the Markov Decision Process. The Deep Q Network algorithm trains the agent and obtains the resolution strategy. The agent uses the command of altitude adjustment, speed adjustment, or heading adjustment to resolve a conflict, and the design of the reward function fully considers the air traffic control regulations. Finally, simulation experiments were performed to verify the feasibility of the strategy given by the conflict resolution model, and the experimental results were statistically analyzed. The results show that the conflict resolution strategy based on Deep Reinforcement Learning closely reflected actual operations regarding flight safety and conflict resolution rules.
Keyword : conflict resolution, deep reinforcement learning, air traffic control, air traffic management, decision support technology, aviation
How to Cite
Sui, D., Ma, C., & Dong, J. (2023). Conflict resolution strategy based on deep reinforcement learning for air traffic management. Aviation, 27(3), 177–186. https://doi.org/10.3846/aviation.2023.19720
This work is licensed under a Creative Commons Attribution 4.0 International License.
References
Brittain, M. W., & Wei, P. (2021). One to any: Distributed conflict resolution with deep multi-agent reinforcement learning and long short-term memory. In AIAA Scitech 2021 Forum (pp. 1–10). Nashville, Tennessee, USA. American Institute of Aeronautics and Astronautics. https://doi.org/10.2514/6.2021-1952
Brittain, M., & Wei, P. (2022). Scalable autonomous separation assurance with heterogeneous multi-agent reinforcement learning. IEEE Transactions on Automation Science and Engineering, 19(4), 2837–2848. https://doi.org/10.1109/TASE.2022.3151607
Cafieri, S., & Omheni, R. (2017). Mixed-integer nonlinear programming for aircraft conflict avoidance by sequentially applying velocity and heading angle changes. European Journal of Operational Research, 260(1), 283–290. https://doi.org/10.1016/j.ejor.2016.12.010
Cai, J., & Zhang, N. (2019). Mixed integer nonlinear programming for aircraft conflict avoidance by applying velocity and altitude changes. Arabian Journal for Science and Engineering, 44(10), 8893–8903. https://doi.org/10.1007/s13369-019-03911-w
Carreno, V. (2002). Evaluation of a pair-wise conflict detection and resolution algorithm in a multiple aircraft scenario. In NASA ™-2002–211963.
Çeçen, R. K., & Cetek, C. (2020). Conflict-free en-route operations with horizontal resolution manoeuvers using a heuristic algorithm. The Aeronautical Journal, 124(1275), 767–785. https://doi.org/10.1017/aer.2020.5
Chen, W., Chen, J., Shao, Z., & Biegler, L. T. (2016). Three-dimensional aircraft conflict resolution based on smoothing methods. Journal of Guidance, Control, and Dynamics, 39(7), 1481–1490. https://doi.org/10.2514/1.G001726
Civil Aviation Administration of China. (2021). 2020 Statistical Bulletin on the Development of Civil Aviation Industry. Beijing, China.
Durand, N., Alliot, J., & Noailles, J. (1996, February 17–19). Automatic aircraft conflict resolution using genetic algorithms. In Proceedings of the 1996 ACM Symposium on Applied Computing (pp. 289–298). New York, NY, USA. American Institute of Aeronautics and Astronautics. https://doi.org/10.1145/331119.331195
Emami, H., & Derakhshan, F. (2014). Multi-agent based solution for free flight conflict detection and resolution using particle swarm optimization algorithm. UPB Scientific Bulletin, Series C: Electrical Engineering, 76(3), 49–64.
Endsley, M. R. (2017). From here to autonomy: Lessons learned from human–automation research. Human Factors, 59(1), 5–27. https://doi.org/10.1177/0018720816681350
Gilles, D., Cesar, M., & Alfons, G. (2001). Tactical conflict detection and resolution in a 3-D airspace. In NASA CR-2001–210853.
Hong, Y., Choi, B., & Oh, G. (2017). Nonlinear conflict resolution and flow management using particle swarm optimization. IEEE Transactions on Intelligent Transportation Systems, 18(12), 3378–3387. https://doi.org/10.1109/TITS.2017.2684824
International Civil Aviation Organization. (2016). Procedures for navigation services – air traffic management. Montreal, Canada.
International Civil Aviation Organization. (2005). Global air traffic management operational concept. Montreal, Canada.
Loft, S., Neal, A., Sanderson, P., & Mooij, M. (2007). Modeling and predicting mental workload in En route air traffic control: Critical review and broader implications. Human Factors, 49(3), 376–399. https://doi.org/10.1518/001872007X197017
Matsuno, Y., Tsuchiya, T., & Matayoshi, N. (2016). Near-optimal control for aircraft conflict resolution in the presence of uncertainty. Journal of Guidance, Control, and Dynamics, 39(2), 326–338. https://doi.org/10.2514/1.G001227
Ma, Y., Ni, Y., & Liu, P. (2013, October 28–29). Aircrafts conflict resolution method based on ADS-B and genetic algorithm. In Proceedings of the Sixth International Symposium on Computational Intelligence and Design (pp. 121–124). Hangzhou, China. https://doi.org/10.1109/ISCID.2013.144
Ministry of Transport of the People’s Republic of China. (2017). Air traffic management rules for civil aviation. Beijing, China.
Omer, J. (2015). A space-discretized mixed-integer linear model for air-conflict resolution with speed and heading maneuvers. Computers & Operations Research, 58, 75–86. https://doi.org/10.1016/j.cor.2014.12.012
O’Neill, T., McNeese, N., Barron, A., & Schelble, B. (2020). Human–autonomy teaming: A review and analysis of the empirical literature. Human Factors, 64(5), 904–938. https://doi.org/10.1177/0018720820960865
Patera, R. P. (2007). Space vehicle conflict-avoidance analysis. Journal of Guidance, Control, and Dynamics, 30(2), 492–498. https://doi.org/10.2514/1.24067
Pham, D. T., Tran, N. P., Goh, S. K., Alam, S., & Duong, V. (2019a, March 20–22). Reinforcement learning for two-aircraft conflict resolution in the presence of uncertainty. In 2019 IEEE-RIVF International Conference on Computing and Communication Technologies (RIVF) (pp. 1–6). IEEE. https://doi.org/10.1109/RIVF.2019.8713624
Pham, D. T., Tran, N. P., Alam, S., Duong, V., & Delahaye, D. (2019b, June). A machine learning approach for conflict resolution in dense traffic scenarios with uncertainties. In Proceedings of the 13th USA/Europe Air Traffic Management Research and Development Seminar (ATM2019) (pp. 17–21). Vienna, Austria.
Ribeiro, M., Ellerbroek, J., & Hoekstra, J. (2020a, December). Determining optimal conflict avoidance manoeuvres at high densities with reinforcement learning. In Proceedings of the Tenth SESAR Innovation Days (pp. 7–10), Virtual Conference. SESAR.
Ribeiro, M., Ellerbroek, J., & Hoekstra, J. (2020b). Improvement of conflict detection and resolution at high densities through reinforcement learning. In Proceedings of the Conference on Research in Air Transportation (ICART) (pp. 1–4), Virtual Conference. http://resolver.tudelft.nl/uuid:d3bf3c0d-16bf-4ca4-b695-2868d761c129
Ribeiro, M., Ellerbroek, J., & Hoekstra, J. (2020c). Review of conflict resolution methods for manned and unmanned aviation. Aerospace, 7(6), 79. https://doi.org/10.3390/aerospace7060079
Soler, M., Kamgarpour, M., Lloret, J., & Lygeros, J. (2016). A hybrid optimal control approach to fuel-efficient aircraft conflict avoidance. IEEE Transactions on Intelligent Transportation Systems, 17(7), 1826–1838. https://doi.org/10.1109/TITS.2015.2510824
Sperandio, J. C. (1971). Variation of operator’s strategies and regulating effects on workload. Ergonomics, 14(5), 571–577. https://doi.org/10.1080/00140137108931277
Sui, D., Xu, W., & Zhang, K. (2022). Study on the resolution of multi-aircraft flight vonflicts based on an IDQN. Chinese Journal of Aeronautics, 35(2), 195–213. https://doi.org/10.1016/j.cja.2021.03.015
Sui, D., & Zhang, K. (2022). A tactical conflict detection and resolution method for en route conflicts in trajectory-based operations. Journal of Advanced Transportation, 2022, 1–16. https://doi.org/10.1155/2022/9283143
Tran, P. N., Pham, D. T., Goh, S. K., Alam, S., & Duong, V. (2020). An interactive conflict solver for learning air traffic conflict resolutions. Journal of Aerospace Information Systems, 17(6), 271–277. https://doi.org/10.2514/1.I010807
Wang, Z., Li, H., Wang, J., & Shen, F. (2019). Deep reinforcement learning based conflict detection and resolution in air traffic control. IET Intelligent Transport Systems, 13(6), 1041–1047. https://doi.org/10.1049/iet-its.2018.5357
Brittain, M., & Wei, P. (2022). Scalable autonomous separation assurance with heterogeneous multi-agent reinforcement learning. IEEE Transactions on Automation Science and Engineering, 19(4), 2837–2848. https://doi.org/10.1109/TASE.2022.3151607
Cafieri, S., & Omheni, R. (2017). Mixed-integer nonlinear programming for aircraft conflict avoidance by sequentially applying velocity and heading angle changes. European Journal of Operational Research, 260(1), 283–290. https://doi.org/10.1016/j.ejor.2016.12.010
Cai, J., & Zhang, N. (2019). Mixed integer nonlinear programming for aircraft conflict avoidance by applying velocity and altitude changes. Arabian Journal for Science and Engineering, 44(10), 8893–8903. https://doi.org/10.1007/s13369-019-03911-w
Carreno, V. (2002). Evaluation of a pair-wise conflict detection and resolution algorithm in a multiple aircraft scenario. In NASA ™-2002–211963.
Çeçen, R. K., & Cetek, C. (2020). Conflict-free en-route operations with horizontal resolution manoeuvers using a heuristic algorithm. The Aeronautical Journal, 124(1275), 767–785. https://doi.org/10.1017/aer.2020.5
Chen, W., Chen, J., Shao, Z., & Biegler, L. T. (2016). Three-dimensional aircraft conflict resolution based on smoothing methods. Journal of Guidance, Control, and Dynamics, 39(7), 1481–1490. https://doi.org/10.2514/1.G001726
Civil Aviation Administration of China. (2021). 2020 Statistical Bulletin on the Development of Civil Aviation Industry. Beijing, China.
Durand, N., Alliot, J., & Noailles, J. (1996, February 17–19). Automatic aircraft conflict resolution using genetic algorithms. In Proceedings of the 1996 ACM Symposium on Applied Computing (pp. 289–298). New York, NY, USA. American Institute of Aeronautics and Astronautics. https://doi.org/10.1145/331119.331195
Emami, H., & Derakhshan, F. (2014). Multi-agent based solution for free flight conflict detection and resolution using particle swarm optimization algorithm. UPB Scientific Bulletin, Series C: Electrical Engineering, 76(3), 49–64.
Endsley, M. R. (2017). From here to autonomy: Lessons learned from human–automation research. Human Factors, 59(1), 5–27. https://doi.org/10.1177/0018720816681350
Gilles, D., Cesar, M., & Alfons, G. (2001). Tactical conflict detection and resolution in a 3-D airspace. In NASA CR-2001–210853.
Hong, Y., Choi, B., & Oh, G. (2017). Nonlinear conflict resolution and flow management using particle swarm optimization. IEEE Transactions on Intelligent Transportation Systems, 18(12), 3378–3387. https://doi.org/10.1109/TITS.2017.2684824
International Civil Aviation Organization. (2016). Procedures for navigation services – air traffic management. Montreal, Canada.
International Civil Aviation Organization. (2005). Global air traffic management operational concept. Montreal, Canada.
Loft, S., Neal, A., Sanderson, P., & Mooij, M. (2007). Modeling and predicting mental workload in En route air traffic control: Critical review and broader implications. Human Factors, 49(3), 376–399. https://doi.org/10.1518/001872007X197017
Matsuno, Y., Tsuchiya, T., & Matayoshi, N. (2016). Near-optimal control for aircraft conflict resolution in the presence of uncertainty. Journal of Guidance, Control, and Dynamics, 39(2), 326–338. https://doi.org/10.2514/1.G001227
Ma, Y., Ni, Y., & Liu, P. (2013, October 28–29). Aircrafts conflict resolution method based on ADS-B and genetic algorithm. In Proceedings of the Sixth International Symposium on Computational Intelligence and Design (pp. 121–124). Hangzhou, China. https://doi.org/10.1109/ISCID.2013.144
Ministry of Transport of the People’s Republic of China. (2017). Air traffic management rules for civil aviation. Beijing, China.
Omer, J. (2015). A space-discretized mixed-integer linear model for air-conflict resolution with speed and heading maneuvers. Computers & Operations Research, 58, 75–86. https://doi.org/10.1016/j.cor.2014.12.012
O’Neill, T., McNeese, N., Barron, A., & Schelble, B. (2020). Human–autonomy teaming: A review and analysis of the empirical literature. Human Factors, 64(5), 904–938. https://doi.org/10.1177/0018720820960865
Patera, R. P. (2007). Space vehicle conflict-avoidance analysis. Journal of Guidance, Control, and Dynamics, 30(2), 492–498. https://doi.org/10.2514/1.24067
Pham, D. T., Tran, N. P., Goh, S. K., Alam, S., & Duong, V. (2019a, March 20–22). Reinforcement learning for two-aircraft conflict resolution in the presence of uncertainty. In 2019 IEEE-RIVF International Conference on Computing and Communication Technologies (RIVF) (pp. 1–6). IEEE. https://doi.org/10.1109/RIVF.2019.8713624
Pham, D. T., Tran, N. P., Alam, S., Duong, V., & Delahaye, D. (2019b, June). A machine learning approach for conflict resolution in dense traffic scenarios with uncertainties. In Proceedings of the 13th USA/Europe Air Traffic Management Research and Development Seminar (ATM2019) (pp. 17–21). Vienna, Austria.
Ribeiro, M., Ellerbroek, J., & Hoekstra, J. (2020a, December). Determining optimal conflict avoidance manoeuvres at high densities with reinforcement learning. In Proceedings of the Tenth SESAR Innovation Days (pp. 7–10), Virtual Conference. SESAR.
Ribeiro, M., Ellerbroek, J., & Hoekstra, J. (2020b). Improvement of conflict detection and resolution at high densities through reinforcement learning. In Proceedings of the Conference on Research in Air Transportation (ICART) (pp. 1–4), Virtual Conference. http://resolver.tudelft.nl/uuid:d3bf3c0d-16bf-4ca4-b695-2868d761c129
Ribeiro, M., Ellerbroek, J., & Hoekstra, J. (2020c). Review of conflict resolution methods for manned and unmanned aviation. Aerospace, 7(6), 79. https://doi.org/10.3390/aerospace7060079
Soler, M., Kamgarpour, M., Lloret, J., & Lygeros, J. (2016). A hybrid optimal control approach to fuel-efficient aircraft conflict avoidance. IEEE Transactions on Intelligent Transportation Systems, 17(7), 1826–1838. https://doi.org/10.1109/TITS.2015.2510824
Sperandio, J. C. (1971). Variation of operator’s strategies and regulating effects on workload. Ergonomics, 14(5), 571–577. https://doi.org/10.1080/00140137108931277
Sui, D., Xu, W., & Zhang, K. (2022). Study on the resolution of multi-aircraft flight vonflicts based on an IDQN. Chinese Journal of Aeronautics, 35(2), 195–213. https://doi.org/10.1016/j.cja.2021.03.015
Sui, D., & Zhang, K. (2022). A tactical conflict detection and resolution method for en route conflicts in trajectory-based operations. Journal of Advanced Transportation, 2022, 1–16. https://doi.org/10.1155/2022/9283143
Tran, P. N., Pham, D. T., Goh, S. K., Alam, S., & Duong, V. (2020). An interactive conflict solver for learning air traffic conflict resolutions. Journal of Aerospace Information Systems, 17(6), 271–277. https://doi.org/10.2514/1.I010807
Wang, Z., Li, H., Wang, J., & Shen, F. (2019). Deep reinforcement learning based conflict detection and resolution in air traffic control. IET Intelligent Transport Systems, 13(6), 1041–1047. https://doi.org/10.1049/iet-its.2018.5357