Natural Language-Based Control of Integrated Aerial Platform Using Large Language Models

This paper proposes a method for controlling IAP (Integrated Aerial Platform) drones based on LLMs (large language models), including both API (Application Programming Interface) calls and locally deployed models, with the aim of generating real-time control commands through natural language input. The system adopts a modular design, consisting of three modules: the Natural Language Instruction Parsing Module, the Control Command Generation Module, and the Admittance Control Module. These modules work in close collaboration to achieve natural language-based human-drone interaction (HDI). To address issues related to cloud-based LLMs, such as network fluctuations and the reliability of generated code, we propose a solution combining both cloud-based and local deployment of LLMs, and validate the generated code through a safety check mechanism. Finally, a series of experiments in the RaiSim simulation environment validate the system's effectiveness in performing both basic flight tasks and complex peg-in-hole tasks. The experimental results show that the system can effectively generate control code and, by integrating drone pose, external force, and torque feedback, collaborate with the control module to complete high-precision tasks, demonstrating the great potential of using LLMs for natural language interaction with drones.