Edge-Deployed Panoramic-PTZ Collaborative Semantic Perception System

Wide-area surveillance and anti-Unmanned Aerial Vehicle (UAV) applications increasingly demand continuous omnidirectional coverage together with fine-grained target verification. While panoramic and Pan-Tilt-Zoom (PTZ) configurations have been explored for such tasks, existing implementations typically employ closed-vocabulary detectors and lack the semantic reasoning required for flexible, language-driven human-machine collaboration. This paper presents an edge-deployed large-model-driven panoramic-PTZ collaborative semantic perception system. A stitched panoramic surround view provides full-scene coverage and candidate target discovery, while a long-range zoom PTZ camera enables high-resolution target confirmation and continuous tracking. Cross-scale collaboration is achieved through panoramic-to-PTZ geometric mapping and closed-loop control, allowing rapid viewpoint switching and coordinated operation. To support natural-language-driven human-machine collaboration, an Automatic Speech Recognition (ASR) front-end and a large-language-model (LLM)-based command semantic parsing module are integrated. Schema-constrained structured output normalizes free-form user commands into executable retrieval and tracking instructions, which orchestrate open-vocabulary object detection, single-object tracking, and monocular depth estimation for cooperative perception. The system further supports re-acquisition and tracking recovery under occlusion or target loss. A prototype is implemented on an NVIDIA Jetson Orin NX platform. Experiments evaluate command parsing accuracy, on-device deployment performance of key vision modules, and collaborative runtime behavior. Results demonstrate a stable real-time closed loop on the edge, enabling panoramic discovery and long-range confirmation with practical engineering utility.