PivotSketch: Control-Ready Semantic Ranking for Adaptive Video Streaming

Adaptive bitrate (ABR) algorithms are central to video-on-demand streaming, yet their QoE objectives remain time-uniform: A stall or quality drop is penalized identically whether it occurs during a climactic reveal or a low-stakes transition. Meanwhile, large language models (LLMs) can now judge which moments matter most to viewers, but their outputs are computationally expensive, lack cross-Timeline comparability, and are risky to inject directly into a control loop. This separation wastes a key opportunity for viewer-Aligned optimization: video semantics often indicate which segments are quality-critical well before playback reaches them.We present PivotSketch, a framework that converts foundation model semantics into stall-safe ABR guidance. It has three components: (1) a pivot-based ranking module that estimates globally comparable chunk-importance percentiles under a fixed inference budget; (2) a calibration module that derives value-of-bits from the encoding ladder; and (3) a reliability estimator that gates semantic influence when confidence is low. Integrated with RobustMPC, the controller prioritizes semantically important moments and reverts to the baseline when semantic signals are unreliable. On Mr. HiSum assets with real throughput traces, PivotSketch aligns best with human importance annotations. For the top 10% most important moments, it reduces rebuffering by 30% and increases delivered quality by 13% versus RobustMPC, while keeping average performance near baseline.