MST-GAN generative model for video enhancement: quality, stability, efficiency

Abstract

The paper considers the improvement of the MST-GAN (Multi-Scale Temporal GAN) architecture for video super-resolution tasks, aimed at ensuring high visual quality, inter-frame stability and efficient operation in real time. The model combines multi-scale feature alignment, temporal aggregation and generative frame generation using a hybrid loss function. Particular attention is paid to ensuring the stability of training: pre-training the generator without a discriminator, the gradual inclusion of temporal consistency, the use of perceptual criteria and regularization techniques allowed to avoid typical problems of generative learning, in particular unstable dynamics and loss of diversity. During training, the model is adapted to realisti c video scenarios, which allows it to maintain quality even in complex scenes with dynamic objects. A number of hardware optimizations are also described, including structural thinning of the model, quantization of weights to INT8 format, compilation in TensorRT and organization of frame streaming processing. As a result, MST-GAN achieved significant inference acceleration without noticeable loss of quality. Qualitative examples on original video data (in particular, a scene with active motion) demonstrate the advantages of the model in preserving textures and smoothness of motion, compared to classical methods of image augmentation. Unlike competing approaches, MST-GAN allows avoiding the characteristic "flicker" and provides a natural transfer of scene dynamics. The obtained results indicate the suitability of MST-GAN for use in practical video enhancement systems, in particular in video streams, monitoring systems, AR applications, where the combination of quality, stability and speed is critically important.