Authors:
Saem Park
1
;
2
;
Donghoon Han
2
and
Nojun Kwak
2
Affiliations:
1
LG Electronics, Seoul, South Korea
;
2
Seoul National University, Seoul, South Korea
Keyword(s):
Video frame interpolation, Opticalflow free, Invertible network, U-Net based, Generative Flow
Abstract:
Video frame interpolation is the task of creating an interframe between two adjacent frames along the time
axis. So, instead of simply averaging two adjacent frames to create an intermediate image, this operation
should maintain semantic continuity with the adjacent frames. Most conventional methods use optical flow,
and various tools such as occlusion handling and object smoothing are indispensable. Since the use of these
various tools leads to complex problems, we tried to tackle the video interframe generation problem without
using problematic optical flow. To enable this, we have tried to use a deep neural network with an invertible
structure, and developed an U-Net based Generative Flow which is a modified normalizing flow. In addition,
we propose a learning method with a new consistency loss in the latent space to maintain semantic temporal
consistency between frames. The resolution of the generated image is guaranteed to be identical to that of
the original images by using an
invertible network. Furthermore, as it is not a random image like the ones by
generative models, our network guarantees stable outputs without flicker. Through experiments, we confirmed
the feasibility of the proposed algorithm and would like to suggest the U-Net based Generative Flow as a new
possibility for baseline in video frame interpolation. This paper is meaningful in that it is the new attempt to
use invertible networks instead of optical flows for video interpolation.
(More)