Coloring With Limited Data: Few-Shot Colorization via Memory-Augmented Networks

Coloring With Limited Data:
Few-shot Colorization via Memory-Augmented Networks

(CVPR 2019)

Seungjoo Yoo
Korea University

Hyojin Bahng
Korea University

Sunghyo Chung
Korea University

Junsoo Lee
Korea University

Jaehyuk Chang
NaverWebtoon Corp.

Jaegul Choo
Korea University

Responsive image — Figure 1: Make the most out of your limited data with our fully-automatic colorization model MemoPainter. (Res-cGAN is MemoPainter without our proposed memory networks.)

Abstract

Despite recent advancements in deep learning-based automatic colorization, they are still limited when it comes to few-shot learning. Existing models require a significant amount of training data. To tackle this issue, we present a novel memory-augmented colorization model MemoPainterunpaired that can produce high-quality colorization with limited data. In particular, our model is able to capture rare instances and successfully colorize them. We also propose a novel threshold triplet loss that enables unsupervised training of memory networks without the need of class labels. Experiments show that our model has superior quality in both few-shot and one-shot colorization tasks.

Paper and Supplementary Material

[Paper] [Bibtex] [Code will be released]

Paper
CVPR, 2019.
Seungjoo Yoo, Hyojin Bahng, Sunghyo Chung, Junsoo Lee, Jaehyuk Chang and Jaegul Choo. "Coloring With Limited Data: Few-Shot Colorization via Memory-Augmented Networks"

Method overview

As illustrated in Fig. 2, MemoPainter consists of memory networks and colorization networks. During training, memory networks learn to retrieve a color feature that best matches the ground-truth color feature of the query image, while the colorization networks learn to effectively inject the color feature to the target grayscale image. During test time, we retrieve the top-1 color feature from our memory and give it as a condition to the trained generator.