Despite recent advancements in deep learning-based automatic colorization, they are still limited when it comes to few-shot learning. Existing models require a significant amount of training data. To tackle this issue, we present a novel memory-augmented colorization model MemoPainterunpaired that can produce high-quality colorization with limited data. In particular, our model is able to capture rare instances and successfully colorize them. We also propose a novel threshold triplet loss that enables unsupervised training of memory networks without the need of class labels. Experiments show that our model has superior quality in both few-shot and one-shot colorization tasks.
As illustrated in Fig. 2, MemoPainter consists of memory networks and colorization networks. During training, memory networks learn to retrieve a color feature that best matches the ground-truth color feature of the query image, while the colorization networks learn to effectively inject the color feature to the target grayscale image. During test time, we retrieve the top-1 color feature from our memory and give it as a condition to the trained generator.