OpenCV  4.2.0
Open Source Computer Vision
Image Inpainting

.2.0+dfsg_contrib_modules_xphoto_tutorials_fsr_for_inpainting

Introduction

In this tutorial we will show how to use the algorithm Rapid Frequency Selective Reconstructiom (FSR) for image inpainting.

Basics

Image Inpainting is the process of reconstructing damaged or missing parts of an image. This is achieved by replacing distorted pixels by pixels similar to the neighboring ones. There are several algorithms for inpainting, using different approaches for such replacement.

One of those algorithms is called Rapid Frequency Selectice Reconstruction (FSR). FSR reconstructs image signals by exploiting the property that small areas of images can be represented sparsely in the Fourier domain. See [GenserPCS2018] and [SeilerTIP2015] for details.

FSR can be utilized for the following areas of application:

  1. Error Concealment (Inpainting): The sampling mask indicates the missing pixels of the distorted input image to be reconstructed.
  2. Non-Regular Sampling: For more information on how to choose a good sampling mask, please review [GroscheICIP2018] and [GroscheIST2018].

Example

The following sample code shows how to use FSR for inpainting. The non-zero pixels of the error mask indicate valid image area, while zero pixels indicate area to be reconstructed. You can create an arbitrary mask manually using tools like Paint or GIMP. Start with a plain white image and draw some distortions in black.

#include <opencv2/opencv.hpp>
#include <iostream>
using namespace cv;
int main(int argc, char** argv)
{
// read image and error pattern
Mat original_, mask_;
original_ = imread("images/kodim22.png");
mask_ = imread("images/pattern_random.png", IMREAD_GRAYSCALE);
// make sure that mask and source image have the same size
resize(mask_, mask, original_.size(), 0.0, 0.0, cv::INTER_NEAREST);
// distort image
Mat im_distorted(original_.size(), original_.type(), Scalar::all(0));
original_.copyTo(im_distorted, mask); // copy valid pixels only (i.e. non-zero pixels in mask)
// reconstruct the distorted image
// choose quality profile fast (xphoto::INPAINT_FSR_FAST) or best (xphoto::INPAINT_FSR_BEST)
Mat reconstructed;
xphoto::inpaint(im_distorted, mask, reconstructed, xphoto::INPAINT_FSR_FAST);
imshow("orignal image", original_);
imshow("distorted image", im_distorted);
imshow("reconstructed image", reconstructed);
return 0;
}

Original and distorted image:

Reconstruction:

Left image: fast quality profile (run time 8 seconds). Right image: best quality profile (1 minute 51 seconds).

Additional Resources

Comparison of FSR to existing inpainting methods in OpenCV

cv::imread
CV_EXPORTS_W Mat imread(const String &filename, int flags=IMREAD_COLOR)
Loads an image from a file.
cv::Scalar_< double >::all
static Scalar_< double > all(double v0)
returns a scalar with all elements set to v0
cv::xphoto::inpaint
void inpaint(const Mat &src, const Mat &mask, Mat &dst, const int algorithmType)
The function implements different single-image inpainting algorithms.
cv::waitKey
int waitKey(int delay=0)
Waits for a pressed key.
cv::IMREAD_GRAYSCALE
@ IMREAD_GRAYSCALE
If set, always convert image to the single channel grayscale image (codec internal conversion).
Definition: imgcodecs.hpp:66
cv::Mat::size
MatSize size
Definition: mat.hpp:2108
cv::imshow
void imshow(const String &winname, InputArray mat)
Displays an image in the specified window.
cv::xphoto::INPAINT_FSR_FAST
@ INPAINT_FSR_FAST
See INPAINT_FSR_BEST.
Definition: inpainting.hpp:95
inpainting.hpp
cv::Mat
n-dimensional dense array class
Definition: mat.hpp:792
cv::Mat::type
int type() const
Returns the type of a matrix element.
cv::Mat::copyTo
void copyTo(OutputArray m) const
Copies the matrix to another one.
cv::resize
void resize(InputArray src, OutputArray dst, Size dsize, double fx=0, double fy=0, int interpolation=INTER_LINEAR)
Resizes an image.
cv::INTER_NEAREST
@ INTER_NEAREST
Definition: imgproc.hpp:246
cv
"black box" representation of the file storage associated with a file on disk.
Definition: affine.hpp:52
cv::gapi::mask
GMat mask(const GMat &src, const GMat &mask)
Applies a mask to a matrix.