Image inpainting is the process of filling missing or fixing corrupted regions in a given image. Intensity values of pixels in a missing area are expected to be associated with the pixels in the surrounding area. Interpolation-based methods that can solve the problem with a high accuracy may become inefficient when the dimension of the data increases. Also, they suffer from finding the underlying texture and pattern in the missing region. The proposed approach produces candidate inpainting results by interpolating to the observed data at the different neighborhoods of the missing region using High Dimensional Model Representation with Lagrange interpolation. Later, a final inpainting decision is given among the candidates for each pixel in the missing region for a texture and pattern preserving inpainting by combining the information obtained from the co-occurrence matrix and from a patch found in the image that fits best to the missing region.
Interpolation-based methods can produce inpainting results with high accuracy if a part of a smooth region is missing as shown in Figure 1. However, in many cases, the underlying structure of the missing region can contain complicated texture and pattern. Such structures cannot be captured by interpolating to whole surrounding pixels of the missing region. If we have prior knowledge about the direction of the texture and pattern in the missing region, interpolating to the observed neighboring pixels in only that direction can help to retain the underlying structure.
 |  |  |
|---|---|---|
| (a) | (b) | |
| Figure 1. Toy example where interpolation produces good inpainting results. (a) The input image with a missing region (shown with black pixels), (b) inpainting result obtained using interpolation. | ||
We illustrate the aforementioned difficulty of using interpolation for texture and pattern preserving inpainting in Figure 2. Let us consider the part of a zebra body shown in Figure 2(a) where a small region (shown by red) on the vertical black pattern of the zebra body is missing. If we perform interpolation only using the pixels on the left and the right parts of the missing region, we lose the vertical black texture in zebra after inpainting (see Figure 2(b)). However, if we use the pixels on the upper and the lower parts of the missing region for interpolation, the interpolation can complete the missing region quite well as shown in Figure 2(d). Inpainting results using the neighboring observed pixels on the upper-right and the lower-left, and on the upper-left and the lower-right of the missing region are also shown in Figure 2(c) and 2(e), respectively.
 |  |  |  |  |  |
|---|---|---|---|---|---|
| (a) | (b) | (c) | (d) | (e) | (f) |
| Figure 2. Toy example that demonstrates motivation of the proposed method. (a) The input image with a missing region (shown by green). Interpolation results using the neighboring pixels on (b) the left and the right (0°), (c) the upper-right and the lower-left (45°), (d) the upper and the lower (90°), (e) the upper-left and the lower-right (135°) parts of the missing region, respectively. (f) Inpainting result of the proposed method. | |||||
Performing interpolation using the observed pixels in the direction of the underlying structure is not trivial, since we do not know the direction of the texture and the pattern in advance. The problem becomes even more complex than the example in Figure 2 when the underlying structure of the missing region consists of more complicated patterns (e.g., different textures and patterns in different directions) as in many natural images. This motivates us to exploit interpolation results obtained from different directions to develop an interpolation-based texture and pattern preserving algorithm for image inpainting.
Our study proposes a texture and pattern preserving interpolation-based algorithm for inpainting missing regions in color images. This work has been supported by the Scientific and Technological Research Council of Turkey (TUBITAK) under Grant 115E424.
EN
