Christmas is already here for image processing folks…

… Because the latest version 1.6.8 « XMas 2015 Edition » of G’MIC has been released last week 🙂 ! G’MIC (GREYC’s Magic for Image Computing) is an open-source framework for image processing that I’ve started to develop in August 2008. This new release is a good occasion for me to discuss some of the advances and new features added to the project since my last digest, published here 8 months ago. Seven versions have been published since then.

I’ll talk about the few improvements done on the G’MIC plug-in for GIMP (which is currently the most used interface of the project). But I’ll also describe more technical developments of the framework, which already permitted to design some cool new filters and above all, which promise great things for the future.

1. Overview of the G’MIC project

(Skip this section, if you already know what G’MIC is !)

gmic_banner_shadow

Fig .1.1. Mascot and logo of the G’MIC project, a full-featured open-source framework for image processing.

G’MIC is an open-source project developed in the IMAGE team at the GREYC laboratory (a public research unit based in Caen / France). It is distributed under the free software license CeCILL. It provides several user interfaces for the manipulation of generic image data. Currently, its most used interface is a plugin for GIMP (the plug-in alone seems to be quite popular, as it has been downloaded more than 2 million times so far).

G'MIC plugin for GIMP Overview

Fig. 1.2. The G’MIC plug-in (version 1.6.8) running inside GIMP 2.8.

G’MIC also proposes other worthy interfaces, as a command line tool (similar to the CLI tools from ImageMagick or GraphicsMagick) which fully exploits its capabilities. There is also a nice web service G’MIC Online to apply effects on images directly from a web browser (check it out, it has been re-designed recently!). Currently, the G’MIC core library (used by all those interfaces) provides more than 900 different functions to perform various image processing tasks. The library alone takes approximately 5.5 MB and has more than 100,000 lines of source code. There is a wide diversity of commands available for geometric and color manipulation, image filtering (denoising and enhancement by spectral, variational or non-local methods), motion estimation / image alignment, primitive drawing (including 3D meshes), edge detection, segmentation, artistic rendering, etc.  It’s a very generic tool, as you can imagine. To learn more about the motivations and goals of the G’MIC project, I suggest you take a look at the presentation slides. You can also meet the developers on the new forum, or on the IRC channel #pixls.us (on Freenode).

2. Improvements of the G’MIC plug-in for GIMP

Three major improvements have been done on the GIMP plug-in interface, these last months.

2.1. Import and export of image data via GEGL buffers

That’s probably the greatest news to announce about the plug-in itself. The G’MIC plug-in is now able to import and export image data from/to GIMP using GEGL buffers, which are the foundations of the current development version (2.9) of GIMP. In practice, the plug-in can process high-bit depth images (for example with pixels stored as 32 bits-floats per channel), without loss of numerical accuracy. Actually, the application of a single filter on an image rarely generates serious visual artifacts only because of the 8 bits quantization of an input image. But people applying many filters and effects sequentially on their images will appreciate that the numerical accuracy is preserved during their entire workflow. The animation below (Fig.2.1) shows the subtle quantization phenomenon occurring when certain types of filters are applied on an image (here, an anisotropic smoothing filter).

Where the application of the same G'MIC filter on an image by 8 bits and 32 bits per channel

Fig. 2.1. Comparison of output histograms generated by the same G’MIC filter applied on an image encoded with 8 bits or 32 bits-per-channel values.

The histograms in Fig.2.1 illustrate the fact we get a greater diversity of pixel values ​​when the filter is applied on an image stored as 32-bits floats rather than 8 bits integers. Applying such filters several times in 8 bits-mode could slowly create undesirable quantization / posterization / banding effects on the processed image. To say it briefly, the G’MIC plug-in seems to be ready for the next major stable release GIMP 2.10 🙂 (a big THANKS to Mitch from the GIMP team, who took the time to explain me how to achieve this).

2.2. Support for UTF-8 strings in widgets

The plugin now handles strings encoded in UTF-8 for all interface widgets, which will ease its internationalization. For instance, we have now a Japanese version of the interface (as well as Japanese translations for some of the filters), as shown in the screenshot below (Fig. 2.2).

The GIMP plugin G'MIC partially translated into Japanese

Fig.2.2. The G’MIC plug-in for GIMP, partially translated in Japanese.

2.3. A more responsive interface

The plug-in now reacts faster to a parameter change, particularly when an image preview is being computed. A cancellation mechanism has been added, and we don’t have to wait the rendering of the filter preview anymore before having the possibility to act on the interface. It might sound obvious, but this greatly improves the user experience.

Also check out the interesting initiative of Jean-Philippe Fleury – a nice contributor – who displays on his web page a gallery of most of the filters and image effects available in the G’MIC plug-in. It’s a very quick and easy way to get an overview of the filters, and to observe how they perform on different images (sometimes with different parameters).

3. The PatchMatch algorithm

PatchMatch is a fast algorithm for computing image correspondences using a block-matching technique. Since a few years, it raises a high interested in the image processing community, thanks to its computation speed and the fact it can be parallelized relatively easily. It has become then a key step in many recent algorithms that need to compute fast comparisons between image patches, including methods for reconstructing missing parts of an imagesynthesizing textures, or performing super-resolutionThus, I’m happy to announce that G’MIC has now a parallelized implementation of PatchMatch (through the native command -patchmatch) both for 2D and 3D images. Based on this recent implementation, I’ve been already able to develop and add three interesting filters in G’MIC, described hereafter.

3.1. Another filter for image Inpainting

A new image inpainting filter – using a multi-scale approach – has been included in G’MIC to reconstruct missing or corrupted image regions. Very convenient to pull your stepmother out of the holidays photos you shot in Ibiza! Note that this is not the first patch-based Inpainting algorithm available in G’MIC (this post on Pat David’s blog already presented such a tool in February 2014). It’s just a different algorithm, thus giving different results. But in this field, having the choice to generate alternative results is definitely not a luxury, given the ill-posed nature of the problem. And if – like me – you have a machine with a lot of CPU cores just waiting to be heated, then this new multi-threaded filter is for you! Here are two examples of results (Fig.3.1 and Fig.3.2) we can obtain with this Inpainting filter.

img

Fig. 3.1. Removing a bear from an image, using the PatchMatch-based Inpainting filter of G’MIC.

img

Fig. 3.2. Removing the Eiffel Tower, using the PatchMatch-based Inpainting filter of G’MIC.

A video showing how this latest example has been obtained (in only 1 minute 07) is visible below. This is a real-time video, no tricks were used, except using a PC with 24 cores! Even to me (the developer), it’s a bit magical to see the algorithm regenerating the whole trees at the feet of the Eiffel Tower. Actually, its does nothing more than cloning an existing tree from elsewhere in the picture. But that is still impressive. Of course, it does not work as well on all images where I’ve experimented the algorithm 🙂 .

This new Inpainting filter is roughly the same as the one Adobe introduced in Photoshop CS5 in 2010, under the (very windy) name “Content-Aware Fill”. The main strength of this kind of algorithms is their ability to reconstruct large coherent textured areas. Note also that my PhD student Maxime Daisy (who defended his thesis successfully two weeks ago, well done pal!) has worked on some nice extensions of such Inpainting methods to remove moving objects in video sequences (see his demo page for examples). These extensions for video Inpainting are not yet available in G’MIC (and are still very demanding in terms of computation time), but it may happen one day, who knows?

3.2. Texture re-synthesis

By using a very similar multi-scale approach, I’ve also added a new texture synthesis filter in G’MIC. It allows to synthesize a texture of arbitrary size from an input template texture (which is usually smaller). Of course, the filter does something smarter than just tiling the template texture: It actually regenerates a whole new texture having similar characteristics by copying / pasting some bits of the template texture so that the synthesized result does not contain visible seams. Fig.3.3 shows such a 512x512 re-synthesized texture (top-right) from an input template image having a smaller size 280x350 (left). A comparison of this result with a more basic tiling process is also shown (bottom-right).

gmic_retexture2

Figure 3.3. Example of re-synthesizing a complex texture using G’MIC and comparison with basic tiling.

Note that we already had a texture re-synthesis algorithm in G’MIC, but it worked well only with micro-textures. On the contrary, the example on Fig.3.3 shows that the new algorithm is capable of regenerating more complex macro-textures.

3.3 Make seamless textures

And here is the third (and last) PatchMatch-based filter added so far in G’MIC. This one allows to transform an input texture to make it appear seamless when tiled. The idea is a bit different than with the previous filter, as most of the original input texture is actually untouched here. The filter only performs some global color normalization (to remove low-frequency gradients due to the non-homogeneous illumination) and locally adds an inner (or outer) frame that makes the texture appear seamless. Look at what the filter can do on the two examples below (textures shown as 2x2 tiled). Not bad at all ! I have to admit anyway that the parameters are still a bit hard to tweak.

gmic_seamless1

Fig.3.4. Make a seamless texture with the third PatchMatch-based filter in G’MIC.

gmic_seamless2

Fig.3.5 Make a seamless texture with the third PatchMatch-based filter in G’MIC.

Believe me, having this PatchMatch algorithm now implemented in G’MIC is a really great news! No doubts many filters could benefit from this method in the future.

4. A more powerful expression evaluator

In my mind, G’MIC has always been a very convenient tool to quickly design and build complex image processing pipelines (possibly with loops and conditionals). However, when I do algorithm prototyping, it frequently happens I want to experiment with quite “low-level” operations. For instance, I may want to write nested (x,y) loops over an image, where each loop iteration performs quite “exotic” things at the pixel scale. This kind of low-level algorithms cannot be expressed easily as pipelines of macro-operators (assuming we don’t already have a macro-operator that does the job, of course!). Trying to express such an algorithm anyway as a G’MIC pipeline was a bit tedious, either because it required some tricks to write, or because it ended up in a very slow implementation (due to the nested loops over all the image pixels that keep being interpreted). Note that this problem is not specific to G’MIC : try to discuss with image processing folks working with Matlab to see how creative they are when it comes to avoid writing explicit loops in a Matlab script ! It’s a fact: image processing often requires a lot of (pixel) data to process, and prototyping heavy IP algorithms only with interpreted languages introduces a speed bottleneck.

My goal anyway was to be to able to prototype most of my algorithms (including those low-level ones) directly in G’MIC, with the constraint they never become ridiculously slow to run (compared to an equivalent C++ implementation, for instance). As you might have guessed, a conventional solution to this kind of problems would consist in embedding a Just-in-Time compiler into my interpreter. So that’s what I’ve done – but only for a subset of G’MIC – namely the mathematical expression evaluator. As one might expect, evaluating math expressions is indeed a key feature for an image processing framework such as G’MIC.

To be perfectly honest, this JIT has been already included in G’MIC for years. But it has been also greatly optimized and improved these last months to allow the parallelized evaluation of more complex expressions. These expressions can now be in fact considered as small programs themselves (containing loops, variables, conditional tests, pixel access, etc.), rather than just plain mathematical formulas. As a result, we now have a very efficient expression evaluator in G’MIC, that appears to be at least 10x faster than the equivalent feature proposed in ImageMagick (with its generic command -fx) !

Thus, G’MIC is now a tool good enough to prototype almost any kind of image processing algorithms, without sacrificing the computation speed. I wrote a quite comprehensive post about this improved expression evaluator a few months ago, so please take some time to read it if you want more details. Here again, this is very encouraging news for the future of the project.

5. An abundance of filters!

This section proposes a bulk list of some other features/effects/filters added in G’MIC since last April.The screenshots below often show those filters running  in the G’MIC plug-in for GIMP, but of course they may be applied also from all the other interfaces (the CLI tool, in particular).

  • Filters search engine: this has been a long-time request from users, so finally we have proposed a first draft for such a tool (thanks Andy!). Indeed, the number of filters in the plug-in keeps increasing and it is not always convenient to find a particular filter, especially if you forgot to put it in your favorites. Anyway, you have still to specify relevant keywords to be able to find you filters with this tool.
img

Fig.5.1. New filter search engine by keywords in the G’MIC plug-in for GIMP.

  • Vector Painting: This image filter is a direct consequence of the recent improvements made on the math expression evaluator. It basically generates a piecewise constant abstraction of an input color image. It’s quite funny to look at the full source code of this filter, because it is surprisingly short (19 lines) !
img

Fig. 5.2. The Vector Painting filter creates image abstractions.

  • Freaky B&W: This filter performs black & white conversion (grayscale to be precise) of an input color image. But it does this by solving a Poisson equation rather than applying a simple linear formula to compute the luminance. Here, the goal is to get a B&W image which contains all of the contrast details present in each color channel of the original image. This filter often generates images that have a kind of HDR look (but in grayscale).
img

Fig. 5.3. The Freaky B&W filter converts a color image into HDR-like grayscale.

  • Bokeh: This one tries to generate synthetic Bokeh backgrounds on color images (artistic blur). It is highly configurable and can generate Bokeh with various kind of shapes (circles, pentagons, octagons, stars ..). It can be used for instance to add some light effects to a color image, as illustrated below.
img

Fig. 5.4. Application of the new Bokeh filter on a color image.

  • Rain & snow: as the name suggests, this filter adds rain or snow on your images (example below is a zoom of the previous image).
img

Fig. 5.5. Adding rain on a color image with the filter Rain & snow.

  • Neon lightning: not much to say about this one. It generates random curves starting from a region A to another region B and stylizes those curves as neon lights. Convenient for generating cool wallpapers probably 🙂
img

Fig. 5.6. Effect Neon Lightning in action!

  • Stroke: This filter decorate simple monchrome shapes on a transparent layer with color gradients. The figure below illustrates the transformation of a simple text (initially drawn with a single color) by this filter.
gmic_stroke2

Fig. 5.7. Applying the Stroke filter to decorate a plain text.

  • Light leaks: this one simulates the presence of undesired light artifacts in pictures. In general, that is rather the kind of effects we want to remove! But the simulation of image degradation can be useful in some cases (for those who want to make their synthesized images more realistic for instance).
img

Fig. 5.8. Simulation of image degradation with the Light leaks filter.

  • Grid [triangular]: This filter converts an image into a grid composed of colored triangles, allowing many choices for the type of grids to apply.
img

Fig. 5.9. Transforming an image in a triangular grid.

  • Intarsia: This is a quite original filter. It allows to turn an image into a knitting diagram for Intarsia. The idea came from a user of the GimpChat forum to help her distributing custom knitting patterns (apparently there are websites that offer such patterns in exchange for hard cash). The filter itself does not modify the picture. It only generate a knitting diagram as an output HTML page. An example of such a diagram is visible here.
  • Drop water: I must admit I particularly like this one. The filter simulates the appearance of water droplets over an image. In its most basic version (with the default parameters), it looks like this:
img

Fig. 5.10. Adding water droplets on an image with the Drop water filter.

But it gets really interesting when the user defines its own droplet shapes by adding a transparent layer containing a few colored strokes on it. For instance, if you add this layer (here represented with pink strokes) to the leaf image:

img

Fig. 5.11. Adding a layer to define the desired shape of water drops.

Then, the filter Drop water will generate this image, with your custom water drops:

img

Fig. 5.12. Synthesis of custom shaped water drops.

Moreover, the filter has the good sense to generate its result as a stack of several ouput layers, each corresponding to the simulation of a different physical phenomena, namely: the specular light spots, the drop shadows, the own shadows, and the light refraction. It is then easy to modify these layers indivisually afterwards to create interesting effects with the image. For instance, if we apply a color-to-monochrome filter on the base image layer while preserving the refraction layer (computed from the initial color image), we end up with this:

img

Fig. 5.13. Applying the Drop water filter, followed by a colorimetric change on the initial layer only.

The short video tutorial below explains how to achieve this effect, step by step, using the G’MIC plug-in for GIMP. It takes less than 2 minutes, even for a complete beginner.

Even better, you can superimpose the result of the Drop water filter on another image. Here is an example where two portraits have been aligned then “merged” together, separated by a kind of liquid interface. It’s really easy to do, and the result looks pretty cool to me.

img

Fig. 5.14. The G’MIC Drop water filter applied to create a liquid interface between two distinct portraits.

Well, that’s it for the newest filters!

6. Other notable points

To end this long post, let me list some other interesting news about the project:

  • First of all, we have been contacted by Tobias Fleischer. Tobias is a professional plug-in developer for Adobe After Effects and Premiere Pro (among other video editing software). He has made ​​a tremendous job in developing a DLL for Windows which encapsulates libgmic (the G’MIC core library). And, he has used this DLL to implement a prototype of a G’MIC plug-in for After Effects. He was kind enough to share some screenshots with us of what he has done recently: you can see below an example of the G’MIC skeleton filter running in After Effects.
img

Fig. 6.1. Prototype of a G’MIC plug-in running in Adobe After Effects.

I can already hear grumpy guys saying: “But why didn’t he try doing that for a free software, as Natron, rather than a 100% proprietary software? “. But in fact, he did it also! And not just for Natron, but for any other video processing software compatible with the OpenFX API. The screenshot below shows a prototype of an OpenFX-compliant G’MIC plug-in running in Sony Vegas Pro. There are probably no reasons why this couldn’t work with Natron too.

img

Fig. 6.2. Prototype of an OpenFX-compliant G’MIC plug-in running under Sony Vegas Pro.

Of course, all this should be still considered as Work In Progress. There are probably still bugs to fix and improvements to do, but this is really promising. I can’t wait to see these plug-ins released and running on Natron 🙂

  • Let me continue with some other good news: Andrea, a nice contributor (who is also the developer of PhotoFlow) managed to understand why G’MIC had frequent crashes under MacOSX when computations were done using multiple threads. He proposed a simple patch to solve this issue (it was actually related to the tiny stack size allocated for each thread). G’MIC under MacOSX should be then fully functional now.
  • The webcam/video interface ZArt of G’MIC has been also improved a lot, with new added filters, automatic detection of the webcam resolutions, and the possibility of having a dual-window (one for monitoring + one for display, on a second screen).
img

Fig 6.3. The ZArt interface running the G’MIC filter Drop water in dual-window mode.

  • A new animated demo has been also added to G’MIC, through the command -x_landscape. The demo itself is of little interest for the user, but was gainful for me to test/improve the speed of the G’MIC interpreter (and isn’t it funny to have this kind of things in G’MIC 😉 ). Note that all animated and/or interactive G’MIC scripts are available via this invokation of the CLI tool:
$ gmic -demo
img

Fig. 6.4. Animated virtual landscape in the G’MIC CLI tool gmic with command -x landscape.

As I had more and more small and funny animations implemented into G’MIC, I’ve decided to put them together as a little intro, entirely coded as a single G’MIC script (750 lines of code). The result smells the divine fragrance of old 8/16 bits machines (the corresponding video is shown below). Remember that all have been generated 100% with the G’MIC interpreter. Isn’t that cool ?

8. So, what’s next?

Now it’s probably time to rest a little bit 🙂 . 2015 has already been a very busy year for the G’MIC project. Christmas is rapidly approaching and I’ll take a (small) break.

To be honest, I don’t have yet a clear idea about the things I’ll focus on. In any case, it appears that G’MIC keeps improving gradually and can potentially raise the interest of more and more people. I wanted to thank especially all the contributors of the project and users of the software who give regular feedback to make it better and better. Also, be aware that each significant addition is usually subject to an announcement on the Google+ page of G’MIC, so it’s easy to stay informed about the project progression, day after day. Dear reader, would you have any proposal for major improvements ?

Let me finally conclude this post by proclaiming it mightily: “Long live open-source image processing ! And Happy XMas everyone !

Cheers,

David.