History

Until now, there was an OTB-Applications package built on top of the library, which provided command-line tools, Qt and FLTK based GUIs, and Qgis C++ plugins. This package was not very popular among our users : it was lacking of documentation, missed really useful auto-generated GUIs and did not provide an interface to interpreted languages. These applications were barely more than a serie of mains decorated with an argument parser. However, these applications were providing a lot of image manipulation utilities and high-level processing chains like ortho-rectification, classification or segmentation. Being too obscure, they simply did not get the attention they deserved.

Six month ago, we started a side project called OTB-Wrapper, which was designed as a sand-box to experiment with all the possible enhancements to make the applications really useful. After a while, we came up with a complete solution and implemented it directly into the OTB library. All applications from the OTB-Applications package have then been migrated to this new framework, also directly into the OTB library.

The new framework in a nutshell

So, what does this new framework provide to the users ? A lot of things ! See :

• Each application is now compiled into a single tiny shared-library,
• Writing new applications in an external project is as simple as inheriting the base class and calling a CMake macro,
• These applications shared libraries are loaded dynamically as plugins of several generic access points :
• Directly from C++
• From an interpreted or compiled higher level language like Python through the SWIG interface,
• From a command-line launcher, like the old applications,
• From a nice auto-generated Qt GUI,
• From QGis through seamless OTB plugins.

The general philosophy of this new framework is to have to write the processing only once, and then use it from whatever environment, programming language or software you want through one of these access points.

Success !

Even before the official release, success is already showing up :

• A simple python script allows to generate documentation for all the applications,
• A simple python script allows to generate applications file descriptors for integration in external tools like the KEO system from ESA,
• A simple python script allows to perform batched processing chaining several applications,
• Qt GUIs are really nice to use, with a lot of useful features like automatic parameters estimation (for instance, the ortho-rectification application provides you with the best-fit parameters for your image),
• The python interface allows nice integration within QGis,
• And the command-line interface is more robust and easier to use.

Learn more in next posts

If you want to give this a try, clone a the OTB mercurial repository and activate the BUILD_APPLICATIONS flag. You are ready to go ! In the following posts, we will focus on key features that makes this new framework so exciting. But for now, it is time for us to wish you all a merry christmas !

Julien, on behalf of the OTB team

With the Pleiades launch a few weeks ahead, there has been a lot of ongoing work from the OTB team to prepare for these new data. Since a lot of discussions happened off the list, either with phone meetings, or on other mailing lists (we will see later on why), and since we now have a comprehensive knowledge of what the support of Pleiades images in OTB will be, it is about time we explain it to users and developers.

Reminder on Pleiades images

We will start with a reminder : Pleiades images will be available in Jpeg2000 format allowing a high compression rate, tiled in 2048×2048 pixels, so a standard Pleiades image will contain a few hundreds of tiles. A typical Pleiades image size is 40 000 x 40 000 pixels (corresponding to a 20 km by 20 km area), and one of the standard product is a pan-sharpened one, which is a merge of the high resolution panchromatic and lower resolution multispectral imagery to create a single high resolution color image. These images would be very heavy without Jpeg2000 compression : for instance, if the Jpeg2000 file weights 1.7 Go, the decompressed file weights 7.3 Go. To decode those JPEG2000 files, there are a few commercial libraries, the most popular beeing Kakadu. There is also some open-source alternatives, among which OpenJPEG seems to be the most advanced. Of course, for Pleiades support in OTB, we need an open-source solution, even if GDAL can be compiled with a driver based on Kakadu (has to be the commercial version, not the trial one available for free under restrictions of use).

Open-source JPEG 2000 implementation

Back in 2007, the OTB people here at CNES already spotted OpenJPEG as the best open-source bet for JPEG2000 support in OTB. Since the library was missing some important features (like partial decoding and MCT support), they set-up a CNES contract with the CS Company in order to add these features to OpenJPEG. But when this contract was over, the new features were not integrated in OpenJPEG trunk but left on a side branch (the so-called v2), because nobody in OpenJPEG community had in-dept knowledge of what had been done in this new version. The development of the trunk went on with bug-fixes and enhancements, while the v2 branch did not evolve. In the meantime, we started a driver in OTB based on the v2 OpenJPEG version, but this was not a sustainable option, because the v2 was barely maintained. Thanks to its more advanced functionnalities, the v2 branch of OpenJPEG also received interest from other FOSS projects in need for advanced decoding capabilities, and got integrated in at least ITK, GDCM and GDAL.

Four months ago, we agreed with the OpenJPEG community that we needed to get the v2 merged with the OpenJPEG trunk. Mickaël Savinaud from the CS OTB team got involved into OpenJPEG development to get the merge between v2 and trunk done, and we now have a full-featured version of OpenJPEG in trunk thanks to his great work and to the support of the OpenJPEG community.

Now, it is time to face the truth : even with this new version, in terms of decoding performances, this state-of-the-art open-source software is way worse than Kakadu : decoding one tile is way slower than with Kakadu, and one tile of 2048×2048 pixels is the atomic unit for decoding (i.e. if you need one pixel, you will still need to first decode the whole tile).

We looked into optimization : reducing file seeking, implementing some macroscopic code optimisation (mainly avoiding pessimisation) of the Tier1 part which is clearly identify as the performances bottleneck (see figure below with profiling reports using kcachegrind). We made some clear progress, but the JPEG2000 standard is complex, and without knowledge of the big picture, we could not gain a lot.

KCachegrind representation of time percentage per functions

What does it mean for Pleiades data ?

It means that simply decoding a full Pleiades image at full resolution will take about 20 minutes on a decent i5 CPU with 4 Go of RAM, while the same image in standard TIF format would have taken 8 minutes and Kakadu takes only 4 minutes. Any OTB processing pipeline streaming the whole image will be limited by this decompression time. And we are not even talking of sub-sampling or estimating statistics at some point in the pipeline.

Of course, we could tell our users to buy a Kakadu licence and compile GDAL with the Kakadu driver enabled, but we can not even tell them how much it will cost, and this is clearly in contradiction with the open-source philosophy of OTB. Still, this remains an option to get high performances JPEG2000 support in OTB, but you are on your own if you choose this solution.

Now, what will it be possible to do using Pleiades data with the open-source solution OpenJPEG in OTB ?

• On OTB side: No specific handling will be added. It is the responsibility of the developer to know that the JPEG2000 driver is not as fast as the other drivers. The developer has the possibility to easily select the appropriate resolution and accordingly set the size streamed region (to reduce the number of decoding operations).  Of course, metadata related to geometric and radiometric calibration will be supported through OSSIM. Moreover, we will support the GMLJP2 box included in the Pleiades to support geo-information in case of ortho-image products.

• On applications side:

• We will provide an application able to convert a Pleiades image, either full or extract, from any JPEG2000 resolution, into another file format (like TIF for instance). Of course, one will need a lot of disk space to handle the decompressed data. Please note that this is not fully supported by Kakadu trial version, which decodes a maximum of 3 channels (and which use is restricted by the way),

• All applications will support Pleiades images, but it will be highly recommended to first decompress the image to disk for reasonable performances and if more than one processing is to be executed on the image,

• On Monteverdi side:

• Pleiades images will open in a dedicated new type of data in Monteverdi interface, and the user will be able to select the resolution level for decoding,

• The viewer module will accept this new type of data and we will be able to efficiently navigate within Pleiades images : fast quicklook computation due to JPEG2000 capabilities, instant navigation into a 4 tiles squared area thanks to caching, and quick-enough refresh when moving outside this cached area thanks to parallel tiles decoding. We expect the navigation experience to be quite good !

• The extract ROI module will accept this new type of data and allow to select an extract of the image using the fast quicklook ability. Caching (which will decompress image to disk in TIF format) will be recommended (this behaviour might be extended to other modules later).

• The writer module will accept this new type of data for conversion purposes.

• All other modules will not accept this new type of data : to use them, the user will first have to use the extract ROI module which will produce a plain image type accepted by all modules. Caching will be recommended for this purpose.

Closing Thoughts

We can hope that the widening use of JPEG2000 in spatial imagery will encourage the improvement of open-source JPEG2000 libraries performances. When and if this becomes available, you will be able to use those datasets in OTB seamlessly, as any other image.

Until then, we can look at the bright side :

OTB will provide an open-source and free access to Pleiades data for those not owning a commercial image processing software  (please note that Gdal is looking into the new OpenJPEG version and might also support Pleiades through it in a near future).

OTB will provide an efficient solution for Pleiades data decompression and visualization. 

Last, the whole set of OTB processing remains fully available, at the expense of extra decoding time or once your Pleiades data have been decompressed to disk.

This has also been a great experience of collaboration between open-source projects towards a common goal. We would like to thank again the members of the OpenJPEG community, as well as the OTB Team at CS for their great work !

Classical scheme

The classical scheme to compute elevation maps from these kind of data is to first resample both images of the pair into the epipolar geometry (with optional refinement of the sensor-to-ground function using GPCs and tie points) , where displacement related to height only occurs in the horizontal direction. A block-matching algorithm (just like otb::FineRegistrationImageFilter does) is then applied to retrieve the disparities. Finally, disparities are casted back to height values using sensor modelling and the elevation map is resampled to a cartographic projection. Several post-processing may be needed to make the produced elevation map usable, which are out of the scope of this post.

However, resampling to epipolar geometry is an heavy process that must be done for both images, and in the case of push-broom acquisition system, this resampling may be tricky. Moreover, this resampling is often done with an average elevation hypothesis, which leads to a wide range of disparities to explore in case of highly varying relief over the scene.

Implicit exploration of epipolar lines

In Orfeo ToolBox, the otb::GenericRSTransform allows to map points from sensor geometry to ground, but it also allows to map points between two sensors geometries : we can therefore use it to build a function that will transform a point of the first image of our stereo pair to its homologous point in the second image of the stereo pair, according to both RPC modelling (possibly refined). This function also uses an elevation value as a parameter (either average or drawn from a Digital Elevation Model), so that we in fact have a function which associate a point with a candidate elevation to its homologous point in the other image of the stereo pair. If we transform the same point with a range of candidate elevation, we are then implicitly exploring its associated epipolar line in the other image (a similar approach is performed by the MicMac software).

A new OTB filter

Using this principle, we developped a new OTB filter called otb::StereoSensorModelToElevationFilter. This filter takes a pair of along-track stereo images, and behave as follows. For each pixel of the first (reference) image:

1. Extract the neighborood of the pixel in the reference image according to the user-defined radius,
2. For candidate elevations in a user-defined range:
1. Transform the patch grid into the secondary image with the sensor-to-sensor function and the candidate height
2. Interpolate the corresponding secondary patch
3. Compute cross-correlation of reference and secondary patches
3. If best correlation is high enough, keep the corresponding best elevation candidate as the final elevation.

The output of the algorithm is directly an elevation map in meters, in the same geomety and grid as the reference image of the stereo pair. There is no need to resample both image in epipolar geometry, we only need to interpolate necessary locations along the epipolar lines in the secondary image. One can then orthorectify the elevation map in any desired cartographic projection.

We can see that a difficult part of this algorithm lies in the definition of the appropriate elevation exploration range. First, it requires to know about the average elevation over the area, and even with this information, elevation might be varying a lot over the scene, so this range has to be pretty wide, and lead to high computation time and lots of matching errors. Therefore, the filter allows to draw the initial elevation from a DEM (like SRTM for instance). The algorithm will then explore elevation values around this intiial local elevation, which allows to define a much more narrow range.

Results

Our first result will be a pair of Worldview 2 images over the city of Toulouse. The following results were obtained on a detail of the multi-spectral images (2.3 meters resolution) over the city stadium.

©CNES

©CNES

Applying the same algorithm to the panchromatic images (0.5 meters resolution) leads to noisier results, but we can still clearly distinguish the shape of the stadium. Image on the left is the correlation mask associated with the result (and produced by the filter). Dark values correspond to low correlation, brighter ones correspond to higher correlation. Please note that in this case, initial height has been subtracted from the result (filter option).

Next results were obtained from a pair of WorldView 2 multi-spectral images (2.3 meters)  over the area of Aurignac, in France, an area of agricultural and natural landscape with lots of valley and hills. The two images were first properly down-sampled at a resolution of around 11.5 meters before the process was applied. The upper-left corner is more blur because it is an area which is only viewed in one of the image, and thus contains only the initial SRTM elevation.

The two images bellow shows the difference between the SRTM (on the left) and the estimated elevation (on the right).

If one does not want to use an initial DEM, the filter can start from a user-defined average elevation. Next image shows the result of using an average elevation on the Aurignac scene. Red pixels are those for which correlation fails. The result looks worse than the one using an initial DEM because in the DEM case pixels for which correlation fails are given the intial DEM value, which leads to a smooth result.

Conclusion

Of course, this block-matching algorithm is still very simple, and we do not claim that these results are better, faster or even valid with respect to those one can produce with similar tools. For instance, it is obvious that the results suffers from the well known fattening effect, and no particular filtering is performed to reduce it. The same holds for occlusions, incorrect matches or stereoscopic effects. Using some work from Ipol would be of great interest in this purpose. Still, this filter provides a very simple tool to perform elevation map estimation from along-track stereo pairs which makes use of the generic implementation of sensor-to-ground and ground-to-sensor transformation in Orfeo ToolBox to avoid the need for explicit epipolar geometry. Built-in access to DEM allows to initialise the height map with sound values and thus to narrow the search space. One can also make use of the filter streaming capability to produce only part of the image, which is more delicate with the standard scheme. Future work is to speed-up the computation, since the filter currently makes an intensive use of the sensor to sensor transform and of interpolation. Reviewers are welcome to read and comment the code which can be found here, whereas users can give it a try using the new otbStereoSensorModelToElevationMap-cli application from the latest revision.

This new functionality has also been included into the Monteverdi application to provide a intuitive way to perform complex band computation the easy way. The module also prevents error in the mathematical command by checking the expression as the user types it, and notifying information on the detected error:

The new Monteverdi Band Math module showing syntax error on mathematical expression.

Correct mathematical expressions are highlighted in green.

This feature is already available in the release 3.6 of the Orfeo Toolbox.

Give it a try!

The new otb::FineRegistrationImageFilter uses the full range of image-to-image metrics available in ITK (see the available metrics here) to perform block matching on a given neighborhood at each location of the reference image. The filter has two outputs, the optimum metric field and the deformation field (i.e. displacement in each image direction for which the optimum has been found), and both outputs support streaming. The filter performs dichotomic sub-pixel refinement up to a user defined accuracy, allows to generate coarser result with respect to a given grid step and can take into account an initial offset between the two images. It can also process reference and moving images with different sizes, origins and spacings. Here are some results of applying this filter to a pair of small stereo patches with two different metrics: the correlation and the mean reciprocal square difference (MRSD).

First image

Second image resampled (correlation)

Second image resampled (MRSD)

Second image of the stereo pair

Correlation field

MRSD field

Displacement (correlation)

Displacement (MRSD)

This new filter can be accessed through Monteverdi using the new Fine Correlation module. This module allow to perform displacement estimation using the standard correlation metric. It also includes an optional gaussian smoothing of the input images, an optional resampling of the second image and allows to tune all the parameters in a clear and compact way. Here is what it looks like.

The new Fine Correlation module for Monteverdi

Here is an example of application of this new feature. Using Monteverdi, we produce two patches of ortho-image from a Quickbird bundle product over a cloudy part of the scene. The panchromatic image has a ground sampling distance of 0.7 meters, whereas the multispectral one has 2.8 meters.

Panchromatic

Multispectral

Due to the space between the panchromatic and multispectral detectors, there is a small stereo effect between the two images. This space also induces a small time delay between the two images, that is why some moving objects (such as cars) show a small displacement. These two effects are well known (see this paper for instance).

Using the new Fine Correlation module, it is possible to estimate displacements between the intensity of the multispectral image and a gaussian-smoothed version of the panchromatic image. In the below image of the estimated displacement amplitude, we can clearly see that the cloud, being a lot higher than surrounding ground, shows a noticeable displacement due to the stereo effect.

Correlation field

Amplitude of displacement

Please feel free to give a try to the new filter and module, and do not hesitate to send your feedback!

Basically, it gets data which are provided as small image files to produce a big image of the area requested. The available source for now are open street map,  landsat background provided by nearmap, and a hill shaded terrain model.

After launching Monteverdi, just pick up the tilemap import:

You can then choose your location, either by providing the latitude and longitude or if you have no idea of it, the name of the city, street, town, country, etc. Also pick up the level of zoom (0 is the entire earth, 18 is really zoomed in) and the size of the image you want as an output.

After that, you get some nice images that you can plug to the other processing algorithms available in Monteverdi:

This is still quite experimental so we are willing to get any suggestion on how to improve that. The tiles are kept locally in cache to minimize the network transfer when working repeatedly on the same area, but try to stay reasonable in term of how much data are requested…

Note that the stuff is working on both direction even if we illustrated only the part which is getting the image. The hill shading part is provided by the OTB server and all the tiles, several hundreds of thousand, have been generated by OTB. This behave (almost) like a standard writer, so tiling your own satellite images (with of without additional processing in between) should be just plumbing some filters together.

For a few weeks now, the OTB development team has been working on this task called refactoring. In software development, the purpose of refactoring is to rewrite parts of the code to enhance its robustness and clarity, to make it easier to maintain and re-usable in other contexts, without modifying its external functional behaviour (source: wikipedia). That is why end users should not notice a lot of changes appart from minor changes in the way it looks. But behind the curtains, it makes a big difference.

The whole visualization module is now fully compliant with the Model-View-Controller architecture in use in the most recent OTB applications. The initial limited set of classes has been splitted into several lighter classes and customization entry points have been carefully introduced in each critical part in order to provide maximum extensibility at minimal cost. Adding a new user interaction, a new type of curve to display or a new eyecandy object rendering onto the image is now as simple as rewriting a light class with about no code overhead.

The otbFeatureExtractionApplication was chosen to beta-test the refactored module, and was therefore modified to use the new visualization tool. Migrating the applications essentially involved removing hundreds of lines of code without compromising any of its functions, and therefore greatly improved its clarity, robustness and maintainabilty. We are already on the way to migrate other applications in order to make everyone benefit from the changes.

The otbFeatureExtractionApplication now uses the new visualization module

Using the new design, we also built a new overlay component to render the content of a shapefile or a kml file over an image in about no time.

Using the new design, a tool to quickly render vector data files (shapefile or kml) has been built

Of course, this does not extend the functionnal scope of the Orfeo ToolBox or its applications, but in the future, it will help us to reduce the cost and improve the robustness of new applications. Developping users will also find it easier to extend it to suit their needs. For further technical details, you might want to read the OTB wiki page on this topic, which includes a small tutorial to start using the new visualization module.

The application has been redesigned from scratch and will be distributed with the next release of OTB (2.8) coming next Monday. You can now load full images, as only the area displayed will be processed. You also have access to all the parameters (you can refer to the ICIP paper for details: E. Christophe, J. Inglada, Robust Road Extraction for High Resolution Satellite Images, ICIP 2007).

Road Extraction Application

One of the main improvement is coming from the changes in the vector data handling. Now, OTB knows how to convert these data between different projection systems (which could be map projections or sensor models). That means that you can now export the results of the main OTB application in shapefile format to load it in you favorite GIS (Quantum GIS for example) or in KML to load it in Google Earth.

For example, with the previous extraction, Google Earth will display:

Road extraction results on Google Earth

You can notice that there is an offset of several meters due to registration problems. Indeed the orthorectification of the image used for the extraction was not done with OTB and images on Google Earth can also suffer from registration problems.

Looking as such results may give you some ideas of new stuff to try with OTB. We are bubbling with ideas but lacking the time to try them all!

One last thing, the algorithm included here is far from being perfect and was designed mostly to be fast enough to allow interaction. There are many ways to improve the results and adapt the algorithm to your needs. The good thing is that this application has been designed following the MVC architecture. That means that the visualization (the View) is decoupled from the processing (the Model). You could quite easily create a new model with a completely different algorithm to have a better road extraction application. Feel free to reuse the code!

Object counting includes a very broad field of problems. For this first version of the application, we have narrowed the problem to the case where the objects to be counted are compact and radiometrically homogeneous. In this case, an operator can select several of the objects she wants to count as training examples and a simple algorithm can find the other objects in the image. After that, the counting step is trivial.

The OTB library offers all the building blocks needed for this kind of algorithm. Actually, 2 different algorithms have been implemented in the application by now.

The first one uses a supervised classification (a One-Class SVM) fused with the result of a Mean-Shift clustering. After that, the connected components of the classifed image are extracted and counted. This approach is not very stable and needs very pertinent examples in order to give good results. However, it can be useful in some cases.

The second approach, which is rather robust is as follows. Using the examples entered by the user, a mean spectral reference is computed (remember that we are assuming that the objects are radiometrically homogeneous). This spectral reference is used to compute a spectral angle for all the pixels in the image. The spectral angle is thresholded. After that, the algorithm is identical as the SVM one: extract the connected components and count.

The following image shows an example of use of the application.

The red polygons are the examples selected by the operator. As you can see, some parameters can be selected on the GUI. The resulting objects are plotted in blue on top of the image.

What is also interesting about this application is that you can load a full scene (say, 30000×30000 pixels) and tune the algorithm on a small area. The tuning and the generation of the results for the small area will take only a few seconds. Once you are happy with the results, you can run the processing on the full image by using the “File” menu. The results can be exported as a raster labeled image or in vector format (shape or kml). With the latter option, and if you are using ortho-rectified images, you can upload the results of the counting to GoogleEarth!

The source code for the application is as usual available in the Mercurial site. You will need to update both the library and the applications in order to compile the application.

Of course, before the registration starts, the resampled moving image is identical to the moving one. Using the widgets on the lower right part of the GUI, the user can tune the parameters for the disparity map estimation. Using the Run button, the disparity map estimation is triggered on the displayed area only. This way, the optimum set of parameters can be found by try and error.

Once the disparity map is estimated, it is displayed on the lower left part of the GUI. A color composition is used as follows: R= horizontal disparity, G= vertical disparity, B=modulus of the deformation field.

The different components of the color composition can be checked or unchecked at will. Also, on the upper part of the GUI, the resampled moving image (the result of the registration using the estimated disparity map) will be displayed on the blue channel.

Once the user is satisfied with the results obtained on the displayed area, she can choose to apply the registration on the whole image. This is done by selecting to save the registered image and/or the disparity map (File menu).

We have chosen to use a disparity map estimation based on normalized correlation, but other choices are available given the fact that many disparity map estimation filters (using PDEs) are available in OTB.

The application still needs some minor tweaks before it is released, but interested users can start playing with it and giving some feedback.