DaVinci's Shadow - Software - Beneath the Waves

DaVinci's Shadow

article by Ben Lincoln, software by numerous developers

DaVinci's Shadow is an experimental fork I created of DaVinci, a command-line data-processing tool written by Arizona State University for NASA's Jet Propulsion Laboratory. The changes I made are minor, but very necessary for some of the core functionality I wanted to use in The Mirror's Surface Breaks.

If you just want to process images with TMSB, you can ignore everything on this page. You don't need any of it. The main reason this page is here is to explain why I needed to fork DaVinci to do what I wanted to do with TMSB, and to comply with the GPL by providing full source that will allow someone who wants to to build it themselves.

Alterations From The Mainline Version

DaVinci's Shadow 2.9.2 is based on version 2.09 of DaVinci (as of version 1.3 of The Mirror's Surface Breaks - prior releases were based on version 2.05 of DaVinci), and has had these modifications performed:

libpng 1.2.3 has been replaced with libpng 1.2.46. This required a minor modification to Makefile.in for the iomedley subcomponent.
The standard DaVinci code has a long-standing bug related to saving 16-bit-per-channel (AKA "48-bit") TIFFs. The data is represented internally as 16-bit signed integers, but TIFFs use a 16-bit signed format. DaVinci shifts the data down by 32768 when TIFFs are read, but does not shift it up by 32768 when saving TIFFs, resulting in corrupted output images. DaVinci's Shadow corrects this.
The standard DaVinci code has a similar bug (also present for some time) related to 16-bit-per-channel PNG files, and in addition, this type of PNG file is not read correctly, because the 16-bit values are read in big-endian order instead of little-endian order. DaVinci's Shadow corrects this as well, although the code has not been tested on a big-endian platform to verify that it hasn't broken the same thing there.
Several changes which were made to DaVinci in an attempt to work around the 16-bit-per-channel image issue in a different way have been reverted to the method I used, because they caused problems for the way I use this software. See the "16-Bit-Per-Channel Image I/O - Detailed Discussion" section, at the end of this article.
The standard DaVinci code introduced a new bug related to TIFFs sometime after 2.05, in which 16- and 32-bit-per-channel images with three or four channels were stored in a way that caused other software to interpret them as a single greyscale channel and multiple alpha channels, instead of an RGB colour image or an ARGB colour image. DaVinci' Shadow corrects this as well.
~~The TIFF output code has been modified to use deflate/zip compression instead of LZW, because it results in much smaller files~~. [ This change was incorporated into the main-line DaVinci. One line of source code I wrote has a non-zero probability of being used for processing images from space probes! ]
Built-in functions for converting values in the RGB colourspace to and from values in the YUV, HSL, HSY, and IHSL colourspaces have been added.
- rgb2yuv()
- yuv2rgb()
- rgb2hsv()
- hsv2rgb()
- rgb2hsl()
- hsl2rgb()
- rgb2hsy()
- hsy2rgb()
- rgb2ihsl()
- ihsl2rgb()
- rgb2ynuv() - same as rgb2yuv(), except returns U and V values in the range 0.0 - 1.0
- ynuv2rgb() - same as yuv2rgb(), except accepts U and V values in the range 0.0 - 1.0
- rgb2pdfhsy() - same as rgb2hsy(), except uses the slightly different conversion coefficients specified in Adobe's PDF specification.
- pdfhsy2rgb() - same as hsy2rgb(), except uses the slightly different conversion coefficients specified in Adobe's PDF specification.
A built-in function for calculating the range (range()), median (med()), and an extended version of the "moment" function (momentext()) which returns the median and range in addition to the standard moments of the original function have been added.
The THEMIS-related code has been disabled in the Windows® build, because I couldn't get the module to load. It is still enabled in the Linux/Unix-like build, although I have not tested it.
Numerous custom (and customized) user-defined functions have been added in the dshadow.dvrc library file.

General-purpose data-manipulation functions:
- anormalize() - performs an aggressive normalization on the input data - this is discussed in detail in TMSB XML Schema Part 6: Transformation Profiles.
- clipdata() - clips the upper and/or lower bound of the input data.
- iiaz() - "invert independently about zero" - inverts the relationship between large and small values, while retaining the distinction between positive and negative values.
- naav() - normalizes the data about an arbitrary value - this is discussed in detail in TMSB XML Schema Part 6: Transformation Profiles.
- todouble() - a shortcut function to convert input data of arbitrary numeric type to double-precision floating point in the specified range (default: 0.0 - 1.0).
- tofloat() - a shortcut function to convert input data of arbitrary numeric type to single-precision floating point in the specified range (default: 0.0 - 1.0).
- tointeger() - a shortcut function to convert input data of arbitrary numeric type to byte (8-bit unsigned), short (16-bit signed), or int (32-bit signed) integer format.
- tobyte() - a wrapper for tointeger() which always converts to byte (8-bit unsigned) format.
- toint() - a wrapper for tointeger() which always converts to int (32-bit signed) format
- toshort() - a wrapper for tointeger() which always converts to short (16-bit signed) format
General-purpose image-related functions:
- applypalette() - applies a palette to a greyscale image. Designed for use specifically with gradients generated by buildgradient().
- blendcolourhsy() - performs a luminance/colour blend similar to the method used in Photoshop® (see Secrets of Photoshop's Colour Blend Mode Revealed (Sort Of))
- blendcolourrgb() - a wrapper for blendcolourhsy() which accepts RGB input images instead of HSY
- buildgradient() - generates a colour gradient based on user-defined control points/colours.
- momentcube() - uses the one-dimensional output of DaVinci's moment() function to generate a derived three-dimensional image cube which contains statistically-calculated greyscale images (see Calculated Greyscale Images).
Specialized image-related functions. For more information on the DCS functionality, see Decorrelation Stretch Images. For more information on the vegetation index/satellite imagery-related functions, see Calculated Greyscale Images.
- dcsf() - a modified version of Noel Gorelick's dcs() function which uses floating-point values instead of 8-bit integers (for higher image fidelity).
- dcss() - the same as dscf(), except it uses 16-bit signed integers.
- rdcsf() - a modified version of C. Edwards' rdcs() function which uses floating-point values instead of 8-bit integers (for higher image fidelity).
- rdcss() - the same as rdscf(), except it uses 16-bit signed integers.
- arvi() - generates Atmospherically Resistant Vegetation Index images.
- asvi() - generates Atmospheric and Soil Vegetation Index images.
- avi() - generates Angular Vegetation Index images.
- evi() - generates Enhanced Vegetation Index images. Note: I added a few optional parameters to allow the output to be tweaked if so desired.
- gemi() - generates Global Environmental Monitoring Index images.
- msavi() - generates Modified Soil-Adjusted Vegetation Index images.
- msavi2() - generates Modified Soil-Adjusted Vegetation Index 2 images.
- ndi() - a shortcut function for generating Normalized Difference images of various types (NDVI, et cetera).
- slavi() - generates Specific Leaf Area Vegetation Index images.
- kt1() - a shortcut function for generating KT1 ("Tasseled Cap") images.
- kt2() - a shortcut function for generating KT2 ("Tasseled Cap") images
- kt3() - a shortcut function for generating KT3 ("Tasseled Cap") images

Known Issues

This application uses a fairly large amount of memory. The use of the 64-bit version is highly recommended. See the TMSB Troubleshooting and Optimization for more information.
32-bit-per-channel RGB and ARGB TIFFs are written in a slightly odd format. Some software (for example, CinePaint) is perfectly happy to read them. Adobe Photoshop® will read the file, but instead of assigning the data to the red, green, and blue channels, it will treat the red channel as a greyscale channel, and everything else as a bunch of alpha channels. I have not been able to find a good fix for this, although it's easy enough to move the channels around in Photoshop® after the fact if necessary. I can't imagine many people are using images with this fidelity, so hopefully it's not a big problem.
The statistical calculations for an image cube are very slow compared to other operations. This becomes extremely noticeable with large window sizes. For example, while the calculations with a window size of 1 may take several minutes, the same calculations with a window size of 81 may take several days. While this is partly due to the large volume of data, it is also significantly exacerbated by DaVinci's relatively slow execution of for loops in user-defined functions. I do not know enough C to create a built-in function equivalent, which is the only reasonable way of addressing this limitation.
Gradient-mapping is also slow compared to other operations, for a similar reason (it involves the use of a for loop in a user-defined function).
DaVinci's Shadow does not include the optional add-ons (GNUPlot, ImageJ, etc.) that the full version does. This is because I didn't want to be obligated to provide source code for them. If you need these add-ons, install the regular version of DaVinci from ASU's site, then copy those add-ons to the DaVinci's Shadow directory.

Building DaVinci's Shadow on Linux

This is an easy process. See DaVinci's Shadow Build Process - Linux.

Building DaVinci's Shadow on Windows®

Prior to version 1.3 of TMSB, I built the Windows® version of DaVinci's Shadow on a Windows® XP VM running MinGW and MSYS. This is a laborious environment to set up, but see DaVinci's Shadow Build Process - Windows for the instructions if you would like to attempt it. You may encounter unexpected issues attempting to use that process for versions 2.9.2 and later of DaVinci's Shadow. Windows® users don't need to do this unless they want to compile the software themselves for some reason - TMSB includes pre-built executables for 32-bit and 64-bit Windows®.

Cross-Compiling DaVinci's Shadow on Linux for Windows®

This is the method I've begun to use more recently for generating the Windows® version of DaVinci's Shadow. It still involves a lot more work to get the build environment up and running than the regular Linux compile, but is much less problematic than the MinGW/MSYS combination under Windows®. See DaVinci's Shadow Build Process - Cross-Compiling For Windows on Linux for the instructions. Windows® users don't need to do this unless they want to compile the software themselves for some reason - TMSB includes pre-built executables for 32-bit and 64-bit Windows®.

Building DaVinci's Shadow on BSD

This is a bit of a headache, but I'm sure BSD users are used to those by now. See DaVinci's Shadow Build Process - BSD.

Building DaVinci's Shadow on MacOS

I don't have access to a Mac, so I can't guarantee this will work. You may be able to use a similar process to the one described in DaVinci's Shadow Build Process - BSD, although you'll probably also need to install most/all of the packages listed in the DaVinci's Shadow Build Process - Linux article (most of them were installed by default in the BSD distribution I used). You should be able to get them from The MacPorts Project. If you are successful (or not), please let me know. If someone wants to donate a PowerBook or other MacOS laptop (it doesn't have to be high-end or brand new, but it does need to have an Intel processor, unless you want me to try to make it work on the older Power-based hardware), I'll see what I can do. I don't have room for a MacOS desktop, even if it's free.

Download Options

There are three downloads available at the bottom of this page: the precompiled Windows® version, the full source code, and the source files that I changed from the stock DaVinci versions. If you really don't trust me, you can download the 2.09 version of the source code from ASU's website, then apply the files in the last archive to it. If you really don't trust me, you can paste in the specific changes from those files into the files of the same names in the stock code. Everything I added or changed is generally contained in comment blocks that look like this:

/* DaVinci's Shadow customization - custom colourspace conversions - part 1 of 2 - Begin */

...code...

/* DaVinci's Shadow customization - custom colourspace conversions - part 1 of 2 - End */

Note: the build_utils directory in the "diff" file is mostly original content from ASU, but it's not included with their normal source package.

Update History

Changes in version 2.9.2

Re-applied all customizations I developed for 2.05 to ASU's 2.09 version of DaVinci.
Using the 2.09 codebase meant that libtiff was updated to 3.9.5, which means that TIFF support works correctly on openSUSE now.
Replaced the old 1.2.3 version of libpng included with DaVinci with 1.2.46.
Created customized build scripts so that the end result actually compiles as dshadow.exe on Windows® and dshadow on other operating systems (instead of davinci. This means that this fork and the main branch of DaVinci can easily coexist on the same system.
Incorporated the memory-use-reducing improvements to the dshadow.dvrc function library that were released as The Mirror's Surface Breaks 1.2.2, plus several similar improvements that were unreleased until now.
Adopted a more standard-Unix-like version number ("2.9.2", as opposed to "2.09.01" or somesuch).
Verified that this version builds correctly on the following specific platforms: openSUSE 11.3, Fedora 16, Kubuntu 10.10, PC-BSD 8.2, and cross-compiles for Windows® (64-bit and 32-bit) using MinGW-w64 under openSUSE 11.3, and that components which have been problematic in the past (TIFF and PNG support) work correctly on all of those platforms.

16-Bit-Per-Channel Image I/O - Detailed Discussion

When I first started monkeying around with DaVinci back in early 2010, one of the first things I noticed is that it had some issues with 16-bit-per-channel ("48-bit") images. It could read 16-bit-per-channel TIFFs just fine, but writing them to disk resulted in a corrupted image. A similar issue plagued the PNG support, but in addition, reading PNGs also resulted in corrupt data being loaded into the tool (although the original file on disk was unaffected).

It turned out that the issue with writing to disk was related to how DaVinci loaded that type of image. A 16-bit-per-channel PNG or TIFF has its data represented as "unsigned integers". That is, each value is between 0 and 65535. DaVinci uses a "signed integer" format for 16-bit numbers, where each value is between -32768 and 32767. There was a step in the file-loading code to convert the unsigned integers in the file into signed integers, but there was no corresponding step during the file-writing code.

The issue with PNGs was related to another exciting aspect of cross-platform development called "endian-ness": each 16-bit value is actually stored as two 8-bit bytes, and those bytes were being handled in the wrong order.

This is a lot easier to understand by looking at the test images I made:

Image I/O Test Images

Greyscale

RGB

ARGB

This is the way they should look...

...and then seeing the results when the unmodified code is used:

Bad Results

16BPC - TIFF

16BPC - PNG

16BPC - TIFF

16BPC - PNG

16BPC - TIFF

...this is the sort of result that will crop up with the unmodified DaVinci code when 16-bit-per-channel TIFF or PNG output is used.

If you'd like to try this for yourself, you can download dshadow_iotest-1.0.tar.bz2 from the bottom of this page. It isn't a comprehensive test, but it is fairly thorough - it actually caught some subtle issues in my code right before I was about to release TMSB 1.3, which is why DaVinci's Shadow is up to version 2.9.2 instead of just 2.9 or 2.9.1.

The DaVinci developers came up with a different approach, which is to upconvert 16-bit-per-channel images to 32-bit integer values. I am not a big fan of this, for two main reasons:

The values are not scaled to fill the full 32-bit range - they are left as 0-65535. This means that once the image is loaded, it is no longer possible (in DaVinci) to determine what the upper and lower bound of the original colourspace were (unless you carefully construct your images to each have at least one pixel of the minimum value and at least one of the maximum value).
Working with 32-bit-per-channel files is problematic at best, these days (Photoshop® barely supports them, for example), and they are of course twice the size of 16-bit-per-channel files.

I don't blame the DaVinci developers for this, though. DaVinci is mostly intended for scientific use, and I doubt most of its main users use PNGs or TIFFs at all. But I still backed out the change in DaVinci's Shadow 2.9.2.

Download
File	Size	Version	Release Date	Author
DaVinci's Shadow (Source Code)	41 MiB	2.9.2	2012-01-15	Numerous
This is the complete source code package for DaVinci's Shadow 2.9.2. If you are compiling DaVinci's Shadow, this is probably the file you want.

Download
File	Size	Version	Release Date	Author
DaVinci's Shadow (Source Code - Differences Only)	997 KiB	2.9.2	2012-01-15	Numerous
This is all of the files that are changed in DaVinci's Shadow 2.9.2 compared to the stock DaVinci 2.09, in case you want to make the task of examining the differences faster.

Download
File	Size	Version	Release Date	Author
DaVinci's Shadow Image I/O Test Script	8 MiB	1.0	2012-01-15	Numerous
A script to test various PNG/TIFF input/output combinations to make sure the data is uncorrupted.

Download
File	Size	Version	Release Date	Author
DaVinci's Shadow (Source Code)	55 MiB	2.05	2011-02-20	Numerous

Download
File	Size	Version	Release Date	Author
DaVinci's Shadow (Source Code) - Differences Only	123 KiB	2.05	2011-02-20	Numerous
The files containing content I added or changed from the stock DaVinci. You will need the 2.05 sourcecode from ASU's website to make effective use of this file.