DaVinci's Shadow Build Process - Windows

article by Ben Lincoln

Warning: this article was written with the older 2.05 release of DaVinci's Shadow in mind. I no longer build the Windows® version using this process - see the DaVinci's Shadow Build Process - Cross-Compiling For Windows on Linux article for the process I'm using now. You may be able to make this process work with version 2.9.2 and later, but I haven't tested it. This article is preserved here as a reference on the off chance someone needs or wants to do something like this again.

This article describes the process of assembling a build environment for DaVinci's Shadow on a Microsoft Windows® system. You probably don't need to do this, but if you do (or want to), here is how.

If you haven't done so already, please read the warning at the top of this article. This content has not been tested with versions of DaVinci's Shadow newer than 2.05.

Build Environment Setup

Putting together a build environment for DaVinci's Shadow on a Microsoft Windows® system is unfortunately something of a headache. If you are old enough to remember "the bad old days" of Linux in the mid-90s, this will be a trip down memory lane. One which you would probably rather not have taken.

Fortunately, the only reason you would need to follow the instructions on this page is if you want to compile the software yourself (for example, if you intend to make further changes to the source code, or you just don't trust me and want to ensure that the software you're running matches the source code). If you just want to use it, the compiled version is included with The Mirror's Surface Breaks and available separately at the bottom of the DaVinci's Shadow article.

In addition, you should only have to follow the steps in this section once (assuming you follow them correctly the first time). After that, you can build DaVinci's Shadow (or the stock DaVinci) as many times as you want. You may wish to follow these steps on some sort of VM, so that once you have a working environment, you can take snapshots and/or replicate it as needed.

As you trudge through this process, you may begin to wonder why I haven't scripted it out. There are three main reasons:

It would be a lot of work, when it is very likely no one else on the planet will ever actually use these instructions in exactly the same way.
Due to the old-school 1990s open source-style precarious house of cards that it is based on, the script would almost certainly fail at some point, and it's better for the user to understand where the procedure has differed from the document.
Having to see how terrible things were back in the 1990s versus how easy they are today on an actual Linux system builds character.

You may also begin to wonder why there is a repetition of many of the library builds, one with the default configure options, and one with --prefix=/mingw. This is because after days of trying to sort out which subcomponent looked in one location versus the other for other libraries, I decided to just brute-force it and build them in both locations. Seriously, this is something that should not be hard, but it is. If you choose to bypass this procedure, well, good luck. I hope the tiny disk space savings is worth the aggravation and frustration.

There are two packages available at the bottom of this page which contain nearly all of the software necessary to create the build environment. You only need one of them. I suggest the smaller package, which contains only the binaries. The larger package is provided for full GPL compliance, which requires that the source code for all components be made available. You do not need this larger package unless you want to build all of the MSYS components from source as well.

Important: you will also need a text editor which can save files with Unix-style newline characters. Wordpad will not work for this purpose, because even though it happily reads files of this type, it saves them with Windows-style newline characters, which will break numerous steps in this process. I personally use Programmer's File Editor, even though it's extremely old and unsupported. If you choose to use that editor, make sure you get the 32-bit version.

MinGW Installation

There is one component that I have not included in either package: the installer for MinGW. I did not include it because even after several hours of rummaging around in the extremely-confusing-organized MinGW SourceForge repository, I couldn't locate the files necessary to perform an offline installation equivalent to the online installation that mingw-get-inst does.

To obtain MinGW (at least as of this writing - the project team seems to be fond of reorganizing their SourceForge site without actually making it easier to find anything), go to the main file page for mingw-get-inst, and click on the link near the top of the page that is near the "Looking for the latest version?" text.

Once you have that executable, run it. Leave all of the options as the default, except for the following:

In the list of compilers to install, select all of the checkboxes.
Do not install MSYS or the Developer Toolkit. You will do that later.

Everything Else

Once you have MinGW installed, you can proceed with the rest of these exciting, non-tedious steps which are 100% guaranteed not to leave you tearing out your hair and cursing at your PC when you forget to perform one of them and thereby end up with strange errors in the actual build process^[1].

Run MSYS-1.0.11.exe. Accept all of the defaults, with these exceptions:
1. When prompted to run the post-install, enter y
2. When when asked if you have MinGW installed, enter y
3. Enter c:/mingw as your MinGW path (unless you changed it, contrary to the instructions about accepting the default options). Note: yes, you must use a forward-slash, because the best thing ever is when developers spend years putting together complex software but still make it incapable of handling the standard directory-separator character on the target platform.
4. Uncheck both checkboxes on the final screen of the wizard.
Run MSYS-DTK-1.0.1.exe. Accept all of the defaults.
In Windows Explorer, create the directory C:\msys\1.0\home\dsb. Note: this is to ensure that the base directory can be referred to consistently throughout the rest of this document.
Copy everything from the MSYS_Components directory to C:\msys\1.0\home\dsb.
Start MSYS (Start -> MinGW -> MSYS -> MSYS).
Enter the following commands in the MSYS shell:

cd /home/dsb
for a in `ls -1 *.tar.lzma`; do tar -xvf $a --lzma; done
for a in `ls -1 *.tar.gz`; do tar -xzvf $a; done
for a in `ls -1 *.tar.bz2`; do tar -xvf $a --bzip; done
exit
Once the MSYS shell window has closed, in Windows Explorer, copy all of the files and directories extracted by the previous step (IE not the tar files that were already there) from C:\msys\1.0\home\dsb to C:\msys\1.0.
In Windows Explorer, delete everything that's still in C:\msys\1.0\home\dsb.
In Windows Explorer, copy everything from the Libraries directory to C:\msys\1.0\home\dsb.
Start MSYS again.
Enter the following commands in the MSYS shell:

cd /home/dsb
for a in `ls -1 *.tar.lzma`; do tar -xvf $a --lzma; done
for a in `ls -1 *.tar.gz`; do tar -xzvf $a; done
for a in `ls -1 *.tar.bz2`; do tar -xvf $a --bzip; done
In your Windows text editor (see the Important note near the beginning of this page), open C:\msys\1.0\home\dsb\PDCurses-3.4\pdcurses\terminfo.c. Find this text:
```
int tputs(const char *str, int affcnt, int (*putfunc)(int))
{
    PDC_LOG(("tputs() - called\n"));
	    return ERR;
}

			
```
...and replace it with this:
```
/*
int tputs(const char *str, int affcnt, int (*putfunc)(int))
{
    PDC_LOG(("tputs() - called\n"));
    return ERR;
}
*/
int tputs(const char *str, int affcnt, int (*putfunc)(int))
{
	PDC_LOG(("tputs() - called\n"));
	while (*str)
	{
		putfunc (*str++);
	}
	return ERR;
}

							
```
Once you have made this change, save the file (making sure to preserve Unix-style newline characters), and exit the text editor.
This step manually applies a fix to PDCurses created by either Andrew Stubbs or Jon Ringle (I forget which I found first), which was rejected by the PDCurses developers, because the other best thing ever is when obstinate open source developers create a package on Windows that works like one on Linux, except that it's broken^[2], then refuse to fix it to work like it does on Linux (even when such a fix is three short lines of source code), because obviously anyone who wants to port Linux applications to Windows must also want to run everything in an MSYS terminal instead of the Windows command prompt, and so must everyone who wants to use that ported application. Join the open source revolution today!
Note: if you do not perform this step, then the backspace key will have bizarre effects in any version of DaVinci or DaVinci's Shadow which you compile.
In your Windows text editor (see the Important note near the beginning of this page), open C:\msys\1.0\home\dsb\PDCurses-3.4\win32\mingwin32.mak.
Perform the following find/replace operations:
- Replace all occurrences of copy with cp
- Replace all occurrences of del with rm
Then save the file and exit the text editor.
In the MSYS shell, execute the following commands:

cd PDCurses-3.4/win32
make -f mingwin32.mak
In Windows Explorer, copy the file C:\msys\1.0\home\dsb\PDCurses-3.4\win32\panel.a to C:\MinGW\lib, then rename it to libpanel.a
In Windows Explorer, copy the file C:\msys\1.0\home\dsb\PDCurses-3.4\win32\pdcurses.a to C:\MinGW\lib, then rename it to libcurses.a
In Windows Explorer, copy all of the files which end in .h from C:\msys\1.0\home\dsb\PDCurses-3.4 to C:\MinGW\include
In the MSYS shell, execute the following commands:

cd /home/dsb/readline-6.1
./configure --with-curses
make
make install
./configure --prefix=/mingw --with-curses
make
make install
In Windows Explorer, copy the files C:\msys\1.0\home\dsb\readline-6.1\shlib\libhistory6.dll and C:\msys\1.0\home\dsb\readline-6.1\shlib\libreadline6.dll to C:\msys\1.0\bin
In the MSYS shell, execute the following commands:

cd /home/dsb/jpeg-8c
./configure
make
make install
./configure --prefix=/mingw
make
make install
cd /home/dsb/cfitsio
./configure
make
make install
./configure --prefix=/mingw --with-curses
make
make install
cd /home/dsb/zlib-1.2.5
make -f win32/Makefile.gcc
In Windows Explorer, copy these files from C:\msys\1.0\home\dsb\zlib-1.2.5 to both C:\msys\1.0\include and C:\MinGW\include. If you receive a message that any of the files already exist, choose to overwrite them:

zlib.h
zconf.h
In Windows Explorer, copy these files from C:\msys\1.0\home\dsb\zlib-1.2.5 to both C:\msys\1.0\lib and C:\MinGW\lib. If you receive a message that any of the files already exist, choose to overwrite them:

libz.a
libzdll.a
Get ready to have some fun with the mingwPORT version of libpng-1.2.8! The readme includes the hilariously misleading statement "./mingwPORT.sh is all that is required to get started", which, I suppose, is literally true, in the sense that "a journey of a thousand miles begins with one step" is.
Once the Flight Director has verified that all systems and personnel are go to let the good times roll, execute the following commands in the MSYS shell:

mkdir /usr/src
mkdir /usr/src/libpng-1.2.8
cp -r /home/dsb/libpng-1.2.8 /usr/src
In your Windows text editor (see the Important note near the beginning of this page), open C:\msys\1.0\src\libpng-1.2.8\mingwPORT\mingwPORT.cleanup.
Remove all of the contents in that file, then save it and exit your text editor. This step is so that if you screw up any of the rest of the libpng build, you won't have to re-copy a bunch of files around again.
In the MSYS shell, execute the following commands:

cd /usr/src/libpng-1.2.8/mingwPORT
./mingwPORT.sh
n
n
[press the enter key on your keyboard]
[press the enter key on your keyboard]
[press the enter key on your keyboard]

You will get a bunch of errors. That's OK! You were expecting them! Because you knew that the other best thing ever is when build scripts allow the user to specify a path, then completely ignore what that path is and just use a relative path!
In Windows Explorer, copy C:\msys\1.0\src\libpng-1.2.8\mingwPORT\libpng-1.2.8\scripts\makefile.mingw to C:\msys\1.0\src\libpng-1.2.8\scripts
In the MSYS shell, experience a sense of deja vu as you execute the following strangely-familiar commands:

cd /usr/src/libpng-1.2.8/mingwPORT
./mingwPORT.sh
n
n
[press the enter key on your keyboard]
[press the enter key on your keyboard]
[press the enter key on your keyboard]
In your Windows text editor (see the Important note near the beginning of this page), open C:\msys\1.0\home\dsb\msys-build-regex.
Locate the following lines, and comment them out (insert a pound sign (#) at the beginning of each line in question):
```
if [ "$MSYSTEM" != "MSYS" ]
then
  echo "You must be in an MSYS shell to use this script"
  exit 1
fi

		
```
...that is, when you are done, those five lines should look like this:
```
#if [ "$MSYSTEM" != "MSYS" ]
#then
#  echo "You must be in an MSYS shell to use this script"
#  exit 1
#fi

						
```
This step is necessary because someone thought it would be helpful to have an MSYS-specific script verify that it is in fact running inside MSYS, but code it so that the check would fail even when it really was running in MSYS. But the danger of someone running this script in a different Linux shell on Windows® was averted! Until you commented it out, that is.
In the MSYS shell, execute the following commands:

cd /home/dsb
./msys-build-regex regex-20090805.tar.xz

The script will error out, but before it does, it will have performed a couple of important steps.
In the MSYS shell, execute the following commands:

cd /home/dsb/regex-20090805
./configure
make
make install
./configure --prefix=/mingw --with-curses
make
make install
In your Windows text editor (see the Important note near the beginning of this page), you will need to perform the following change to each of these files:

C:\msys\1.0\home\dsb\hdf5-1.8.6\test\Makefile.in.
C:\msys\1.0\home\dsb\hdf5-1.8.6\tools\h5copy\Makefile.in
C:\msys\1.0\home\dsb\hdf5-1.8.6\tools\h5diff\Makefile.in
C:\msys\1.0\home\dsb\hdf5-1.8.6\tools\h5dump\Makefile.in
C:\msys\1.0\home\dsb\hdf5-1.8.6\tools\h5ls\Makefile.in
C:\msys\1.0\home\dsb\hdf5-1.8.6\tools\h5stat\Makefile.in
C:\msys\1.0\home\dsb\hdf5-1.8.6\tools\misc\Makefile.in
1. Locate the line which begins with TEST_SCRIPT = (it will read something like TEST_SCRIPT = $(top_srcdir)/test/testerror.sh depending on which of the files you are editing). There will be one - and only one - of these lines in each of the indicated files.
2. Delete everything on that line after the equals sign (including the space). That is, the line should end up reading TEST_SCRIPT = with no additional text on that line.
3. Save the file and exit.
In the MSYS shell, execute the following commands:

cd /home/dsb/hdf5-1.8.6
export LIBS=-lws2_32
./configure --with-zlib=/usr/local --prefix=/usr/local --enable-shared=no
make
make check
make install
./configure --with-zlib=/usr/local --prefix=/mingw --enable-shared=no
make
make check
make install
Close the MSYS shell and reopen it to clear out wacky environment variables set during this process which will cause build failures in DaVinci's Shadow.

You should now have a functioning build environment for DaVinci's Shadow (or the stock DaVinci). If you ever want to relive the excitement of the above process, just uninstall MinGW and MSYS and start over from scratch. Note that some of the more esoteric (and enormous) components like ISIS are not included. Please see ASU's documentation if you need these features. Unless you actually work at NASA, you probably don't.

Building DaVinci's Shadow

Download the source code package from the DaVinci's Shadow page, and place it C:\msys\1.0\home\dsb. These instructions assume you are using version 2.05, so you should end up with a file named C:\msys\1.0\home\dsb\dshadow2.05.tar.gz
If you already have an MSYS shell running, close it and reopen it to make sure there are no extra/changed environment variables present.
In the MSYS shell, execute the following commands:

cd /home/dsb
tar -xzvf dshadow2.05.tar.gz
cd /home/dsb/dshadow2.05
export PATH=.:/usr/local/bin:/mingw/bin:/bin
export LD_LIBRARY_PATH=.:/usr/local/lib:/usr/lib:/mingw/lib
cd libltdl
./configure
make
cd ..
cd iomedley
./configure
make
cd ..
./configure --with-hdf5=/usr/local
make clean
make
cd build_utils
./makewin.sh

The make clean command is because there are either some bad files included with ASU's source code, or because I've managed to screw up the build process in some way during every one of my tests, so by the time I get to that point, there is a bad file there anyway.
Assuming everything worked, you should now have these two files and one directory in C:\msys\1.0\home\dsb\dshadow2.05\build_utils. Copy them to the location where you want to run DaVinci's Shadow from.

davinci.exe
davinci.dll
modules
Copy these additional library files to the same location:

libgcc_s_dw2-1.dll (from C:\MinGW\bin)
libhistory6.dll (from C:\msys\1.0\bin)
libreadline6.dll (from C:\msys\1.0\bin)
libregex-1.dll (from C:\msys\1.0\local\bin)
msys-1.0.dll (from C:\msys\1.0\bin)
msys-z.dll (from C:\msys\1.0\bin)
If "the location" is not the same as "a directory that previously contained a precompiled distribution of DaVinci's Shadow, then copy the following files and directories there from the precompiled distribution (available at the bottom of the DaVinci's Shadow page):

library
davinci.bat
davinci.ico
davinci.png
dv.gih
LICENSE.TXT
README.TXT

Troubleshooting

If you failed to follow any of the steps correctly, you may experience one or more of the following symptoms:

One or more make commands generate unexpected errors.
The backspace key does not work as expected in the compiled software.
PNG support is missing from the compiled software.

If you experience problems similar to this, the easiest thing to do is to uninstall MSYS and MinGW, remove both C:\MinGW and C:\MSYS, and start over from scratch. I highly recommend not attempting to build a "general-purpose" MinGW/MSYS environment that can be used for other purposes. In my case, I built an entire Windows XP® VM in VirtualBox which has a dedicated MinGW/MSYS installation for purposes of building DaVinci's Shadow.

If you follow the instructions to the letter, you should end up with a working build environment. The only exception is the MinGW compiler suite itself. Since I couldn't find a way to cobble together an offline install of the specific version I used (which was based on GCC 4.5.2), if you use a different version you may end up encountering problems even if you follow the instructions perfectly. Sorry.

Acknowledgements

These instructions are based (extremely loosely) on ASU's Building DaVinci for Windows page.

The existence of a Windows version of this utility is the result of the dedication of numerous open source developers. Even though I may complain about the lengths it's necessary to go to to get some of the packages to compile, it's a lot better than not having them at all.

Download
File	Size	Version	Release Date	Author
Windows Build Environment Package (Compact)	39 MiB	1.0	2011-02-21	Numerous
This is the file you want unless you are trying to verify that I'm complying with the GPL.

Download
File	Size	Version	Release Date	Author
Windows Build Environment Package (GPL Compliance)	268 MiB	1.0	2011-02-21	Numerous
This file contains all of the source code for the binary distribution of DaVinci's Shadow (except for DaVinci's Shadow itself). This is almost certainly not the file you are looking for.

Footnotes

1.	Guarantee not valid in all jurisdictions.
2.	It's broken if the same software as compiled on Linux functions differently for an end user on Windows. End users don't care (and shouldn't have to care) about stupid spats regarding functionality (or lack thereof) in shared libraries.