The Impossible Journey of a Simple Task

Contrary to perhaps popular opinion, I think the most defining feature of an expert problem-solver is not how they handle large and difficult tasks, but instead, how they navigate a cascade of tiny problems.

Source: Malcolm in the Middle, Fox Broadcasting Company

While I do not by any means consider myself an expert problem-solver, as a Masters of Engineering graduate and current PhD student, I feel some professional obligation to be constantly bettering my innate ability to solve problems, whether in work or in life.

Over the last 24 hours, I have battled against such a cascade of tiny problems, and somehow emerged victorious. These cascades seem all too common in programming and related fields, so I wanted to document one while it is fresh in my brain. More importantly, if I ever need to solve this problem again, I will have little choice other than to endure the same gauntlet unless I have record of the cryptic solution.

The problem:

My PhD involves medical image processing. These images are stored in DICOM format; a common medical imaging format. My PhD also involves machine learning algorithms, which are created in Python.

Python cannot read DICOM images by default, so an external module is required. The two I was able to find were PyDICOM, and SimpleITK. PyDICOM is user-friendly and allows easy manipulation of image metadata (eg. what time the X-ray was taken), but cannot (by default) handle the compressed DICOM images I am working with. SimpleITK, while having relatively poor documentation and being far less common, can handle my compressed DICOM images out of the box. When I was deciding which to use and simply needed to access the pixel data roughly a year ago, SimpleITK was an easy choice.

However, I recently had an issue. In performing my machine learning training on large datasets, I noticed that a small but measurable portion of my images were displaying as inverted; that is, the black parts of the image were white, and vice versa.

 

 

 

 

 

 

 

Example image source: stepwards.com

Depending on the machine learning method being used, and the goal of the algorithm (in this case, detect the location of anatomical features), this inversion can of course have drastic effects on the correctness of the algorithm’s predictions.

Based on my prior experience with DICOM images, my suspicious was that the DICOM Lookup Table was either not being read, or was missing. The Lookup Table aims to make the X-rays look the same on different monitors/different printers by allowing accurate correlation between image brightness and pixel intensity, and stored as metadata in the image.

My small problem had reared its little head. I ruffled its hair condescendingly as I googled how to interact with Lookup Table information using SimpleITK, my DICOM reading package of choice. However, due to the lackluster documentation and relatively low usage of the package, I was unable to find any information on how to even read the Lookup Table. While I did know the metadata keys of the Lookup Table, exploring them with PyDICOM revealed no interesting patterns.

However, this got me thinking; viewing the images in other DICOM viewers, such as the Windows program MicroDICOM, shows no such inversion. Perhaps the issue is with SimpleITK?

I googled to check if I had any alternative packages I could use, and found that PyDICOM had recently been updated to 1.1.0, a version which allows reading of compressed DICOMs! I thought great, let’s just update my PyDICOM package and be on my merry way. I winked at the tiny problem ahead of me as I strolled past it.

PyDICOM is updated, no problems. I quickly change my code to use PyDICOM to read the images instead of SimpleITK, and give it a test. I get an error:

NotImplementedError: No available image handler could decode this transfer
syntax JPEG Lossless, Non-Hierarchical, First-Order Prediction 
(Process 14 [Selection Value 1])

I think “did I misread something? PyDICOM works with these images now, doesn’t it?” So I google and find a similar question https://github.com/pydicom/pydicom/issues/532 ,  in which the answer given was to simply include the package GDCM (Grassroots DICOM), which PyDICOM uses to read those particular compressed images.

“No worries!”, I thought. Too. Easy. Just install another package and be on my way! I google GDCM, and find that it cannot be installed via pip or easy_install, as most python packages can these days. Not to worry, though; the creators have a pre-packaged Windows binary file I can use. I download the package, and am careful to not make any silly mistakes in the few installation option choices I am given. I make sure to add it to PATH (a Windows environment variable) for all users. I make sure to include all the install options, just to be safe.

I finish the installation, I go back to my IDE and try in the shell: “import GDCM“;

Traceback (most recent call last):
 Python Shell, prompt 1, line 1
builtins.ModuleNotFoundError: No module named 'gdcm'

Hmm, I wonder why it isn’t finding it. I double check the GDCM Installation Guide, which suggests running “gdcmdump” from the command line to ensure the binary files are installed correctly. I do so, and I get:

C:\Users\Jase>gdcmdump
'gdcmdump' is not recognized as an internal or external command,
operable program or batch file.

Hmm, so the environment PATH variable isn’t set correctly? I double check; it is set correctly. I google as to why an environment PATH variable might not be working, and am told that one simply needs to reset the “session” after adding to the PATH variable, by logging out or restarting the computer. AH, the classic solution! Aren’t computers wacky! I chuckle to myself as I tie the bow on this problem, and restart my computer.

Once the computer is back on, I test “gdcmdump” from the command line again; it works! GDCM is found by my computer and working great. I go back into my IDE and run “import gdcm”, but get the same error:

Traceback (most recent call last):
 Python Shell, prompt 1, line 1
builtins.ModuleNotFoundError: No module named 'gdcm'

Hmm…so Python can’t see the module, but cmd can…

The problem looks straight at me and winks right back. It seems more confident than before.

I google why this might be the case. A common theme is that GDCM is really not the most common piece of software in the world, so help does not exist for every little issue as it may for many other pieces of software. However, I remember the concept of PYTHONPATH; an environment variable for Python. That’s it! I set the Python path to where the GDCM files were installed – Nothing. I restart – Nothing. I add all the subdirectories to the Python path – Nothing. I add everything to the PYTHONHOME path (because I don’t know, why not…) – Nothing.

So I get back to googling, trawling through the message board posts and two StackOverflow posts about GDCM on Windows via Python. One has the answer “you need to switch from Python 2 to Python 3”, which isn’t an issue for me as I am using Python 3.6.5, and the other has no answer.

However, I scroll to the bottom of the no answer post and find the user kindly edited the bottom of the post to show their solution!

It was given as follows:

I finally solved this problem:

1. clear my py enviroment: here
2. install x86 GDCM: here (current is GDCM-2.8.4-Windows-x86.exe)
3. copy ‘_gdcmswig.pyd, gdcm.py, gdcmswig.py’ to ‘C:\Python27\DLLs

“Hallelujah!”, I thought. I have already spent far too long on this problem. And of course, I should’ve known, some of the files need to be put into a Python directory for me to use them. You nearly got me, you pesky little problem, but not today! Heck, this person is even pointing out how they had to use the x86 version, when we are both using x64 computers; that would’ve caught me out for sure!

I “clear” my py environment, uninstall the x64 GDCM, install the x86 version, copy the files to C:\Python36\DLLs, run “import gdcm” – still Nothing. I start to get the feeling this may be a long night.

I google even more incessantly for solutions. Every combination of GDCM, Python, Windows, the specific error message, etc. . I find a crucial nugget of information (which I can no longer locate (that’s how deep into Google I was)); an installation guide for a Conda package (something I am not using) which at the bottom, under “Windows”, mentioned copying the files to “PythonXX\site-packages”, not to DLLs.

I try copying the previously mentioned files to that directory – It works!

Well, sort of.

It doesn’t work, but in a DIFFERENT WAY!

When I try to import gdcm, I get the following error:

Uncaught Error: %1 is not a valid Win32 application

1% of what??

My immediate thought is: Oh, of course, I have to be using x64, not x86. But then how did that  StackOverflow poster get it working on their x64 machine, specifically mentioning that you need to use x86? I’m not so sure anymore. Upon reading it again, I think: they’re using version 2.8.4; I’m using 2.8.7. It couldn’t be that easy, could it?

I uninstall GDCM 2.8.7 x86 and install GDCM 2.8.4 x86 – Nothing. I triple check all my PATHs, PYTHONPATHs, etc., making repeated futile changes, resetting my computer randomly just in the hope that it will start working even though I know it shouldn’t – Nothing. I find a post on the internet that suggests installing GDCM not on my C-drive might be an issue; uninstall, reinstall to C-drive – Nothing.

In the midst of this I considered going back to the x64 version, but the very first StackOverflow post I googled using the error message “1% is not a valid Win32 application” makes very clear that this message simply means Python could not find a DLL file, not that it is incompatible, and that that is a very common misconception. However, at this point, I’m trying everything.

I uninstall the x86 version, I install the x64 version:

ImportError: DLL load failed: The specified module could not be
found

So wait, is this better or worse than before?

I use my better judgement to decide this must be further along the right track, because “not a valid Win32 application” meaning “use x64, not x86, you dummy” just makes so much sense to me.

Yeah but which?? Which DLL??

I try running python in verbose mode; it gives me no clues as to which DLL is failing to load. I google this, thinking this should be easily accessible information – apparently it isn’t.

I try moving combinations of the many DLL and non-DLL files of GDCM to different Python directories, in an attempt to get them detected – no dice. I keep googling, and keep getting taunted by the StackOverflow poster, who has already solved the problem and been on their merry way.

Eventually I stumble upon a nugget in the Grassroots DICOM / Mailing Lists; a mailing list in which the creator of GDCM himself (presumably) answers these kinds of specific questions. Much of what they are discussing is entirely unfamiliar to me, though I manage to find a couple more people having issues with GDCM on Windows. This one, aptly titled [Gdcm2] Magic to get GDCM+Python working on windows ?, helped me feel a little less bad about spending so long on this (but I still felt pretty bad):

On Wed, 2009-02-25 at 17:24 +0100, Mathieu Malaterre wrote:
> Finally I would also suggest you inspect the dll gdcmswig.pyd with
> depends (http://dependencywalker.com) for any missing dll.

Excellent advice:
yes it shows missing dependency on PYTHON24.DLL
(also MSJAVA.DLL but dep.walker FAQ says that's expected)

[I was originally thinking it wasn't finding 
  _gdcmswig.pyd
but doing 
  import imp
  imp.find_module("_gdcmswig")
shows it knows exactly where it is.]

Will probably try reverting to 2.4 first.

Thanks
Tim

Dependency Walker, you say? Tells me exactly what DLL is causing the problem, you say? Sounds good to me. This post was written in 2009, and I managed to find an even better, modern dependency walker which I’d highly recommend.

Upon using it, I found that of the 10 or so DLLs that _gdcmswig.pyd directly depends on, 6 were missing. 5 were GDCM DLLs, which were located in C:\Program Files\GDCM 2.8\bin . Through some completely unrepeatable process of random restarting, copying the named DLL files to C:\Python36\Lib\site-packages, and inscribing a pentagram on my forehead and performing a ritualistic sacrifice to the DLL Gods (praise be unto Them),  I managed to have Dependencies show those 5 DLLs as found upon my last restart.  The other, final missing DLL, was PYTHON34.DLL. I thought, “that must be it!”, along with “how could that possibly be it?”, along with “oh God I hope that’s not it…”

You see, my hunch was that the Windows executables provided for GDCM had been compiled and built for Python 3.4, a quite old version of Python (I think we’re up to 3.7 now). I initially thought this could not be the case, as the famed StackOverflow poster, had reportedly already solved the problem on Python 3.6. However, given that they also said to use x86 on an x64 machine, which turned out to be a waste of time, I had lost faith in that particular solution.

I had an idea: “Could I just take my Python36.DLL, copy it, rename it to Python34.DLL, and be on my (increasingly less) merry way?

I tried, and while it didn’t work, it confirmed my suspicions. I have since lost the error message given to me upon running “import gdcm”, but it was something along the lines of “PYTHON34.DLL is not compatible with your version of Python”.

Dangit. Am I seriously going to have to compile and build this package from scratch? I don’t even have a C compiler installed on this computer.

I download the GDCM source code and look through it. I find a file called CMakeLists.txt; that sounds familiar. I google it; I need a program called CMake to “make” the build. I download CMake, which thankfully comes with a pretty GUI for package-building novices like myself.

It asks for a source and destination directory; no worries, I know those. I tell it where it can find the code, and where it should put the makefiles. I click “Configure”; it asks me what compiler I want to use, and presents a list of about 200 options. Hmm…

My only previous experience with compilers is MinGW, and thankfully “MinGW makefile” is an option I can select. I set out on my (decidedly not) merry way, installing MinGW. I fudge my way through the installation options. In the midst of all this, I’m googling around for any GDCM-specific instructions, and find an instruction page (which I can no longer find) which confirmed that MinGW is perfectly good for compiling the GDCM makefiles.

I go back to CMake, select “MinGW makefile” as my choice, and watch.

I’m both glad and exhausted to see that several options now pop up. What do they do?? I don’t know. This is something that an end user such as myself should not be doing.

Pictured: Me not knowing what I’m doing at all

By this point the problem is about the size of a tractor trailer and dunking my head in the toilet, while I try to merely focus my eyes between each dunk (it is well past bedtime at this point).

I try ticking all the boxes; CMake says “what are you doing, moron? You don’t have Doxygen installed. How are you going to build Documentation without Doxygen?”
I humbly apologise, untick GDCM_DOCUMENTATION, and try again. I get a similar error about VTK, and then JAVA. I’m really just interested in the Python wrapping, so I untick both of these.

I get even more errors about the TESTING and EXAMPLES options, so I untick those too. I configure…finally, it succeeds! I have the makefiles. Now…uh…how do I make things again?

A quick embarrassing Google reminds me that you use make.exe, dummy. I check in MinGW/bin; I don’t have make.exe, but I have mingw32-make.exe. Is that the same? I google. Here’s what 3am me tried to interpret:

The "native" (i.e.: MSVCRT dependent) port of make is lacking in some 
functionality and has modified functionality due to the lack of POSIX 
on Win32. There also exists a version of make in the MSYS distribution 
that is dependent on the MSYS runtime. This port operates more as make 
was intended to operate and gives less headaches during execution. 
Based on this, the MinGW developers/maintainers/packagers decided it 
would be best to rename the native version so that both the "native" 
version and the MSYS version could be present at the same time without 
file name collision.

All I wanted was to look at some X-rays…

I try using mingw32-make.exe anyway. I get some error. I don’t even remember what it was. At some point I try doing all of this using Cygwin shell, because the creator uses it in testing GDCM on Windows; doesn’t change anything. By this point I have littered my relatively clean installation with a mess of random unhelpful programs. I find MSYS, which apparently has make.exe. I download MSYS. I install MSYS. At the end of the installer I get another error; apparently I can’t even install it correctly. I go back through the minGW installer options; I sleepily choose between these:

This is about 5% of the page.

I correctly choose msys-make (bin); it starts working! I see it completing, openjpeg8, done, 10%. openjpeg16, done, 20%. But I know it’ll mess up. I’m deep in it now. I’m staring at the screen, jaded, waiting for an error to pop up.

And it does.

> undefined reference to `gdcm::System::StrCaseCmp(char const*, char
> const*)’

I can’t even tell you the rabbithole this sent me down. Suddenly I’m programming in C again. Some I checked whether the mentioned files included string.h (the C package that holds stuff like strcasecmp as far as I remember); they do. I checked if they <include string>s; they did. I check the StrCaseCmp function in the gdcm::System namespace: it looks like this:

Google tells me that strcasecmp doesn’t fit the POSIX standard (oh I remember that word, from 3am…), but something like _stricmp should be fine. I try forcing _stricmp by editing the source code of the package – _stricmp can’t be found either. This is apparently a limitation of minGW, but using different compiler options (eg. std = c++11) should fix it. Not only does my minGW installation not have that option, but upon finally finding out how to set those compiler flags, nothing changed. Upon another recommendation, I try downloading a release of minGW called x86_64-4.9.0-release-posix-seh-rt_v3-rev2.7z (yeah great, I’ll take one thanks); didn’t change anything. The curious case of strcasecmp continues.

I was back to aimlessly googling. Reading the Grassroots DICOM / Mailing Lists; provided a lot of red herrings that didn’t apply to my situation, but the pattern that I picked up which was of importance was that the creator uses Visual Studio, not minGW, to build the GDCM Windows binaries. I do not have Visual Studio, and have never used it, and was pretty sure it cost money.

I find the “community” version is free, but gigabytes in size. My internet speed caps out at around 500kbps, and I generally dislike the way Microsoft force their telemetry and user-tracking in their services… but at this point I was willing to try anything.

I set Visual Studio to download. I get some sleep. I wake up. Restless, I continue. The problem engulfs not just me, but the entire Earth; the Sun begins to revolve around it.

I recompile using CMake. There is no Visual Studio Community option, but thankfully the first Visual Studio choice works.

My compiled “makefiles” look different now; they’re all accessible as Visual Studio projects. Of the many,

, I choose “ALL_BUILD”. Because that sounds like what I want to do.

I’m presented with the foreign interface of Visual Studio. One of the tab menus is “Build”, which is what I want to do! Under it, I select the first option: “Build Solution”.

It starts building. A log screen shows me the progress, but it is zooming past so quickly that I cannot even read what it is doing. Finally, after a considerable time, I’m presented with an error screen: 14 Errors, 93 Warnings. Ouch.

The first 4 mention the min() and max(); stuff like

Error C2039: 'max' : is not a member of 'std'

I google it, and apparently it’s due to a conflict between Windows and C++ min/max functions. The page suggests I edit the offending file (gdcmLookupTable.cxx) to add “#include <algorithm>”. But is that really possibly the solution? This is a package, used in PyDICOM, built for Windows many times; how could the solution possibly be that I have to edit their code??

Well I don’t have an answer to that, but I do know that…it worked. It was the solution. The problem cackles in laughter, each burst causing simultaneous inter-galactic earthquakes and tsunamis, destroying several trillion planets that would’ve otherwise gone on to develop intelligent and altruistic life and save us from Global Warming.

In you go. Does this mean I’m a contributor for GDCM now??

I CMake again. I attempt to build in Visual Studio again. The errors, ladies and gentlemen, they keep-a comin’. This time it’s FATAL. Ooh.

fatal error LNK1112: module machine type 'x64' conflicts 
with target machine type 'X86'

Hmm. You know, weirdly…this feels…almost…manageable? Like, a not completely impossible error?

I give it a good Google, as I always do, and by this point I’m hardened. I see the most common solution:

Check your properties options in your linker settings at: 
Properties > Configuration Properties > Linker > Advanced > Target Machine. 
Select MachineX64 if you are targeting a 64 bit build, or MachineX86 
if you are making a 32 bit build.

And then check my Visual Studio installation:

No “Linker” option. No “Advanced” option. That’s fine, I didn’t expect anything helpful anyway. I start clicking around myself and find the “Configuration Manager”, separate of this menu. It clarifies that “win32” (aka x86) is my current “Active solution platform”, and gives me the option of creating a new ARM or x64 solution platform.

I do so.

I re-CMake, making sure this time to choose “Visual Studio 15 64-bit”. I re-build in Visual Studio. A million warnings wash over me, but no errors. It compiles, with this final message:

========= Build: 98 succeeded, 0 failed, 0 up-to-date, 7 skipped =========

7 skipped…yeah I don’t like my odds.

But I stick to my guns. I check the newly built files in my Dependency Walker; it says Python36 instead of Python34, and none are missing. I copy the three files mentioned at the start of this whole endeavour into site-packages, and try to “import gdcm”; I get the same DLL missing error. I copy all of the DLLs from the build, just all of them, into the site-packages folder and try again, and you know what?

It worked.

I did it. I loaded an image using PyDICOM. It. Just. Worked. I couldn’t believe it. I’d finally solved the problem, and I’d built the package myself to do it.

And then I realized that the same inverted image problem was happening in PyDICOM so I just thought a bit more about the problem and solved it in SimpleITK with these lines of code:

Turns out it was the Photometric Interpretation setting, not the Lookup Table…

So…yeah.

But in seriousness, as much as I’d like to joke that I wasted hours of my life for nothing, I really did need to go through the effort. PyDICOM’s tools made it possible to debug and determine what was causing the inversion, and I doubt any other than those very familiar with DICOMs (the wisdom of which I unfortunately could not locate using Google without the keywords I now have after solving the problem) would’ve known it was the “Photometric Interpretation” setting being “MONOCHROME1” instead of “MONOCHROME2”, stored at key (0x28, 0x04).

(On top of that, I purely guessed the “GetMetaData” SimpleITK method, as I really could not find even that simple documentation. Perhaps someone more in-the-know than me could point me to a better source of SimpleITK knowledge.)

Anyway, there’s no moral to this story. I just wanted to record something that happens to me in solving these kinds of problems, the magnitude of which always seems truly impossible to convey.

That and, if I end up wanting to use PyDICOM on my work computer, I don’t want to be wasting any more time remembering any of these steps…

P.S: To all of the developers involved in the projects I’ve mentioned, and expressed frustration around: Sincere thanks for your continued unpaid work on free software. You are at a far higher level than I am in all of this. I hope none of this is construed as me suggesting fault in your work, at all, because without your work, people like me could not be doing research on thousands of DICOM images stored in uniquely-compressed formats.