About this page.

I'd like to talk about photograph storage strategy, and the use of software to keep track of what I've photographed and the details of the photographs. As an ex-software engineer I know my way around software programming and databases and so on, and I've used my knowledge and skills to create what I think is efficient strategies to store and understand my pictures.

Storage of the material

I often read in various forums about people who take lots of photographs but are really unsure about where and how to store the material. Others will reply, stating how they store and process their work. They'll talk about individually named and dated folders such as Las Vegas June 2017 or 062017LasVegas. All very well, but after a few months of this sort of convention things are getting quite busy in the master folder, and it's becoming increasingly difficult to find that elusive picture of dear old Margaret wearing the silly hat. Things are even more difficult if you shoot RAW. You end up with 062017LasVegas-RAW and 062017LasVegas-JPG and possibly 062017LasVegas-SmallForEmail.

How and what I shoot

I've been through numerous cameras in my time, but my current favourites are Sigmas, the dp0q and the dp2q - wide angle and normal. The native RAW format of a Sigma is X3F. These are complex multi-layered RAW files and get, I believe, the most out of these wonderful cameras, which produce what I'm sure are the sharpest results imaginable. Sigma is a niche brand, quite often despised by many, but highly beloved by afficionados. I don't care what others think, my Sigmas, a little wacky in many ways, produce stunning results.

But X3F? Nothing understands this format. Lightroom, Capture One, Luminar, etc, etc, all turn up their noses and declare that there are no photographs in the folder you're attempting to import. So what do we do? Right, Sigma cameras may be wacky, but wait till you behold Sigma's own post-processing software, Sigma Photo Pro (SPP). This is the ONLY software that understands an X3F apart from a strange and rather unuseable product called Kalpanika, which we will ignore and never mention again.

But I don't think SPP should be used to produce the finished article, the processed and finished JPG. It's just too weird a piece of software to be used in that way, and I'm really not sure it gets the ultimate result out of an X3F.

But if SPP is the only thing that can understand an X3F, but it's not ideal for fully post-processing our RAW pictures, then we're a bit stuck aren't we?!?!

No. We use SPP as an intermediate step, a middleman

This is what we do: Simple!

So we end up with three versions of the same picture. X3F, TIFF and JPG. This is almost as bad as Margaret's hat!

This is where a good storage strategy comes into play

We make no attempt to put the results of our latest outing into meaningfully-named folders. Our software takes care of that. We just place the pictures into serially numbered folders. This is what I do:
  1. Copy the X3Fs into a new folder under the main Sigma folder. The new folder will be called just a number, one higher than the previous one, eg, Sigma/217.
  2. Process the X3Fs into TIFFs using SPP. The default in SPP is to put the TIFFs into the same folder as the X3Fs. Thus we end up with two versions of each picture in Sigma/217, X3F and TIFF.
  3. Move the X3Fs into another new folder within the /X3F main folder, thus they go to Sigma/X3F/217, leaving just the TIFFs in Sigma/217
  4. Import Sigma/217 into LR. Post-process them. In my case the full size JPGs go into a temp folder on the desktop call fullsize. If I think a particular photograph may come to this, my website, I'll also output a 1500 pixels wide version into another desktop folder called 1500. LR adds my CallMeAlan watermark to these
  5. The full-sized versions get moved into a dedicated JPG folder which is Sigma/Devs/217 and the web-sized will go into Sigma/Web/217
So these are what folders we have:

So we end up with three or four versions of the same picture, all in different places. X3F, TIFF and JPG. We still can't find Margaret's hat!

This is where the software makes an entrance.
As a Mac user, my primary programming language is called XOJO. Similar in many way to Visual Basic, it's a powerful professional system which produces efficient and fast program code behind good-looking graphical user interfaces (GUIs). Please don't write in and mention XCode, Apple's own programming environment. Tried it, spent weeks trying to learn it; dismissed it - grossly and needlessly complex.

PhotoDB, a product of CallMeAlan Software

PhotoDB, a XOJO application with some Python backend, uses SQLite and MySQL databases, which are backed up to a MongoDB database. Complex in database terms, but I'm fanatical about backup and security. This is PhotoDB:

The options are quite explanatory.

Some screenshots:

The folder creation and picture import function. The system already
knows what the next new folder number will be, so all it needs to know
is the name of the first picture to be imported. Then it does the rest

Recording details of a new import.
By this stage we have created the TIFFS with SPP,
moved out the X3Fs to their proper home, and post-processed
the TIFFS into JPGs in Lightroom.

Again, the system knows what the next folder number will be.
The next step is to click on Get EXIF and continue.

The EXIF discovery has provided the date, the
number of pictures, the first and last picture
number and the ISO used. If it detects that
various ISOs were used it puts a zero in that field.

All we have to do is give the location and
add some initial notes - they can be edited and added to
elsewhere in the system

A sample of the main view screen.
Columns are folder, date, number of pics,
first and last pic name, location, camera, ISO
and the first 50 or 60 characters of the notes

Double-clicking a row opens up an expansion window.
Textual editable notes are at the top. Emojis at the right
may be clicked, whereupon they are inserted into the notes
at the cursor position.

For a brand new folder we will click Append Exposure Data,
which will discover and paste in the details of individual
pictures and provide averages. This data is saved with the notes

So there you go, my photo storage strategy and the software I wrote to keep track of it all. I hope at least some of this has proved to be useful.