About this page.
I'd like to talk about photograph storage strategy, and the use of software to keep track of what I've photographed and the details of the photographs.
As an ex-software engineer I know my way around software programming and databases and so on, and I've used my knowledge and skills to create what I
think is efficient strategies to store and understand my pictures.
Storage of the material
I often read in various forums about people who take lots of photographs but are really unsure about where and how to store the material. Others
will reply, stating how they store and process their work. They'll talk about individually named and dated folders such as Las Vegas June 2017 or
062017LasVegas. All very well, but after a few months of this sort of convention things are getting quite busy in the master folder, and it's
becoming increasingly difficult to find that elusive picture of dear old Margaret wearing the silly hat. Things are even more difficult if you shoot RAW.
You end up with 062017LasVegas-RAW and 062017LasVegas-JPG and possibly 062017LasVegas-SmallForEmail.
How and what I shoot
I've been through numerous cameras in my time, but my current favourites are Sigmas, the dp0q and the dp2q - wide angle and normal. The native
RAW format of a Sigma is X3F. These are complex multi-layered RAW files and get, I believe, the most out of these wonderful cameras, which produce
what I'm sure are the sharpest results imaginable. Sigma is a niche brand, quite often despised by many, but highly beloved by afficionados. I don't
care what others think, my Sigmas, a little wacky in many ways, produce stunning results.
But X3F? Nothing understands this format. Lightroom, Capture One, Luminar, etc, etc, all turn up their noses and declare that there are no photographs
in the folder you're attempting to import. So what do we do? Right, Sigma cameras may be wacky, but wait till you behold Sigma's own post-processing
software, Sigma Photo Pro (SPP). This is the ONLY software that understands an X3F apart from a strange and rather unuseable product called Kalpanika, which
we will ignore and never mention again.
But I don't think SPP should be used to produce the finished article, the processed and finished JPG. It's just too weird a piece of software to be
used in that way, and I'm really not sure it gets the ultimate result out of an X3F.
But if SPP is the only thing that can understand an X3F, but it's not ideal for fully post-processing our RAW pictures, then we're a bit stuck
No. We use SPP as an intermediate step, a middleman
This is what we do:
- Import the X3Fs into SPP
- Don't touch anything, set everything to default
- Batch export the whole lot as TIFFs
- Lightroom understands TIFFs. Essentially they may be considered to be RAWs
- Do all the post-processing in LR - exposure, contrast, colour temperature, clarity, and so on
- Export from LR to JPG
So we end up with three versions of the same picture. X3F, TIFF and JPG. This is almost as bad as Margaret's hat!
This is where a good storage strategy comes into play
We make no attempt to put the results of our latest outing into meaningfully-named folders. Our software takes
care of that. We just place the pictures into serially numbered folders. This is what I do:
- Copy the X3Fs into a new folder under the main Sigma folder. The new folder will be called just a number, one higher than the
previous one, eg, Sigma/217.
- Process the X3Fs into TIFFs using SPP. The default in SPP is to put the TIFFs into the same folder as the X3Fs. Thus we end up
with two versions of each picture in Sigma/217, X3F and TIFF.
- Move the X3Fs into another new folder within the /X3F main folder, thus they go to Sigma/X3F/217, leaving just the TIFFs in Sigma/217
- Import Sigma/217 into LR. Post-process them. In my case the full size JPGs go into a temp folder on the desktop call fullsize. If
I think a particular photograph may come to this, my website, I'll also output a 1500 pixels wide version into another desktop
folder called 1500. LR adds my CallMeAlan watermark to these
- The full-sized versions get moved into a dedicated JPG folder which is Sigma/Devs/217 and the web-sized will go into Sigma/Web/217
So these are what folders we have:
- Sigma/217 containing TIFFs
- Sigma/X3F/217 containing X3Fs
- Sigma/Devs/217 containing finalised JPGs
- Sigma/Web/217 containing finalised reduced size JPGs
- Sigma/OOC/217 containing out-of-camera JPGs. I shoot RAW+JPEG, the latter to be used for a quick preview prior to post-processing the RAWs
So we end up with three or four versions of the same picture, all in different places. X3F, TIFF and JPG. We still can't find Margaret's hat!
This is where the software makes an entrance.
As a Mac user, my primary programming language is called XOJO. Similar in many way to Visual Basic, it's a powerful professional system
which produces efficient and fast program code behind good-looking graphical user interfaces (GUIs). Please don't write in and mention
XCode, Apple's own programming environment. Tried it, spent weeks trying to learn it; dismissed it - grossly and needlessly complex.
PhotoDB, a product of CallMeAlan Software
PhotoDB, a XOJO application with some Python backend, uses SQLite and MySQL databases, which are backed up to a MongoDB database. Complex
in database terms, but I'm fanatical about backup and security. This is PhotoDB:
The options are quite explanatory.
- View database - shows a large grid giving all the essentail information about each folder
- Create folders and import images - this module expects the camera's memory card to be loaded and asks for the name of the first picture
to be imported (as there will probably be older pictures on the card which have already been imported before). It then calls Python scripts
to create the various folders, as explained above, and copies the material off the card to the appropriate folders, ie, Sigma/217 and
- Record a new photo folder - having imported the images we now want to create a database record of the pictures. This module assumes the latest
folder number and reads the EXIF data to derive: date number of pictures, first and last picture name and ISO. You then manually indicate location,
and add free-text notes. This is where the mention of Margaret's hat goes.
- Search notes - we can search for one or two terms, using either AND or OR. Thus Margaret and hat. It then shows us which folder(s) have those
terms in the notes
- Backup and quit - backs up the SQLite database to a local instance of MySQL as well as to the local MongoDB database, then quits
The folder creation and picture import function. The system already
knows what the next new folder number will be, so all it needs to know
is the name of the first picture to be imported. Then it does the rest
Recording details of a new import.
By this stage we have created the TIFFS with SPP,
moved out the X3Fs to their proper home, and post-processed
the TIFFS into JPGs in Lightroom.
Again, the system knows what the next
folder number will be.
The next step is to click on Get EXIF and continue.
The EXIF discovery has provided the date, the
number of pictures, the first and last picture
number and the ISO used. If it detects that
various ISOs were used it puts a zero in that field.
All we have to do is give the location and
add some initial notes - they can be edited and added to
elsewhere in the system
A sample of the main view screen.
Columns are folder, date, number of pics,
first and last pic name, location, camera, ISO
and the first 50 or 60 characters of the notes
Double-clicking a row opens up an expansion window.
Textual editable notes are at the top. Emojis at the right
may be clicked, whereupon they are inserted into the notes
at the cursor position.
For a brand new folder we will click Append Exposure Data,
which will discover and paste in the details of individual
pictures and provide averages. This data is saved with the notes
So there you go, my photo storage strategy and the software I wrote to keep track of it all.
I hope at least some of this has proved to be useful.