iOta - Photo Organization, Archiving and Management Software (text)

Introduction to the text-only version of iOta

iOta is a tool to organize, index, archive, and web publish your digital photography collection. Whether you use a simple point-and-shoot digital camera, or scan your photographs from a megabuck film scanner after having captured it on your megabuck film SLR, this ImageMagick and Berkeley DB based tool will help you organize your digital photography. iOta allows the user to generate uniquely rename and index images to an embedded database, generate thumbnails and browsable HTML indexes, archive directories of images onto CDR, and caption and describe each photograph in the user's collection. In addition, all this information also travels with the image itself in the thumbnail EXIF header. For documentation on the KDE/Qt version of iOta or on other aspects of image organization, please visit the main iOta page.

iOta consists of a main menu (MM), and two sub-menus called the Image Preprocessing menu (IP), and the Database Access menu (DA). Hence, effectively, iOta has two parts to it

The Image Preprocessing Part

The Database Access Part

For book-keeping, config options, and download sites, you could refer to Other Sundry Information.

The image preprocessing menu has the following entries:

Image Preprocessing Menu ------------------------- 0 - Exit Image Preprocessing Menu. 1 - Rename images in a directory. 2 - Prepare directory for archiving. 3 - Archive a list of directories. 4 - Web publish a directory. 5 - Border a set of images for the web. 9 - Execute shell command.

The database access menu has the following entries:

Database Access Menu -------------------- 0 - Exit Database Access Menu. 1 - Create database from images. 2 - Delete a record. 3 - Edit a record. - Retrieve a record. 41 - Search Category. 42 - Search Name. 43 - Search Description. 44 - Search Keywords. 45 - Search on Date Range. 5 - Insert records from a file. 6 - Get database statistics. 9 - Execute shell command.

To make life simpler, all throughout this document, I will refer to the menu items in a short form which looks somewhat like this. MM->IP->3 would refer to "Archive a list of directories" option (Option 3) in the Image Preprocessing menu, which is under the Main menu. Similarly, MM->DA->43 would refer to the "Search description" option under the Database Access menu.

Part 1 - The Image Preprocessing Menu

1.1 - The Cataloging Process

The first step along the road to better organization is Cataloging. But even before that, we would like to rename the downloaded images to a unique identifier, which can then be accessed through database operations.

Lets say you have used your favorite download or scanning program and have a bunch of image files in a directory called KingsDownloads. For the purposes of this discussion, lets assume that KingsDownloads contains just two files, pic01.jpg and pic02.jpg.

Directory check

KingsDownloads: pic01.jpg pic02.jpg

Let us first rename these files to be globally unique so that you have no problem in indexing/searching for the image years later. iOta uses the date and time stored in the EXIF header of a JPEG file to rename its images uniquely. Not only are your photos uniquely numbered, the time and day glares out at you instantly. The way to do this is to go MM->IP->1. You are prompted for the download directory name. Type in the relative or absolute path for KingsDownloads.

At this point, you should have files named 2001-10-10++14-10-20, and 2001-10-10++15-15-32 (instead of pic01.jpg, pic02.jpg) in your KingsDownloads directory.

Directory check:

KingsDownloads: 2001-10-10++14-10-20 2001-10-10++15-15-32

2001-10-10++14-10-20

2001-10-10++15-15-32

The second step is the most painful, but the most important - giving captions. The menu entry to perform this step is "Prepare directory for archiving" and can be reached as MM->IP->2.
If you have read the iOPlan outline properly, you would notice that its a must to have captions for each and every image you photograph if you want to have a really well indexed database. A year later, if you would like to search for an oil spill at Kings Canyon National Park, the category, keywords, and description information for the image will help you in retrieval instantly. Fail to do this, and you will have to browse through all your Kings Canyon images to get to the one you want.
Having said this, there are lazy guys and there are method freaks. Moreover, the same person could be a method freak now and lazy some other time. I myself oscillate between these two modes. Hence, there is an Abbreviated option for captioning and a Comprehensive option. Take your pick.
- 1.1.1 - Abbreviated Captioning
  In this method, we store 4 things with the image and therefore, into the database: The Category, The Equipment (camera, lens), The Film Used (Smartmedia, Kodak Gold ISO 100 etc.), and the Image Copyright (usually your name).
  On entering MM->IP->2, you are prompted for all of the above fields. Usually, the category is a descriptive line for the current set of photographs. It should also be descriptive enough for you to search for months later, as that is your only "search string". For the two photographs that we are dealing with in this example, the category could be Kings Canyon National Park, California.
  iOta will prompt you as a captioning file is absent. Just say "y" to continue the process.
- 1.1.2 - Comprehensive Captioning
  In addition to the information stored in Abbreviated Captioning as discussed above, you could store much more information per image to help you search more accurately later. That is the whole point of having a database in the first place. I have found that the following extra pieces of information tends to be useful in searching for an image accurately:
  - A Description: This is a small description of the image (maybe just one line), which you think best describes the photograph. This could be used as a caption to the image if you were displaying them in an image gallery.
  - Relevant Keywords: This is a list of all the keywords that you can think of while looking at the photograph - this is to make the photograph easier to search for later, while not spoiling or lengthening the caption itself.
  In the download directory (KingsDownloads), open up a flat text file called txtfile, in any editor, and enter these two pieces of information, preceded by the image name. Separate each entry (imagename, description, keywords) by the delimiter #.
  An easy way of doing this is to first create txtfile thus: unalias ls; ls > txtfile, and then open it in an editor. View your images one by one in any image or thumbnail viewer. A download tool such as gPhoto, or a file browser such as Konqueror will show you a thumbnail view easily.
  So for our two photographs, txtfile could look somewhat like:
  
  2001-10-10++14-10-20# Moss, minerals and mountain, Kings Canyon National Park, California # abstract colors green red rocks # 2001-10-10++15-15-32 # Oil swirls over water, Parking lot, Kings Canyon National Park, California# colors abstract yellow puddle reflection #
  
  where, the description for the first photo is "Moss, minerals and mountain, Kings Canyon National Park, California", and the keywords for it are "abstract colors green red rocks". Similarly, the description for the second image is "Oil swirls over water, Parking lot, Kings Canyon National Park, California" and the keywords are "colors abstract yellow puddle reflection".
Armed with this information in the txtfile, iOta prepares the directory KingsDownloads for archiving. It first creates an index.html of all the images in this directory which could be easily perused using any web browser.
It also creates a Thumbs directory in your KingsDownloads directory, and places nicely bordered thumbnails of all your images in this directory with the category, description, keywords, equipment, date and copyright information embedded in these thumbnails as EXIF image comments. Each thumbnail in the index.html points to the actual image, which has been left untouched and has been placed in a directory called Pictures.

Directory check: After the above step, your directory should appear like this:

KingsDownloads:     Thumbs/     Pictures/     index.html
KingsDownloads/Thumbs:     2001-10-10++14-10-20     2001-10-10++15-15-32
KingsDownloads/Pictures:     2001-10-10++14-10-20     2001-10-10++15-15-32

You need to perform these steps every time you download a new set of photographs. I like to label each download directory with some sort of descriptive name (such as KingsDownloads). You may follow any other naming convention as it really doesn't matter anymore, since all your images carry all the relevant information and can be searched through with ease.

To see the outcome of these steps, you can download the two images shown above, and use "strings [imgname]|more" to see how this information is embedded into the image. A sample index.html file containing these images can also be found here.

1.2 - Storing and Archiving

Suppose you have a bunch of directories, say, KingsDownloads and DeathValleyDownloads, which have been cataloged as described earlier, and are now ready to be archived onto CD-R. Also, lets say that these directories are placed under the top-level directory MyPhotos.

Directory check: At this point, your directory should appear like this:

MyPhotos: KingsDownloads/ DeathValleyDownloads/ MyPhotos/KingsDownloads: Thumbs/ Pictures/ index.html MyPhotos/KingsDownloads/Thumbs: 2001-10-10++14-10-20 2001-10-10++15-15-32 MyPhotos/KingsDownloads/Pictures: 2001-10-10++14-10-20 2001-10-10++15-15-32 MyPhotos/DeathValleyDownloads: Thumbs/ Pictures/ index.html MyPhotos/DeathValleyDownloads/Thumbs: 2001-11-10++14-10-20 2001-11-10++15-15-32 MyPhotos/DeathValleyDownloads/Pictures: 2001-11-10++14-10-20 2001-11-10++15-15-32

Its time for the third step in the Image Preprocessing menu. MM->IP->3 will archive these directories, which you could then burn onto CD-R. This step creates two directories, Thumbs and Pictures under MyPhotos. It then creates hard links from your actual images and thumbnails in your sub-directories (KingsDownloads and DeathValleyDownloads) to Pictures and Thumbs, respectively. It also creates a hard link from all the index.html files within these directories to the top-level MyPhotos. Finally, it creates a master index.html file in MyPhotos, which is sort of a Table-of-Contents file and has links to each sub index.html files along with their category information.

The reason for this elaborate procedure of having the original directories (KingsDownloads and DeathValleyDownloads) as well as top-level Pictures and Thumbs with hard links is as follows. The original structure enables you to add in or remove download directories from your archive. On the other hand, a database shouldn't really care what directory an image is in. It should search on its absolutely unique image name, or on other attributes like category or keywords. Therefore, adding hard links to a top-level Pictures and Thumbs directory will make database retrieval of images quite straightforward and uniform from any archived CDR.

As a final directory check, after the archiving step has completed, you should have the following in MyPhotos:

MyPhotos: KingsDownloads/ DeathValleyDownloads/ Pictures/ Thumbs/ index.html index1.html -> KingsDownloads/index.html index2.html -> DeathValleyDownloads/index.html MyPhotos/Pictures: 2001-10-10++14-10-20 -> KingsDownloads/Pictures/2001-10-10++14-10-20 2001-10-10++14-15-32 -> KingsDownloads/Pictures/2001-10-10++14-15-32 2001-11-10++14-10-20 -> DeathValleyDownloads/Pictures/2001-11-10++14-10-20 2001-11-10++15-15-32 -> DeathValleyDownloads/Pictures/2001-11-10++15-15-32 MyPhotos/Thumbs: 2001-10-10++14-10-20 -> KingsDownloads/Thumbs/2001-10-10++14-10-20 2001-10-10++14-15-32 -> KingsDownloads/Thumbs/2001-10-10++14-15-32 2001-11-10++14-10-20 -> DeathValleyDownloads/Thumbs/2001-11-10++14-10-20 2001-11-10++15-15-32 -> DeathValleyDownloads/Thumbs/2001-11-10++15-15-32 MyPhotos/KingsDownloads: Thumbs/ Pictures/ index.html MyPhotos/KingsDownloads/Thumbs: 2001-10-10++14-10-20 2001-10-10++15-15-32 MyPhotos/KingsDownloads/Pictures: 2001-10-10++14-10-20 2001-10-10++15-15-32 MyPhotos/DeathValleyDownloads: Thumbs/ Pictures/ index.html MyPhotos/DeathValleyDownloads/Thumbs: 2001-11-10++14-10-20 2001-11-10++15-15-32 MyPhotos/DeathValleyDownloads/Pictures: 2001-11-10++14-10-20 2001-11-10++15-15-32

A sample master index.html file can be found here. You can then go through the usual steps of making an image and writing it to disk. I personally use the cdrtools package (mkisofs+cdrecord) and would do the following to burn MyPhotos to CDR:

mkisofs -R -J -o /tmp/cdimg path_to_directory_MyPhotos cdrecord -v speed=4 dev=0,0,0 /tmp/cdimg

This would create a CDR which would be readable on Windows as well as FreeBSD/Linux systems.

1.3 - Web Publishing With Ease

So far, you have cataloged your entire photo collection and archived them on CDR. You just spent your savings investing in a domain name and are planning on spending your pocket money on a hosting service. Now you want to publish your work on the Web. At this point, let us assume that you have a directory called Publish where you would like to store your publish-able photographs.

1.3.1 - Bordering images

If you have followed the Comprehensive option while captioning, you already have all the required information embedded in your images, so the hard part is over. You can now just border your images and upload them to your site. To do this, copy all the images (from the Pictures directory into Publish, and just go MM->IP->5. You will be prompted for the directory (Publish), and the copyright information, and you shall be all set.

Upload all the images in Publish to your web server, and place the corresponding Thumbs directory also on your web server (since the thumbnails carry the caption information). A sample CGI script, image.cgi, would generate outputs like you see when clicking on any image on this web-site.

1.3.2 - Web Publishing

On the other hand, if you eased out and followed the Abbreviated option while captioning, you still have some work to do before uploading your images. Specifically, you have to provide captions for your upload-able images now. If you want the images to appear without captions, you can always just border them as described above.

As before, lets assume that all your upload-able images are in a directory called Publish. You can copy them into Publish from your archived CDRs or from the archived directories on disk. In this Publish directory, create a flat text file called txtfile which contains the following two lines per image: (1) The image filename, and (2) A short one line caption for the image. Therefore, for the two images in our running example, 2001-10-10++14-10-20, and 2001-10-10++15-15-32, txtfile could look somewhat like this:

2001-10-10++14-10-20 # Moss, minerals and mountain, Kings Canyon National Park, California # 2001-10-10++15-15-32# Oil swirls, Parking lot, Kings Canyon National Park, California#

Now, just go MM->IP->4, provide the directory (Publish), equipment and film (camera, lens and film used), and the copyright holder's name, and you are all set. After the step is over, you can upload the Publish directory to your web server.

A sample CGI script, image.cgi, would generate outputs like you see when clicking on any image on this web-site. This script needs the image path (on your website) and the filename from which it is being called, and could be written up in HTML as:

a href=/cgi-bin/image.cgi?Kings/2001-10-10++14-10-20:/iOta.html img src=/Photos/Kings/Thumbs/2001-10-10++14-10-20 border=0 /a

where, the path for our images is /Photos/Kings/2001-10-10++14-10-20, and we are calling the script from this file, i.e., iOta.html.

Click on any thumbnail in this website to see the parameters being passed to /cgi-bin/image.cgi, and of course, you can also evaluate the output that this program produces.

Part 2 - The Database Access Menu

The image database has been built around Berkeley DB version 4 from SleepyCat Software. Its very basic in its functionality right now, and allows searches on the Name (e.g., 2001-10-10++14-10-20), parts of the Category (e.g., Kings Canyon), parts of the Description (e.g., Oil swirls parking), and parts of the Keywords (e.g., abstract colors). It also allows you to retrieve images over a date range for times when you would want all the photos shot over the Memorial Day weekend, for example.

The database modeling part for iOta resulted in the following fields which I thought were useful for any photograph. You may want to use all or a subset of these.

Name: [Char 30] The unique image name, based on date and time.

Category: [Char 255] The broad category of images, such as Kings Canyon National Park.

Description: [Char 255] A more specific description for each image. Something which could be displayed as a caption to the image.

Keywords: [Char 255] Phrases and words describing the image which may not appear as part of the caption/description, such as "patterns green cloudy"

Equipment: [Char 128] All information about equipment ranging from camera used to lenses, whether filters were used etc.

Film: [Char 80] Film type used.

Copyright: [Char 80] Copyright information about the image. Usually this will be your name.

Access Tags: [Short Int] The access control for the image. I have three access types.
1. Private images - for my eyes only.
2. Personal - for a smaller audience.
3. Global - for everyone to see.

Image Tags: [Short Int] The type of image. Currently, I have 4 different types.
1. Normal - nothing special. The default.
2. PanoSingle - Part of a panorama.
3. PanoStitched - A Panorama stitched.
4. Duplicate - A duplicate image because of modifications such as sepia toning, or De-saturation to black and white.

Quality Tags: [Short Int] The quality of the image. Makes it easy to search for good images when you have terabits of other lower grade images lying around.
1. Normal - A regular image.
2. Better - Better than average images.
3. Best - Some of my best photographs. In this was it is easy to search for your best photographs when you desperately need them.

CD Archive Number: [Short Int] This is your CD volume number (after archiving and burning), or your Kodak PhotoCD or PictureCD number. You can play tricks like keeping the first digit to represent the type of CD (Digital camera, Kodak PhotoCD or Kodak Picture CD), and the rest as a CD number.

By default, the AccessTag is set to Personal, ImageTag is set to Normal and QualityTag is set to Normal when you import a set of images into the database. You can later edit the images as you deem necessary (MM->DA->3), and change these fields. Amongst the other fields, Name, Category, Equipment, Film, Copyright are a must and are prompted for by iOta when you prepare a directory of images for archiving. On the other hand, Description and Keywords are extra pieces of information which are imported only if you have followed the Comprehensive captioning guideline.

2.1 - Insertion

prepared for archiving

MM->IP->2

archived on CDR

MM->IP->3

To insert a directory full of images, just go MM->DA->1 and provide the download directory name, e.g., KingsDownloads [1].

2.2 - Deletion

MM->DA->2

2.3 - Editing

MM->DA->3

attributes

iOta

, the old value is retained. This feature is especially useful for updating fields like the Image Access tag, which are left default when the database was created.

2.4 - Searching

Searching is quite intuitive. The results of any search are spewed onto the screen as a list of image filenames along with all their attributes. In addition to this text list which whizzes past you on the screen, the entire result set of ANY search operation is stored in a graphical form in a well formatted HTML file called .iOta.html in your HOME directory.

Therefore, when running iOta and searching through the database, it is almost essential to fire up a browser and point it to $HOME/.iOta.html. Keep refreshing the browser after every search request.[2].

Part 3 - Other Sundry Information

Under any menu, MM, MM->IP or MM->DA, you can run a shell command by entering 9 and then providing the command on being prompted.

There are three files that iOta works with, which are important to know about.
1. The Database File: By default this is $HOME/.iOta.db. Your entire image database is stored here. So, remember to take backups of this file! It is a flat file database, so copying it somewhere should be good enough.
2. The Configuration File: A configuration file similar to httpd.conf or other .conf files. By default it is stored as $HOME/.iOta.conf. Currently, you can change the database file name (from $HOME/.iOta.db to something else), the result file name (from $HOME/.iOta.html to something else), and the root directory of your images (cdddev). If you are storing your images on hard disk starting at /home/photographs, then a sample config file could look like:
  # Sample iOta Config file for a system where # photographs have been archived on disk at /home/photographs database .iOta.db resultfile .iOta.html cddev /home/photographs
  If you are archiving all your photographs on CDR, then cddev should be /mnt/cdrom for Linux systems, and /cdrom for FreeBSD systems:
  # Sample iOta Config file for FreeBSD systems # where the photos are archived on CDR only. database .iOta.db resultfile .iOta.html cddev /cdrom
Licencing: This program has been released under the GNU Public Licence, a copy of which could be found here.

Downloads: Sources and binaries (for Mandrake, RedHat and FreeBSD) can be downloaded from http://www.varp.net/downloads. Please let me know if you have compiled it for other systems, and I will put the binaries up here.

Reporting Bugs: For now, since I don't have a proper Bugzilla kind of mechanism set up for reporting bugs, please use the feedback form for submitting them. Please include a short summary of the bug in the subject line. Also, unless it pertains to FreeBSD, Mandrake or RedHat, please do not submit portability bugs, as I will definitely not have the time to deal with those. On the other hand, help for portability is happily welcome.

Footnotes:

This step takes the information you have entered in your Image Preprocessing steps (which is embedded into your thumbnails for each image), extracts it into the database fields and dumps it into the image database. By default, the database is always stored under your home directory as $HOME/.iOta.db.

The result HTML file $HOME/.iOta.html displays the resulting thumbnails of the search as links to /Pictures/. By default the CDROM mount path is /mnt/cdrom for Linux systems and /cdrom for FreeBSD. You could configure this path to be anything else, such as /Photographs if you are storing all your photos under that directory on your hard disk.
It might so happen that you may get broken thumbnails to some or all the thumbnails in the result HTML file. If it happens, please load the appropriate CD number (displayed in the HTML result file) and reload the HTML result page to view the thumbnails.