Tuesday, June 10, 2008

What Do You Need To Back Up?

I'm enjoying the new series on backup strategies at Outback Photo and it got me thinking.

I have three kinds of images.

a) those from film negatives

b) raw files which are then edited in Photoshop

c) stitching jobs which come from several raw files and are then edited in Photoshop.

If I have the negatives, then I only have to decide if the edited version took enough trouble to create and would be a great deal of trouble to recreate and is therefore worth the trouble preserving.

With images from a single raw file, the obvious thing is to back up the raw file and redo the editing. Now, Ansel Adams did exactly that with every print he made, and arguably styles of printing change, we get better at editing and so perhaps there are advantages to not having the edited version to recreate.

With stitched images, I'd need to remember which original raw files were used to create the stitch (currently that would be difficult for me), or I'd need to save the stitched image (which is the same size as the edited image) so I might as well save the edited image.

There are some edits which were so complex that recreating the edit would be next to impossible and there is enough about the edit that I would be lothe to lose it.

Lastly, since often disasters befall entire hard drives, not just one or two images, if I lost all my edits and had to start over, each edit taking hours of work, would I be willing to take that risk.

Given that hard disks fail and the issue is not if but when, it sounds like I need the following:

1) an absolutely reliable way of preserving the raw files since anything that took out both the edits and the raw files would mean total loss of images - NOT ACCEPTABLE.

2) a pretty reliable way to protect the edited images so that the odds of loss of many edited images is low. I would interpret this as a hard drive detached from the power between uses to store all edited images and two backups of raw files, one onsite, the other away (that could be with a relative or at the office or even online). Any really tricky edited images and stitches should perhaps be saved the same as raw files.

By raw files, I mean those raw files which have led to useful edited images. Whether there is a role for saving the hundreds if not thousands of raw files which never inspired you enough to do anything with them is a whole other matter. People sometimes quote the newspaper photographer who happened to capture a picture of someone completely unknown but who years later becomes famous or infamous and suddenly that old image becomes important. Frankly, if I haven't found a use for a raw file within a couple of years, it isn't very likely that I ever will, and the simple fairly reliable backup will be just fine thank you.

Risk Analysis

The risk of a hard disk failure should be considered 100%. It may well be that most times you buy a new computer before a hard disk fails, but frankly that's mostly luck, like seeing storm clouds and not taking your umbrella.

The risk of a power surge getting past your rudimentary power bar is basically the risk of a nearby lightning strike. While such risks vary with climate and power grids, you should plan that once in your life you are going to have a power surge that destroys any storage device currently hooked to the electrical grid.

The risk of losing both the computer and the hooked up backup drive and the disconnected backup drive isn't zero. All you'd need is something to go wrong in doing the backup which said problem gets propagated to the backup before it gets disconnected - not impossible, so you don't have to think house fire.

I'm a family doctor. My medical records are irreplaceable. My income depends on them staying intact. If your income depends on your images, then you need a pretty darn full-proof system to protect them and in that situation you probably won't have time to re-edit your images so all decent edited images need saved connected, disconnected and off site.

One issue I struggle with is whether it's easier to back up all raw files securely or to select out only those that pertain to important images for backup - all my edited images that come from a single raw file retain the raw file number so it wouldn't be impossible to search out those. My stitched images don't, and I have a lot of them - so this is a problem, meaning that saving all raw is certainly easier, if not as cheap. Easy beats cheap every time for me!

It's possible to break down backups another way.

Always backed up - basically we're talking a raid drive or similar, and while that will help in the future, won't deal with the 4 drives I have almost full already. I'd need a big expensive raid system to deal with those. But it's probably more important to start backing up continuously with new images and do intermittent backups of the old stuff.

Intermittent backups - if it's nightly, then I don't want to be plugging and unplugging a drive - as likely to cause problems as fix with all that wear and tear, shaking and static. So nightly backups need to be kept plugged in. A suitably big backup drive(s) is what's needed. If it has redundancy, so much the better, though arguably redundancy in a backup isn't needed.

Occasional backups - given the concerns of long term stability of dvd's, not to mention the hassles of burning a hundred of them, this really means an unpluggable hard drive. How often you do this comes down to how much time you are willing to commit to do it, and how much work you are willing to lose in case of a power surge sneaking past your protection.

How do I do it? Badly and not often enough, but at least I do have a system.

Raw files are offsite and backed up seldom - I should do this monthly.
Edited images are backed up and then unplugged - again it should be monthly

As for continuous backup - well it hasn't happened yet and sooner or later it's going to cost me. Raid systems are cheap enough that I will do something to correct continuous backups within the next month or so. As I'm running out of drive space, a new large raid system seems the way to go.

Nightly backups, well if I back up new work continuously, this isn't as needed but as I'm constantly editing old images, I guess I'm going to do a nightly backup of changed files on the existing drives.

Sure is a hell of a lot more complicated than throwing your negatives in a drawer - though the number of famous photographers who lost negatives to processing errors, floods and darkroom fires is pretty impressive, even after the days of nitrocellulose film stock. All my wedding pictures succumbed to a humid basement in Kentucky.


My Camera World said...

My images files all version the good, bad and ugly are all very important to me and I guess priceless in that some you can never replace.

I have had several drivesfail an I have never lost an image.

MY backup strategy is as follows

RAID C drive (Dual drives) one did fail once. I could work of the other and all I had to do was insert a new drive and it would take of all duplicating. The files rated less than 3 stars are removed from C and only available on backup drives.

Weekly backup to external 500GB always connected. ( had one fail)

Once a month I back up to 2 – 500GB external drives not normally connected to power or computer in ase of electrical surge.(Had a 250GBail before I upgrade to 500GB)

DVD of the best RAW and edited in safety deposit box in case the e gets destroyed.

Every now and then a little prayer just in case.

I am, a little over 300 GB of images and when it start to get full I getr Terabyyte drives.

We buy house insurance and the cost of drives a few hundred every year to upgrade is better insurance for peace of mind.

Niels Henriksen

Mike Mundy said...

"Sure is a hell of a lot more complicated than throwing your negatives in a drawer - though the number of famous photographers who lost negatives to processing errors, floods and darkroom fires is pretty impressive, even after the days of nitrocellulose film stock."
Thinking back to the film era . . . you know, I don't think we HAD the "backup" concept. Except maybe to make duplicate prints. If your film was ruined by X-rays, that was it!
One wonders: at what point does backup concern turn into backup paranoia? Some of the backup routines I've seen are extremely elaborate, to say the least.

George Barr said...

You have that right, Mike. It's a pain in the deriere. Problem is, it's just so much easier than it was to wipe out thousands of important images. Perhaps protecting off-site is a bit much - after all few houses burn down these days and if you are hit by a tornado, you have bigger issues to worry about than saving your images (like, did you get a picture of it). On the other hand I personally have had a software install go so badly wrong that I could not repair the hard drives. Unfortunately the experts who did, weren't able to repair a good half of my images. As this was early in my digital career, it wasn't that big an issue, but it did teach me. Once, during a thunderstorm; I went to turn off the TV and the house was hit the exact second I touched the button - for the first few seconds I thought I'd somehow blown up the TV - but it wasn't me who did it. At several million volts and the ability to jump to the ground from thousands of feet up, I'm not convinced that any piddly little circuit is going to protect my computer.

I think that regular backups to deal with disk failure combined with occasional backups that are then taken off line is the minimum protection anyone should consider if their images are important to them.


KeithAlanK said...

The main thing is to have a convenient system.

My originals and edits stay in the same folder and go to an external drive as soon as the first wave of photoshop is done, and any stray re-edits might get put there, or not. I can live with a few lost edits.
I still use CDs since I only shoot around 8000 jpgs a year, and burn them when I can fill them--every three weeks or so.

Until they are on CDs, my photos aren't really backed-up in my mind.
The external drive is only about easy instant access to everything, plus one layer of protection against a system HD failure.

Lightning strike scenario is that I lose a few images worth selling, but they can usually be re-shot.
House fire scenario is that I lose years of work since I don't trust anyone I know to care for several boxes of CDs correctly.

I really don't look forward to the day I start shooting RAW and the storage problem gets out of control.

Donncha O Caoimh said...

My most recent photos live on my measly internal laptop drive for quick access

I have two external drives. Each is 1TB in size and connected via USB cable so access to them is slightly slower. One drive contains all my older photos, and I sync that drive with my "backups" drive using rsync regularly. I should cron that so it happens every night.

I backup my websites and my internal drive to the backup drive using a piece of free software called Backuppc available at http://backuppc.sourceforge.net/
That keeps 7 days of incremental backups of everything that is likely to change including my photo's work directory, and web servers.

I've lost images in the past due to drive failure, but haven't lost any in 5 years. My one liability is my Macbook which isn't backed up because WiFi is simply too slow to backup. I hardly use it for anything anyway.

Hope that helps, I'm using Linux, but rsync and backuppc are available on Windows and Mac OS X too.

George Purvis said...

Working on Windows, I have 4 internal physical drives configured as 3 logical drives. The first drive, C:, contains Windows and program files. The second logical drive is a Raid 0 pair that contains all of my data, including My Documents, the Desktop and Documents and Settings and every photograph or raw file (15,000+). The third drive is an internal backup drive that contains backup images of my C: drive and my data drive.

I run scheduled backups of the C: and Data drives that create full backup images on the Backup drive. At my convenience, I copy the Backup images to external USB backup drives.

Raw files are downloaded from compact flash using Lightroom. One copy goes to the Lightroom library on the Data drive and a second copy goes to a folder on the C: drive. At my convenience (when I have 4GB of images in the folder), I write the raw files from the folder on the C: drive to DVD and delete them from the C: drive.

To reduce drive failures, my system is on an APS UPS (uninterruptible power supply) that provides some spike protection and eliminates power loss crashes that can result in corrupted file systems. The APS UPS cost less than a single large drive and are worth their price in increased uptime. My system no longer crashes when the lights flicker as the power company makes a change in their distribution grid.

Disks are cheap, my time is not, whether it is time I spend in photography or any other time I spend including time spend installing and updating software. Information is stored willy-nilly on Windows. Do you know which preferences might be in the Registry or an arcane settings file? What about the printer profiles? Where are your monitor calibration files? The time required to build a system for photography is long. A backup strategy should be designed to restore the full system as quickly and easily as possible.

For the same reason, I keep all of my images available on hard drives. I find that I often need to refer to older work to answer questions such as "How did I do this before?" or "How do these shots differ from when I was there last time" or "Ah! now that I know a new way of processing, could I improve that old image?" I don't have time to look through old DVDs, contact sheets, etc, Consider the complication of loading the images back onto the hard drive, making changes and then figuring out how to archive the updated versions so that next time I needed them I could find both the updated versions and the original version.

As far as photo organization is concerned, my Lightroom library contains folders for years and within each year a folder for each day shot. Tags and keywords are used for cross-referencing content. Changes to image filenames are made by appending information, never changing the original in-camera provided first portion of the name.