On-set data wrangling for digital cinematography
This is based on a forum post at cinematography.com where the best approach to reliable data handling for digital cinematography was being discussed. I'm absolutely terrified by what's being proposed by some productions as regards data handling procedures. Everyone seems to agree a need for backups and redundancy, but there's often a huge holes in data security procedures which people try to fill in by simply making a lot of copies. This helps a bit, but the brute-force approach can be excessively expensive and will rarely provide as much protection as something that's been carefully thought through. Material should come off the camera, be backed up to duplicates, one of which should go to the salt mines for long term storage. Second duplicate goes to post and is verified as being what was expected, complete, and bit for bit verifiable. Only then do you nuke the mag and start again. This means you need more cards. They're cheap. You can deal with it. But the biggest issue, the one that is most overlooked, is ensuring first that the camera original material is reasonable. Hard disks do not have the human beings inside that film labs do, and nobody will run an experienced eye over your footage and say "hang on, this is all bright pink". It's staggeringly easy to take a bad original, carefully duplicate it and back it up, only to find months later, when reshoots are expensive, that all you've done is carefully store gibberish. Problems I have seen with this sort of thing include:
- Material is verified, but it's the wrong material. Specify, retrieve and compare timecode ranges and roll numbers to prevent this, but also check a written description of the first-take picture content, sketch of the frame, or other details to enable an eyeball sanity check. Ask the question: Is it a reasonable-looking image, and is it the right reasonable-looking image.
- All the backing up of hard disks and mag tapes is often done by one person in one room on one computer. Programme interchange problems still exist (disk/tape/card only readable on the computer that wrote it, etc). Redundant backups should be redundantly verified - send the primary to post and have them verify it, verify the secondary yourself, on a different computer, and have the production office compare the numbers then sign a form to let you delete the mag. Don't do it all yourself - humans have an unpleasant tendency to write down what seems right, rather than taking a step back and reevaluating the correctness of what they're seeing. In any case, making the decision to delete camera original material is way above the pay grade of most of the people who end up doing it.
- Both sets of a redundant backup are sent to post, while both sets of another redundant backup are sent to the salt mines. One backup, usually representing a whole reel, appears to go missing completely, causing panic, whereas another has a dangerous hole in its data security. Ask the question: who is handling my media, how am I labelling and controlling where it goes, and how do I keep it and its paperwork together.
- Secondary media are often not subjected to any form of testing, with the result that, if there's a problem with the primary, you won't find out if the backup works until you really need it. Both sets of backups from the first few days of shooting should go to post for checking. This particular problem is endemic in all forms of data backup and archiving in all industries. Ask the question: do I know if my secondary backups are actually good, or do they only get tested when I find I need one, which is already a failure situation?
- Image is thought to be OK, but includes horrible problems such as swapped R/G/B channels (dual-link SDI cables misconnected), bad LUTs, or some other issue. Material should be watched and compared to frame grabs, photos, or written notes describing the image content. Ask the question: it's a DPX file, it's readable, it's verifiably identical to the one I backed up, but is it reasonable.