Essential knowledge on handling the digital negative
To ensure the secure transfer of your original camera data, avoid using simple file copy methods (e.g. macOS Finder, Windows Explorer, or basic command line copy). Instead, follow these steps to ensure your are not loosing a day worth of footage because missing or corrupt files weren't noticed before the original recording media had been erased – your minimum standard should be checksum-verified data backups.
- Checksums: A checksum is a small block of data representing a file, used to detect errors that may have been introduced during file transmission. Different types of checksums exist, calculated using various algorithms, but they all serve the same purpose.
- Checksum Verification: A data management tool offering "checksum verify" will calculate and compare the checksum of a file with the checksum of its copy. The verification result (verified or failed), along with additional information, can then be stored in a checksum directory listing called a media hash list (MHL).
A Media Hash List (MHL) is an XML file which contains:
The next time the files will be copied, the hash values from the MHL file will be used as the reference for the verification. If any file was modified in any way (i.e. corrupted or edited), the checksums will no longer match and the verification will fail. Checksums therefore should be created as early as possible, i.e. when the original camera data is copied from the recording media and should be kept along with the original files all the way into archival.
Though most agree that checksums are vital, different tools, companies or individuals may utilize any of the verification methods/checksum types. This makes it difficult to track where a corruption or unintended modification occurred within the production’s lifespan.
To overcome this issue, a sub-committee of the American Society of Cinematographers (ASC), together with members from studios, software companies, DITs, and post facilities, developed an extension to the original MHL concept, the ASC MHL (version 2).
In a nutshell, the ASC MHL refines the XML structure of the MHL file and adds a so-called "chain file", which serves as an MHL history. The chain file keeps track of the path and file hash of each ASC MHL files so their integrity, in turn, can be verified. This enables productions to exactly determine, where in the chain the problem occurred.
More information on the ASC MHL can be found on the website of the American Society of Cinematographers.
The following is a list of data management tools for ensuring secure handling and integrity verification of your original camera data. Those offer the recommended level of security though verification. Please note that this list cannot be exhaustive!
ARRIRAW is ARRI’s format for uncompressed, unencrypted, and uncompromised sensor data. It can be considered a digital version of the camera negative. ARRIRAW is the only format that fully retains the camera's natural color response and great exposure latitude as unprocessed sensor data.
Like film negative, ARRIRAW data has to be developed – or rather processed. The processing step converts the sensor readout into a color image suitable for normal viewing. ARRIRAW conserves the originally recorded raw data for later use, providing the flexibility to go back and refine the results at any time. This makes ARRIRAW the perfect format for digital cinematography and high-quality visual effects production.
Our camera line-up supports in-camera recording of ARRIRAW or MXF/ARRIRAW data. MXF/ARRIRAW is ARRIRAW within a container format (*.mxf), allowing each clip to be contained in just one file, as opposed to several thousand files, because with the regular ARRIRAW format (*.ari) every single frame is a separate file.
Older ALEXA cameras (ALEXA Classic, ALEXA Plus/Plus 4:3, ALEXA M, and ALEXA Studio) require an ARRI-certified external ARRIRAW recorder which understands the T-Link signal.
Debayering – The first phase of ARRIRAW processing is the most compute intensive. ARRIRAW images (like all CMOS sensor raw images) have only one ‘color’ channel (in fact, it’s a color-coded luminance channel). A color reconstruction algorithm calculates the missing components for each pixel based on the type and position of colored filters on the camera sensor. ARRI cameras use the Bayer pattern color filter array. The term 'color reconstruction' therefore is also known as ‘debayering’. The Bayer pattern filters the light hitting the sensor so that 50% of the sensor’s photosites are used to represent green, 25% of the photosites represent red, and the remaining 25% represent blue. Therefore, the debayer algorithm needs to reconstruct 75% of red, 50% of green, and 75% or blue color information:
The image above shows a single-channel capture from the sensor on the left, and the reconstructed image on the right. Utilizing the information about its color filter array half of the reconstructed image's green values are interpolated from the surrounding photosites rather than captured, as are three-quarters of the red and three-quarters of the blue.
The output quality of the image depends on the debayering algorithm. Generally speaking, a simpler algorithm will process faster, but will also involve a higher probability of color errors. ARRI developed and maintains its own debayer algorithm (ARRI Debayer Algorithm, or ADA for short). ADA is avaialble in the ARRI Reference Tool and 3rd party applications using our SDK.
For VFX, however, the images are often processed using the native sensor pixel count and then downscaled to e.g. 4K or 2K at a later stage. Using this approach takes advantage of the luminance resolution, which correlates to the sensor pixel count.
ARRIRAW SDK and Third-Party Implementations
In many cases, ARRI's SDK is fast and adaptable enough to satisfy the processing needs of an application. But in some cases, especially when the product uses custom hardware (or standard hardware in a non-standard way), Partner Program members may want to implement the ARRIRAW processing pipeline themselves. In this scenario, the SDK serves as a reference against which the member's developers can test their results before submitting evaluation imagery to ARRI's Workflow Group. In a few applications, the vendor's product offers both processing solutions, giving the user the option of maximum throughput with a very good match to the SDK, or bit-for-bit matching of other products that use the reference SDK implementation.
ARRI has built long-term relationships with postproduction equipment manufacturers through the ARRI Partner Program. These relationships have facilitated all of the leading compositing and color correction tools being able to process ARRIRAW files out of the box.
ARRIRAW in 3rd-party implementations
In many cases, ARRI's software development kit (SDK) is fast and adaptable enough to satisfy the processing needs of an application. In some cases however, especially when the product uses custom hardware (or standard hardware in a non-standard way), Partner Program members may want to implement the ARRIRAW processing pipeline themselves. In this scenario, the SDK serves as a reference against which the member's developers can test their results before submitting evaluation imagery to ARRI's Workflow Group. In a few applications, the vendor's product offers both processing solutions, giving the user the option of maximum throughput with a very good match to the SDK, or bit-for-bit matching of other products that use the reference SDK implementation.
The ARRI Reference Tool (ART) is a software application (for Windows or Mac) that provides a graphical user interface for ARRI’s reference SDK. ART is a combined tool for viewing, rendering, metadata, and look files. It combines and therefore replaces the ARRIRAW Converter, ARRI Color Tool, and ARRI Meta Extract.
Learn more and download your copy of the ARRI Reference Tool!
Once you have chosen your tricks of the trade for an upcoming production, ARRI recommends running tests rather than to anticipate that things should work. So, define:
ARRI provides sample footage of all our cameras via FTP server or cloud service.
HDE or CODEX High Density Encoding is a lossless, variable bitrate encoding scheme for ARRIAW data. It reduces the data footprint of ARRIRAW by up to 40 – 50%, but delivers a bit-exact match to the original files when it's decoded. So, HDE does not compromise on quality.
Supported by leading content providers and postproduction applications, HDE is widely used on ARRIRAW data from any ARRI camera. The conversion to HDE takes place during offload to a storage or later on via ARRI's HDE Transcoder (ALEXA 35 only).
HDE, like ARRIRAW, comes in two file types:
We've started transitioning to MXF-wrapped files in ALEXA Mini. This brings speed advantages within the camera and postproduction: Static metadata, information that does not change throughout a clip) is only written once. File systems do not need to go back and forth looking for data to debayer an image but read that information once.
Single frame ARRIRAW and HDE therefore is only available for earlier camera models such as ALEXA XT, ALEXA SXT and ALEXA LF.
ALEXA 35 gives you two options to turn MXF/ARRIRAW into MXF/HDE:
ALEXA 265 MXF/ARRIRAW clips can be converted to MXF/HDE upon download: the CODEX Device Manager, also needed to operate our download stations/docks, takes care of MXF/HDE creation as well.
The software can either present only an ARRIRAW volume or a virtual MXF/HDE volume next to the actual CODEX drive on the desktop. The data on that virtual drive can be copied similar to MXF/ARRIRAW using industry standard media management tools. The HDE files are created on-the-fly by the time they get transferred.
For drives containing ARRIRAW data from any earlier ARRI camera, the only option to apply HDE is CODEX Device Manager. Customers with an ALEXA Mini or AMIRA in addidtion need a CODEX CFast HDE License ("HDE for any CFast 2.0 reader") unless you still have a CODEX USB CFast 2.0 card reader to access the cards.
In 2010, ALEXA was the first digital motion picture camera on the market offering in-camera recording of Apple ProRes files onto SxS PRO cards, providing the full range of codecs from Apple ProRes 422 Proxy to Apple ProRes 4444 and later 4444 XQ. We called this "direct-to-edit", becuase MacBooks of that time featured an internal card reader to offload the SxS PRO cards.
In 2019, ALEXA Mini LF introduced a new method for recording Apple ProRes: sound, images, and metadata are no longer wrapped inside a QuickTime *.mov container but use the common MXF (Material eXchange Format) container.
MXF is an open standard that is widely supported and allows for more flexible access to metadata. Using the same container also allows us to record ARRIRAW and Apple ProRes to the same drive without reformatting. Apple is supporting MXF/Apple ProRes. They have published the "SMPTE RDD 44:MXF – Mapping and Application of Apple ProRes" (on IEEE.org).
ARRI cameras that currently support QuickTime/Apple ProRes will continue to use QuickTime.
Recording clips in Apple ProRes 4444 XQ preserves the full sensor quantization in logarithmic encoding, with the same range of colors available in ARRIRAW. Images recorded in a 4:4:4 codec are almost indistinguishable from uncompressed HD or UHD material. This makes internal recording attractive to feature film productions for the big screen, too. Recording in any of the high-end 4:2:2 codecs provides perfect source material for web or TV applications.
Please Note:
Apple ProRes is a variable bitrate codec, therefore a longer recording time than initially indicated on the camera may be possible.
Apple ProRes White Paper (2022)
