A commercial ZFS solution is (still) coming to Mac OS X, thanks to former Apple filesystem and OS engineer Don Brady (who previously worked on the abandoned internal Apple project to port ZFS). Brady and his company, Ten's Complement, just launched a limited private beta in hopes to have the software polished and ready for a summer launch this year.
Ars spoke with Brady, who has a long history engineering filesystems for Mac OS and Mac OS X, to find out a little about his previous work with ZFS at Apple, and what Mac users can expect to gain from Ten's Complement's port of ZFS.
ZFS versus HFS+
ZFS, aka Zettabyte File System, is a 128-bit file system originally designed by Sun in 2004. The main advantage of 128-bit is that it is possible to have files as large as 16 exabytes on total storage volumes up to a theoretical 256 quadrillion zettabytes. A zettabyte of storage might be hard to comprehend—it's equivalent to over a billion terabytes, or roughly enough space to store 251 billion single-layer DVD movies without further compression. ZFS creator Jeff Bonwick noted that filling a 128-bit file system "would exceed the quantum limits of earth-based storage." So ZFS is designed with massive future capacity in mind.
But ZFS also includes a number of features designed to simplify disk management and maintain data integrity. All disks can be assigned to a single pool of storage, and ZFS keeps track of where all data is on the physical hardware. The storage pools can be arranged as mirrors or RAID-Z groups with data redundancy. Each data block written to a device has an automatic checksum which can be examined for data integrity. If any data gets corrupted, which can happen "silently," ZFS can use redundant mirror or parity data to repair the corruption without any user intervention or lengthy rebuilding.
To prevent the sort of errors that can happen during write operations, ZFS (like BtrFS and others) uses the concept of copy-on-write (COW). This means that a copy of the filesystem structure is made when in the process of writing new blocks. If a drive failure happens during the write, the original data is still accessible and the file system knows that the write operation didn't complete. Because of COW, the file system can automatically make snapshots of files as they are created and edited, recording only the changes for more efficient performance.
Though ZFS was originally developed for Sun's commercial Solaris operating system, it was made open source in 2005. Now part of Open Solaris, it has been ported to other UNIX-like operating systems, including Linux and FreeBSD.
The current file system used on Mac OS X is HFS+, an evolution of the original Hierarchal File System designed for the original Mac OS. HFS was 16-bit, which at the time of Mac OS 8 in the late '90s was becoming a critical constraint on the hardware team. Larger, multi-gigabyte hard drives began to exceed the capacity of HFS. A next-generation file system was planned, but HFS+ was designed as a stop-gap measure until that project was finished.
Brady was the technical lead on the original HFS+ team. "The initial HFS+ was primarily about addressing the block count problem," he told Ars. "Since we believed it was only a stop gap solution, we just went from 16 to 32 bits. Had we known that it would still be in use 15 years later with multi-terabyte drives, we probably would have done more design changes!"
The HFS+ team had about six months to re-write HFS in C from the original 68K assembly code before adding changes. Besides the increased file capacity and ability to access larger disk sizes, the team also added in Unicode support at the behest of engineers on the international team.
Brady later worked on the Mac OS X team, and was involved in much of the work to bring HFS+ to Mac OS X. "Since Mac OS X was Unix and HFS+ wasn't, I had to start adding all the missing features—symlinks, hardlinks, POSIX conformance—and in a way that was backwards compatible to Mac OS 8," he said.
"In hindsight, I'm amazed at how we were able to extend HFS+ so much," Brady added. That includes the extensible metadata used by Mac OS X's Spotlight search, live partition resizing used for Boot Camp, and the Adaptive Hot File Clustering used to reduce seek times for oft-used system files. "I think the last change I was involved with was the directory hard links necessary for Time Machine. Since then, Apple engineers have managed to add even more features, to my amazement."
Skunk works: ZFS at Apple
After working on porting and optimizing HFS+ for Mac OS X, Brady was ready to move on to something new. Yet another next-generation file system had been planned for Mac OS X, but Brady wasn't involved in that team. But, he told Ars, "at the time I had 2 terabytes of family videos and photos, and I was motivated to get a file system that addressed data integrity. So Chris [Emura] and I started a ZFS port as sort of a skunk works project."
Brady and Emura originally worked on porting ZFS on the side before it became an official project within Apple. "When I heard of the FreeBSD port, I was sort of motivated to beat them and get up and running first. I think Pawel [Jakub Dawidek] ended up winning by a week," Brady said.
Work on ZFS at Apple continued even as Amit Singh's MacFUSE project enabled users to load ZFS as a user space module. That eventually lead to read-only support being included in Mac OS X 10.5 Leopard.
"If I had any knowledge of unannounced projects, I couldn't talk about them. But look in Lion—I'm sure you won't find anything but HFS+ in there."
"By early 2007, people inside Apple took notice," Brady told Ars. "We talked about it at WWDC that year. The talk wasn't even advertised in the official schedule, but the room was packed." Source code for a command line only version was made available on Apple's "Mac OS Forge" open source repository, and developer previews of Leopard included the code. By the time Leopard launched, it was solid enough to support reading ZFS storage pools, even if new ones couldn't be created or managed, nor could any new files be written. "We just suppressed the write code paths," Brady explained.
At one time, there was a lot of hope that Apple might transition from HFS+ to ZFS as a more modern replacement. In particular, ZFS's automatic snapshot feature was believed to be the perfect fit for Apple's Time Machine backup tool. When Snow Leopard was first announced in 2008, it was slated to have full read-write ZFS support, at least in the server version.
But when WWDC 2009 rolled around, all mention of ZFS support was scrubbed from Apple's website. A patent infringement lawsuit was still pending between NetApp and Sun, with NetApp claiming that it held patents on copy-on-write. Furthermore, Sun's CDDL open source license was also believed to make ZFS incompatible with Mac OS X, and that Apple couldn't reach suitable license terms with Sun. (Brady told Ars he was not at liberty to share any details on that particular point.) Whatever the reason, ZFS support from Apple was dead.
(The open source community has continued work on the open source code that was available from Apple before it shut down Mac OS Forge. That project, MacZFS, is hosted on Google Code.)
"As you can imagine, it was tough on the team to have all the uncertainty and not be able to talk about it," Brady said. So he left the file system group in the fall of 2009 for the kernel group, focusing on core OS features. And those other internal projects for a new file system? "If I had any knowledge of unannounced projects, I couldn't talk about them," Brady said. "But look in Lion—I'm sure you won't find anything but HFS+ in there."
reader comments
122