How does one go about running intensive business-intelligence (BI) database queries on rapidly changing data? We found ourselves pondering this question while helping Etsy do periodic reporting on their massive BI database. We harnessed the power of ZFS to create lightweight, yet writable, clones of production data that could be deleted when the report was done. The side effect was that we could also stand up staging environments at any time to test upgrades, schema changes and anything else that needed to be validated against real data.
A panel at ZFS Day covering the state of ZFS on Linux, FreeBSD, illumos, and more, moderated by Deirdré Straughan wih Matt Ahrens, Justin Gibbs, Will Andrews, Don Brady, Brian Behlendorf
For over 30 years, hard drives have designated the smallest storage location as 512 bytes. In January 2011, all major hard drive manufactures began shipping their hard drive platforms using a new standard called Advanced Format. To aid in the transition, these new hard drives provide a 512 byte emulation mode that allows the drives to advertise themselves as a 512 byte addressable devices. This can severely impact write performance resulting in the need for read-modify-write operations for any misaligned or partial writes that are issued.
The problem is not limited to just physical hardware. Other storage platforms may also provide LUNs (logical unit number) that presents themselves as a 512 byte addressable devices when, in fact, they use a 4K sector size internally. Although ZFS has built-in support for 4K sectors, it has no automatic way of dealing with the lies that the storage devices tell. This talk will focus on the methods that have been developed to work around the lies that hard drive storage platforms tell and will discuss the challenges and drawbacks that come with using 4K sectors.
In 2008, the ZFS Storage Appliance (nee the Sun Storage 7000) was one of the first architectures to add flash SSDs between the existing tiers of disk and memory. The Hybrid Storage Pool (HSP) offered completely new economies, but with both known and unknown idiosyncrasies. Adam Leventhal, the inventor of the HSP, will talk about their functionality and utility, pitfalls and shortcomings, as well as the next steps for ZFS amidst an ever changing landscape of technologies and economics.
