I like big server rooms, I cannot lie. One of the things Dan Maslowski showed me on my first tour of Sun’s Broomfield campus in 2007 was the server room – an entire floor of a large building, chock full of Sun hardware, with thousands of fans whirring and lights blinking (data centers are LOUD). I was enthralled, and continued to be every time I got near big hardware in my subsequent career.
One such occasion was when I tagged along with Brendan and a bunch of other performance engineers on a trip to visit Facebook’s data center in Prineville, Oregon. The whole thing, including a weekend at a resort “ranch,” was paid for as part of an effort to recruit them to work for Facebook. Multiple members of Brendan’s team from Netflix (as well as other companies) were invited – yes, their manager knew about it. No, he was not concerned, and in the event no one left Netflix for Facebook.
Most of us went out of curiosity to see the hardware underpinning Facebook, and to be able to tour a data center – they’re rarely open to the public. Data centers have been remote from headquarters locations for many years, and as more and more companies move their operations to the cloud, it’s even less likely that you’d be able to point to a rack of servers and say: “My data is here.”
Most companies are not even managing their own data centers remotely anymore, preferring to leave the wrangling of hardware to cloud providers (AWS, Azure, Google Cloud, Alibaba). The exceptions are a few companies with huge data center needs for their own operations, including Google (everything underlying Google’s search and advertising businesses) and Facebook.
The tour was led by a Facebook VP of data centers whose name I do not remember. He had to shout his explanations over the constant noise. It was all very impressive, with a lot of intelligent engineering to reduce energy consumption. Data centers generate a lot of heat, so cooling is a constant problem. The Prineville location was chosen because its dry climate allows the data center to be cooled with enormous desert coolers – much more energy-efficient than air conditioning. Other companies have located data centers there for the same reason. (Another favored location is Nevada.)
Another piece of engineering cleverness was to save money by keeping long-term data on slower storage. Somewhere in its Terms & Conditions, Facebook has or had a clause about storing everything you put on there in perpetuity. That’s an expensive promise to keep – very expensive, when you consider the billions of people sharing stuff on Facebook every day. (It’s a pity so much of what they share is damaging disinformation and other kinds of rubbish.) Facebook do have all your data, but it can take a while to fetch the older material from the slower, cheaper storage drives. If I remember correctly, these are also in a slower part of the data center – still connected, though possibly not by high-speed networks. (NB: This information is five years old – it’s possible that everything has changed.)
I was not able to capture the scale of this data center – we saw small parts of one or two buildings, and I was not equipped to do, say, a long dolly shot down the rows and rows of server racks. Keeping all these parts running, making replacements and repairs when needed, etc. kept a large crew busy. At the time, all data center operators were starting to look into robotics to help with that labor, as I knew because I was then working for Ericsson, one of the companies becoming interested in robotized data centers which they could sell to their customers. I imagine some of those projects have come to fruition by now, but it’s been a while since I’ve been in a data center.
Prineville was not Facebook’s only data center location in 2016, and more were planned as Facebook’s user base and global footprint grew. So, to the conspiracy theorists who made wild claims during Facebook’s recent outage about Facebook’s headquarters being raided and “all its servers carried off by the government,” I can only say: LOL.
top: Brendan recreates his famous shouting in the datacenter scene
