Fishworks | Countries Beginning with I

(Part 3 of Resistance is Futile: The Oracle Acquisition)

Note: This slight detour in what is rapidly becoming a long series on the last years of Sun is another very personal angle on a particular piece of Sun history. As you’ll see, I didn’t even know what Fishworks was until it launched, and can’t claim direct insights into what being on that team was like. But I now work elbow-to-elbow every day with six of the former Fishworks team, so I’m sure they’ll correct and expand as they see fit.

I first heard about Fishworks early in my Sun career. It was probably in the late summer of 2007, during my first extended stay in Colorado, that Lynn Rohrer took me to a meeting between engineering and marketing. This was my first time meeting several people whom I would later come to know better, including Brian Wong and Matthew Baier. (The latter made an impression with his stunning good looks, loud checked jacket, and inexplicable accent.) The meeting was about Solaris storage: the engineering group and set of technologies for which I’d been hired to help with content, blogging, social media, and (eventually) video.

I remember little of what was said, except that someone brought up “Amber Road” – the code name for an engineering project I’d not heard of. A palpable chill descended on the room. “Oh, well, that will never be completed,” someone said loftily. I was surprised at the venom aroused by its mere mention, and asked Lynn about it afterwards, but she didn’t tell me much except that it was a storage project being run by some people I had not yet met. This surprised me – I thought I was working for the Solaris storage group, hadn’t realized there were more people doing storage elsewhere at Sun.

Feeling some ownership of anything to do with storage, I kept my ears pricked. From time to time the name Amber Road came up, but always as something remote and mysterious; I was unable to get any details. (I now know that this was absolutely intentional – it was a stealth project, though clearly word had leaked out. To further confuse matters, the team never referred to themselves as Amber Road – it was the name of a product they had effectively killed that had been foisted upon them.)

In March of 2008 I was hired full time by Sun, on condition that I move to the US. Though I initially chose to live in Colorado, I traveled a great deal, filming Sun people at conferences, and had contact with many areas of Sun engineering – first storage, then HPC, grid and cluster, etc. But never that tantalizing Amber Road project. (In retrospect, it was somewhat insulting that they never got in touch – I was running blogs.sun.com/storage, a top destination among the thousands of Sun blogs.)

By September, 2008, the project had publicly taken its true name, Fishworks (for “fully integrated software and hardware”, and partly inspired by Skunk Works). There was a preview in San Francisco for some members of the press. Someone in marketing had the bright idea to have me fly out and film this (I was cheaper than the official Sun media crew), which I was happy to do. Ashlee Vance, familiar to me from his coverage of enterprise storage in The Register, was present. (For some reason, perhaps his horizontally-striped shirt and boyish look, he reminded me of Dennis the Menace.)

I filmed several hours of presentations: Jeff Bonwick, who theatrically abandoned his slide deck and spoke off-the-cuff; Adam Leventhal; Mike Shapiro; one or two beta customer case studies. After the formal presentations, we all adjourned to a nearby bar for a not-very-well-attended party. I cast about for something useful to do; Matthew suggested that I film Stephen O’Grady interviewing Mike Shapiro. We had to do this on the street outside the bar with buses roaring by and bar patrons chattering in the background; I was pleased that the directional Rode Videomic coped so well.

Later that month, the small OpenSolaris community team of which I was a member ran the first Open Storage Summit, at which Mike Shapiro, Ben Rockwood, and many others spoke. I filmed everything, though some of that video was later lost in the Oracle transition.

Going Viral

Part of my job was to encourage Solaris engineers to blog. One of the ideas we came up with, around October, 2008, was a contest. Soon afterwards I heard that Fishworks was about to be officially launched, and I seized the opportunity.

I wrote to the Sun bloggers’ alias (an internal mailing list which reached the thousands of Sun bloggers), as follows: the first 20 Sun bloggers to post about Fishworks after the official lauch would receive “I’m blogging this for Sun” t-shirts. At the end of a week, the three who received the most traffic would also get iPods engraved with the OpenSolaris logo (we had had hundreds of these made as incentive giveaways for the OpenSolaris community).

It didn’t occur to me to ask anybody’s permission to do this. Bryan Cantrill didn’t know who I was, and I think he was caught by surprise when the bloggers’ alias suddenly exploded with discussion about the launch, including basic “What is Fishworks?” questions. This was probably useful: it gave the Fishworks team a chance to spread the word within Sun about a project that had hitherto been hidden from the world. Bryan responded to the thread with some official wording, so that all the potential bloggers would know what they were talking about.

Others were not so happy about my initiative. Sun was very protective of its bloggers’ autonomy, and some people who had nothing to do with Fishworks feared that I was trying to impose a marketing message. Nothing could be less my style, but these people didn’t know me: one said that lazy bloggers would simply copy the press release Bryan had given us – my silly contest would encourage bad blogging! (I had more faith in our colleagues.)

I was trying to manage all this during an insane travel week: (from Twitter) “San Diego (FAST) Sat-Mon, Las Vegas (CEC) Tues-Weds, MPK (Forum) Thurs, Austin (HPC Consortium/SC08) Friday.” I sat in a weird-smelling hotel room in San Diego, reading the emails and then eagerly following the blog posts at midnight on launch day. Not one of the Sun bloggers embarrassed us: of the few dozen who blogged about the launch, everyone found something original to say, even those who had had no contact with the project beforehand.

I could take no credit – but some pleasure – in the fact that the Fishworks team themselves were well prepared, with multiple posts ready to go. I had jokingly told Bryan that only one of his team could win one of the iPods; in fact, he won it himself. (CEO Jonathan Schwartz won one, too, but his blog always got the most traffic, so I gave out a fourth iPod for that contest.)

A couple of days later I was at CEC, Sun’s big internal conference in Las Vegas, but was too busy filming other stuff to see any of the Fishworks launch activities. I did join an informal “birds of a feather” session on blogging – I made a point of showing up when I realized that two of the people I’d been arguing with on the bloggers’ alias would be there: “anyone at CEC, come to the blogging BoF this evening 8-9. I suspect the fur is going to fly. ; ) [It did.] 1:28 PM Nov 12th“.

During the ensuing heated discussion, one of them tried to overwhelm me with his vast online experience. I cut him short: “I’ve been online since 1982, and had my own site since 2001.” That finally got me some cred with him. (It’s ok, we’re all friends now.)

I also said: “my friends in marketing owe me a drink. Or three. 2:09 PM Nov 12th “I took a lot of crap over the contest to get people blogging.”

After all this, I still had no real contact with the Fishworks team, but I started tracking statistics on their blogs, alongside dozens of other Solaris engineering blogs. This meant that they were included in my monthly increased-traffic contest to win the remaining t-shirts.

I became aware of Brendan Gregg early in 2009, after he and Bryan posted a video that went viral on YouTube. Brendan’s post about it got a tremendous amount of traffic, making him the easy winner of the January t-shirt contest. I read the post, saw the video, thought: “Yeah, that’s pretty funny.” It was actually a bit annoying: I’d put so much work into so many videos, none of which had gotten more than a few hundred or a few thousand views, and then this very casual little piece was a runaway geek sensation. Which just proved the point I had tried over and over to explain to my marketing colleagues: it makes no sense to say: “Let’s make a viral video,” because it’s next to impossible to “be viral” without a very clever creative team and a fair amount of money. If you’re very, very lucky, it just happens. It happened to Brendan, who is now known worldwide as “that shouting guy”.

I didn’t think much one way or the other about Brendan himself – Aussie, geek, round-faced, nothing to get excited about – though apparently other women thought him cute.

I sent Brendan his t-shirt. He took pictures of himself wearing it, then forgot to send them to me.

The Solaris Schism

In February, 2009, our community team ran the second Open Storage Summit, at which a couple of the Fishworks team spoke. This was even bigger than the first one, with dual tracks resulting in many hours of footage – which I was then unable to publish, for an extremely stupid reason.

Unbeknownst to me, there had been a power struggle between the Fishworks team and the execs in charge of the rest of Solaris engineering. The week before our summit, the VP of the group I worked for (who shall remain nameless here) had partly lost the battle: everything storage was placed under Mike Shapiro, who by then was managing Fishworks (to the extent that they put up with any management at all). Said VP would probably have stopped the summit from happening if he could, but it was too late – sunk costs for venue, etc. However, he ordered us never to do anything to help “those guys” afterwards. This meant that I had to sit on some very useful video, indefinitely. I also had to close down the extremely successful Sun storage blog in which I collected links and published storage-related videos. Remember this: it comes up again later.

I did publish one talk that had nothing to do with Sun storage: SETI For The People: Addressing the Challenge of Massive Data Sharing. I had invited Tucker Bradford to speak after hearing his boss Jill Tarter speak at TED (via video, which we got to watch live because Sun was a sponsor), hoping that someone in the storage industry would step up to help solve SETI’s massive data storage problem.

I reluctantly dropped storage from my roster of activities, but found plenty of other things to keep busy with. The Fishworks team worked from a “secret” office in San Francisco, so there was little chance that I would casually run into any of them in Broomfield or Menlo Park (nor recognize them if I did).

Half a World Away

Brendan is Australian, but had moved to San Francisco in 2006, at Bryan’s invitation, to join the Fishworks team. It was therefore bizarre, but perhaps fitting, that he and I did finally meet in Brisbane, Australia.

Sun’s James McPherson had organized Australia’s first kernel conference in July, 2009. I was sent to film it, and Brendan (native son made good) was one of the speakers. Someone at Sun had described the Fishworks team to me as: “A bunch of egotistical trolls holed up in their batcave, despising the rest of us.” That was certainly not my first (nor second) impression of Brendan, nor (as I later learned) did it accurately describe the rest of the team. For the most part. 😉

The conference was a success, thanks partly to my efforts – for which I nearly got fired. Here’s what happened:

The keynote speech for the conference was to be given by Jeff Bonwick and Bill Moore, on ZFS. I knew that this would be the big draw, particularly for the audience I was trying to get onto the live video stream at a difficult time of day for most people outside of Australia. Yeah, I wasn’t supposed to help those guys in storage, but this was for the greater good of Solaris and Sun. I publicized the event on Twitter and my Sun video blog, and got some PR help from people like Ben Rockwood. But I knew that the biggest audience I could reach in a hurry would be found on the now-inactive storage blog (readers had been asking me why there had been no posts on it for months).

So I put a two-line announcement on the blog, telling people where they could see the live video stream of a new Bonwick and Moore ZFS talk. I figured Mr Nameless VP wouldn’t notice, or wouldn’t care, or would understand that this was the best thing for the company – right?

Wrong. Apparently he had a screaming fit that could be heard over half a floor in Menlo Park. My manager, Lynn, saved my job, but I was warned not to help storage again. I have to admit I went right on doing so at that very conference: a new friend with a big storage problem happened to be visiting Australia’s Gold Coast that same week, doing location prep for a movie. He’d been asking me for months to get him together with some of Sun’s storage experts, a number of whom were right there with me. So he came down to the conference, met Jeff and Bill, and joined me at a dinner at which four Sun engineers, including Brendan, happily got their teeth into his problem.

As it turned out, I wasn’t able to film Brendan’s two talks on the final day of the Kernel Conference, because I had to be on a plane back to the Bay Area to attend the Community Leadership Summit and OSCON (still more filming!). I left that camera with Garrett D’Amore so others could film the final day and later send me the tapes.

Hanging out and talking during kernel conference social activities, Brendan and I had discovered that we thought similarly about some things, such as the value of capturing technical material on video. In October, 2009, he came to Broomfield to do performance training of Sun field engineers who were selling and supporting the Fishworks appliances. I arranged for him to give a talk to the Front Range OpenSolaris User Group, titled Little Shop of Performance Horrors (the date was close to Halloween). This was surprisingly well attended, considering we’d had a blizzard with about two feet of snow on the ground.

That was just the first of our joint video productions. On one of my San Francisco trips in March, 2010, we managed to gather Brendan, Roch Bourbonnais, and Jim Mauro – that is, several of Sun’s top performance experts – in one room for a series of short videos. We also got Jim and Brendan talking about the DTrace book that they were then writing. Sometime after that I, too, became involved in the book, first coordinating technical reviews, and eventually editing all 1100 pages of it (minus the code, which I was not and am not qualified to comment on). By this time I was living in San Francisco; I’d chosen an apartment in SOMA to have easy access to Caltrain for my commute to Sun’s Menlo Park office.