Tag Archives: SmartOS

How the DTrace Book Got Done

In the last few months, I’ve spent a lot of time on the DTrace book: copy editing, managing the review process, and (ongoing) marketing – keep an eye out for video! Also provided care and feeding and a quiet place to work for one of its authors, Brendan Gregg, shown above.

Brendan and co-author Jim Mauro did an insane amount of work for this book, spending many late nights and long hours to write nearly 1000 pages (before final publishing layout) of mostly original material: 57 topics covered in over 150 new scripts and 150 new one-liners (beyond Brendan’s existing DTrace Toolkit), requiring a lot of new thought and invention. It all adds up to a comprehensive DTrace cookbook that will be useful to sysadmins, developers, students, or anyone needing to provide support and/or performance troubleshooting for systems running Solaris, MacOS X, or FreeBSD.

Here’s the Table of Contents:

1. Introduction
2. The D Language
3. System View
4. Disk I/O
5. File Systems – sample chapter available
6. Network Lower Level Protocols
7. Application Level Protocols
8. Languages
9. Applications
10. Databases
11. Security
12. Kernel
13. Tools
14. Tips and Tricks

Appendices:

A. DTrace Tuneables
B. D Language Reference
C. Provider Args Reference
D. FreeBSD Guide
E. USDT Example
F. Error messages
G. DTrace Cheatsheet
Glossary

The picture below was taken in the final throes of writing, when Brendan got down on the floor to plug in his laptop, and didn’t have time to get up again: he was chatting online with Jim, ironing out last-minute details. The manuscript is now with the publisher for final editing and layout – and we can all breathe a sigh of relief.

NB: This is a companion post to How ZFS Really Gets Done, in what might be an ongoing series about “My Life Among the Geeks” (hey, I’m allowed to use the term – I’m a geek, too).

video:

Why Film Engineers?

In the last three and a half years, working for Sun Microsystems and now Oracle, I have produced over 300 video assets, ranging in length from 10 minutes to 3 hours. Most of this material is software engineers talking about deeply technical topics.

By YouTube standards, our audience isn’t large: my videos have had a few hundred to a few thousand views apiece. So why bother? What’s the ROI?

This breaks down into two underlying questions:

  • Why share this kind of information at all?
  • Why do it in the form of video rather than, say, technical white papers?

Why Share Technical Information?

Although most of my videos have a limited potential audience, those who watch them are the system administrators, developers, and other techies who use our technology in jobs revolving around large, complex systems for hugely complex computing and storage tasks. They are influencers, if not direct decision-makers, in major IT purchases. They prefer to get their information from those who know it best – the engineers who create the products – and they don’t want any marketing spin on it. To these folks, great engineers are gurus, and access to our engineers’ knowledge is a selling point. My technical videos never say: “Buy this technology, it’s great!” They don’t need to, because they feature the engineers who designed it telling you why it’s great.

Why Use Video?

More Effective Learning

Although the percentages on the “Cone of Learning” are open to question (in fact, the author of the original, Edgar Dale, disavows any such numbers on his original diagram), the hierarchical concept itself is common sense. Think back to your own education. Most people find it hard to learn simply by reading. (Otherwise, why take notes from textbooks?) You absorb more by seeing and hearing an engaging teacher. Better still are small, seminar-style classes in which you actively participate. Next on the hierarchy is hands-on learning where you do something yourself (how vivid are your memories of dissecting frogs in high school science?).

Seminar-style learning and hands-on training are beyond the scope of my current job (Sun/Oracle offers classes through another department). But we can certainly do more to engage and instruct our audiences than plain old text.

More Efficient Information Transfer

Top engineers are extremely valuable people whose working hours, from their employers’ point of view, are best spent coding. Even those who have the (quite different from coding) skills and inclination to write papers or blog posts, often simply don’t have the time.

It didn’t take me long at Sun to realize that the most efficient way to get information out of engineers was to film them. This was even easier when, as often happened, they were already creating presentations for conferences or internal seminars. It cost them no extra time or effort for me to film, edit, and publish video of that same presentation. (Video also extends the reach of such presentations to people around the world who cannot attend them in person.)

Outside of formal presentations, you can still tap engineering expertise for video. Sometimes it’s just a matter of getting the right people in a room and letting them talk. Back in March, we had a rare confluence of three of our top performance engineers (Jim Mauro, Brendan Gregg, and Roch Bourbonnais) in the same city at the same time. With Dominic Kay chaperoning, we spent about four hours in a conference room, resulting in at least three hours of usable video (not all of it published yet). As Brendan later pointed out, among all possible forms of technical information transfer, this was by far the most efficient return on his time.

Update: IBM thinks, so too: IBM is Turning to Video to Make Its Point

The Numbers

You may be wondering how much it cost to produce all this video. I have a very complicated spreadsheet in which I track details on every event I’ve filmed. I determined costs, calculating in my time (shooting, editing, managing) and travel, the cases where I did the shooting but paid someone else to edit, or I received tape that someone else had shot and I did the editing, etc. I then divided by the amount of video finally published from each specific event (some trips/events resulted in hours of published video, some only minutes).

With all the variables calculated for, costs ranged from $150 to $1400 per hour of published video. For comparison, I asked our marketing folks how much they were paying for “professional” video production. Their estimate was $3,000-5,000 per video (talking heads in the studio, usually), none of which was longer than 15 minutes. So figure $12,000 to $20,000 per hour.

Guerrilla video is definitely more cost-effective.

And its reach can be larger than anticipated:

I can’t claim any credit for this (though I did post a making of long after). It was filmed and posted by Bryan Cantrill in about half an hour on New Year’s Eve, 2008. Between the humorous presentation, the technical content and the discussion it raised, this thing went viral in a hurry, becoming the most-viewed video ever made about Sun Microsystems (650,000 770,929 over 900k views on YouTube to date). It continues to generate conversation in the market and with current and potential customers – which is, in the end, the real point.

So the major reason to put geeks engineers in front of video cameras is this: IT WORKS.

 

^ top: The Perf Trio: Jim Mauro, Brendan Gregg, and Roch Bourbonnais

Related: