A conversation between Rick Ramsey of OTN and Darren Moffat of Oracle Solaris engineering on a book useful to Solaris sysadmins (and others). Discussion includes the fact that this book will be useful with Solaris 11 as well.
Tag Archives: what I do
How the DTrace Book Got Done
In the last few months, I’ve spent a lot of time on the DTrace book
Brendan and co-author Jim Mauro
Here’s the Table of Contents:
1. Introduction
2. The D Language
3. System View
4. Disk I/O
5. File Systems – sample chapter available
6. Network Lower Level Protocols
7. Application Level Protocols
8. Languages
9. Applications
10. Databases
11. Security
12. Kernel
13. Tools
14. Tips and Tricks
Appendices:
A. DTrace Tuneables
B. D Language Reference
C. Provider Args Reference
D. FreeBSD Guide
E. USDT Example
F. Error messages
G. DTrace Cheatsheet
Glossary
The picture below was taken in the final throes of writing, when Brendan got down on the floor to plug in his laptop, and didn’t have time to get up again: he was chatting online with Jim, ironing out last-minute details. The manuscript is now with the publisher for final editing and layout – and we can all breathe a sigh of relief.
NB: This is a companion post to How ZFS Really Gets Done, in what might be an ongoing series about “My Life Among the Geeks” (hey, I’m allowed to use the term – I’m a geek, too).
video:
Why Film Engineers?
In the last three and a half years, working for Sun Microsystems and now Oracle, I have produced over 300 video assets, ranging in length from 10 minutes to 3 hours. Most of this material is software engineers talking about deeply technical topics.
By YouTube standards, our audience isn’t large: my videos have had a few hundred to a few thousand views apiece. So why bother? What’s the ROI?
This breaks down into two underlying questions:
- Why share this kind of information at all?
- Why do it in the form of video rather than, say, technical white papers?
Why Share Technical Information?
Although most of my videos have a limited potential audience, those who watch them are the system administrators, developers, and other techies who use our technology in jobs revolving around large, complex systems for hugely complex computing and storage tasks. They are influencers, if not direct decision-makers, in major IT purchases. They prefer to get their information from those who know it best – the engineers who create the products – and they don’t want any marketing spin on it. To these folks, great engineers are gurus, and access to our engineers’ knowledge is a selling point. My technical videos never say: “Buy this technology, it’s great!” They don’t need to, because they feature the engineers who designed it telling you why it’s great.
Why Use Video?
More Effective Learning
Although the percentages on the “Cone of Learning” are open to question (in fact, the author of the original, Edgar Dale, disavows any such numbers on his original diagram), the hierarchical concept itself is common sense. Think back to your own education. Most people find it hard to learn simply by reading. (Otherwise, why take notes from textbooks?) You absorb more by seeing and hearing an engaging teacher. Better still are small, seminar-style classes in which you actively participate. Next on the hierarchy is hands-on learning where you do something yourself (how vivid are your memories of dissecting frogs in high school science?).
Seminar-style learning and hands-on training are beyond the scope of my current job (Sun/Oracle offers classes through another department). But we can certainly do more to engage and instruct our audiences than plain old text.
More Efficient Information Transfer
Top engineers are extremely valuable people whose working hours, from their employers’ point of view, are best spent coding. Even those who have the (quite different from coding) skills and inclination to write papers or blog posts, often simply don’t have the time.
It didn’t take me long at Sun to realize that the most efficient way to get information out of engineers was to film them. This was even easier when, as often happened, they were already creating presentations for conferences or internal seminars. It cost them no extra time or effort for me to film, edit, and publish video of that same presentation. (Video also extends the reach of such presentations to people around the world who cannot attend them in person.)
Outside of formal presentations, you can still tap engineering expertise for video. Sometimes it’s just a matter of getting the right people in a room and letting them talk. Back in March, we had a rare confluence of three of our top performance engineers (Jim Mauro, Brendan Gregg, and Roch Bourbonnais) in the same city at the same time. With Dominic Kay chaperoning, we spent about four hours in a conference room, resulting in at least three hours of usable video (not all of it published yet). As Brendan later pointed out, among all possible forms of technical information transfer, this was by far the most efficient return on his time.
Update: IBM thinks, so too: IBM is Turning to Video to Make Its Point
The Numbers
You may be wondering how much it cost to produce all this video. I have a very complicated spreadsheet in which I track details on every event I’ve filmed. I determined costs, calculating in my time (shooting, editing, managing) and travel, the cases where I did the shooting but paid someone else to edit, or I received tape that someone else had shot and I did the editing, etc. I then divided by the amount of video finally published from each specific event (some trips/events resulted in hours of published video, some only minutes).
With all the variables calculated for, costs ranged from $150 to $1400 per hour of published video. For comparison, I asked our marketing folks how much they were paying for “professional” video production. Their estimate was $3,000-5,000 per video (talking heads in the studio, usually), none of which was longer than 15 minutes. So figure $12,000 to $20,000 per hour.
Guerrilla video is definitely more cost-effective.
And its reach can be larger than anticipated:
I can’t claim any credit for this (though I did post a making of long after). It was filmed and posted by Bryan Cantrill in about half an hour on New Year’s Eve, 2008. Between the humorous presentation, the technical content and the discussion it raised, this thing went viral in a hurry, becoming the most-viewed video ever made about Sun Microsystems (650,000 770,929 over 900k views on YouTube to date). It continues to generate conversation in the market and with current and potential customers – which is, in the end, the real point.
So the major reason to put geeks engineers in front of video cameras is this: IT WORKS.
^ top: The Perf Trio: Jim Mauro, Brendan Gregg, and Roch Bourbonnais
Related:
“Delivering Happiness”
Delivering Happiness: A Path to Profits, Passion, and Purpose
When I heard Tony Hsieh speak at SxSWi in 2009, I was only vaguely aware of Zappos.com, and that mostly because of their advertising and wifi sponsorship in Denver airport. I couldn’t imagine wanting to buy shoes online, so never bothered to look at the site.
However, Hshieh’s talk caught my attention. Here was a company visibly practicing many of the principles of customer service – and its importance in a company – that I had developed for myself in the course of my career, but had had little opportunity to implement. I realized long ago that truly great customer service would require a corporate culture and mindset quite different from the ones I was working in or dealing with, but I alone could not effect the necessary changes.
My History in Customer Service
I have cared about good customer service, both receiving and giving, for a long time. Starting in 1993, I provided great customer service for Incat and Adaptec by being visible and responsive in early online forums (Usenet, email, discussion lists, newsletter). Eventually I was allowed to scale my efforts by hiring a couple of guys who could answer questions as well as I could. But there were huge obstacles to great customer service, such as the lack of communication and data-sharing between the internal groups at the company who dealt with customers. This siloing within companies seems to happen a lot and be considered normal, so it was difficult to get my colleagues to understand that customers neither know nor care about a company’s internal systems and power struggles. To customers, it’s all one big entity, and they are understandably frustrated when the customer service team doesn’t know what sales or support are doing or, worse, they all contradict each other.
There was a smart VP who immediately understood the problem when I presented a graphic showing all the different information stores at the company and how they did (and didn’t) connect to each other. He wanted to try to fix the situation, but soon we were all sidetracked by the spinoff of Roxio, which required us to build new systems and sites from the ground up. As webmaster, I spent 14 hours a day on that for many months, but (as I dimly recall – it was a crazy time) still tried to keep in sight my long-term goal of creating a great customer experience right across the company.
I was ahead of the curve. Long before “user community” became a fashionable term, I realized that our community of customers could help each other far more effectively than we could ever help them: there were simply far more of them than us, and they were using our products in ways than we couldn’t test in the lab. We just needed to help them to communicate with each other, and put some human intelligence into organizing the information they produced.
The kinds of online forums in widespread use today were not well known in 2000, but I designed some nascent social media features into the new Roxio website, and tried to make it a showcase for our users rather than just brochure-ware for our products. I also worked closely with our new tech support organization, trying to keep us all heading in the same direction.
For personal and professional reasons, I left that job in 2001 and went back to Italy (where my family was), so I never got to see whether my ideals of customer service were realized at Roxio.
A few years later, at TVBLOB, I chose “Director of Customer Experience” as my title because I hoped to instill in this new company, from the beginning, a culture of great customer service. We were trying to introduce an entirely new kind of technology to a mass audience, and I felt we would succeed or fail on how our end-user customers felt about us. This would, in turn, be a function of their entire user experience, from product interfaces to sales calls to technical support; it would be critically important to have all these areas aligned behind the same vision of how things should look to customers. I’m not sure whether this could have been achieved in that particular company, but personal and professional considerations took me out of that job before we even had end users.
In summary: I have long cared a a great deal about treating customers well, but realized that truly great customer service would require a corporate culture and mindset quite different from the ones I found myself immersed in. Though I tried very hard for years, I alone could not effect the changes needed. I was left with a vague hope that, somewhere, someday, some company might get it right, and I might even be part of making it happen.
So, back to Tony:
Zappos sounded like a great place to work: they were getting customer service right, and this was proving out for them, financially and otherwise. When I heard Hsieh speak a SxSWi I was intrigued enough to consider having a look at their job openings. But my job at Sun was exciting and fun, and I had no desire to move to Las Vegas, Zappos’ home base. Seven months later Zappos became part of Amazon, a piece of news I somehow missed (had quite a lot going on in my own life at the time).
Still, I was now aware of Zappos and what it was really about (hint: it’s not shoes!). So, when someone recently tweeted about the Delivering Happiness project around Hsieh’s new book, I signed up. I had an ulterior motive: we, too, have books to sell
So I was surprised a couple of weeks ago to receive a package with two advance copies of the Delivering Happiness book, with the request that I post a review on my blog during the week of June 7th, the book’s release date. This week has been tumultuous (on a par with just about every other week in my life in, oh, say, the last three years…), but here I am.
About the Book
(Yes, this is meant to be a book review.)
Delivering Happiness, at 244 pages, is a quick read written in a chatty, friendly style – quite different in both tone and content from the usual “How to Succeed in Business” books.
In the first section, Hsieh covers his own life as an entrepreneur (since childhood!) through the creation and sale of LinkExchange, a dot.com success from which he walked away with quite a lot of pocket change, and some even more valuable lessons. When we can’t learn by direct experience, the next best way to absorb and retain information is via stories, and Hsieh generously shares his business lessons through engaging stories from his own life.
Though I was working in high tech during the boom, I never managed to get rich. I imagined that those who did went off with their millions, bought fancy houses and cars, and never worried about anything again. This wasn’t the case with Hsieh. He mentions that “experiences were much more important to me than material things.” Well, his post-LinkExchange experiences included nearly going broke during the early years of keeping Zappos afloat.
The second section of the book is a history of Zappos and how the founders came to believe that their company was really about customer service, rather than the specific products they sold – after all, anyone could sell the same products. Zappos would differentiate itself by providing the best customer service in the world, and they realized that this would require a different kind of company from top to bottom, bottom to top, outside in and inside out. I won’t spoil the story by telling you how; the book offers many examples of how Zappos creates and sustains the necessary corporate culture.
Some of these examples are presented in the form of short essays written by employees. That, in itself, gives you a clue: employees are encouraged to think with their own heads, use their own voices and words, and decide for themselves how best to please a customer, rather than read from canned scripts and act from a limited list of options.
The Hollywood climax of the story is that Zappos “married” Amazon. You can read something very close to that section of the book here. Like most people in high tech, I’ve been through several corporate buyouts/mergers, and they always make for, um, interesting times. What’s interesting about Zappos and Amazon is that, while both do great customer service (I wrote about my love for Amazon years ago), they approach it quite differently. They chose for Zappos to continue to operate independently, using Amazon’s technical and financial resources to be able to do some things (I’m betting with the website) that they couldn’t have done as quickly on their own. I’ll be curious someday to read a follow-up book about what each company is learning from the other.
In the final section of the book, Hsieh ventures into even bolder territory, sharing research he’s done on the new science of happiness and how that can be applied in a corporate environment. All good stuff, I shall certainly do some of the reading he recommends. The new happiness research quoted by Hsieh shows that people are happy when they feel that they: have control over their lives, are making progress towards a goal, are connected to other people, and are part of something bigger than themselves.
All of this can be applied to running a company, as Zappos is doing brilliantly. But applying it to the act of a customer receiving a shipment is a stretch in a direction I don’t like. I’m wary of the idea that happiness can be delivered in a box in the form of “the perfect pair of shoes or the perfect outfit”. Consumer culture has conditioned us to believe that happiness lies in stuff (or experiences) that can be bought. This may be an inescapable contradiction of capitalism. Companies and economies grow because people buy more and more stuff. To keep them buying, you have to make them believe that your stuff is going to make them happy. But buying stuff is not really what makes people happy, and you are lying to your customers by telling them it will. And it’s not just a little white lie, either: “material goods will make us happy” is the biggest lie the human race has ever told itself. Voracious consumption is destroying our societies, ourselves, and our planet.
Which won’t make anyone very happy.
Tony Hsieh is a smart man and has no doubt seen this contradiction, but for the moment he’s still in the business of selling stuff, so he can’t state this even if he wants to. However, he now has deep pockets and a large soapbox to go with his considerable talents, brains, and charisma. I’ll be watching to see what happens when he turns all this to the question of what, ultimately, is really going to create global happiness.
ps. I already gave away one of the advance copies of the book, but I have an extra hardback copy that they sent me after it was released. If you comment on this post, you’re entered into a contest: one week from today I will randomly choose one commenter to receive one of those copies.
Woodstock School Memories: The Student Survival Kit
This was the first manual/user guide I ever wrote, a student handbook for Woodstock School, incorporating the school’s rules and policies, survival tips, a fill-in class schedule and diary pages. I wanted to make the rules easier to understand, and more accessible, by applying simpler, clearer language and a sense of humor. Apparently it worked. I heard from students who worked on revisions 20+ years later that my style was still the template for student handbooks at Woodstock.
Unfortunately, I haven’t figured out how to get the pages in order in this gallery.
The cartoon illustrations are mostly by my classmate, Jeet Singh.
