yield thought

it's not as hard as you think

An Honest Superstar: Always One Level Deeper

with 7 comments

So the other day, coffee in hand, I opened up one of our bug tickets about some unusual error messages in our C++, GUI-based desktop application and was confronted with this little gem from another dev:

Compiler bug.

The code does this:

00000000000002f0 <petscinitialize_>:
2f0: 48 81 ec e8 01 00 00 sub $0x1e8,%rsp
2f7: 48 89 5c 24 18 mov %rbx,0x18(%rsp)
2fc: 48 89 f3 mov %rsi,%rbx
2ff: 48 89 6c 24 10 mov %rbp,0x10(%rsp)

but the compiler doesn’t emit the correct FDE. It claims %rbp is not
stored on the stack frame, even though it’s stored at 0x10(%sp) by the code.

00000038 0000001c 00000000 FDE cie=00000000 pc=000002f0..00000add
DW_CFA_undefined: r6
DW_CFA_advance_loc: 7 to 000002f7
DW_CFA_def_cfa_offset: 496
(r6 = %rbp)

I responded instinctively by spraying my keyboard with droplets of delicious dark caffeine. “Kr-ala-fskwag-yak,” I spluttered, “Uh wha?”

I shouldn’t have been surprised – this kind of bug resolution isn’t all that uncommon from the developer in question, although this was a particularly fine example at a sensitive time of the morning. The developer is, I somewhat grudgingly admit, the best I’ve ever worked with. She* is also prone to patch the windowing system we work with, or indeed any other open source API we have to use.

(* The sex of the developer in question has been changed to protect the innocent)

You might argue that tracking a bug all the way from the GUI symptoms down through assembler and DWARF output is overkill, a waste of time that could better have been spent elsewhere, but you’d be wrong, wrong, wrong. The amount of time it took Janine* to do this? 54 minutes.

(* Yes, the name has been changed as well – there’s no such thing as overkill, after all)

Fifty. Four. Minutes. From start to end. I couldn’t have done that. More to the point, I wouldn’t have. I cut my coding teeth on the Windows API and I’m used to finding what appears to be a bug^H^H^Hundocumented API behaviour and thinking “fine, let’s not waste any more time there, I’ll find another way around.”

Everyone does that, right? Sometimes things just don’t work. The universe just says ‘no’. But the other way of looking at it – Janine’s way – is that computers are logical systems that follow simple instructions. There’s *always* a reason they’re doing what they’re doing. If you keep looking, you can find it.

Having watched Janine at work for a while now, I can see three sweet bonuses for doing this regularly.

Bonus 1: You actually fix the bug, resulting in more elegant application code, free of awkward workarounds.

Bonus 2: You build up a set of tools and techniques for following bugs right down to the assembly level, which makes tracking down problems in the future easier and faster every time. Eventually it takes even less time than getting a cup of coffee and complaining about the API to anyone who can’t get away fast enough.

Bonus 3: You look like a superstar without turning to the dark side.

But these are all just bonuses compared to the best reason to hunt bugs relentlessly through the layers of API right down to their source:

Superstar bonus: You understand your software stack at many levels of abstraction.

Let me explain why this is awesome, because I know it sounds, well, dull. Abstraction is such a Java word. I know.

It’s like this: when you learn to drive a car, you’re told that one pedal makes it go faster, one slower and the wheel turns. Maybe you learn to change gears. That’s pretty much enough to be able to drive a car from A to B.

A racing driver, on the other hand, knows – even groks – that pushing the accelerator increases the flow of fuel through the injectors. He knows that the exhaust gases produce back-pressure on the engine while the turbo is still spinning up, before the boost effect starts to kick in. He knows that his suspension setup distributes this amount of pressure on the tyres during a tight corner, which is just below the coefficient of friction with this road surface.

To put it another way, he understands driving a car at a deeper level; he sees behind the first layer of abstraction and this makes him a much better driver.

In the same way, Guido van Rossum is a much better Python programmer than I’ll ever be, because he knows the system literally inside out. Sure, I have this idea that the garbage collection system uses reference counting rather than the more modern generational systems, but I don’t know that it’ll need to do some cleaning up exactly here, or that using tuples instead of lists just there will result in two fewer indirections per loop in the bytecode.

Although I don’t believe in Joel and friends’ concept of ‘superstar’ programmers (a subject for a whole post on its own), it seems clear to me that you can be 10x as productive in a particular domain that you understand deeply, thoroughly.

That’s why I’m convinced that anyone who wants to become great should take the extra time to peel back that comfortable layer of abstraction and learn about the unseen world beneath.

It’s the only honest way I know to become a superstar.


Written by coderoom

February 1, 2010 at 8:24 pm

Posted in Programming

Tagged with ,

7 Responses

Subscribe to comments with RSS.

  1. The 5 Stages of Programmer Incompetence left out the critical first stage, i.e Using programs and thinking you invented them.


    March 21, 2010 at 3:57 pm

  2. Also…..good article. I’ve gotten the name work around Chris at work, cause I hate debugging deeper into the API. Mostly because I don’t understand the coding environment I’m in. But looking like a ninja is clearly better. *Thumbs up*

    Christopher Roy

    March 26, 2010 at 6:46 pm

  3. Great post, inspirational even.


    March 26, 2010 at 8:27 pm

  4. I’ve started deliberately digging a lot deeper these days. For example, I have the Ruby source, I’ve compiled it, run irb, and started down through the call stack with main().

    Also, had to laugh, Petsc? Just guessing on that one… only compiled once, turned out to be overkill… but it sounds like you’re into something I’d enjoy.

    David Doolin

    January 18, 2011 at 4:53 pm

    • I know, this is kind of lame, replying to my own comment, but this article and the previous on “how to be a superstar” have lodged into my thoughts and won’t let go.

      Anyway, these days, collaborative development, agile, all that stuff is all the rage. Kumbaya programming. There is no box. And not even cubicles.

      But it occurred to me, this kind of mastery doesn’t occur without a lot of, well, individual focus and concentration.

      The cognitive dissonance I’m experiencing is that employers seem to say they want this kind of mastery, but they do not provide an environment encouraging it.

      Or is it just me? I’m open to that, I’m past the cynicism, now I’m simply curious how many people can develop such mastery working in a noisy environment with a lot of distraction.

      David Doolin

      January 18, 2011 at 6:30 pm

      • Hi David, no, not Petsc: I work for Allinea developing petascale development tools (linked from the current blog at http://yieldthought.com). Nice try though 😉

        I’ve always been very lucky in having a lot of control over my working environment, both physically and organizationally – we have what must be the bare minimum of ‘noisy’ interference from e.g. management or other internal sources, so I can’t really comment on its effect.

        On the other hand, I never started developing a deeper understanding until I was shown that it was not just possible but worthwhile to dig into someone else’s code running at the level below your own. Generally, all the code in your stack was written by someone who thought it made sense and wants you to be able to understand it. Educated guesses can take you a long way!


        January 19, 2011 at 9:03 am

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: