An Honest Superstar: Always One Level Deeper

So the other day, coffee in hand, I opened up one of our bug tickets about some unusual error messages in our C++, GUI-based desktop application and was confronted with this little gem from another dev:

Compiler bug.

The code does this:

00000000000002f0 <petscinitialize_>:
2f0: 48 81 ec e8 01 00 00 sub $0x1e8,%rsp
2f7: 48 89 5c 24 18 mov %rbx,0x18(%rsp)
2fc: 48 89 f3 mov %rsi,%rbx
2ff: 48 89 6c 24 10 mov %rbp,0x10(%rsp)

but the compiler doesn’t emit the correct FDE. It claims %rbp is not
stored on the stack frame, even though it’s stored at 0x10(%sp) by the code.

00000038 0000001c 00000000 FDE cie=00000000 pc=000002f0..00000add
DW_CFA_undefined: r6
DW_CFA_advance_loc: 7 to 000002f7
DW_CFA_def_cfa_offset: 496
DW_CFA_nop
DW_CFA_nop
(r6 = %rbp)

I responded instinctively by spraying my keyboard with droplets of delicious dark caffeine. “Kr-ala-fskwag-yak,” I spluttered, “Uh wha?”

I shouldn’t have been surprised – this kind of bug resolution isn’t all that uncommon from the developer in question, although this was a particularly fine example at a sensitive time of the morning. The developer is, I somewhat grudgingly admit, the best I’ve ever worked with. She* is also prone to patch the windowing system we work with, or indeed any other open source API we have to use.

(* The sex of the developer in question has been changed to protect the innocent)

You might argue that tracking a bug all the way from the GUI symptoms down through assembler and DWARF output is overkill, a waste of time that could better have been spent elsewhere, but you’d be wrong, wrong, wrong. The amount of time it took Janine* to do this? 54 minutes.

(* Yes, the name has been changed as well – there’s no such thing as overkill, after all)

Fifty. Four. Minutes. From start to end. I couldn’t have done that. More to the point, I wouldn’t have. I cut my coding teeth on the Windows API and I’m used to finding what appears to be a bug^H^H^Hundocumented API behaviour and thinking “fine, let’s not waste any more time there, I’ll find another way around.”

Everyone does that, right? Sometimes things just don’t work. The universe just says ‘no’. But the other way of looking at it – Janine’s way – is that computers are logical systems that follow simple instructions. There’s *always* a reason they’re doing what they’re doing. If you keep looking, you can find it.

Having watched Janine at work for a while now, I can see three sweet bonuses for doing this regularly.

Bonus 1: You actually fix the bug, resulting in more elegant application code, free of awkward workarounds.

Bonus 2: You build up a set of tools and techniques for following bugs right down to the assembly level, which makes tracking down problems in the future easier and faster every time. Eventually it takes even less time than getting a cup of coffee and complaining about the API to anyone who can’t get away fast enough.

Bonus 3: You look like a superstar without turning to the dark side.

But these are all just bonuses compared to the best reason to hunt bugs relentlessly through the layers of API right down to their source:

Superstar bonus: You understand your software stack at many levels of abstraction.

Let me explain why this is awesome, because I know it sounds, well, dull. Abstraction is such a Java word. I know.

It’s like this: when you learn to drive a car, you’re told that one pedal makes it go faster, one slower and the wheel turns. Maybe you learn to change gears. That’s pretty much enough to be able to drive a car from A to B.

A racing driver, on the other hand, knows – even groks – that pushing the accelerator increases the flow of fuel through the injectors. He knows that the exhaust gases produce back-pressure on the engine while the turbo is still spinning up, before the boost effect starts to kick in. He knows that his suspension setup distributes this amount of pressure on the tyres during a tight corner, which is just below the coefficient of friction with this road surface.

To put it another way, he understands driving a car at a deeper level; he sees behind the first layer of abstraction and this makes him a much better driver.

In the same way, Guido van Rossum is a much better Python programmer than I’ll ever be, because he knows the system literally inside out. Sure, I have this idea that the garbage collection system uses reference counting rather than the more modern generational systems, but I don’t know that it’ll need to do some cleaning up exactly here, or that using tuples instead of lists just there will result in two fewer indirections per loop in the bytecode.

Although I don’t believe in Joel and friends’ concept of ‘superstar’ programmers (a subject for a whole post on its own), it seems clear to me that you can be 10x as productive in a particular domain that you understand deeply, thoroughly.

That’s why I’m convinced that anyone who wants to become great should take the extra time to peel back that comfortable layer of abstraction and learn about the unseen world beneath.

It’s the only honest way I know to become a superstar.

The Best Comment Ever (No, I Didn’t Write It)

Disclaimer: I didn’t write this and take no credit for it whatsoever. edw519 wrote it as a comment on Hacker News, which you can see here: 3 Simple Rules That Will Make You a ‘Superstar’ Developer – or at the time of writing on the wonderful HN Best Comments page. However, I’m terrified that this zen-like jewel will become lost in comment-page obscurity. Also I want an easy way to link to it, so here it is without further commentary:

A smart accountant once told me that the answer to “How much money did you make?” is always, “Who wants to know?” If it’s an investor, the answer is “A lot.” If it’s a customer, the answer is “A little.” If it’s the IRS, the answer is “None.”

Same thing here. The answer to “Who is a superstar developer?” is always, “Who wants to know?”

To a project manager, the programmer who hits every deadline (regardless of quality) is a superstar.

To a customer, the programmer who solves their problem quickest is a superstar.

To a business owner, the programmer who makes them the most money is a superstar.

To a PHB, the programmer who makes them look the best is the superstar.

To a journalist, the programmer who tells the best stories is the superstar.

To a junior programmer, the best mentor is the superstar.

To another programmer, the programmer they are most likely to want to go into battle with is the superstar.

— edw519, writing an incredible comment on an otherwise unworthy Hacker News story.