Posts Tagged ‘project management’
Don’t stop listening until you’ve heard my personal message around 1:44 ;-) In fact, it keeps on getting better!
If anyone makes a nice cut summarizing the software development sections (or just a loop ending at 1:16) send me a link!
Anyone know of any more songs secretly about programming and startups out there?
I start in the middle of a sentence and move both directions at once.
— John Coltrane
Newspaper reporters are taught to write fractal articles: summarize the entire article in the title. Elaborate a little in the first sentence, then fill out the most important details in the first paragraph. A reader should be able to stop at any point without having missed any important details. We should approach programming projects in the same way.
As a child – after some experimentation and a lot of thought – I decided the best way to eat cornflakes was as follows:
- Pour the cool milk over the golden roasted flakes
- Sprinkle the one allowed teaspoon of sugar over the top
- Start eating around the edges, saving the sugary middle section for one last big spoonful of joy at the end
I stand by that decision. In fact, I’ve noticed I do similar things in other areas of my life. I’m sure a psychologist would talk for hours on the subject. Luckily for you I’m not a psychologist, I’m a programmer. And it turns out that this is an awful way to work on software projects.
Has this ever happened to you? You wake up one day with a great new idea for applying bayesian filtering to twitter streams to filter out the
pictures of Joel’s new puppy spam. You’re totally convinced it’s what the world needs. It’s the startup that’s finally going to help you to break out of your day job maintaining PHP payroll software stock supermarket shelf stockers. So what do you do? You do this:
- Fire up your IDE and start a new website project
- Whip up a login page and get the user account basics set up
- Decide OpenID’s really where it’s at these days and hit stackoverflow for a good OpenID provider plugin
- Run into problems getting it to accept Google accounts and spend half the night debugging it
Wait, what? How did this happen? Getting OpenID working isn’t fun. It’s almost the definition of not fun.
I didn’t want to do all this, I just wanted to make an awesome bayesian twitter filter, but somehow there’s all this stuff I have to get through first.
– Me (swear words redacted)
My hard disk is littered with projects that I started, got half way through setting up without ever really getting to the good bit, then abandoned. I suspect yours is, too.
The right way to start a bayesian twitter filter is to apply a bayesian filter to content from a twitter stream. I know. It looks like this:
- Google for some bayesian filter code
- Dump whatever’s in your twitter client logs to a file and write three lines of python to parse it into a form the bayesian filter can work with
- Train the filter and see what happens
Compared to the original approach it looks awesome, right? So what stops us approaching all projects like this? Well, there’s something beguiling about wanting to get the framework right from the start this time. It’s more comfortable starting with something we already know how to solve. Sometimes we have a clear vision of how it should end up in our heads and simply start to create that vision from the beginning through to the end.
Start in the middle
– Paul Graham (lightly paraphrased)
Lean startups and the Minimum Viable Product are all about starting in the middle. Paul Graham’s advice for startups can be summed up as ‘first solve the interesting part of the problem, then build the business around it’, but the process is also fractal – starting in the middle applies right down to the level of writing a new class, or a single function. First write some code that solves the problem even if it’s imperfect or partial, then expand it out with your favourite blend of accessors, inheritance and polymorphism (Note: don’t even bother with that bit unless you hate yourself).
I’ve seen four key benefits to starting in the middle:
Benefit 1: Ideas that turn out to be impossible or just plain bad are discovered early. This is classic lean startup advice: fail early.
Benefit 2: Spend most of your the time solving interesting problems and not fine-tuning framework skills. Which would you rather get better at?
Benefit 3: Discover interesting things while your project is still young and flexible enough to adapt to them.
Benefit 4: Once you’ve solved a problem, you’re so motivated to use it that you finish up the surrounding areas in no time. You add extra users because you want to show it to your friends; you add keyboard shortcuts because you’re getting tired of using the mouse all the time. This is programming the right way around – first the need, the desire, and then the solution.
I’ve recently seen all of these benefits while working on my own side-project-turned-startup. Ages ago I had this great little idea for making profiling so simple that it just told you which line of code was slowest in a nice little report and I whipped up some C# code to do just that. The results weren’t making much sense, so I tried plotting the data to a canvas to see what was going on. Pretty soon I was looking at a poor man’s sketch of this:
Instantly I knew I’d been working on the wrong thing; seeing the execution of a program laid out before me in all its glory was so rich and so interesting; something I had no hope of summarizing in a small table of figures. I just had to explore it – I added function names, colour, a breakdown of time spent in each and over time it grew into such a valuable part of my toolkit that I’ve started sharing it with the rest of the world.
Would I have changed direction if I had already created a website, a login system, a messaging layer, a database schema all geared around the original idea? No. I’d have reached the interesting part of the problem with a half-finished framework and close to zero energy and enthusiasm. The discouragement at seeing the futility of my cherished profiling-via-pdf idea would’ve seen me put the whole thing back on the shelf and go play Altitude to forget about it.
So start in the middle, start with the interesting, challenging, core of the problem you’re trying to solve. Cut down everything else to ridiculous minima and see what happens; you may create something fascinating.
Note: Yield Thought has moved to http://yieldthought.com – check there for the latest posts!
Incompetence is the cause of many ills in the software world, but increasingly I’m seeing a certain kind of competence as being just as destructive. You see, there are a lot of programmers who care deeply about their code. I did, but it turns out there are good reasons we shouldn’t do that; reasons why we should hate our code…
Reason 1: It keeps you open to change
The most productive way to write software is to release early and iterate quickly with feedback from real users, right? It saves us from implementing the wrong features, that it helps build a community and so on. What’s people don’t talk about as often is the impact it has on our relationship with the code. Putting a program in front of users is terrifying because they might not like it. The more lovingly we craft and polish every class before release, the more carefully we plan our abstractions and structures, the more emotionally attached we are to the way we have made it.
The result is that we instinctively resist making significant changes to the code based on early feedback, even though this was the whole point. It’s an unconscious, emotional reaction that we rationalize away to ourselves by claiming the users just don’t understand the feature, or that it just needs a bit more polish when, seen objectively, the best thing to do might be to restructure it completely do do something rather different.
In fact, the best way to remain responsive to your users is to hate your code.
Reason 2: Jeff Atwood says that you should
You can tell a competent software developer from an incompetent one with a single interview question: “What’s the worst code you’ve seen recently?”
If their answer isn’t immediately and without any hesitation these two words: “My own,” then you should end the interview.
– Jeff Atwood
Every now and again people attack Jeff for his assertion that we write bad code, for his arms-wide-open group humility. I think people misunderstand him. The message is not you are a bad programmer, it’s that part of being – of becoming – a great programmer is writing code that you’re ashamed of. If you think all the code you write is wonderful, you’re probably not learning anything.
Reason 3: If you don’t hate your code, you’re wasting too much time on it
Andy Brice famously wrote that if you aren’t embarrassed by v1.0 you didn’t release it early enough – you spent too much time polishing it to meet your expectations instead of giving it to the user and finding out what they think of it. A corollary of this is that if you aren’t embarrassed by your own code then you spent time crafting and reworking it until you loved it. This is time that you should have spent trying to make a product the user will love.
Nobody cares if you refactor to use the Chain of Responsibility pattern or not; they want simple, elegant features that work first time. We should spend our time delivering that, instead.
Reason 4: It’s what Jesus would do
No one can serve two masters. For you will hate one and love the other; you will be devoted to one and despise the other. You cannot love both your code and your users.
– Matthew 6:24 (lightly paraphrased)
Reason 5: Loving your code for its own sake will make your product suck
Loving your own code is a kind of hubris. It presumes that you’re writing the code so that you can work on it in the future, or can show off a neat trick to your colleagues.
This love is misplaced; your code isn’t going to be judged on its internal beauty and consistency. It’s going to be judged by people who just want to get their work done. They’ll love it if it makes that easier and they’ll hate it if it distracts them from that task, if it makes their lives harder.
Creator: I love the power of Unix/AJAX/C#/whatever.
Customer: I want to finish my work and go play outside.
– Scott Belkun
I once spent six months crafting a beautifully-constructed version 2.0 of our company’s product. We’d started again so we could use unit tests throughout, the code was completely modular and pleasant to work on. There was none of the spaghetti-code complexity that dogged the first version.
We demoed an early prototype to management and marketing who said:
Great, so it’ll be ready next month?
Aghast I pointed out we still had to add a whole bunch of features and do a ton of testing. Their faces fell, and at that point I realized that it didn’t matter how beautiful the new codebase was. It would cost too much to bring it up to the feature level of the old one.
Its intrinsic beauty wasn’t worth anything; only user-visible features weighed in the balance. The project was cancelled, and rightly so.
To worry about code aesthetics more than the aesthetics of the product itself… is akin to a song writer worrying about the aesthetics of the sheet music instead of the quality of sounds people hear when the band actually plays
– Scott Belkun
Reason 6: Gandhi would want you to
Hate the User class, love the user.
– Mahatma Gandhi (lightly paraphrased)
Reason 7: It will make you a better developer
Like most people, I got into programming because I wanted to write things for myself and for my friends. During this enthusiastic newbie phase I was a happy user and a happy developer. My code sucked but I didn’t know that yet.
Later on I got into programming for its own sake, I ended up caring deeply about the kind of design patterns I was applying and whether this for loop could be refactored a third time. Eventually this abstraction freak phase left me almost incapable of creating any real value because I invariably got lost in the upper echelons of the software design and never finished any useful features.
Of course, when I started writing professionally I had to care about the features, and had to see my beautiful code mutilated again and again in the name of profit, of getting this feature out in time for the next release. This is enough to make anyone into a bitter veteran.
Eventually I started discovering the joy of programming again, via dynamic languages and web frameworks that made it possible to get useful programs out of the door on day one. I started writing side projects I actually wanted to use. I became free to write the minimum necessary to make something useful. I’d become the ‘guru’, who mistakenly believes he’s better than everyone else because he’s finally become productive again.
But I haven’t become great yet. What might the next step be? I don’t know. Reading more code? Writing less code? Perhaps. But at the moment I have a feeling that to grow in another dimension – creating awesome software – I need to sacrifice the pride I’ve built up in my art:
The seventh step of humility is that a monk not only admits with his tongue but is also convinced in his heart that he is inferior to all and of less value, humbling himself and saying with the Prophet: I was exalted, then I was humbled and overwhelmed with confusion and again, It is a blessing that you have humbled me so that I can learn your commandments.
– The Rule of St Benedict 7.51-54
Until today I found this quote somehow ridiculous or inhuman, yet now I’m beginning to believe that overcoming our egos and admitting that we are writing unworthy code but writing it anyway is a vital step in becoming truly great programmers. Perhaps we can let go and accept that, ultimately, the code never really mattered at all.
Postscript: I’m not saying we should set out to write bad code. It’s vitally important to have an idea of how the code should end up, but instead of working and working until we achieve that and then shipping the product we must write the least we can get away with while gradually approaching the ideal. Understanding this distinction is crucial.
Note: Yield Thought has moved to http://yieldthought.com – check there for the latest posts!
Different environments give rise to different programming styles. Over the last few years there’s been a massive trend for software to move from:
Work for a whole year then release a fixed binary, then write for another year.
to the more dynamic, web 2.0 ideal of:
Whip up a web application and upload each feature as soon as it’s done.
This is a much more efficient way to do business, but it’s a very different kind of programming environment and benefits from a very different programming style – not the one we’ve all picked up by copying API conventions and old-fashioned corporate practices.
If you’re able to make your software available to customers instantly, there is one optimal strategy: code for flexibility.
The flexibility of your code is defined by the ease with which you can modify it to fulfill some purpose you hadn’t envisaged at the time you wrote it.
This is important, because when you give your customers a rapidly-iterating product you’re going to find your initial guess at what they wanted was completely wrong. In fact, all your guesses are probably wrong. You’re going to spend years learning from your customers and adapting your software in all sorts of unexpected ways to fit their needs as precisely as possible.
For a long time I confused generality with flexibility; I was always worrying about how I might want to use a class in the future and trying to abstract out all the common use cases right form the start. As it turns out, this is the opposite of flexibility; this builds rigidity into the project. The moment you realise that actually it makes much more sense to have flip your design so that users vote for *each other’s* questions a little part of you dies inside, because you now have five levels of abstraction and four different database tables to rewrite.
In fact, flexible code is specific code. We want to write code that expresses, as succinctly as possible, a solution to a problem. It’s quicker to read and understand code with fewer syntactic elements, such as variables, functions and classes. It’s also quicker to repurpose.
We also build flexibility – or rigidity – into the large-scale structure of our programs. I’ve often fallen in love with the idea of encapsulation for its own sake; I thought that each part of my program should be as well-hidden from the rest as possible, with only one or two well-specified interfaces connecting things. This is a very tempting dream that results in a kind of tree-like structure of classes. This is extremely rigid, because by design it’s difficult to get access to classes in another branch of the tree.
The best way to build for flexibility is drawn from the Lisp world and championed by Paul Graham: when working on a problem, write code that builds up into a language for describing and reasoning about the problem succinctly.
You can magnify the effect of a powerful language by using a style called bottom-up programming, where you write programs in multiple layers, the lower ones acting as programming languages for those above.
This way the solution ends up being a clear, readable algorithm written in this language – the language of the problem domain. In non-lisp languages you end up with a set of functions and classes that make it easy to reason about the problem, a little like the standard string and mail classes make it easy to work with strings and email.
Building up code for reasoning about the problem domain is vital for flexibility, because although our solution to the problem might change drastically as we get extra insights from our customers, the domain of the problem will probably only change incrementally as we expand into new areas. Adopting this flatter, domain-language appropach to program design almost always increases flexibility.
Another aspect of flexibility is robustness. A function or class that makes assumptions about the circumstances it is called in is going to break as soon as we repurpose it in some other part of the code; sooner or later I always forget that the caller needs to check the file exists, or that the user isn’t null and already has a validated address. There are two schools of thought on how to deal with this – contract programming (essentially visible, machine-verified preconditions) and defensive programming (make no assumptions, handle all the exceptions you can locally and try hard not to destroy any persistent state if things go wrong). It doesn’t really matter which you use, but use one of them.
Many of the extreme programming principles help us write flexible code – the c2 team recognized that:
Change is the only constant, so optimize for change.
They looked at this from a project management perspective, but code guidelines like Don’t Repeat Yourself and You Ain’t Gonna Need It are well-recognized as being fundamental to producing flexible code. In fact, these two oft-quoted principles are worth looking at more closely in terms of how they create flexibility.
Don’t Repeat Yourself: When modifying a bit of code it doesn’t help if there are three or four undocumented places it also needs to be changed; I never remember them all. A perfectly good way to deal with this is to add a comment in each block mentioning the link rather than to refactor right away. Over-zealous refactoring tends to merge lots of bits of code together that look kind of similar but ultimately end up going in different directions. At this point very few people take the logical step of splitting a class or subsystem back into two more specialized units; instead it’s over-generalized again and again until it’s completely unmaintainable
Refactoring should always be Just In Time, because the later you leave it the better you understand the problem; refactor too soon and you’ll combine things in a way that restricts you later, building rigidity into the system.
You Ain’t Gonna Need It: This doesn’t just apply to extra functionality. It applies to all elements of your code, from accessors (a waste of time in most cases) to overall class structure. You’re not going to need to handle N kinds of book loan, you’re going to handle at most two, so don’t write some complicated code to handle the general case. You’re not going to need three classes, one interface and two event types for your “please wait” animation because there’s only ever going to be one and it’s always going to be while printing. Just write a function with a callback and release – the simpler it is, the quicker we can write it today and the more easily we can modify it tomorrow.
There is a tension here. On the one hand, we want to write our software by constructing recombinable blocks that represent the problem domain, yet on the other we want to write the simplest thing possible – and writing reusable blocks of code isn’t usually the simplest thing possible.
This tension is resolved when we stop looking at a program as a snapshot of its state at any one time and instead look at it as an evolving codebase with many possible futures. So we write the simplest thing possible, yet while doing so we keep an eye on its potential for being refactored into nice, reusable blocks and avoid doing making poor choices that will make that process difficult. The potential for refactoring is just as valuable as the refactoring itself, but by deferring the refactoring we keep our options open.
The ability to see the code you’re writing today and its possible evolution through time is something you can focus on learning, but it also grows with experience. It’s one of the things that makes a great hacker’s code subtly better than a newbie’s code – both start writing simple code that addresses the problem directly, but the newbie makes all sorts of small structural mistakes that get in the way of the code’s growth, whereas the hacker knows when it’s time to turn this set of functions into a class, or to abstract out this behaviour, and has already prepared the way for doing so. The effect is to make programming look effortless again, naturally flowing from the simple into the complex.
I’m not a great hacker who does this as naturally as breathing. I have tasted it, have seen glimpses of it in my own work and in other people’s, but I still make mistakes; I mistime my refactoring – sometimes too early, sometimes too late. I still bind objects to each other too tightly here, or too loosely there. Despite my failings, just aiming at this goal makes me far more productive than I’ve ever been before. It’s not an all-or-nothing premise. The closer you come, the smoother and lower-friction your development will be.
We can gather all these heuristics together into a concise manifesto:
A Manifesto for Flexibility
- Write new code quickly, as if you’re holding your breath. Cut corners and get it in front of users as quickly as possible.
- Keep your code specific, with a clear purpose. Don’t over-generalize and don’t refactor too early – it should be as simple as possible, but not simpler.
- Stay aware that all code is thrown away or changed dramatically. Hold the image of the refactoring you’d like to do in your mind and avoid doing things that will make it more difficult – this is slightly better than doing the refactoring now.
- Recognize the point at which simple code becomes messy code and refactor just enough to keep the message clear. Just In Time refactoring is not the same as no refactoring.
- Build up a language for reasoning about your domain and express your application in that language. Avoid building up a large, rigid hierarchy of over-encapsulated classes that embody your particular solution to the problem.
Postscript: You should do this in a startup and you should do it in personal project, but you shouldn’t do it everywhere. If, say, you’re writing the Java or .NET API then none this applies to you because:
- Millions of people will abuse your API in every way possible the very second it’s released.
- Every time you change something a howling mob will descend upon your office, making it difficult to get a good parking spot.
So you’ve read Joel’s posts about how 1% of developers are 10x as productive as the rest; the superstar developers; the only ones worth hiring. You know you can program, that you’re reasonably talented, but that’s not enough. You want to be 10x as productive as everyone else in your company? You want to be the guru everyone else comes to when they’ve got a problem, the guy who takes one glance and puts his finger on the problem right away?
You can. Just follow these 3 simple rules and within 12 months all this and more will be yours.
Rule 1: Write lots of code. Have to fix a small bug in an area someone else has written? Don’t waste time trying to understand it or their motivations for writing it that way. Just rewrite the lot as you think it ought to work. Call it refactoring if anyone asks.
Rule 2: Write your code quickly. Touch lots of files, and include every one of them in the ChangeLog. Don’t worry about accidentally introducing hard-to-find bugs; they’ll actually help you later on, as long as they’re actually hard to find. Avoid introducing trivial bugs.
Rule 3: Don’t take time to document your code, or add little comments explaining potential pitfalls in modifying some of the less clear statements you’ve introduced. You don’t need them, you wrote the code.
Dilligently following these three rules will make you the superstar in your team within 12 months. Write them out and stick them up. Not in the office! Somewhere private. Repeat them to yourself every day before you start work. They may seem controversial, but trust me, everything has a deeper underlying reason.
How do I know this works? Because I’ve done it myself and because I’ve seen others do it. Of course, I didn’t realise I was adhering to the Three Rules at the time – I was young and naive and thought I was just trying to do my best for the project. It’s only now, many years later, that I am able to reflect and draw upon the wisdom of the experience to share with you like this.
The 3-Rule Programming Supremacy system is based on one technical principle and one social principle.
Technical Principle: You’re 10x as productive when you’re working on code you wrote as on code you didn’t write.
Everyone understands their own code best. It’s part memory, part convention – that is, everything follows your convention. You expect a function to be called X and it is. You look in class Y to find the functionality and of course it’s there. It’s even at the bottom of the file, where you’d have added it. All these little things add up. You don’t need to worry so much about unintended side-effects when you modify something, because you remember the places that were somewhat risque and know instinctively where you would do careless things and where not.
In short: if you wrote the code, you have an almost perfect model of the code in your head, because that’s what you used to write it.
Social Principle: Your programming ability is judged by how much code you write, how quickly you finsh features and fix critical bugs and how often your insights are necessary to solve problems.
These two principles define the problem space – or, as we shall call it – The Game. You win The Game by improving your reputation to superstar guru levels. To improve your reputation, you need to improve the speed with which you write code, fix bugs and help others. The technical principle tells us how to achieve that – by working in our own code as much as possible. Indeed, by converting as much of the project to our own code as possible.
The 3-Rule System is a foolproof strategy for achieving that.
By following rule 1, you will quickly get to know a large amount of the codebase. Most importantly, you won’t waste time trying to understand other people’s code, which is a difficult and time-consuming process.
By following rules 2 and 3, you maximize your rate of code-conversion by avoiding the most time-consuming parts of writing good code. It’s vitally important that you are touching more code than the rest of the team combined.
There are also beneficial social aspects to working like this:
Benefit 1: Everyone will see how quickly you’re committing huge chunks of code and will start to respect you – especially your boss, who has no other metric to judge you on other than commit frequency and volume.
Benefit 2: Although you’ll introduce lots of new bugs like this, it’ll be several months before they start showing up, by which time your reputation as an expert programmer will already be assured. You can now profit a second time from them by fixing them faster than anyone else. Your colleagues will take 10x longer than you to track each one down. Increasingly, they’ll ask you for advice or help because you wrote that bit of code. Be friendly, humble, happy to help. You’ll spot the bugs very quickly. Your guru reputation will grow day by day.
You can view the process as analagous to a real-time strategy game. The world map is your project’s codebase. The regions you wrote are generating you resources; the regions owned by others are generating them resources. Your resource, here, is programming efficiency, as you’re 10x as productive when dealing with your own code as with someone else’s. Being seen to be 10x as productive gains you reputation, which is the point-scoring mechanism in The Game.
Clearly if you own very little code, you aren’t going to be gaining much reputation because most of the time you’ll be working with other people’s code at the same rate as anyone else would. Time to change that.
Rule 1 is the basis for your economy. You become a superstar by owning more code in the project than anyone else. You invest in your economy by writing new code, or converting existing code to your own, because every area of code you own generates you productivity and reputation.
Rule 2 is aggressive. Expand into other people’s areas and modify their code to make sense to you. Let people see that you’re touching a lot of code – it is good for your reputation. However, you have to be careful when going on the attack. Rewriting code carries a certain reputation penalty, but more about that later.
Rule 3 is defensive. Make it difficult for other people to work in your code or fix bugs in it – for every hour they spend fixing bugs you’ve accomplished another hour of fresh code rewrite somewhere else, thus increasing your control over the board.
Never forget that this is all about gaining reputation. It’s critical not to over-extend yourself and appear as if you’re harming the project. It’s equally important to remain on excellent terms with your opponents^H^H^Hcolleagues, always being polite, helpful and humble. Your reputation determines how much of the codebase you can convert and which parts you can convert without incurring negative reactions.
To this end, you must start small, in the most hated, ugly area of code in the project. Just go in there and rewrite it all to make sense. Don’t worry about all the bugs you’ll introduce; people will be glad that you’re brave and bold enough to take the project on and will agree that it needed doing and that bugfixing the new code will be easier than maintaining the old code.
When you’ve finished, you’ll find your reputation has increased. You’ll be able to rewrite slightly less hideous areas of code without people asking if that really needs to be done. Eventually your reputation will grow so large that you can rewrite core application functionality at will. By then, you’ll know far more about the rest of the system than anyone else and will introduce surprisingly few bugs. This will assure your guru staus as superstar of the team. This is your end-game. You have won!
Postscript for the naive: This post is a mild satire on programming in teams. These three rules, while undoubtedly effective, are evil. They harm overall project progress for your own benefit. They don’t make you a better programmer intrinsically, only compared to the rest of your team. You may, like I and countless others, have done something like this completely innocently in the past, when you didn’t know better. Now you know better.
Postscript for project managers: If your environment meets the grounds for the Two Fundamental Principles, then you will get programmers playing The Game and your project will suffer. Change the rules. Make sure that programmers are recognised for playing nicely with each other’s code, for working in small teams on larger problems. That rewriting for the sake of it is frowned upon, or that the bugs it introduces are traced back to the rewrite that caused them. I don’t know what the right way to get away from this is. If you do then please, for the love of all projects everywhere, leave a comment!
Note: Yield Thought has moved to http://yieldthought.com – check there for the latest posts!