yield thought

it's not as hard as you think

Criminal Overengineering

As programmers we’re continually accused of doing a sloppy job. There are countless programs in the wild, crashing, locking up and accidentally writing “I am a fish” a million times over someone’s mid-term essay. The effect? Something like this:

This damn computer and excel r fuckin my life up! Hatin life right now
MissAlauren (and everyone else at one time or another)

It’s experiences like this that cause people to rant about Microsoft and curse the anonymous programmers who suddenly (yet inevitably) betrayed them. We all know this; it’s burned into our very souls by countless hours of tech support provided to family and friends. Time after time we see that programmers who do quick, sloppy work make other people suffer. And so we try, we try so damn hard not to be like that. We try to be the good programmer who checks every return value and handles every exception.

If we stopped at competent error handling and sufficient testing, all would be well. In truth, we actually go too far and, it has to be said, largely in the wrong direction.

vast proportion of software at work today is horribly over-engineered for its task. And I’m not talking about the interfaces, about having too many controls or options for the users. These are, indeed, terrible sins but they are the visible ones. The worst of the overengineering goes on under the surface, in the code itself.

You’re Doing It Wrong

Have you ever seen someone using the strategy pattern when they should’ve used a 5 line switch statement? There are a million ways to turn something like this:

switch(operation)
{
case OP_ADD: return a + b;
case OP_SUBTRACT: return a - b;
case OP_MULTIPLY: return a * b;
default: throw new UnknownOperationException(operation, a, b);
}

… into a hideous, malformed mutant beast like this one, which I haven’t inlined because it’s far too long.

The most insidious cause of overengineering is over-generalizing. We will over-generalize anything given half a chance. Writing code to work with a list of students? Well, we might want to work with teachers and the general public someday, better add a base People class and subclass Student from that. Or Person and then EducationPerson and then Student. Yes, that’s better, right?

Only, now we have three classes to maintain each with their own virtual methods and interfaces and probably split across three different files plus the one we were working in when a one-line dictionary would have been fine.

Perhaps we do it because it’s relaxing to rattle off three classes worth of code without needing to pause and think. It feels productive. It looks solid, bulletproof, professional. We look back on it with a comforting little glow of self-satisfaction – we’re a good programmer, no messy hacks in our code.

Except, this doesn’t make us good programmers. Overengineering like this isn’t making anyone’s lives better; it’s just making our code longer, more difficult to read and work with and more likely to contain or develop bugs. We just made the world a slightly worse place. It lies somewhere between tossing an empty drinks bottle on the street and grand theft auto.

The extra effort caused by our overengineering carries a hefty opportunity cost:

  1. Less time spent refining the user experience
  2. Less time spent thinking about the meaningful implications of the feature we’re working on
  3. Less time available to look for bugs and – with harder-to-read code – more time spent debugging them

Yes, by overengineering the Student class you indirectly ruined MissAlauren’s day.

We have to stop championing each ridiculous feat of overengineering and call it what it is. It’s not ‘future-proof’, because we can’t see the future. It’s not robust, it’s hard to read. Applying a generic solution to a single case isn’t good programming, it’s criminal overengineering because like it or not somebody, somewhere will pay for it.

Don’t Worry, Be Happy

I suspect all the best programmers have already realized this, but they’re not shouting about it loudly enough for everyone else to hear. Paul Graham is completely right when he suggests that succinctness is valuable:

Use the length of the program as an approximation for how much work it is to write. Not the length in characters, of course, but the length in distinct syntactic elements– basically, the size of the parse tree. It may not be quite true that the shortest program is the least work to write, but it’s close enough… look at a program and ask, is there any way to write this that’s shorter?
– Paul Graham, The Hundred Year Language

He’s actually talking about language design here; indeed, in Succinctness is Power he’s careful to note that it’s clearly possible to write a program that’s too succinct. This is because, these days, Paul Graham is more a language designer than a working programmer. Otherwise he might have said:

If you’re about to take a hundred lines to write what you could in ten, stop and ask yourself this: what the fuck?
– Mark, Criminal Overengineering

When I feel tempted to over-generalize or over-engineer a bit of code, it’s often because of fear. Fear that someone will find a really good reason I shouldn’t have done it the easy way. Fear that I’ll have to rewrite the code again. Fear of finding myself on the wrong side of an argument about the merits of the visitor pattern. But fear does not naturally lead us to the most elegant solutions.

Next time you feel the compulsion to write a nice, general solution to a simple case, stop and ask yourself what’s stopping you just writing it the simple, specific, succinct way:

  1. Am I worried I’ll have to rewrite it?
  2. Am I worried someone will criticize it or that I’ll look bad?
  3. Am I worried that it’s not professional enough?

Are any of these true? Then relax. Don’t worry. You worry, you call me, I make you happy.

Just write the code the simple, specific way and then add a short comment, something like: Replace with the Strategy pattern if this gets any bigger.

This is the perfect solution. It’s a great reminder to you next time you come here about what you wanted to do. It shows other programmers on your team that you considered the ‘correct’ way to do it and have a good reason not to do it just yet. It’s very hard to argue with a comment like that, because you’re not arguing about the strategy pattern vs the switch statement, you’re arguing about whether to use the strategy pattern after 3 cases or after 4 cases – not a discussion that can reflect badly on you, in any case.

A few months later you can go back and look at how many of your comments eventually turn into more complicated, engineering code. I’ll bet you it’s not very many. That’s how much time and effort you’ve saved, right there. That’s setting yourself free to pursue the solution and that’s making the world a slightly better place.

Note: Yield Thought has moved to http://yieldthought.com – check there for the latest posts!

Written by coderoom

June 23, 2010 at 8:15 am

Posted in Programming

Tagged with , ,

103 Responses

Subscribe to comments with RSS.

  1. Great post.

    Programs should be as simple as possible, as short as possible, and as understandable as possible. That’s what makes code truly modifiable.

    There is indeed usually no benefit in “future-proofing” code by overgeneralizing. Most of the time the actual future scenarios end up being nothing like you predicted anyway, and you have to refactor and rewrite code.

    So don’t bother to predict and generalize based on that. Instead, make the code modifiable, easy to refactor, and easy to rewrite.

    Ville Laurikari

    June 23, 2010 at 11:10 am

  2. Amen! You said it much better than I would.

    From experience and discussing with fellow programmers that do this, I can’t help but think that this is result of people taking abstraction and object-orientation too seriously, as in, they bought the idea too much, to the point it is almost a religion. While fear plays a big role, like you said, they seem to have this irrational belief that anything that deals with a single case, the one at hand, is bad. “If you make it abstract, you’ll be solving an entire class of problems, not just this one!”, they usually say.

    This is also rather common with Java programmers that picked it up using formal training (usually at college), so I do believe it’s partially a flaw in the teaching of high-level programming and object orientation. Developers I know that started down with C, Perl, BASIC or ASM never seem to have this problem…

    Anyway, thanks for the post. I’ll be sure to pass it over next time I need to explain this to someone.

    Lucas

    June 23, 2010 at 11:13 am

    • “This is also rather common with Java programmers” . I believe that you can find over-engineering in any programming language; however, I do absolutely agree that in the Java world we have a huge number of so called framework (I called them shamework) that is a mess on over-enginerring.
      Great post.

      harringf

      June 25, 2010 at 2:14 am

  3. Great read.

    I do want to point something out. Even though your example is right on (switching a “switch” statement with the strategy pattern is *really* ridiculos), that doesn’t mean there aren’t cases where the “Strategy” design pattern doesn’t come in handy. Obviously, the Wikipedia page shows a very short and therefore unhelpful example, but that’s mostly because a longer one would be too hard to understand easily.

    I still agree with all your points, but want to make sure no one gets the impression that the Wikipedia article is actually suggesting using this pattern in this case.

    Edan Maor

    June 23, 2010 at 11:47 am

    • Thanks Edan, yes, this is worth being clear about – there’s nothing wrong with the strategy pattern per se; it’s knowing when to apply it and when not to that’s important.

      coderoom

      June 23, 2010 at 11:56 am

  4. I kind of agree, but it feels like you’re playing the devil’s advocate in your sample switch statement to explain why you shouldn’t overengineer. It seems like your comparing half of an apple with an apple tree. The provided “perfect” code doesn’t do half the job the beast code does. Give us some better examples when you want to argue against overengineering.

    Benny Thomas

    June 23, 2010 at 11:53 am

    • Yes and no; the sample switch statement is a rather obvious strawman, that’s true, but it’s only there to illustrate a point. I didn’t want to turn the entire post into an in-depth real-world case study, for example. I trust people can recognize overengineering in their own code.

      Regarding not doing half the job the ‘beast’ does – this is a dangerous line of argument. It does exactly the same thing in terms of features provided. If all you needed was a way to add, subtract or multiply then all the extra flexibility and complexity of the strategy pattern are completely wasted, which is entirely the point of the article.

      If you just need a few bites of apple then growing an entire apple tree is overengineering.

      coderoom

      June 23, 2010 at 12:03 pm

      • Sorry, it was harsh of me. But when I say half the job, I’m referring to the fact that you don’t have the main, how the context is set up, and all the things that makes the choice of the strategy pattern looking like a beast against your nice little switch statement when you’re compeering them.

        Benny Thomas

        June 23, 2010 at 12:21 pm

      • That’s true enough, although the main will be half the length of the java example because you can just call a function instead of instantiating a context object each time :-)

        coderoom

        June 23, 2010 at 1:08 pm

  5. Oh man, overengineering sucks. I am a product manager for an embedded piece of enterprise networking software, and for us it’s far worse than just criminal. Here it’s lethal as we are constantly working to optimize down to bytes (I am not even kidding) of program footprint and RAM. Tolerance for overengineering is a lot lower in embedded C code, but Java, C++, and C# give you so much more pattern cruft that it’s almost a given that you will end up with a deeply nested set of interfaces and classes that does nobody any good.

    codercofounder

    June 23, 2010 at 12:11 pm

  6. Or even shorter: YAGNI.

    Tierlieb

    June 23, 2010 at 12:12 pm

  7. heh, the doom of patterns. you must not give in the temptation to make your code fit patterns at all costs.

    jrk

    June 23, 2010 at 12:15 pm

  8. Test Driven Development can really help with this. It’s one of many incidental unexpected benefits of TDD that end up being more significant than the expected benefits.

    There is emphasis placed in the process on writing the simplest possible code that can make your tests pass. In my experience, this isn’t just rhetoric: TDD makes a substantial impact on programmer’s behavior, with a consequent improvement in readable code, simple designs, low coupling, etc.

    Also, there’s an acronym, used as a justification for not using the over-engineered solution: YAGNI: You ain’t gonna need it.

    Jonathan Hartley

    June 23, 2010 at 12:28 pm

    • Yes this is definitely a great TDD benefit and I think is a fundamental part of the agile movement. Have you read TDD without the T? Kent Beck (the man himself!) seemed to enjoy it!

      coderoom

      June 23, 2010 at 1:07 pm

      • I hadn’t seen that – many thanks.

        It seems to make sense – I consider tests the most significant invention in software since the invention of the compiler, but even I don’t write tests for some parts of small, personal projects. I kick into TDD for the fiddly bits, and do 100% TDD for any project that requires high quality or is going to be long-lived.

        Jonathan Hartley

        June 23, 2010 at 1:44 pm

    • The reality of testing is that, at least with a strong, staticly typed language like java, it requires one to use interfaces more so that one can decouple dependencies and swap in test implementations of various pieces.

      Jesse

      June 24, 2010 at 2:22 am

  9. So true. An common artifact of overengineering in my experience is the generalization fallacy: For example being stuck with a large codebase built around the Person class which Teacher and Student are built on.

    Suddenly you realize that your program actually has to treat these cases differently in ways you hadn’t imagined. Perhaps there are data protection laws that require you not to store students ID numbers but for teachers it is mandatory because the payment system requires it (or whatever).

    The logic band-aids that one ends up with separating these instances post facto can be remarkably complicated.

    Anyway, it is not that simple that “overengineering” is always wrong and “underengineering” is always right. The spaghetti result of systems that grew for a long time where nobody had a vision can be used to frighten children. It has often been remarked that one sign of intelligence is the ability to keep conflicting ideas in one’s mind and reason with them. Whatever that may be, such is the work of the programmer.

    Jonas B.

    June 23, 2010 at 1:07 pm

  10. We build the software equivalent of jumbo jets in order to do simple things like crossing the street even though a unicycle might have been the ideal solution for the task. Most conscientious software developers will go to great lengths to make sure their jet is over engineered so that it does not crash and burn midflight.

    The fact that using a jumbo jet in such a scenario is so insanely ludicrous may be the fault of:

    - tools that give you a vast framework to solve a simple problem when all you needed was a for loop

    - a marketing department or product head that thinks the only way to justify yet another product release to customers is to make it super duper capabable just in case one day the customer decided to make a continental flight even though the customer only ever uses your code to cross the street

    - the sense that complexity leads to greater job security, which is ever so true these days

    - and the general notion that bigger is always better

    Mario

    June 23, 2010 at 1:33 pm

    • All true – I’d also add insufficiently expressive languages to the list of causes. A lot of the boilerplate in C++, Java and (although decreasingly) C# comes from trying to implement things like first class functions or multiple dispatch in languages that don’t directly support them.

      Not all languages are created equal!

      coderoom

      June 24, 2010 at 6:50 am

  11. Sometimes, using primitives (like the dictionary of strings example) can SEEM easier than the corresponding abstractions, but the initial bloat of introducing the abstraction later allows unanticipated collapses via generalization.

    It is valuable to have a step in your process (immediately after getting to green in TDD, for example) to experiment with various refactorings to attempt to improve the code. If they don’t pay off pretty soon, roll them back. However, constraining yourself to refactorings that IMMEDIATELY shrink the code will lead to premature convergence to local optima, and worse results overall.

    Johnicholas

    June 23, 2010 at 1:42 pm

  12. If you are over-engineering and/or over-generalizing, it is a sign that you don’t know enough about what your code is expected to do. This means you didn’t write any tests first.

    If your code is supposed to deal with a list of students, you just write the tests for that. When they pass, you are done. And if you have to come back later and make it work for teachers, then you add the appropriate tests, rinse and repeat.

    Matt V

    June 23, 2010 at 1:42 pm

  13. I agree that brevity is the soul of code wit. I don’t agree that patterns are the root of all evil, as some comments have expressed (and is somewhat implied by your example).

    A pattern in the hand of an under-experienced engineer is just as dangerous as a pencil in the hand of a naive building architect, but only a fool blames his tools.

    If a pattern is applicable, it should be used. If it’s overkill, it should be avoided. But only an experienced engineer knows the difference, and clearly no experienced software guy would implement the operator example as you show above using Strategy. Being succinct is the trait of experience.

    A friend of mine once said, you can add one layer of inheritance for every 4 years of OO experience you have because you finally have enough knowledge to understand the implications of that layer. The same is true here…Naivete is dangerous, no matter the language, the pattern, or the person.

    Dave Rodenbaugh

    June 23, 2010 at 2:13 pm

    • I’d have said you add one layer of inheritance for every year of experience until you reach 8 or so, then you start taking them away again :-)

      coderoom

      June 24, 2010 at 6:51 am

  14. This article just rocked my world. I have no formal CS training and I write scientific and numeric software (mostly in modern Fortran). I constantly over engineer my codes and it makes my life so miserable. Next time I write a piece of code I will definitely be thinking about this article.

    Also, I think I have seen a similar article in the past named something like: ‘How I learned to stop worrying and love the singleton.’ I should try to find this and have another look.

    SciGuy

    June 23, 2010 at 2:23 pm

  15. Adding unnecessary complexity to your code is very different from over-engineering or generalizing. Doing the exact same thing with more code doesn’t really help you, but writing something general enough that you can use it in multiple places saves time, avoids repetition and helps to enforce consistency. The biggest trouble with coding these days is that programmers are writing far too much of it. If you stop and properly abstract the problem, you can save yourself a massive amount of effort. If you just pound out the most specific, most brittle code you can, as fast as you can, you are only burying yourself in technical debt.

    Paul.

    Paul W. Homer

    June 23, 2010 at 3:02 pm

  16. Of course, generalization versus workability is an age-old challenge to any design work, ever. Both sides of the question can be correctly argued for. But actually neither is right. The only “right” solution is actually to find a way to become both more general and more workable. Which, again, is an age-old torn. But it is possible. Probably not adequate at the moment (at almost all moments), but possible nevertheless. Worse is better. It has been since the Epoch. But even then complexity rises, often without added complicatedness. Maybe you are very close to overengineering yourself with a general rule instead of a simple observation ;-). But, that said, the extreme overengineering of current software is indeed alarming…

    Marcio Rocha Pereira

    June 23, 2010 at 3:29 pm

  17. My guess is that the author couldn’t understand what abstraction is, got frustrated, then wrote this.

    prindle

    June 23, 2010 at 3:37 pm

  18. Abstraction is a good thing – whereever it makes sense. But clearly, not every beginner is able to make good ones, I think that is what you’re talking about.
    Bad abstractions are purely structural and will bring more work in the future without any savings.
    Good abstractions offer new standalone pieces, easy implementation of new related things you didn’t think of when building the base and some kind of schema.

    I was currently given the task to create a new generic solution for a business management solution. The first implementation was fast, but it now incapable of improving. Abstracting some patterns will make that better for future enhancements.

    frevd

    June 23, 2010 at 3:46 pm

    • I think of code as being much more fluid. Just because I’ve written a function doesn’t mean it’s finished. There’s no harm in going back and repurposing code to apply to a general case when you find you need one.

      Of course, as you hint at, writing code in such a way as to make this easy or hard is the difference between great and new programmers…

      coderoom

      June 24, 2010 at 6:53 am

  19. Great post.

    It wasn’t until I stop over engineering I could finish projects.

    Generalization comes from observing patterns in specific examples. The other way leads to unnecessary constraints.

    Now days I wait till I have a handful of examples before I refactor into a generic method.

    Jonathan Fischoff

    June 23, 2010 at 3:51 pm

  20. Thank you! I am constantly trying to profess YAGNI to my team but it’s a hard sell. Everything has to look generalized and reusable and “professional” – I say “look” because it never actually *is* those things, we just burn cycles trying to pretend it is, when in reality we don’t even have a reuse case planned anyway.

    I sometimes overgeneralize and overengineer too – I think that it comes from a desire that many developers have to build something really great that really can be reused and adored the world over. It’s nice to be able to revel in the cleanliness, layered design and loose coupling of one’s code. In reality, there’s often very little payoff of any kind for that kind of work, simply because it’s not what’s required. If you need to write some files to disk and show some data, *just do it.* Worry about tomorrow’s requirements tomorrow.

    NW

    June 23, 2010 at 4:44 pm

    • You’re welcome – YAGNI for the win!

      coderoom

      June 24, 2010 at 6:54 am

  21. [...] the rest here: Criminal Overengineering « yield thought 23 June 2010 | Uncategorized | Trackback | del.icio.us | Stumble it! | View Count : 0 Next Post [...]

  22. You forgot something: *Every design pattern comes with a downside*.

    The problem isn’t that the strategy pattern is used for a small case, it’s that it’s used without looking at the tradeoff; a switch is fine if it’s used only in one place and is a fixed list of options — even if it’s long.

    Which brings us to the comment: Comments should be about assumptions, intent, or justification, preferably. (Good job on not falling into /* use a switch here */ -type comments — they’re useless!) A better comment would have been something more like: /* The set of operations is only used here and is fixed. */

    The general thrust of the essay is a good one: overengineering and trying to predict the future is wasteful. I just think the perspective is a little off. Shorter is generally better, but better still is to sit back and *think* about the problem for a minute. What’s the tradeoff of my intended course of action? What are my assumptions?

    The hard part of programming (after choosing good names) is selecting the right abstraction. The BEST rule of thumb should be “What would I want to see if I had never been exposed to this code before and I was told to maintain it?”

    Another useful rule of thumb about generalization is that you shouldn’t abstract code for reuse[1] until you’ve used it three times. The third time you write the *same* code is about the right place to sit back and say “I’m tired of doing this, can’t I abstract this out somehow?”

    That doesn’t mean you shouldn’t write your code *as* *if* you want someone to reuse it. I’ve had to rewrite too much code from scratch because the programmer who wrote the original code never gave a thought to reuse; after a while, overgeneralization and over-abstraction don’t look nearly so terrible.

    Remember, it’s “high” cohesion and “low” coupling. :)

    [1] As opposed to abstracting code to clarify the problem and solution. You can use a hashtable and well-known keys, or you can abstract out a Student class. You might have less code with the hashtable, but that doesn’t necessarily mean it’s a good idea when it comes to clarity.

    SJS

    June 23, 2010 at 5:53 pm

  23. Here’s a better version of the Strategy pattern in Java, from the book Effective Java. The goal of the program is to calculate how much a person should be payed, based on the day of the week in a flexible manner (ie. you get payed more on weekends). I think it’s pretty slick.


    public enum PayrollDay {
    MONDAY(PayType.WEEKDAY),
    TUESDAY(PayType.WEEKDAY),
    WEDNESDAY(PayType.WEEKDAY),
    THURSDAY(PayType.WEEKDAY),
    FRIDAY(PayType.WEEKDAY),
    SATURDAY(PayType.WEEKEND),
    SUNDAY(PayType.WEEKEND);

    private final PayType strategy;

    PayrollDay(PayType strategy) {
    this.strategy = strategy;
    }

    double pay(double hours, double rate)
    {
    return strategy.pay(hours,rate);
    }

    //A strategy enum pattern
    private enum PayType {

    WEEKDAY {
    double overtimePay(double hours, double rate) {
    if(hours <= HOURS_PER_SHIFT)
    return 0;
    return (hours-HOURS_PER_SHIFT)*rate/2.0;
    }
    },
    WEEKEND {
    double overtimePay(double hours, double rate) {
    return hours * rate / 2.0;
    }
    };

    private static final int HOURS_PER_SHIFT = 8;
    abstract double overtimePay(double hours, double rate);

    double pay(double hours, double rate) {
    double basePay = hours * rate;
    return basePay + overtimePay(hours,rate);
    }

    }
    }

    z4srh

    June 23, 2010 at 5:59 pm

    • Thanks for the example, it is a better one. I still think it’s horribly overcomplicated; perhaps I’ll write a follow up post sometime comparing it to a simple version!

      coderoom

      June 24, 2010 at 7:02 am

    • Well, here’s a simpler way to express the same algorithm (hopefully still clear even though the narrow comment field breaks lines in all the wrong places):

      # strings instead of enums for clarity in the blog post
      weekday = ['monday','tuesday','wednesay','thursday','friday']
      HOURS_PER_SHIFT = 8
      weekday_overtime = lambda hours,rate: 0 if hours < HOURS_PER_SHIFT else (hours-HOURS_PER_SHIFT)*rate/2.0
      weekend_overtime = lambda hours,rate: hours * rate/2.0
      def pay(day,hours,rate):
        overtime = weekday_overtime if day in weekday else weekend_overtime
        return hours*rate + overtime(hours,rate)

      coderoom

      June 24, 2010 at 7:10 am

      • There are a couple of problems that I see with yours:

        If someone says pay(‘january’, 234, 234) they’ll get paid the weekend amount. You could fix this by adding a weekend array or a valid days array but even then you get a crash on runtime. I guess this is what you meant with using enums, but still worth mentioning.

        If we go with that strategy, suppose that we have a dozen categories, so now you have a dozen arrays of enums. If you forget to add an enum to one of the arrays, it’ll fail silently at runtime, giving either the default switch money or no money at all. Both of ours have the problem that we could give the wrong strategy (ie. TUESDAY(PayType.WEEKEND)), but mine (well, Effective Java’s) doesn’t have the other problems.

        Plus it’s still quite concise for the user – they would just call PayrollDay.MONDAY.pay(5, 3), so the extra overhead in the internals doesn’t really show. I could make it work with date objects to be a bit safer.

        Also, sorry about the code formatting in the above post – not sure how to do it.

        z4srh

        June 24, 2010 at 7:14 pm

      • Yes, that is what I meant about using an enum or whatever the rest of the program uses to represent days.

        Suppose we have a dozen categories? For a week of seven days? Which has been divided into week / weekend for longer than computers have existed? This is exactly my point about overengineering – the chance of needing that flexibility is vanishingly small. In the very unlikely circumstance that you do, you can still refactor at the time of need because the implementation is all hidden behind the pay function.

        coderoom

        June 25, 2010 at 6:38 am

    • This payment example, is only relevant to closed-source programmers. :D

      dick_dastardly

      June 24, 2010 at 10:24 am

  24. Hey, just found your blog today via reddit, you have some really good points here about programming and I agree on much of it. Thanks!

    bugshake

    June 23, 2010 at 6:38 pm

    • You’re welcome, I hope you enjoy the blog!

      coderoom

      June 24, 2010 at 6:55 am

  25. YAGNI!

    Ed Summers

    June 23, 2010 at 6:41 pm

  26. I can tell you who kills the simple solutions: managers.

    vanyali

    June 23, 2010 at 7:07 pm

  27. Some of this comes down to simply not having enough time to reflect and refactor in most commercial situations. The human brain has not changed significantly in 10,000 years but we try to accelerate the pace of everything blithely ignorant of this.

    JG

    June 23, 2010 at 7:11 pm

  28. [...] As programmers we're continually accused of doing a sloppy job. There are countless programs in the wild, crashing, locking up and accidentally writing "I am a fish" a million times over someone's mid-term essay. The effect? Something like this: This damn computer and excel r fuckin my life up! Hatin life right now — MissAlauren (and everyone else at one time or another) It's experiences like this that cause people to rant about Microsoft and cu … Read More [...]

  29. Wouldn’t not “overengineering” just be “engineering”? :) My approach is pretty simple: A. can I benefit from a more general solution now. B. do my plans for the immediate future benefit from the same solution. C. will using the general solution reduce the overall complexity of the code – in terms of dependency, points of failure, maintainability, etc.

    If !(A & B & C) return false.

    Justen Robertson

    June 23, 2010 at 8:24 pm

    • Nicely put, although you could simplify that to:
      return A and B and C :-)

      coderoom

      June 24, 2010 at 6:56 am

  30. Of course you are right that over-engineering is a sin committed by many. Inheriting to specialize behavior, for instance, is far more heinous than encapsulating variation that maybe didn’t need to be encapsulated.

    I think the two real crimes here are:
    1. Propagating a misunderstanding of what complexity and simplicity actually are
    2. Suggesting that people add insult to injury by slapping a “fix this later” comment on top of their bad design

    Max Guernsey, III

    June 23, 2010 at 11:00 pm

  31. This is an excellent post, and the method of adding a comment is pure genious (and some good derriere covering for the ‘unprofessional’ nay-sayers as well)

    I have to admit, that the more Object Oriented code I write – the more I want to go back to Assembly, or at least PL/I…

    Nimo

    June 23, 2010 at 11:09 pm

    • Have you tried python, ruby or Lisp? I think they might be just what you’re looking for…

      coderoom

      June 24, 2010 at 6:57 am

      • Played a bit with Lisp a few (20…) years back. Will give it another go – cheers :)

        Nimo

        June 24, 2010 at 12:18 pm

  32. The Zen of Python, by Tim Peters

    Beautiful is better than ugly.
    Explicit is better than implicit.
    Simple is better than complex.
    Complex is better than complicated.
    Flat is better than nested.
    Sparse is better than dense.
    Readability counts.
    Special cases aren’t special enough to break the rules.
    Although practicality beats purity.
    Errors should never pass silently.
    Unless explicitly silenced.
    In the face of ambiguity, refuse the temptation to guess.
    There should be one– and preferably only one –obvious way to do it.
    Although that way may not be obvious at first unless you’re Dutch.
    Now is better than never.
    Although never is often better than *right* now.
    If the implementation is hard to explain, it’s a bad idea.
    If the implementation is easy to explain, it may be a good idea.
    Namespaces are one honking great idea — let’s do more of those!

    Estebab

    June 24, 2010 at 12:16 am

    • The Zen of Python is awesome, I just wish is had a punchier last line somehow :-)

      coderoom

      June 24, 2010 at 6:57 am

  33. [...] Criminal Overengineering « yield thought (tags: programming development design.patterns)   Leave a Comment [...]

  34. Hi,

    well as a guy who is sometimes quilty of the very crime described here I feel like I have to defend myself a bit ;)
    First you are perfectly true – BUT as a FP guy I just love generalization – I honestly sometimes give a $hit about wrong/right, I just need some non-trivial task from time to time or I just get bored to the max ;)

    … to make some kind of point: guess what. From time to time these “generalized” pieces of code are not only shorter and more to the point – no they truely might help you implement those SOLID principles.
    Example? Well let’s keep it simple and say you need to compute the average of some list of numbers. Of course you can inline a loop and do it right there but IMHO some LINQ-like FP-magic is so much better suited and better readable – but of course you just have to be guilty of the crime mentioned here ….

    We all seem to forget, that we are using heavy frameworks to do our everyday job and those frameworks are nothing but the very crime we are ranting about here – so yeah you are all right but somebody has to write those general frameworks anyhow … maybe not the everyday LOB developer but thank god – that’s not the only kind of programmer left.

    Carsten

    June 24, 2010 at 4:28 am

    • I think we’re talking about different kinds of generalization (indeed, I should have been more specific ;-)). Introducing unnecessary abstractions to handle possible future specializations is over-generalizing in my book.

      Writing, say, a median function that will find the median element of any sortable list rather than just an array of doubles is – as long as it doesn’t complicate the syntax or implementation – a wonderful thing. This is why I love dynamic typing; you get a lot of this for free, without the massive and unnecessary boilerplate you’d need to add if you decided up-front you wanted to do this in, say, Java.

      By simple I mean ‘fewer syntactic elements’ or ‘easier to see the purpose’ and not “avoiding useful language features that new programmers haven’t learned about”. By this definition, inlining a loop to work out the average of a list of numbers is a horrible crime. There’s almost certainly a library function called “average” to do that.

      coderoom

      June 24, 2010 at 7:20 am

  35. Quite right, at the first sight.

    Yet, if this “anti-overengineering” philosophy it taken to the extremes, we’re back to the ol’ good days of Pascal submitting to the full power of structured programming in which you write a system once and you rewrite it 90% over again every time you have to add a half dozen new data inputs to the application.

    Marius Filip

    June 24, 2010 at 6:13 am

    • I mean to write a post about code reuse sometime. It’s a commonly-held assumption that more general code or more abstracted code is more reusable code, but a mistaken one.

      coderoom

      June 24, 2010 at 6:59 am

  36. Great post! Some programmers love to complicate their own lives. I like mine simple :)

    Quang Ngo

    June 24, 2010 at 6:16 am

  37. “If you’re about to take a hundred lines to write what you could in ten, stop and ask yourself this: what the fuck?”

    Well said! And great bit of linking to the original source!

    rachel

    June 24, 2010 at 6:39 am

  38. Quite right.
    But sometimes you also need to resist the requests by your end-users.
    It is my experience that end-users requests are also systematically over-engineering requests. I often ask them about the functionalities they have used: the statistics is always below 50% !

    Lalbatros

    June 24, 2010 at 6:53 am

    • This is a pretty interesting take, flipping it to the functionality side of things. Others have drawn parallels to 37signals’ work in this respect too.

      coderoom

      June 24, 2010 at 7:00 am

  39. While I agree with the general meaning of your article, I have a few remarks. I work for a company which often makes software for other people on a project basis. Some of these customers are not the brightest when it comes to software engineering (they are great in their own business, mind you). They think that programming is “taking a bunch of components from the internet and throwing them together”. They also think “hey could you add a button here that does this task?” while not realising that adding the task takes a few days sometimes. Furthermore, while they love easy GUIs, they also look at your software and say “is that all I get for my money?”. Sometimes they don’t say it out loud, but that’s always the feeling I get from them. So as a job protection kind of thing, programmers often add a more complicated GUI, just to show the client that the software is capable of quite a lot.

    What I want to say, is that internally software should be simple and just work, but externally, you have customers who always demand more functions (as Lalbatros mentioned). You have to be able to change your code easily. Sometimes over-engineering already does this (because you thought of the feature before the customer did).

    And as a sidenote: I don’t think extra classes and generalisation are bad for understandability per se. All they need is documentation ;-)

    Tom

    June 24, 2010 at 10:07 am

    • This sounds like a tough gig to work. It’s hard when the paying customer asks “Is That All?”. Perhaps adding an ‘advanced’ mode that hides away the features would placate them somewhat, since changing people’s minds is one of the hardest (albeit most valuable) things to do.

      I completely agree that you have to be able to change your code easily.

      Extra classes => extra documentation => more to read before I understand the code I need to modify. Reading code is hard enough without pushing half a dozen classes onto your mental stack while you work out what each one’s callees do!

      coderoom

      June 24, 2010 at 10:18 am

  40. Ok, Advocatus Diaboli time here. I’ve been a developer for over 30 years–and I can’t count the number of times I’ve had to maintain, repair, or rewrite an insufficiently-thought-out piece of work because “it was just a simple hack”, so they **didn’t** engineer it.

    There’s an important middle ground here. If you *know*, without a doubt, that your code won’t live on for decades–it’s a quick knockoff to do a one-off analysis of some data, for instance–sure, just whack something together. Hell, that’s what was so wonderful about Unix; that deep, powerful toolchest of little commands that made no sense when you saw them stand-alone, but when used in a shellscript, could create a powerful tool you’d have taken much longer to have to write in C or C++.

    But if you *know* your code is going to be reused, *know* it’s going to live for years if not decades, it’s criminal to *under*-enginer it.

    Intelligent application of the tools at hand to the problem being solved is the key. Be a *problem solver*, not a “coder”.

    Dave Ihnat

    June 24, 2010 at 1:17 pm

  41. I found it interesting to read this article right before this other one:
    http://programmingzen.com/2010/06/23/how-microsoft-is-changing-the-programming-world/

    Both were included in today’s CodeProject mailout:
    http://www.codeproject.com/script/Mailouts/View.aspx?mlid=8014

    Dan Korn

    June 24, 2010 at 1:36 pm

  42. I didn’t even finish reading the post before I completely agreed with the author.

    As a senior software engineer with 35+ years in the field I can categorically state that business people do not have a clue about how to promote a credible system into their organizations. This is combined with technical managers that believe that a system isn’t user-friendly unless it has everything thrown in for the “user experience” with the exception of the kitchen sink…

    I have always believed that you make your applications as simple as possible. And this is why I attempt to limit the features available to a user to single set, which can be learned quite quickly. For example, you only need one way to select an item in a list-box, not a duplicate set of list-boxes with extra buttons that provide add & remove (all or a single item). It is thinking like this that has turned our work into just so much garbage that users often do not require or even use.

    Internally or under the surface, our code is designed to primarily support the interface, so if we limit the interface to you substantially reduce what code you need in the middle-tiers.

    A recent example is an application I completed where I provided up two column-based filters for GridView that had about 8 columns. I thought that would be enough for what most users would need. Than one of my client’s “underlings” walks in and simply says make it 5 filters. Did the user need this? No, she was never asked. Was there a specific reason for this extra filtering? No, no one had come up with one except that maybe the user may need it…

    And that brings me to my last point. Young technicians today are ensconced with the belief that you have to think about every possibility an application may face. And then they proceed to dump everything into it not realizing that most of those possibilities will never happen.

    Write for today’s requirements and forget about the future unless you are working with a team of developers who understand intrinsically where their project is expected to go. Trying to encompass future possibilities in an application is a good recipe for failure, which I have seen quite a bit of in my career…

    Steve Naidamast

    June 24, 2010 at 1:49 pm

  43. Great post, but one thing that I didn’t see mentioned is that “overengineered” is an entirely subjective term. So whether or not you may have over or under engineered a solution depends greatly on the audience. If the audience have not been exposed to closures (by choice or circumstance), then if your code uses them, then you are guilty of overengineering.

    Again I agree with the post, but writing YAGNI or “simlest thing that could possibly work” is much different than writing least commn denominator code.

    -EM

    Edwin McConnel

    June 24, 2010 at 1:55 pm

    • I wouldn’t consider using closures overengineering just because others on your team haven’t used them before. Learning is a part of our lives as programmers. To stretch a point, perhaps using closures everywhere because OMG I JUST LEARNED FP AND ITS TEH WIN!!!! could easily result in an overengineered mess of functions that really just wanted a singleton somewhere.

      coderoom

      June 24, 2010 at 6:26 pm

  44. BRAVO!!!!! as simple as that.

    Andres Urena

    June 24, 2010 at 1:56 pm

  45. I had the same problem when I started coding. My programs would become huge and unreadable.
    Now, whenever I feel like I’m overdoing it, I take a 10 min break and then look at the problem (not the code) again – it works wonders!

    Robin

    June 24, 2010 at 2:41 pm

  46. All I have to say is that there is merit in the above article. However I find programming like Chess… There are times to make a quick kill and times to think many moves ahead. I think learning the strategy of when to apply various kinds of strategies, techniques, when to optimize, when to just crank out something that works etc… It’s all important. The good books says there is a “Time for eating and a Time for drinking” … and I would say there is a time for intense engineering and a time to use a goto statement!

    One thing I found that helps me along is the fact I got started with writing video Games… Unlike Commercial Software – you constantly have to worry about the balance of readability and PURE TABOO coding to optimize the heck out of parts of the system… failure to do so is a game whose performance stinks! Coupling the Game Maker Approaches with the more pragmatic approach of business software developers is one way that might help one know the when and where to do the appropriate coarse of action.

    Jason P Sage

    June 24, 2010 at 2:54 pm

    • You mean the excellent poem at the start of Ecclesiastes 3? “There is a time to tear down and a time to build… a time to keep and a time to throw away”

      I’d quite enjoy seeing a programmer’s mantra in this format – I think you may be on to something :-)

      coderoom

      June 24, 2010 at 6:31 pm

      • Yeah and Shame on me for not knowing it verbatim!

        Unfortunately – As much as I think this would be a great “Read” I can’t begin to take the time out for it.. If someone is going to do it – it should done well!

        I did an article in “100 Things all Programmers should know” by O’Reilly” – and the funny thing is my article is about “Rewriting the Wheel Often”… Of course there was another fellow with “Never Rewrite the Wheel” story… his wasn’t published. B^)

        Jason P Sage

        June 24, 2010 at 6:54 pm

  47. Great Post, definitely applies to real software development.

    Only thing I would add is that oftentimes the “overengineering” habit occurs in prototype code (code used to explore possible solutions, or code used to get user feedback to refine requirements), and that this is the most dangerous thing.

    In a prototyping phase, the programmer should be focused on getting a runable result that is capable of getting the feedback needed to answer the currently known design questions.

    I use the term prototyping, but this is functionally the same as test-driven development (the tests form a set of questions, successful completion answers the question, and the exploration is determining how many and what type of tests do you need to solve the problem).

    Once the requirements and problem are fully understood, a second design phase will naturally weed out the “overengineered” parts of the code.

    I would also fully agree with the author that code needs to be written for the NOW, not the unpredictable future. The rule of three also works well in practice as a way to determine when it is worthwhile to develop a general solution.

    (The rule of three is simply stated as don’t design a general solution until you have written a specific solution three times. The reasoning is that a) you won’t understand the features you need for a general solution until you have 3 good application examples, b) after 3 examples, the probability of needing to reuse the code a fourth time is high enough to make it worth the general design effort).

    -Kent

    Kent

    June 24, 2010 at 2:57 pm

    • Yes, three strikes and you refactor seems to work pretty well in practice.

      coderoom

      June 25, 2010 at 6:41 am

  48. Must be nice to work somewhere that gives you unlimited time and resources for rewrites. If I had to rewrite every time a new feature was requested or a requirement changed (like adding support for teachers when students were the only original requirement) I would be out of a job. Every single rewrite introduces danger that you won’t correctly re-implement all the original business rules. Every rewrite requires much more regression testing.

    Even if you are a TDD shop and convince yourself that all the regression is already handled by your unit tests, when it comes to UAT and your users are told they have to completely test the entire app rather than the little piece they wanted changed you are going to be facing a lynch mob.

    Tim Gray

    June 24, 2010 at 3:18 pm

  49. You are so right. I rememeber when I was relatively young and starting out as a programmer in the sixties I would use every trick I could think of that would show how clever I was but then I realized that simple design, simple code and simple documentation is what counts in the end. Now I am running into situations where programmers take weeks to find problems because of their “clever” code. Your code doesn’t have to be complex to solve complex problems.

    George Lang

    June 24, 2010 at 4:08 pm

  50. The problem is more complex then the simplification that developers are just too afraid to “do the simple thing”.

    “Good design” is inherently linked to the problem being solved. In judging any design, we look at the suitability in comparison to what is needed.

    When a developer does not understand the domain, they will retreat into defining the problem as technology generalizations. Thus the source of the “overengineering” is not an inability to find the appropriate design, but an inability to define the problem in its most succinct nature which will always be a domain context, not a technical one.

    It’s this step of “decide what problem to solve” (or as Kent Beck would say, “decide what you’re going to learn”) is that is more often then not the source of over-engineering

    One final observation, the design mantra “find what varies and encapsulate it” implies that implementation proceeds refactoring. Preemptive refactorings can be dangerous in that they often corrupt variation in the name of adhering to a generalization.

    Marty

    June 24, 2010 at 5:24 pm

    • Nicely put Marty; I forgot to mention that over-generalization can also be a reaction to starting coding before understanding the problem properly, thanks for pointing it out.

      coderoom

      June 25, 2010 at 6:42 am

  51. This is what exactly i had been thinking about for the past 1+ years.The thing is I cannot argue with my Manager and justify the reason for not over engineering.Whenever we start on a new application for a particular need, he talks about a lot of different ways that a software can shape up in the future and to write accordingly so that it can accommodate all those changes.He applied the same thought process even when we wrote a simple file copy program.He insisted it be so generalized that it should do Volume shadow,robocopy,command line hooks,logs,email start and stop and all the possible configuration parameters that will be stored in database.Its been 2 years now and we never used all those options but resulted in an awful program hard to maintain and debug.My experience tells me that whatever programs that is writen keeping generalization in mind has not worked well and it sucks both in terms quality and user experience as well.

    Right

    June 24, 2010 at 5:35 pm

    • Ugh, I feel your pain! :-/ Have you tried looking for a job at a less psychopathic place?

      coderoom

      June 25, 2010 at 6:44 am

  52. All I have to say is, I’m glad I read this post. I was beginning to believe that maybe I were missing something because I try to keep my coding simple.
    Thanks.

    Kevin

    June 24, 2010 at 9:10 pm

  53. The rule I use for abstraction is to avoid it until I encounter 2-3 variations on the class. Then I will abstract out the common behavior from the existing class. This avoids doing abstractions that seem like a ‘good idea’ but never end up being needed.

    russell

    June 24, 2010 at 11:14 pm

  54. [...] Criminal Overengineering As programmers we’re continually accused of doing a sloppy job. There are countless programs in the wild, [...] [...]

  55. Great article, feels like getting some fresh air.

    I think over-engineering is in part caused by an urge to impress or a sense of insecurity.

    David Liu

    June 25, 2010 at 7:24 pm

  56. Great article with great points, however, I think it is an example of misdemeanor over-generalization.

    Each programmer writing their own version of the exact same process is not comforting nor simple to debug. You’ve effectively taken an X number of potentially bug-ridden lines of code and per-mutated it to the power of the number of programmers.

    Just think if coderoom (the author) had to write all the lines of code needed to under-engineer the function of wordpress, scorecardresearch.com, googleanalyticssyndication.com, wp.com, quantserve.com, skimlinks.com, the network firmware and software, my operating system, and the browser used for me to view this article.

    ccody

    June 25, 2010 at 10:39 pm

    • Code reuse is a different issue to over-generalizing the code you do write; I’m planning to talk about it in a future post. There’s a very common misconception that more general code is more likely to be reused!

      coderoom

      June 26, 2010 at 11:53 am

  57. By the way, you forgot to break; your case statements, assuming you didn’t intend fall-through

    MikeN

    June 26, 2010 at 6:31 pm

    • Not at all; each line returns from the function. There’s no possibility to fall through and no need for an entirely meaningless and somewhat confusing break statement in addition! :-)

      coderoom

      June 26, 2010 at 7:15 pm

  58. When I overengineer, it’s usually because I assume another case will require the same pattern. I’m right about 60% of the time as a guestimate. Using the Strategy as an example is good; I’ve seen game developers overdo this to stay in line with thwe architect’s best practices. We need these kinds of reminders — thanks for this sanity check. It’s a book unto itself since GoF assumes you know to to avoid overengineering (or have the good sense to avoid it).

    Mike Britton

    June 26, 2010 at 8:25 pm

  59. [...] of the most common signs of smartness overload is overgeneralization (nice recent summary here). While it may make sense in some cases (when writing public library), in our world (gamedev), [...]

  60. Just wondering why you went through the trouble of writing this whole post? It’s like 20 paragraphs long. Couldn’t you have just posted this:

    “Don’t write bad code!”

    …and be done with it?
    =)

    Richard

    June 28, 2010 at 12:44 pm

  61. I am a #C .Net programmer, but I approve of this blogs message :)

    Michael Washington

    June 28, 2010 at 12:55 pm

  62. The title of the article should be “Criminal Overengineering in small projects” – try to architect a complex project without appropriate abstraction: You will be surprised how quickly and frequently your adventure of getting a mere “method” re-written will turn into a rewrite of thousands of lines of code.

    Sam Records

    June 28, 2010 at 2:13 pm

    • Encapsulation is a very different matter to generalization; increasing layers of abstraction doesn’t usually decouple hidden dependencies within the program, but it does (1) make them less obvious and (2) increase the amount of code you have to modify in order to maintain a dependency over a functionality change. Spending the best part of a decade building and maintaining a ~million line codebase is what’s brought me to many of these conclusions!

      coderoom

      June 29, 2010 at 7:45 am

  63. [...] introduces few new bugs in some other rarely happening use cases). Some nice summer reading: Criminal Overengineering yield thought __________________ http://day2.no-ip.org/ "Commy64 – The problem there is that Oprah was on. MP [...]

  64. [...] Just a few days ago we talked within a team about this subject, and now I bumped into Criminal Overengineering article says it all (well, almost all) about the subject of over-engineered design/code. IMHO, this [...]

  65. [...] Today, I read the Criminal Over-engineering entry on the coderoom blog (http://coderoom.wordpress.com/2010/06/23/criminal-overengineering/). [...]

  66. [...] stuff.Finally I’d like to mention two fantastic programming articles that I read today. This one is about over-engineering, and this one is about automated tests. If you’re into programming, [...]


Comments are closed.

Follow

Get every new post delivered to your Inbox.

Join 32 other followers

%d bloggers like this: