The Joel test is antiquated

I recently stumbled upon references to the Joel test in job postings on the Stack Overflow Careers 2.0 website (careers.stackoverflow.com). Joel Spolsky (one of the founders of Stack Exchange, the company that publishes Stack Overflow and Careers 2.0) published his “Joel Test: 12 Steps to better code” in a blog entry over 11 years ago – in the fall of 2000.

Some of the steps in his test are still relevant, but not as much of an issue today. Some are antiquated. These are yes/no questions. He suggests that if a team has a score of 10 or lower, the team has serious problems. Here’s my take on his 12 steps, particularly in light of the popularity of more collaborative and agile software development methodologies, like XP and Scrum.

  1. Do you use source control?
    • Still relevant, but I’d say if you don’t have source control, you have serious problems, regardless of the rest of the questions. There’s simply no excuse today with all of the open source options.
  2. Can you make a build in one step?
    • Still relevant, and many folks have caught up with this capability with the tools available today compared to 11 years ago. In fact, the problem has moved beyond dev builds (which I presume he is discussing here) to deployment to different environments (QA, UAT, etc.). Deployment in one step is rare, but desired. This should be the new test, I think.
  3. Do you make daily builds?
    • This should say “at least” daily. Continuous Integration is the current practice for evolved teams.
  4. Do you have a bug database?
    • I don’t see this as relevant. On agile teams, for instance, the team is collectively responsible for managing quality. A card on the card wall representing the bug can be a great mechanism for managing your bugs. I suspect this made the cut because Joel’s company’s flagship product at the time was fogbugz – a bug tracking tool.
  5. Do you fix bugs before writing new code?
    • I don’t like this one simply because the priority of what gets done is based on business needs. If the bug you want to fix first is a typo in the copyright statement, sure it should be fixed, but not necessarily right away. It should be prioritized against everything else.
  6. Do you have an up-to-date schedule?
    • Agile projects emphasize continuous planning. I would rephrase this to ask if planning is done on a regular basis and if everyone on the team understands where the project stands and what needs to be done to get it to the next release.
  7. Do you have a spec?
    • I’m not quite sure what kind of spec he’s talking about. Spec is very generic, and could refer to a functional spec (i.e. requirements), architecture spec, design spec, security spec, etc. In an agile environment, we emphasize evolving the requirements and design over the course of the project, rather than stating it in a document up front. I might reframe this to say “Are requirements and design decisions managed and properly understood by the whole team?”
  8. Do programmers have quiet working conditions?
    • Many agile teams prefer an open collaborative environment over quiet corners for programmers to crank out code on their own. Pairing is a great model for programming collaboration and certainly makes for a less-than-quiet environment.
  9. Do you use the best tools money can buy?
    • I think the reference here to money is antiquated. Back in the 90’s, open source was not nearly as prevalent. I think I would say “Do you use the best tools for the job?” or something like that. Then again, ask an IntelliJ guy to debate Eclipse, or Maven vs. Ant, or … you get the picture. “Best” is interpreted through the eyes of the technology religion sometimes.
  10. Do you have testers?
    • I would ask “Are testing responsibilities understood and covered by members of your cross functional team?” for an agile environment. “Having” testers does not imply much – particularly if they test the code at the 11th hour – right before release.
  11. Do new candidates write code during their interview?
    • I like this very much. But I wouldn’t say it’s necessarily “during” the interview, perhaps simply “part of the interview process” (e.g. code submissions in advance). Still – sitting down with someone to write code is a great test of a programmer’s skill.
  12. Do you do hallway usability testing?
    • I would ask instead: “Is the customer represented within your process to provide regular feedback on your product?” Hallways are interesting, but at a software firm, you get a “software developer” slant on the feedback. This is probably good if you’re developing Fogbugz or an IDE, but if you’re creating a website to appeal to the retired community, your usability testing will likely be misguided.

I’ve seen companies in Careers 2.0 convey their Joel Test as “8 out of 12” and “10 out of 12”. Joel’s perspective is that anything below an 11 suggests you have serious problems. If you want to use his test as a metric, OK, but Joel’s view (11 years ago, anyway), is that it represents serious problems. I’m not sure you want to convey that to your potential candidate pool. Or… maybe you do. It certainly is more candid and honest than not pointing out the issues.

Aside: I wonder if Joel would still tout Microsoft’s preeminence as he does in the article (“companies like Microsoft run at 12 full time”). I wonder what Joel would say.

This entry was posted in Agile, Hiring. Bookmark the permalink.

4 Responses to The Joel test is antiquated

  1. Pingback: Openness for engineering teams - Antoine Augusti

  2. David Miller says:

    I’m not sure that #4 is a fair evaluation. In an age of remote development teams it seems to me that physical boards are not particularly useful. If the boards are virtual then you’re on your way to having a database. When I first started at Fog Creek it was very valuable to have so much institutional knowledge – easily searchable and tagged, located in one place. Understanding, what a dev. team decides to fix and having some of what the fix was gives a valuable view of the strengths and weaknesses of your organization, its products, and the products features. I don’t think the fact that Joel built a bug tracker makes any of the above less true.
    #5 Sure, deciding which bugs to fix is always an estimation of the return on investment. That said, it’s important to understand that allowing defects to persist in code is often a tacit assumption of technical debt. Whether or not that debt grows to the point of paralyzing the development team is all about making smart decisions about when and what defects need to be fixed. As a rule of thumb Joel’s rule looks sound.

    Your rephrasings of 6,7,9, don’t seem to add much although you do make the core ideas shine with a bit more detail.
    10 is a strong point, your re-phrasing points to an organizational rigor toward testing that is often missing in software development.
    12 seems reasonable, but 1) I wonder if you’re not selling short the inhabitants of a retirement community. 2) Software developers are not the only people who work in Software development firms. 3) The idea is that most problems in usability are gross enough so that demographics are less important. Clearly there will be exceptions like does your audience read left to right or right to left. Is this print too small. All things which are relevant, but more nuanced than most of the mistakes coders make in building interfaces.
    In general, the Joel test is avowedly a “sloppy” way to estimate the health of a software development organization. I think it holds up as a general framework – there’s lots of good thinking that isn’t substantially debunked by Agile methodology, but the Joel Test is not as an empirical measure of the success or doom of a software development team.
    Thanks for thinking about this topic since it is important to the main point, which is to figure out ways to facilitate writing high quality code.

  3. Jeremy says:

    Over five years later, this remains a highly-relevant post because the original 12-question Joel Test (now 17 years old) inexplicably remains part of Stack Overflow Talent.

    I think your criticisms of 5, 6, and 7 are particularly cogent and important. Repurposing these questions to assess the organization’s effective use of Agile would be a better indicator of health.

    As a hiring manager interested in using the Stack Overflow Talent platform, I could represent my company as a 5 out of 12 according to a strict reading of Joel’s original criteria, or I could apply an updated understanding of how software development works and give us an 11 (we still fail on the spirit of the testing question, but that one’s on me, not Joel).

    Cheers,
    Jeremy

  4. shawn says:

    The suggestion for number 8 misses the point in my opinion. Joel was talking about whether software engineers can establish and remain within a “zone”. It can still be done in a collaborative environment. Can two people working together still get into a zone and code for a few hours uninterrupted? If yes, then great! If the pair is being interrupted every 15 minutes by loud crashing sounds, phone calls, or a ridiculously loud ringtone, then the answer is no. Some amount of noise is tolerated in collaborative environments. The question has to do with whether it is excessive and prevents teammates from being productive. It’s still a very fair and relevant question in my opinion. In general, this blog post is a good example of how teams can start with something simple, and customize it as a good evaluation. I also agree with you though that advertising to potential employees that your processes are sub-standard (scoring less than 10) is typically bad unless the goal is to bring in experience developers that can help to fix that. Sometimes that is exactly what a company is hiring.

Leave a Reply

Your email address will not be published. Required fields are marked *

Time limit is exhausted. Please reload the CAPTCHA.