I borrowed a title of this post from Paul Graham’s essay The 18 Mistakes That Kill Startups:
So how do you pick good programmers if you’re not a programmer? I don’t think there’s an answer. I was about to say you’d have to find a good programmer to help you hire people. But if you can’t recognize good programmers, how would you even do that?
It follows logically from here that software engineering startup should have at least one good programmer to begin with.
This was the case (at least we, founders, believed so) for our first startup in 1991. But we got an opposite and yet analogous problem to solve “How can our new startup can pick a good accountant if we are not accountants?” We didn’t find a better solution at that time than to pick one among us who will learn accounting. However, this general problem is common to many new startups: “how can a group of people pick a good expert in an area where none of the existing members has any expertise?” The previous solution becomes more difficult if you need many experts in different areas and only few founders.
I found another solution unexpectedly several years later. I was using multiple-choice tests to screen candidates for employment. I was unsatisfied with the inability of such tests to identify creative people — exactly the kind of people we wanted to hire. The problem is that multiple choice question offer no opportunity for a creative answer. Therefore, I was experimenting with ways of accepting and efficiently grading of write-in answers. My friends also asked me to prepare some finance questions to test candidates for their company. While helping them and writing questions, I got an idea that this work can be easily outsourced to candidates themselves. Not only it will save my time, but it also will save their time: instead of solving useless test problems that I made up for them, they will solve real problems that their peers need to solve anyway. In other words, it won’t be testing just for testing, but a collaborative problem-solving process that tests their expertise as a side effect. In other words, we invite many candidates and let them ask each other questions about problems they are facing, submit solutions to those problems, and peer-review and evaluate the submitted solutions concurrently and anonymously, using a web service.
At the very beginning, I was not sure that the idea will work. But when it was finally implemented, it turned out surprisingly effective. Not only it discovered good experts, but it also created a useful repository of problems and solutions, sometimes very innovative. Some participants suggested that knowledge exchange and learning experience that this system provides is no less important than the test results. This led to the creation of this website in summer 1998. 3form Free Knowledge Exchange was created to give everyone the opportunity to participate in this process. Over several years, I was doing research that attempts to explain the operation of this system in terms of human-based evolutionary computation and identify the factors that contribute to the efficiency of such computation. The model I had developed is equally applicable to describe the processes happening in another evolving knowledge repository—Wikipedia.