I'm glad to hear this works for you. But I think the process emphasizes technical skills over what really matters when hiring: how the person works with the team and within the business.
Giving someone a toy project unrelated to the business tells you nothing about how they will perform in the real environment. Letting the candidate take the work home to work on it in isolation tells you nothing about how they will work with your team, or within the company.
In his study "People and methodologies in software development" Alistair Cockburn found that “People’s characteristics, which vary from person to person and even from moment to moment, form a first-order driver of the team’s behavior and results." (http://alistair.cockburn.us/People+and+methodologies+in+soft...). Other studies have come to similar conclusions.
In a technical job, actual skills and experience are necessary but not sufficient for success. Programmers fail in their jobs because they can't work with or fit in with the team, and the larger organization. Any competent programmer can learn a new language, framework, or technique like TDD in a week or two. But people who don't pay attention to the business environment and requirements, and can't work with their team, group, etc. will fail, often costing a lot of money.
Presumably you have weeded out the unqualified before offering to pay them to work for a few hours, but this seems like offering an unnecessary incentive to learn very little about the candidate's likely on-the-job performance.
I don't know any scientific way to measure how someone will fit in, and I don't think anyone else does either. That's why having several people from different parts of the organization interview is the norm -- the interviewers compare notes and reach a consensus because everyone understands they are going mostly by their subconscious (gut) feelings.
I spend time describing the business requirements and then ask candidates how they might solve an actual business problem. I don't expect a complete or deep answer; what I'm after is finding out if the candidate can think about a business problem at all. Examples: Our inventory system frequently doesn't match what's in the warehouse. We get a lot of people abandoning their cart at checkout. We have customers gaming our promotion codes.
Testing for things like "functional reactive principles" is an example of ignoring the business environment and favoring technical trivia. No business ever had the problem "We need 2,000 lines of Ruby code by next week," or "We need a system to organize a small movie library, you have two hours." What I find is that far too many programmers don't even hear business requirements and can't do basic analysis or think of the right questions to ask. They want to start writing code. Having expensive people writing code that doesn't address business requirements is costly, and very common. If a candidate isn't asking questions about the business and the team that's a big red flag for me. Programmers who just want to be left alone to do their own thing are easy to find but they aren't necessarily a good fit, or productive.
I also spend time getting as many people in the organization as I can to chat with the candidate, so we can compare our impressions. That is just recognizing human nature -- we all tend to form opinions of other peole within a few seconds, and then look for things that reinforce our impression. Of course I want to see evidence of actual technical skill, but I can do that with a few actual programming problems -- even something on the level of FizzBuzz will tell you if the person can solve a problem in code. And I want to see how they solve the problem, and what kinds of questions they ask.
If someone is uncomfortable working alongside another programmer, or solving problems on a whiteboard, or talking to a group, I get it, but those are all necessary skills too. Just like learning a new framework, a motivated person can learn to work in a group, listen and ask questions, speak to a room. They may be a genius when left alone in a corner not interacting with anyone but they probably aren't going to be a good fit in most organizations. I often ask candidates to tell me how they would go about hiring someone for the same job they are interviewing for -- suppose we have two positions open and you get to hire a peer. That can tell me a lot about what they value in a co-worker.
Giving someone a toy project unrelated to the business tells you nothing about how they will perform in the real environment. Letting the candidate take the work home to work on it in isolation tells you nothing about how they will work with your team, or within the company.
In his study "People and methodologies in software development" Alistair Cockburn found that “People’s characteristics, which vary from person to person and even from moment to moment, form a first-order driver of the team’s behavior and results." (http://alistair.cockburn.us/People+and+methodologies+in+soft...). Other studies have come to similar conclusions.
In a technical job, actual skills and experience are necessary but not sufficient for success. Programmers fail in their jobs because they can't work with or fit in with the team, and the larger organization. Any competent programmer can learn a new language, framework, or technique like TDD in a week or two. But people who don't pay attention to the business environment and requirements, and can't work with their team, group, etc. will fail, often costing a lot of money.
Presumably you have weeded out the unqualified before offering to pay them to work for a few hours, but this seems like offering an unnecessary incentive to learn very little about the candidate's likely on-the-job performance.
I don't know any scientific way to measure how someone will fit in, and I don't think anyone else does either. That's why having several people from different parts of the organization interview is the norm -- the interviewers compare notes and reach a consensus because everyone understands they are going mostly by their subconscious (gut) feelings.
I spend time describing the business requirements and then ask candidates how they might solve an actual business problem. I don't expect a complete or deep answer; what I'm after is finding out if the candidate can think about a business problem at all. Examples: Our inventory system frequently doesn't match what's in the warehouse. We get a lot of people abandoning their cart at checkout. We have customers gaming our promotion codes.
Testing for things like "functional reactive principles" is an example of ignoring the business environment and favoring technical trivia. No business ever had the problem "We need 2,000 lines of Ruby code by next week," or "We need a system to organize a small movie library, you have two hours." What I find is that far too many programmers don't even hear business requirements and can't do basic analysis or think of the right questions to ask. They want to start writing code. Having expensive people writing code that doesn't address business requirements is costly, and very common. If a candidate isn't asking questions about the business and the team that's a big red flag for me. Programmers who just want to be left alone to do their own thing are easy to find but they aren't necessarily a good fit, or productive.
I also spend time getting as many people in the organization as I can to chat with the candidate, so we can compare our impressions. That is just recognizing human nature -- we all tend to form opinions of other peole within a few seconds, and then look for things that reinforce our impression. Of course I want to see evidence of actual technical skill, but I can do that with a few actual programming problems -- even something on the level of FizzBuzz will tell you if the person can solve a problem in code. And I want to see how they solve the problem, and what kinds of questions they ask.
If someone is uncomfortable working alongside another programmer, or solving problems on a whiteboard, or talking to a group, I get it, but those are all necessary skills too. Just like learning a new framework, a motivated person can learn to work in a group, listen and ask questions, speak to a room. They may be a genius when left alone in a corner not interacting with anyone but they probably aren't going to be a good fit in most organizations. I often ask candidates to tell me how they would go about hiring someone for the same job they are interviewing for -- suppose we have two positions open and you get to hire a peer. That can tell me a lot about what they value in a co-worker.