Over the past 6 years I've interviewed lots of Quality Assurance candidates. Both when I was Quality Assurance Manager for Pirates of the Burning Sea, and now as a QAE Lead working on mobile applications. One of the questions I always ask is "How do you know when a project is ready to ship?" While I did not choose this question purely for it's comedic value, I often find myself laughing internally when I hear answers like "When all the bugs are fixed" or "When my leads tells me to sign-off". While there is some accuracy to each of these answers, they are not quite the well-spoken thought out answer I am looking for.
I recently proposed the same question to the 8 testers that work for me. These testers had anywhere from 0 to 10+ years of QA experience before there employment here. We spent a half hour or so kicking around the question and came up with the following. You likely will not agree with all that is below, and you can probably come up with additional items we left out, and that's fantastic; b'coz that means you gave it some thought.
In short: you have to track and trend the numbers while conducting your bug triage meetings,all the while keeping an eye on budget & schedule and listening to your gut.
The closer you get to the end of a project the more you should be tracking and trending your bug statistics. The most common in the industry seems to be find \ fixed rate. It simply entails creating a query that tells you how many bugs were filed in a specific timeframe versus how man bugs were fixed in the same timeframe. During your last few milestones you might track this week to week, as you get close to ship it will likely become day to day or more often.
*** I though about putting an example here, but math is hard. Checkout stickyminds.com or some other great QA resource for more find\fix stuff ***
However the numbers in and of themselves is not the whole picture. I was recently on a project for a customer that was clearly all about the numbers. When we finished our last milestone, and began our month of freeze 'n fix and UAT testing, we received a very concerned email trending our recent find\fix rate and claiming that at the current rate we would ship a month late. What was not taken into consideration was priority of the issues trended as well as our Triage process.
My motto for triage is "Not everything has to be fixed, but everything has to be addressed!" Addressed can mean fixed, won't fix, postponed, can't fix, works for me ... whatever. But everyone needs to be in agreement. By everyone of course I mean at least the "Big 3"; PM, Dev Lead, and QA Lead; although others can attend Triage meetings as well. Sometimes Triage guidelines help, something along the lines of ...
- Priority 1 issues must be fixed
- Priority 2 & 3 issues must be addressed
- Priority 4 issues are ok.
Now I know that the 3rd bullet above goes against my triage motto; but you gotta be flexible or you're gonna go INSANE! Especially on a larger project with thousands of active bugs; do you really think the PM and Dev Lead are going to walk through them all with you?
BUDGET & SCHEDULE
It's an unfortunate fact that extenuating circumstances will sometimes effect the triage process and guidelines. Any tester worth his or her weight in Hot Pockets and Mountain Dew will be of the opinion that budget and\or schedule is one of the worst reason to ship a project. But alas we know that's the way the world works. At some point you have to stop the developers from developin', polish the sucker up, and get it out the door. All I can say to you is lean heavily on your triage process, and your gut feeling ...
TESTER GUT FEELING
I had a PM in a past project that would regularly ask me "to do a gut check on a milestone"How's your gut feel?"". She knew that a senior tester, knowledgable about the project, process, co-workers, customers, genre, and industry, could give a valuable, if unqualifyable, opinion on the quality of a project. However once the developers got wind of the it they want me to prove it, qualify it, and document it. That's understandable, developers deal in math and hard facts all day long, they should not be ok with a gut feeling being a determining factor in the fate of a project. However as a tester I don't want to lose that gut feeling. It's not likely to stop the show, but it will encourage me to focus my testing in a particular area, or even make me to work a few extra hours or ask for a few more resources. I assure you there are times I should have listened to my gut and didn't.
So, how do you know when a project is ready to ship?!? You have to track and trend the numbers while conducting your bug triage meetings,all the while keeping an eye on budget & schedule and listening to your gut.
Next topic? I'm thinking about brainstorming all the non-funtional testing that one should execute on a mobile project and how oven each suite should be run.