« i just want pmr | Main | Non-Functional Testing »
Thursday
Nov032011

exploratory testing

many times i have been at a daily scrum standup and heard a qae report something along the lines of “yesterday i focused on exploratory testing and will likely continue to do so today”.  while that may be 100% accurate, it is not very informational.  the cynical side of me thinks all a pm or other project team member hears when we say that is “i poked the app a bit yesterday and spent the rest of the day on facebook, it was pretty fun so i’m going to do it again today”. 

i’m no expert on exploratory testing, I leave that to the kaner's and bach’s of the world.  i do however want to get my test team thinking about how they can better give status at standup meetings, as well as train other disciplines exactly why exploratory testing is, as my qa director put it, “the most productive type of testing we do”. so we had a combination brown-bag and brainstorming session, about 10 testers and I, to kick this topic around a bit.

what is exploratory testing?  the standard description seems to be …

simultaneous learning, test design and test execution

a slightly more verbose description is …

testing to the extent that the tester actively controls the design of the tests as those tests are performed and uses information gained while testing to design new and better tests.

or as cem kaner, who originally coined the term, would put it …

"a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.”

continually ask “what’s the best test I can perform, right now?”  exploratory tests are continually optimized throughout, as opposed to scripted testing which can become less fruitful over time. the chance of finding a new issue with a scripted test diminishes each time it is executed, whereas exploratory testing can focus both broad and deep as needed.
   
one of my regular interview questions is: “is software testing an art or a science?”  the answer is usually somewhere in the middle, then i gauge how the interviewee is able to support their opinion.  everyone seems to have a different way of thinking about the ratio, I tend to think of it as a spectrum, with art on one end and science on the other …


art <------> science


for the sake of this analogy let’s say that exploratory testing is somewhere in the middle … 


art <-----et-----> science


now add the extremes.  monkey testing simulates random touch screen input regardless of what is displayed on screen, just like you’d given the device to a monkey.  i put automation as the other extreme; what’s more opposite of a monkey than a machine?


art <-mt-----et-----auto-> science


filling in the lower end a bit would be adhoc testing.  i remember when i worked at that large software empire in redmond washington they often used the concept of both adhoc general and adhoc directed. 


art <-mt-----ag-----ad------et-----auto-> science


filling in the top end i separate exploratory into freestyle exploratory (which bach discusses on his website) and directed exploratory (which is a term i made up assuming that bach’s use of freestyle exploratory implies that there is another more structured version, as exploratory testing often has a charter, giving the tester structure within which to execute their exploratory testing). 


art <-mt-----ag------ad-----fe-----de-----auto-> science


one last test methodology I would add to the art\science spectrum is your standard everyday test case execution, or scripted testing …


art <-mt-----ag-----ad-----fe-----de-----scripted-----auto-> science


i think we should also make this spectrum show a greater distance between directed exploratory testing and test cases execution, perhaps like this …


art <-mt-----ag-----ad-----fe-----de----->...<-----scripted-----auto-> science


of all the methodologies listed above, exploratory testing makes best use of tester knowledge, without being overly rigid thereby getting in the way of using that knowledge.  


when is it good to utilize exploratory testing?

  1. You need to provide rapid feedback on a new product or feature.
  2. You need to learn the product quickly.
  3. You have already tested using scripts, and seek to diversify the testing.
  4. You want to find the single most important bug in the shortest time.
  5. You want to check the work of another tester by doing a brief independent investigation.
  6. You want to investigate and isolate a particular defect.
  7. You want to investigate the status of a particular risk, in order to evaluate the need for scripted tests in that area.
at my company we also use it quite a bit for test case creation.  yes, we know that this is not historically “best practices”, as cases should be written prior to the code being testable.  all i can say is we are a fairly agile company, which does not always provide the type of detailed design documentation that we would like to base our test cases upon. 


that’s that on the subject of exploratory testing for now.  feel free to comment on, poke holes in, or perhaps even to support the above; it’s all part of the brainstorming process in the effort of knowledge sharing and professional development.  


resources:

http://en.wikipedia.org/wiki/Exploratory_testing
http://www.satisfice.com/articles/what_is_et.shtml
http://www.satisfice.com/articles/et-article.pdf


ToddS

Reader Comments

There are no comments for this journal entry. To create a new comment, use the form below.

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>