Test Automation Blog

Is it really possible for non-programmers to create automated tests?

Keyword-driven test design is a popular approach in testing teams in big corporations, where the skill levels of QA people are vastly different and most of them are not programmers. The problem of QA managers in such organizations who want to invest in test automation is how to best utilize their available resources. I think that now almost everyone realizes that capture and replay tools fail to create useful and reliable tests, so another approach is needed.

Keyword-driven frameworks rely on a library of small standalone reusable and parametrized building blocks, that represent simple and atomic actions of the application under test. Those are usually developed in a scripting language by the test automation developers. Then those building blocks (or keywords) can be used in some graphical or table-driven interface as reusable actions by testers to create automation without coding. In a way this is very similar to the action-word scripting approach that most automation tools provide (like QTP or the table view in the Selenium IDE).  The difference is that the keywords that the testing tools provide out of the box (usually things like 'click_button' and 'type' are both very generic (in the sense that they can be used in any application) and very narrow (in the sense that they do very small things like clicking a web button or typing text in a textbox). The successful keyword-driven approach requires keywords that are defined in the semantic level of the tested application (like for example 'login' or 'register_user').

Here is a quote from my favorite testing book
Lessons Learned in Software Testing: A Context-Driven Approach

“In contexts that require many nonprogrammers to create automated tests, we believe this is one of the better solutions One drawback of this approach is that you can't write a test unless supported keywords already exist for the tasks you require. Defining and implementing the task functions can become a major undertaking.”

You can build such a framework in-house, using for example excel tables to contain the test scripts with  the keywords and their parameters, or try to use the keyword capabilities of some of the commercial tools on the market (like the reusable actions in QTP). JSystem is an open source test automation framework oriented towards API testing that can also interface with Selenium for creating  keyword-driven UI tests.

While keyword-driven sounds wonderful, it is not a magical methodology that will solve all automation problems and cure world hunger.  I worked on a keyword-driven project while I was an employee of a big corporation. We had an elaborate in-house tool, that could compose the keywords into larger blocks of actions, which were also reusable in tests. The project was a failure. The library of keywords became so huge that no one could figure out which keyword should be used in which context. The tool did not have control flow mechanisms. You could not do loops or if-then-else constructs, which  meant only very simple straightforward test scenarios were possible. The tests were breaking too often and the maintenance effort was higher in my opinion, compared with traditional scripting automation (if done well).

I would like to hear from you about your experiences with keyword-driven automation.

Opensource might actually be better.

In case you are in the process of evaluating test tools, this review by Forrester research might give you a good initial overview of the current commercial offerings - The Forrester Wave™: Functional Testing Solutions, Q3 2008

It looks at solutions that support the whole testing and quality management process and the conclusion is not surprising:

"In today's market, HP is the default choice"

I find the whole thing objective and if you read it carefully you will be able to find the key facts about the automation tools.
The evaluated vendors and the corresponding tools are HP (ex Mercury) with QuickTest Professional, IBM (ex Rational) with Rational Functional Tester, Compuware with TestPartner, Borland (ex Segue) with SilkTest plus also tools from Empirix and Seapine.

The review has its main emphasis on test management and not on test automation:

"Testing teams looking to purchase just a single product should focus on test management first, since mature practices in this area are a prerequisite for success with test automation."

While this sounds logical at first, I somehow can not agree with it entirely. But what follows is really interesting:

"There are exceptions, though: most notably, development teams looking to automate their own regression testing. Commercial test automation tools, however, tend to be less effective here than open source alternatives like FIT/FitNesse or Selenium."

and:

"Both open source testing tools and SOA testing tools are relatively small parts of the functional testing tools market, but they’re growing in importance. Why? Because iterative development methodologies demand that test automation start earlier in the life cycle, and the testing organizations that purchase commercial tools struggle to make this a reality."

It seems that the analysts that wrote this really have some insignt in the business. Overall it is a very interesting read. Did you know that the element Selenium is the antidote to Mercury poisoning?