Keyword-driven test design is a popular approach in testing teams in big corporations, where the skill levels of QA people are vastly different and most of them are not programmers. The problem of QA managers in such organizations who want to invest in test automation is how to best utilize their available resources. I think that now almost everyone realizes that capture and replay tools fail to create useful and reliable tests, so another approach is needed.

Keyword-driven frameworks rely on a library of small standalone reusable and parametrized building blocks, that represent simple and atomic actions of the application under test. Those are usually developed in a scripting language by the test automation developers. Then those building blocks (or keywords) can be used in some graphical or table-driven interface as reusable actions by testers to create automation without coding. In a way this is very similar to the action-word scripting approach that most automation tools provide (like QTP or the table view in the Selenium IDE).  The difference is that the keywords that the testing tools provide out of the box (usually things like 'click_button' and 'type' are both very generic (in the sense that they can be used in any application) and very narrow (in the sense that they do very small things like clicking a web button or typing text in a textbox). The successful keyword-driven approach requires keywords that are defined in the semantic level of the tested application (like for example 'login' or 'register_user').

Here is a quote from my favorite testing book
Lessons Learned in Software Testing: A Context-Driven Approach

“In contexts that require many nonprogrammers to create automated tests, we believe this is one of the better solutions One drawback of this approach is that you can't write a test unless supported keywords already exist for the tasks you require. Defining and implementing the task functions can become a major undertaking.”

You can build such a framework in-house, using for example excel tables to contain the test scripts with  the keywords and their parameters, or try to use the keyword capabilities of some of the commercial tools on the market (like the reusable actions in QTP). JSystem is an open source test automation framework oriented towards API testing that can also interface with Selenium for creating  keyword-driven UI tests.

While keyword-driven sounds wonderful, it is not a magical methodology that will solve all automation problems and cure world hunger.  I worked on a keyword-driven project while I was an employee of a big corporation. We had an elaborate in-house tool, that could compose the keywords into larger blocks of actions, which were also reusable in tests. The project was a failure. The library of keywords became so huge that no one could figure out which keyword should be used in which context. The tool did not have control flow mechanisms. You could not do loops or if-then-else constructs, which  meant only very simple straightforward test scenarios were possible. The tests were breaking too often and the maintenance effort was higher in my opinion, compared with traditional scripting automation (if done well).

I would like to hear from you about your experiences with keyword-driven automation.

You might not know that Selenium provides a feature which allows you to capture the network traffic during a test. This might be handy in some cases - for example if you want to check the HTTP response headers returned by the server for some reason or to verify that a specific HTTP request is triggered during a test. I use it for example to verify that a web analytics request is properly triggered on a particular web page.
To use this feature you must start your tests by providing a special option to the start method, i.e.  selenium.start("captureNetworkTraffic=true");

Another useful Selenium feature I use is the ability to start the RC server by providing a specific Firefox profile for running the tests. Your normal Firefox settings are not used when running tests in *chrome mode. If you want to have a specific Firefox plugin active, or if for example you must use a proxy to access the tested pages, you have to use the -firefoxProfileTemplate setting when starting the Selenium server and provide the path to a custom Firefox profile folder that contains all the settings and plugins you want to use.

The problem is that those two features do not work together - when I use the captureNetworkTraffic=true option, I always get some generic Firefox profile and not the one I provide in the firefoxProfileTemplate parameter.

Here is my workaround for this problem. It turns out that inside the Selenium RC server jar there is a folder called customProfileDirCUSTFFCHROME, which contains the small generic firefox profile, used in this case and probably also in the case when you don't provide a custom Firefox profile.

If you want to capture the network traffic and still use a custom Firefox profile,  here  is how you can hack your selenium-server.jar to include a  custom profile.

• Extract the customProfileDirCUSTFFCHROME folder from selenium-server.jar
• Close all Firefox processes. Start Firefox with the -profilemanager option.
• Choose 'Create Profile' from the dialog that appears, click 'Choose Folder' and select the folder extracted  from the jar file.This way you will use the profile provided by selenium as a starting point. You can then make configuration changes, add plugins etc.
• When you exit firefox, pack all the changed files from the extracted profile folder back into the jar file. Now selenium will always use your custom profile

I tested this with Selenium 1.0.3. Works like a charm.

Having used the 1.0.3 version of Selenium RC for several weeks  I now have an overall impression of the new release. The thing that bugged me most was that the upgrade was not as smooth as I hoped  - several tests that previously worked fine with 1.0.1  started to fail with 1.0.3.  We were able to relatively quickly  find workarounds for what seem to be bugs introduced in the new release. Here are two links to the Selenium issue database  in case you run into the same or similar problems - 400 and 408; look for 'XHR ERROR' exceptions in your logs when opening URLs.

With the workarounds our tests became as stable as before – not better and not worse. The new version did not really bring any noticeable improvements for me – I have not yet moved to Firefox 3.6 and I am not a Mac user. So my advise would be – If you have your tests running smoothly on 1.0.1 and you don't need Snow Leopard and/or FF 3.6 support yet, don't bother upgrading for now.  There might be a new release on the horizon.

One of the reasons I find Selenium a great test automation tool for the web  is that it lets you easily run your test scripts on different browsers without too much effort. Most scripts will run smoothly on IE, Firefox and even on more exotic browsers like Safari, Opera and Chrome without needing browser-specific code or having to tweak the test for a particular browser.  One of the few areas where you are likely to face problems is the handling of pop-up windows.  If the application you are testing tends to open new browser windows using the target="_blank" technique, you might have some problems with Selenium. The standard .selectWindow() method provided by Selenium requires either a title of the pop-up window (which is often the same as the title of the main window) or a name or a variable name (which are both missing, when the window is opened using a target="_blank" link).

You will need a workaround to select such windows and unfortunately the workarounds do not always work the same for all browsers. The following code snippets might be useful if you run into the same problem. They are tested on the *chrome and *iehta Selenium modes (with Internet Explorer 7 and Firefox 3).

Here is how you can select the pop-up window which is opened after a target="_blank" link is clicked. This actually combines several hacks and workarounds, but works at the end (at least for me):

        //A hack to select unnamed pop-up windows
        if (browser.equals("*iehta")){
            //Find out if Selenium substituted the blank target...
            if (selenium.isElementPresent(
                       "//a[starts-with(@target,'selenium_blank')]")) {
                // ...and make it notice the new window on IE
                selenium.openWindow("", selenium.getAttribute(
                      "//a[starts-with(@target,'selenium_blank')]@target"));
            }
        }
        String [] winNames = selenium.getAllWindowNames();
        //the last one is the pop-up window we want
        selenium.waitForPopUp(winNames[winNames.length-1], timeout);
        selenium.selectWindow(winNames[winNames.length-1]);

And here is how you can go back to the main window after you finished working with the pop-up:

        String [] winTitles = selenium.getAllWindowTitles();
        selenium.selectWindow(winTitles[0]);

I was thinking about the failed automation projects I’ve been involved in during the last few years and I seem to find a pattern. All those projects were very, very buggy. I realize now that you should only aim to automate functionality which is fairly stable, so that you do not expect to see a test failure in more than about 1 of 10 test runs. If you see that your tests are failing 50% of the time, then automation is not the cure for your problems. Try to find out why this is happening in the first place.

The quote comes from one of the best books on software quality I have read - Lessons Learned in Software Testingby Cem Kaner, James Bach and Bret Pettichord.  Be sure to read it if you haven't already.
I can not agree more with the authors. I spend a lot of my time in test automation trying to find workarounds for issues with the tool. Some of those issues can be considered 'functional idiosyncrasies', but most are just plain and simple bugs. During my work in the field of software quality I have noticed that users are normally quite tolerant of bugs, unless the bugs block them from doing their jobs. So for me there are two kinds of bugs in the test tools - those you can find a workaround for and those you can not.
If the tool is flexible enough and you know how to use it, most of the time you can make it do what you want - like for example using keyboard shortcuts and the clipboard to read the text from an unsupported GUI control. This is one advantage of tools that offer the flexibility of a real programming language for scripting.
There are also such things, for which you can not do much except waiting for the vendor to provide you a fix, and this can take long time. A few years ago I tried to upgrade to the latest TestPartner version (it was 5.2.3 back then). After the upgrade I found that all my automated scripts were causing random crashes in Internet Explorer. I just stayed with the old version until 5.6 came out, where this problem was fixed.

 I realized that I was not fair in my previous test,  as I used the last version of Firefox, which unfortunately is not officially supported by TestPartner. To give it a second chance, I decided to repeat the test with Internet Explorer.  I re-recorded the same scenario, but this time in IE and tried to replay it.

 I got a better result, but only slightly. The recorded script was able to log-in, but could not click the button to create a new mail:

The playback failed at step 12, where the tool tries to ‘Attach’ to an HTML frame, identified with ‘Name=c36gxk9p2bo3a’. Attaching to a UI control in TestPartner means simply declaring that the subsequent actions in the script will be executed within this control.  c36gxk9p2bo3a is a generated name which changes each time when you enter GMail.  This was the reason the script could not find the frame, when trying to replay.

 If I had to code this script by hand, I would omit this step altogether. Usually it is sufficient to only attach to the browser window and not to specific frames within it. Also, I would not locate the other controls by ID and screen coordinates as the recording did in steps 13 and 14. This will likely fail too, even if the problematic step 12 is fixed.

If you have ever had anything to do with UI test automation, you should know about recording.  It is an ability that most tools (commercial or open source) have - you start the recording session in the tool and then just happily click through the test scenario on the application you are testing.  Meanwhile the tool sits silently and records your actions, getting ready to replay them automatically when you want to. The ability to record the UI interactions is a marketing highlight for most automation tools. There are even some tools, where recording is the only way you can create test scripts.

On the surface it can really look like the best way to automate - you will be testing manually anyway,  so why don't just record your tests to replay them at a later stage? Unfortunately the view that test automation is something as simple as pressing the record button has spread widely, mostly fueled by the hype of the tool vendors.

If you do not have any hands-on experience with test automation, you might miss a very important fact about the whole thing - it is not only about creating the  scripts.  In fact the actual script creation rarely takes more than 20-30% of the time you invest in automation. Here is a rough outline of all the things you have to do:

• Prepare the test environment
  With automation you want to have a very consistent environment, with known data and configuration, where you execute your tests. Sometimes this is not possible  - maybe  you have to test in a live system, or you don't control every part of the system landscape.  In such cases you can succeed only if you create smart automated tests that can anticipate and cope with all the expected changes in the environment. I will write more about this topic in the future, but for now let's just say that recording is not smart at all.
• Create the test scripts
  This is the actual thing that recording addresses. It might seem that it is a very efficient way of creating test scripts, however in reality it is not. You will make errors while recording (like misclicks or mistypings). Correcting those errors in recorded scripts is most often difficult or even impossible.  Also if you limit yourself to just repeating recorded sequences, you will not benefit from all the good things that automation can offer compared with manual testing (like data-driven tests for example).
• Execute the scripts
  Or maybe I should say try to execute the scripts. My experience shows that unless you are testing something that has a very simple UI, recording tools often fail to create working scripts out of the box, even if you have managed to ensure absolutely consistent environment for the test execution. Recording lacks the intelligence to understand what you are doing and what your actual intentions are in a complex UI.
• Analyze the results
  In case you are executing recorded scripts, you will be hoping desperately that they pass successfully. In case something goes wrong (which will be most of the time), you will have extremely hard time trying to understand if it was a bug in the product, an issue with the test script or maybe a change in the application UI.
  The time spent in analysis of test results is probably the greatest time-waster in test automation and you can optimize this only by having proper logging on a higher semantic level than UI interactions. Again this is something that I intend to write about.
• Update your scripts
  Software changes and you will either have to adapt your tests or throw them away. With recording it is mostly about throwing away. It is true that most of the tools now provide some ways to cope with simple changes in the UI, but those are still very limited. For example if a caption of a button changes from 'New' to 'New...',  in most cases you will be able to adjust your scripts. With the good tools you will have to add the '...'  only in one place, no matter how many times you click the button in your test.  However,  imagine that this button is replaced by a context menu.  Most  tools will let you somehow change this in their recorded scripts, but you will have to do this change every single place where the button was clicked in your scenario. Most probably you will just want to re-record it from scratch.

If you plan to do any serious automation, do not base the choice of your tool on how well it handles record and playback. In my opinion writing tests using the scripting language (and doing it right) is the only way to make UI automation really work.
I find recording only useful when I am testing with a new UI technology or a new automation tool and I want to quickly find out how it recognizes  a particular UI control.  In such cases recording a few mouse clicks and then looking at the generated script can be useful.