Keyword-driven test design is a popular approach in testing teams in big corporations, where the skill levels of QA people are vastly different and most of them are not programmers. The problem of QA managers in such organizations who want to invest in test automation is how to best utilize their available resources. I think that now almost everyone realizes that capture and replay tools fail to create useful and reliable tests, so another approach is needed.

Keyword-driven frameworks rely on a library of small standalone reusable and parametrized building blocks, that represent simple and atomic actions of the application under test. Those are usually developed in a scripting language by the test automation developers. Then those building blocks (or keywords) can be used in some graphical or table-driven interface as reusable actions by testers to create automation without coding. In a way this is very similar to the action-word scripting approach that most automation tools provide (like QTP or the table view in the Selenium IDE).  The difference is that the keywords that the testing tools provide out of the box (usually things like 'click_button' and 'type' are both very generic (in the sense that they can be used in any application) and very narrow (in the sense that they do very small things like clicking a web button or typing text in a textbox). The successful keyword-driven approach requires keywords that are defined in the semantic level of the tested application (like for example 'login' or 'register_user').

Here is a quote from my favorite testing book
Lessons Learned in Software Testing: A Context-Driven Approach

“In contexts that require many nonprogrammers to create automated tests, we believe this is one of the better solutions One drawback of this approach is that you can't write a test unless supported keywords already exist for the tasks you require. Defining and implementing the task functions can become a major undertaking.”

You can build such a framework in-house, using for example excel tables to contain the test scripts with  the keywords and their parameters, or try to use the keyword capabilities of some of the commercial tools on the market (like the reusable actions in QTP). JSystem is an open source test automation framework oriented towards API testing that can also interface with Selenium for creating  keyword-driven UI tests.

While keyword-driven sounds wonderful, it is not a magical methodology that will solve all automation problems and cure world hunger.  I worked on a keyword-driven project while I was an employee of a big corporation. We had an elaborate in-house tool, that could compose the keywords into larger blocks of actions, which were also reusable in tests. The project was a failure. The library of keywords became so huge that no one could figure out which keyword should be used in which context. The tool did not have control flow mechanisms. You could not do loops or if-then-else constructs, which  meant only very simple straightforward test scenarios were possible. The tests were breaking too often and the maintenance effort was higher in my opinion, compared with traditional scripting automation (if done well).

I would like to hear from you about your experiences with keyword-driven automation.

You might not know that Selenium provides a feature which allows you to capture the network traffic during a test. This might be handy in some cases - for example if you want to check the HTTP response headers returned by the server for some reason or to verify that a specific HTTP request is triggered during a test. I use it for example to verify that a web analytics request is properly triggered on a particular web page.
To use this feature you must start your tests by providing a special option to the start method, i.e.  selenium.start("captureNetworkTraffic=true");

Another useful Selenium feature I use is the ability to start the RC server by providing a specific Firefox profile for running the tests. Your normal Firefox settings are not used when running tests in *chrome mode. If you want to have a specific Firefox plugin active, or if for example you must use a proxy to access the tested pages, you have to use the -firefoxProfileTemplate setting when starting the Selenium server and provide the path to a custom Firefox profile folder that contains all the settings and plugins you want to use.

The problem is that those two features do not work together - when I use the captureNetworkTraffic=true option, I always get some generic Firefox profile and not the one I provide in the firefoxProfileTemplate parameter.

Here is my workaround for this problem. It turns out that inside the Selenium RC server jar there is a folder called customProfileDirCUSTFFCHROME, which contains the small generic firefox profile, used in this case and probably also in the case when you don't provide a custom Firefox profile.

If you want to capture the network traffic and still use a custom Firefox profile,  here  is how you can hack your selenium-server.jar to include a  custom profile.

• Extract the customProfileDirCUSTFFCHROME folder from selenium-server.jar
• Close all Firefox processes. Start Firefox with the -profilemanager option.
• Choose 'Create Profile' from the dialog that appears, click 'Choose Folder' and select the folder extracted  from the jar file.This way you will use the profile provided by selenium as a starting point. You can then make configuration changes, add plugins etc.
• When you exit firefox, pack all the changed files from the extracted profile folder back into the jar file. Now selenium will always use your custom profile

I tested this with Selenium 1.0.3. Works like a charm.

Having used the 1.0.3 version of Selenium RC for several weeks  I now have an overall impression of the new release. The thing that bugged me most was that the upgrade was not as smooth as I hoped  - several tests that previously worked fine with 1.0.1  started to fail with 1.0.3.  We were able to relatively quickly  find workarounds for what seem to be bugs introduced in the new release. Here are two links to the Selenium issue database  in case you run into the same or similar problems - 400 and 408; look for 'XHR ERROR' exceptions in your logs when opening URLs.

With the workarounds our tests became as stable as before – not better and not worse. The new version did not really bring any noticeable improvements for me – I have not yet moved to Firefox 3.6 and I am not a Mac user. So my advise would be – If you have your tests running smoothly on 1.0.1 and you don't need Snow Leopard and/or FF 3.6 support yet, don't bother upgrading for now.  There might be a new release on the horizon.

When it comes to  tools I use for work I usually prefer to wait a little before upgrading, so that any potential problems that might be there are discovered by other people and not by me (the quick release of 1.0.3 seems to confirm that this is actually a good idea). On the other hand my previous experience with upgrading test automation tools shows that I sometimes run into problems no one else has found.

So this time I decided to upgrade to 1.0.3 early and I will blog about my experience.

By the way, there is also a new official Selenium blog.

XPather is a small Firefox extension, which I find indispensable (together with Firebug) when writing Selenium tests.

It adds a 'Show in Xpather...' item to the context menu for all the UI elements on the web page. When selected this option opens a small 'XPather Browser' window showing a very, very long XPath expression which selects the element you chose.

Like /html/body/table[2]/tbody/tr[2]/td[2]/div[@id='login']/form[@id='gaia_loginform']/div[@id='gaia_loginbox']/table/tbody/tr/td/div/table[@id='gaia_table']/tbody/tr[8]/td[2]/input, which is the expression you get when you select the 'Sign in' button on the login page of Gmail. Of course you can write your own expressions and test to see what is selected on  the page.  So you can opt to use for example //input[@name='signIn'], which is much shorter and better way to locate the same button. I use XPather mostly to interactively test complex expressions whenever I am trying to locate some hard-to-find object on the web page.

Several months ago I started working on a new test automation project for a web site and one of the first things to do was to evaluate and choose a tool for the task. The two main contenders were Selenium and IBM Rational Functional Tester. I chose Selenium and since then never got to regret this choice. Here are some of the main reasons I lke this opensource tool much better than the expensive commerial alternatives for web automation:

Selenium acknowledges the fact that to create useful tests you will have to code them

Yes, it gives you a tool that can record sequences of user interactions, but as a whole Selenium is not built with the assumption that this will be the main approach you will use to create your tests. The recording tool (which is a very easy to install Firefox plugin) is useful as a learning aid and maybe for very simple tests. But what I like most about it is that it can covert your recorded sequence to code in any of the supported scripting languages.  And this brings me to the second point:

You get to choose the scripting language

Most commercial automation tools force you to use some Visual Basic dialect as a scripting language. If you are lucky you get something relatively functional like VBA (with TestPartner) or even VB.NET (with Rational Functional Tester). Otherwise you might be forced to code your tests in VBScript, which is really, really painful (the main reason why I don't like QuickTest Professional). Rational Functional Tester also lets you use Java, which makes it my favorite among the commercial tools.

Selenium is language-agnostic. You can use almost any programming language you want. C# and Java are popular and PHP is also an option if you are automating a PHP web app and want to stick to the same language for testing. Ruby is a great language that makes coding fun and the Selenium library for Ruby provides some domain-specific idioms for testing, similar to what Rails does for web development. If you are a Perl, Python or Groovy fan you can use them too. As an additional bonus you will be able to use any development environment you are comfortable with to write your scripts – be it Eclipse, Microsoft Visual Studio or even Vi or Emacs if you want to.

Selenium has a powerful approach for locating UI objects

Most automation tools are surprisingly limited in the way they identify the UI objects to interact with. They let you locate a control by some unique property (like id) or by a collection of properties which together identify it uniquely on the web page. In my experience this approach is sufficient in only about half of the real-world cases.

Imagine a web page containing 3 identical OK buttons. The buttons are exactly the same and have no id's or anything in their properties to distinguish them. Lets say that the first button is located in a form with id 'signup', the second is inside 'order' and the third is inside 'invoice'. Most automation tools do not let you easily identify a UI object in relation to other objects in the page, i.e. you can not say I want to click the button with caption 'OK', which is located inside the 'signup' form. Normally the most you get is I want to click the second button with caption 'OK' on the page. This will work, but only until you get a new version of your application with added additional OK button above the one you are interested in. The XPath expressions in Selenium can locate objects in the DOM document hierarchy (“//form[@id='signup']//input[@type='button' and @value='OK']”) and are very powerful – for instance you can specify that you want to click the checkbox, which is located inside a table row next to a cell with some specific text in it.

Your tests will run on any browser without changes (well, almost)

While you should not expect that automated tests will catch most browser-dependent bugs (like broken layout), the option to run your tests on all browsers that you support is definitely useful. As with programming languages, Selenium offers you a wide choice of browers to run your tests on – not only specific versions of IE and maybe Firefox (the choice you are likely to get from commercial test automation tools), but also Safari, Chrome and Opera. What is more important – about 95% of your tests will run without any changes on all supported browsers. I don't know other automation tool that does this.

It is surprisingly stable and robust for a 1.0 version

My experience with test automation is that if you want to run lots of long tests, you have to expect that something will go wrong at some point – random crashes are not that uncommon in this business. Selenium is at least as stable as anything else I have tried and we routinely run hour-long tests without any problems.

Right after the recent release of Windows 7 Microsoft also released a platform update for Windows Vista and Windows Server 2008. The update brings in some of the new technologies from Windows 7 to those operating systems. Besides 'Windows Graphics, Imaging, and XPS Library' (which  actually gives you  new DirectX), 'Windows Portable Devices Platform' and 'Windows Ribbon and Animation Manager Library' (I hate the ribbon interface), the update also includes something called 'Windows Automation API', which "enables accessibility tools and test automations to access the Windows user interface in a consistent way across operating system versions"
Since Active Accessibility Microsoft has been providing programmatic interfaces for UI automation in Windows.  So if you are starting a Windows automation project, think about using this approach.  You can script with the Automation API in any of Microsoft's programming languages (Visual Basic, C# and even C++)  even with the free Express Editions of Visual Studio. This will not only be a really cost efficient approach, compared with the prices of automation tools like QTP or RFT, but probably will produce results which are more stable too.  After all this is the technology Microsoft uses to create UI tests for their own applications.

One of the reasons I find Selenium a great test automation tool for the web  is that it lets you easily run your test scripts on different browsers without too much effort. Most scripts will run smoothly on IE, Firefox and even on more exotic browsers like Safari, Opera and Chrome without needing browser-specific code or having to tweak the test for a particular browser.  One of the few areas where you are likely to face problems is the handling of pop-up windows.  If the application you are testing tends to open new browser windows using the target="_blank" technique, you might have some problems with Selenium. The standard .selectWindow() method provided by Selenium requires either a title of the pop-up window (which is often the same as the title of the main window) or a name or a variable name (which are both missing, when the window is opened using a target="_blank" link).

You will need a workaround to select such windows and unfortunately the workarounds do not always work the same for all browsers. The following code snippets might be useful if you run into the same problem. They are tested on the *chrome and *iehta Selenium modes (with Internet Explorer 7 and Firefox 3).

Here is how you can select the pop-up window which is opened after a target="_blank" link is clicked. This actually combines several hacks and workarounds, but works at the end (at least for me):

        //A hack to select unnamed pop-up windows
        if (browser.equals("*iehta")){
            //Find out if Selenium substituted the blank target...
            if (selenium.isElementPresent(
                       "//a[starts-with(@target,'selenium_blank')]")) {
                // ...and make it notice the new window on IE
                selenium.openWindow("", selenium.getAttribute(
                      "//a[starts-with(@target,'selenium_blank')]@target"));
            }
        }
        String [] winNames = selenium.getAllWindowNames();
        //the last one is the pop-up window we want
        selenium.waitForPopUp(winNames[winNames.length-1], timeout);
        selenium.selectWindow(winNames[winNames.length-1]);

And here is how you can go back to the main window after you finished working with the pop-up:

        String [] winTitles = selenium.getAllWindowTitles();
        selenium.selectWindow(winTitles[0]);

This is the final part of my hands-on review of TestPartner. Here I will try to demonstrate the old-style Visual Basic scripting approach, which I find to be more reliable.   I will again use my GMail scenario.

As I mentioned before, the only place I find recording useful is in such cases when you want to get to know how the tool ‘sees’ the UI.  I have no experience with the UI of GMail,  so I tried to record the scenario first.

This is the script that I got:

Sub Main()
 
' Begin record on Monday, 13. September 2009 at
' 11:17:07 by Admin
 
    ' Attach to Gmail Email from Google -
    ' Microsoft Internet Explorer IEWindow
    IEWindow("Gmail Email from Google - IEWindow").Attach
 
    ' Attach to Caption='Gmail: Email from
    ' Google'
    HTMLBrowser("Caption='Gmail: Email from Google'").Attach
        HTMLEditBox("Name=Email").SetText "testautomationblog"
        HTMLEditBox("Name=Passwd").SetPasswordText "***"
        HTMLButton("Name=signIn").Click
 
    ' Attach to Gmail - Inbox (2) -
    ' testautomationblog@gmailcom - Microsoft
    ' Internet Explorer IEWindow
    IEWindow("Gmail - Inbox (2) - IEWindow").Attach
 
    ' Attach to Name=cgudh0fxq7w2n
    HTMLFrame("Name=cgudh0fxq7w2n").Attach
        HTMLSpan("ID=':r3'").Click 43, 12
 
    ' Attach to Gmail - Compose Mail -
    ' testautomationblog@gmailcom - Microsoft
    ' Internet Explorer IEWindow
    IEWindow("Gmail - Compose Mail - IEWindow").Attach
 
    ' Attach to ID=':kl'
    HTMLFrame("ID=':kl'").Attach
        HTMLFrame("ID=':kl'").Click 33, 19
        HTMLFrame.Type "Lorem Ipsum Dolor"
 
    ' Attach to Name=cgudh0fxq7w2n
    HTMLFrame("Name=cgudh0fxq7w2n").Attach
        HTMLImage("ID=':ia'").Click
        HTMLDiv("ID='' Index=589").Click 48, 7
        HTMLAnchor("Caption='Drafts (1)'").Click
        HTMLSpan("ID='' Index=38").MouseDown 42, 7
        HTMLDiv("ID='' Index=702").MouseUp 358, 25
        HTMLDiv("ID='' Index=733").Click 35, 12
        HTMLAnchor("Caption='Sign out'").Click
 
' End record on Monday, 13. September 2009 at
' 11:18:22 by Admin
 
End Sub

As I expected, the recorded script could not be run successfully at all.  So what is wrong with it?

The recording has used two ways of addressing the controls. The first one is used with controls that TestPartner attaches to, those seem to be the controls it sees as more important,  for example see line 12. Those controls are identified using something called object maps. The object maps are artifacts that represent definitions of UI controls. When you access a control in a script via an object map, you provide only the name of the map. The actual properties used to identify the control are stored in the map itself. This is the definition of the object map from line 12:

Basically this says that the window we are interested in is an Internet Explorer window with a caption 'Gmail: Email from Google - Microsoft Internet Explorer'. The idea of object maps is that they try to separate how controls are identified from the test logic. If something changes in some particular control, you will have to modify only the relevant object map, but not the script itself.    This looks like a nice idea, but I found that in practice it does not always work that well.

The second way of addressing controls is called 'Raw attach names'. You can see it for example in rows 12-15.  Here you just provide sets of properties and their values as a string.  Notice how TestPartner identifies controls with expressions like "Name=cgudh0fxq7w2n"  and "ID=':r3'".  Unfortunately as we  saw before, those IDs and Names are dynamically generated and change each time you log in GMail. We need a better way to locate those controls. Unfortunately GMail does not provide easy and reliable way to locate some of them (like persistent names or IDs for example), and TestPartner on the other hand does not know how to identify controls based on their relative location in the page.

Still, with some dirty hacks,  I was able to quickly come up with this version of the script which can be executed reliably:

Sub Main()
 
    sText = "Lorem Ipsum Dolor"
 
    ' -- Login
    IEWindow("Caption='Gmail*'").Attach
    HTMLEditBox("Name=Email").SetText "testautomationblog"
    HTMLEditBox("Name=Passwd").SetPasswordText "********"
    HTMLButton("Name=signIn").Click
 
    ' -- Go to compose mail and type the text
    HTMLSpan("InnerText='Compose Mail'").Click
    HTMLFrame("Name='' Index=1").Click
    IEWindow.Type sText
 
    ' -- Center and save
    HTMLImage("Width=20 Height=20 Index=16").Click
    Window.TextSelect "Save Now"
 
    ' -- Wait for the draft to get saved
    Sleep 2
    Window.TextSelect "Draft saved"
 
    ' -- Go to Drafts
    HTMLAnchor("Caption='Drafts*'").Click
    HTMLSpan("InnerText='*" & sText & "'").Click
 
    ' -- Find the text and discard the draft
    Window.TextSelect sText
    Window.TextSelect "Discard"
 
    ' -- Logout
    HTMLAnchor("Caption='Sign out'").Click
 
End Sub

This is a quick summary:
At line 6 I attach to the IE window. This is the only place I attach in the script as I only work in one window. I used raw attach instead of object map. Note how I used a wilcard for the window caption. Lines 7-12 are self explainotary. What you see in lines 13 and 14 is really a small hack. The editor area in GMail is contained in a HTMLFrame.  TestPartner can not find anything meaningful in it to identify it, so I used HTMLFrame("Name='' Index=1"), which means basically the first HTMLFrame with empty name.  Because the HTMLFrame object does not support the SetText method (like the HTMLEditBox in line 7 for example), I first click on it (line 13) and then just type in the text (line 14). I do not feel very good about this solution, but it was the first quick hack I could think of.
Identifying the center button in the toolbar of GMail was also problematic (line 17). All the buttons in GMail also lack unique IDs or Names, so I was able to identify the button only as HTMLImage("Width=20 Height=20 Index=16") - the 16th HTMLImage with Width=20 and Height=20 in the page.
Clicking the 'Save Now' button (line 18) also proved more difficult than I anticipated. For some reason it appeared as an empty unidentifiable Div when I tried to locate it with the object browser.  I used the Window.TextSelect trick, which is the most universal way to make TestPartner click on some text you can not identify otherwise.  This is actually a helpful tip - use Window.TextSelect if you can not identify something.  After clicking the 'Save Now' button we have to wait for the backend to actually save the page before moving on with the test. A quick and somewhat dirty way to do this is shown on lines and - wait for 2 seconds and then verify that the "Draft saved" text appears on the page.  I think the rest of the script needs no explanation.

To sum it up - TestPartner can be buggy and confusing, but after all, it is powerful enough to let you automate almost anything.  Maybe you'll need some time to find the right workarounds. Probably they will be ugly. Overall there are much better tools, especially if you want to do web automation. There are worse too, but I am happy that I don't have to use TestPartner anymore.