Test Automation Blog

XPather is a small Firefox extension, which I find indispensable (together with Firebug) when writing Selenium tests.

It adds a 'Show in Xpather...' item to the context menu for all the UI elements on the web page. When selected this option opens a small 'XPather Browser' window showing a very, very long XPath expression which selects the element you chose.

Like /html/body/table[2]/tbody/tr[2]/td[2]/div[@id='login']/form[@id='gaia_loginform']/div[@id='gaia_loginbox']/table/tbody/tr/td/div/table[@id='gaia_table']/tbody/tr[8]/td[2]/input, which is the expression you get when you select the 'Sign in' button on the login page of Gmail. Of course you can write your own expressions and test to see what is selected on  the page.  So you can opt to use for example //input[@name='signIn'], which is much shorter and better way to locate the same button. I use XPather mostly to interactively test complex expressions whenever I am trying to locate some hard-to-find object on the web page.

Several months ago I started working on a new test automation project for a web site and one of the first things to do was to evaluate and choose a tool for the task. The two main contenders were Selenium and IBM Rational Functional Tester. I chose Selenium and since then never got to regret this choice. Here are some of the main reasons I lke this opensource tool much better than the expensive commerial alternatives for web automation:

Selenium acknowledges the fact that to create useful tests you will have to code them

Yes, it gives you a tool that can record sequences of user interactions, but as a whole Selenium is not built with the assumption that this will be the main approach you will use to create your tests. The recording tool (which is a very easy to install Firefox plugin) is useful as a learning aid and maybe for very simple tests. But what I like most about it is that it can covert your recorded sequence to code in any of the supported scripting languages.  And this brings me to the second point:

You get to choose the scripting language

Most commercial automation tools force you to use some Visual Basic dialect as a scripting language. If you are lucky you get something relatively functional like VBA (with TestPartner) or even VB.NET (with Rational Functional Tester). Otherwise you might be forced to code your tests in VBScript, which is really, really painful (the main reason why I don't like QuickTest Professional). Rational Functional Tester also lets you use Java, which makes it my favorite among the commercial tools.

Selenium is language-agnostic. You can use almost any programming language you want. C# and Java are popular and PHP is also an option if you are automating a PHP web app and want to stick to the same language for testing. Ruby is a great language that makes coding fun and the Selenium library for Ruby provides some domain-specific idioms for testing, similar to what Rails does for web development. If you are a Perl, Python or Groovy fan you can use them too. As an additional bonus you will be able to use any development environment you are comfortable with to write your scripts – be it Eclipse, Microsoft Visual Studio or even Vi or Emacs if you want to.

Selenium has a powerful approach for locating UI objects

Most automation tools are surprisingly limited in the way they identify the UI objects to interact with. They let you locate a control by some unique property (like id) or by a collection of properties which together identify it uniquely on the web page. In my experience this approach is sufficient in only about half of the real-world cases.

Imagine a web page containing 3 identical OK buttons. The buttons are exactly the same and have no id's or anything in their properties to distinguish them. Lets say that the first button is located in a form with id 'signup', the second is inside 'order' and the third is inside 'invoice'. Most automation tools do not let you easily identify a UI object in relation to other objects in the page, i.e. you can not say I want to click the button with caption 'OK', which is located inside the 'signup' form. Normally the most you get is I want to click the second button with caption 'OK' on the page. This will work, but only until you get a new version of your application with added additional OK button above the one you are interested in. The XPath expressions in Selenium can locate objects in the DOM document hierarchy (“//form[@id='signup']//input[@type='button' and @value='OK']”) and are very powerful – for instance you can specify that you want to click the checkbox, which is located inside a table row next to a cell with some specific text in it.

Your tests will run on any browser without changes (well, almost)

While you should not expect that automated tests will catch most browser-dependent bugs (like broken layout), the option to run your tests on all browsers that you support is definitely useful. As with programming languages, Selenium offers you a wide choice of browers to run your tests on – not only specific versions of IE and maybe Firefox (the choice you are likely to get from commercial test automation tools), but also Safari, Chrome and Opera. What is more important – about 95% of your tests will run without any changes on all supported browsers. I don't know other automation tool that does this.

It is surprisingly stable and robust for a 1.0 version

My experience with test automation is that if you want to run lots of long tests, you have to expect that something will go wrong at some point – random crashes are not that uncommon in this business. Selenium is at least as stable as anything else I have tried and we routinely run hour-long tests without any problems.

Opensource might actually be better.

In case you are in the process of evaluating test tools, this review by Forrester research might give you a good initial overview of the current commercial offerings - The Forrester Wave™: Functional Testing Solutions, Q3 2008

It looks at solutions that support the whole testing and quality management process and the conclusion is not surprising:

"In today's market, HP is the default choice"

I find the whole thing objective and if you read it carefully you will be able to find the key facts about the automation tools.
The evaluated vendors and the corresponding tools are HP (ex Mercury) with QuickTest Professional, IBM (ex Rational) with Rational Functional Tester, Compuware with TestPartner, Borland (ex Segue) with SilkTest plus also tools from Empirix and Seapine.

The review has its main emphasis on test management and not on test automation:

"Testing teams looking to purchase just a single product should focus on test management first, since mature practices in this area are a prerequisite for success with test automation."

While this sounds logical at first, I somehow can not agree with it entirely. But what follows is really interesting:

"There are exceptions, though: most notably, development teams looking to automate their own regression testing. Commercial test automation tools, however, tend to be less effective here than open source alternatives like FIT/FitNesse or Selenium."

and:

"Both open source testing tools and SOA testing tools are relatively small parts of the functional testing tools market, but they’re growing in importance. Why? Because iterative development methodologies demand that test automation start earlier in the life cycle, and the testing organizations that purchase commercial tools struggle to make this a reality."

It seems that the analysts that wrote this really have some insignt in the business. Overall it is a very interesting read. Did you know that the element Selenium is the antidote to Mercury poisoning?

Yes, it is about capture/replay again.

I decided to start my automation tool reviews with TestPartner 6  for two reasons - the first is that it is the UI automation tool that I am currently using on my project.  The second reason is because the 'Visual Tests'  (also called 'Visual Navigator') feature, introduced with the last major release of TestPartner, supposedly "allows application users—who are intimately familiar with the application under test, but often lack the technical expertise to use automated testing tools—to comfortably and confidently capture business transactions, verify the proper functionality is tested and assist in updating the test when inaccuracies are found or the application changes". The quoted text comes from the product review on the site of Compuware. With TestPartner 6 they are basically trying to catch up with the competition, because HP/Mercury QuickTest Professional (QTP) had this feature first - you record your UI actions and what is created is not code, but descriptive and supposedly less frightening to non-technical people list of entries in a table.

I do not actually use this feature and if you have read my previous post you know why, but I came up with this simple scenario to test how it works overal:

1. Open Gmail

2. Log-in

3. Go to 'Compose Mail'
   

4. Enter some text in the mail body

5. Center the text, using the toolbar button

6. Click ‘Save Now’ to save the email as draft

7. Go to the 'Drafts' folder

8. Click on the draft email you just created

9. Use the features of the test tool to check that the contents
   of the text in the email body are the same as entered

10. Discard the draft message

11. Log-out

I decided to use Gmail as a test application, because I wanted to challenge the tool a little by using some AJAX instead of static HTML and also because almost everyone is "intimately familiar" with Gmail.  The scenario is very short and I think that it is reasonable to expect that automation tools should be able to handle well this level of complexity.

I recorded the sequence with TestPartner.  In the screenshot below you will see how the recorded script looks like:

(click image to enlarge)

It basically borrows from QTP a  lot – for each step of the recording you see a ‘Screen Preview’. The steps are shown as a sequence in the so called ‘Storyboard’ and the currently selected step is shown enlarged in the preview. This is not just a screenshot, but contains the metadata for the UI objects in the application at this particular moment.  It is a little like having the real application in front of you at each step and can be nice in case you want to change something little in the middle of a long test.

To the right of the preview you will see the table with the recorded test steps. The descriptions are verbose like Click the text ‘Sign In’.
Using the ‘Logic Toolbox’  to the right you can insert control flow constructs  like loops and If statements, using a series of wizards. You can also insert checks to verify things (like in step 9 in my sequence) and add additional UI interactions.

I am a little skeptical about the whole idea, because to me it seems that if you can handle this king of wizard-assisted programming, then you can also learn to write code and in the long run you will be happier in terms of your efficiency if you do so.

I wanted to replay the script without any changes in the test environment first.  I expected this to work, but afterwards wanted to see if the recorded script would manage to execute successfully if I have other draft emails in the folder (with other subjects and body texts). My intention was to check how well the tool chooses which properties of the UI elements to use for their identification.