January 17th, 2010 §
XPather is a small Firefox extension, which I find indispensable (together with Firebug) when writing Selenium tests.
It adds a 'Show in Xpather...' item to the context menu for all the UI elements on the web page. When selected this option opens a small 'XPather Browser' window showing a very, very long XPath expression which selects the element you chose.
Like /html/body/table[2]/tbody/tr[2]/td[2]/div[@id='login']/form[@id='gaia_loginform']/div[@id='gaia_loginbox']/table/tbody/tr/td/div/table[@id='gaia_table']/tbody/tr[8]/td[2]/input, which is the expression you get when you select the 'Sign in' button on the login page of Gmail. Of course you can write your own expressions and test to see what is selected on the page. So you can opt to use for example //input[@name='signIn'], which is much shorter and better way to locate the same button. I use XPather mostly to interactively test complex expressions whenever I am trying to locate some hard-to-find object on the web page.
December 12th, 2009 §
Service oriented architecture is not something new or cool anymore. From enterprise software to web sites, software exposing its functionality via web services is becoming increasingly common.
The question is how to test those services. One option is to use a unit testing framework and create tests calling the services in any programming language you are familiar with (Java + Junit, C# + Nunit, etc). Whether such tests can be called unit, functional or even integration tests is not that important, but this approach is probably best suited for developers creating tests themselves. It is also not quite convenient for ad-hoc or exploratory testing.
A lot of commercial Java application servers such as SAP NetWeaver provide built-in web-based tools for browsing and testing the deployed services on the server. Using such tool is a good option for ad-hoc testing, because usually the services are readily accessible in it without any need for additional configuration. If you don't have access to something like this or if you need more flexibility, like maybe saving the test parameters and creating whole test suites, you might want to look at SoapUI. This is a tool for web service testing that has a free (open source) version and a commercial 'Pro' version with additional features. The most useful 'pro' feature for me is the form-based interface for entering the input parameters for the service call. SoapUI allows you to create whole scenarios consisting of services called in sequence and exchanging data between them. It even has some performance / load testing features.
SoapUI seems to be the most popular tool for web service testing, but it has its quirks and is not always as robust as I would like it to be. I am curious if anyone knows something better out there.
October 31st, 2009 §
Right after the recent release of Windows 7 Microsoft also released a platform update for Windows Vista and Windows Server 2008. The update brings in some of the new technologies from Windows 7 to those operating systems. Besides 'Windows Graphics, Imaging, and XPS Library' (which actually gives you new DirectX), 'Windows Portable Devices Platform' and 'Windows Ribbon and Animation Manager Library' (I hate the ribbon interface), the update also includes something called 'Windows Automation API', which "enables accessibility tools and test automations to access the Windows user interface in a consistent way across operating system versions"
Since Active Accessibility Microsoft has been providing programmatic interfaces for UI automation in Windows. So if you are starting a Windows automation project, think about using this approach. You can script with the Automation API in any of Microsoft's programming languages (Visual Basic, C# and even C++) even with the free Express Editions of Visual Studio. This will not only be a really cost efficient approach, compared with the prices of automation tools like QTP or RFT, but probably will produce results which are more stable too. After all this is the technology Microsoft uses to create UI tests for their own applications.
April 11th, 2009 §
Opensource might actually be better.
In case you are in the process of evaluating test tools, this review by Forrester research might give you a good initial overview of the current commercial offerings - The Forrester Wave™: Functional Testing Solutions, Q3 2008
It looks at solutions that support the whole testing and quality management process and the conclusion is not surprising:
"In today's market, HP is the default choice"
I find the whole thing objective and if you read it carefully you will be able to find the key facts about the automation tools.
The evaluated vendors and the corresponding tools are HP (ex Mercury) with QuickTest Professional, IBM (ex Rational) with Rational Functional Tester, Compuware with TestPartner, Borland (ex Segue) with SilkTest plus also tools from Empirix and Seapine.
The review has its main emphasis on test management and not on test automation:
"Testing teams looking to purchase just a single product should focus on test management first, since mature practices in this area are a prerequisite for success with test automation."
While this sounds logical at first, I somehow can not agree with it entirely. But what follows is really interesting:
"There are exceptions, though: most notably, development teams looking to automate their own regression testing. Commercial test automation tools, however, tend to be less effective here than open source alternatives like FIT/FitNesse or Selenium."
and:
"Both open source testing tools and SOA testing tools are relatively small parts of the functional testing tools market, but they’re growing in importance. Why? Because iterative development methodologies demand that test automation start earlier in the life cycle, and the testing organizations that purchase commercial tools struggle to make this a reality."
It seems that the analysts that wrote this really have some insignt in the business. Overall it is a very interesting read. Did you know that the element Selenium is the antidote to Mercury poisoning?
April 9th, 2009 §
"Testers get indignant when they realize that the quality tools purchased at a serious price are themselves full of bugs. Indeed, test tools are often buggier than comparable (but cheaper) development tools. Plan to test your tools and spend time finding workarounds for bugs."
The quote comes from one of the best books on software quality I have read - Lessons Learned in Software Testing
by Cem Kaner, James Bach and Bret Pettichord. Be sure to read it if you haven't already.
I can not agree more with the authors. I spend a lot of my time in test automation trying to find workarounds for issues with the tool. Some of those issues can be considered 'functional idiosyncrasies', but most are just plain and simple bugs. During my work in the field of software quality I have noticed that users are normally quite tolerant of bugs, unless the bugs block them from doing their jobs. So for me there are two kinds of bugs in the test tools - those you can find a workaround for and those you can not.
If the tool is flexible enough and you know how to use it, most of the time you can make it do what you want - like for example using keyboard shortcuts and the clipboard to read the text from an unsupported GUI control. This is one advantage of tools that offer the flexibility of a real programming language for scripting.
There are also such things, for which you can not do much except waiting for the vendor to provide you a fix, and this can take long time. A few years ago I tried to upgrade to the latest TestPartner version (it was 5.2.3 back then). After the upgrade I found that all my automated scripts were causing random crashes in Internet Explorer. I just stayed with the old version until 5.6 came out, where this problem was fixed.
March 16th, 2009 §
Record/Playback does not work, but we all know this, don't we?
If you have ever had anything to do with UI test automation, you should know about recording. It is an ability that most tools (commercial or open source) have - you start the recording session in the tool and then just happily click through the test scenario on the application you are testing. Meanwhile the tool sits silently and records your actions, getting ready to replay them automatically when you want to. The ability to record the UI interactions is a marketing highlight for most automation tools. There are even some tools, where recording is the only way you can create test scripts.
On the surface it can really look like the best way to automate - you will be testing manually anyway, so why don't just record your tests to replay them at a later stage? Unfortunately the view that test automation is something as simple as pressing the record button has spread widely, mostly fueled by the hype of the tool vendors.
If you do not have any hands-on experience with test automation, you might miss a very important fact about the whole thing - it is not only about creating the scripts. In fact the actual script creation rarely takes more than 20-30% of the time you invest in automation. Here is a rough outline of all the things you have to do:
- Prepare the test environment
With automation you want to have a very consistent environment, with known data and configuration, where you execute your tests. Sometimes this is not possible - maybe you have to test in a live system, or you don't control every part of the system landscape. In such cases you can succeed only if you create smart automated tests that can anticipate and cope with all the expected changes in the environment. I will write more about this topic in the future, but for now let's just say that recording is not smart at all.
- Create the test scripts
This is the actual thing that recording addresses. It might seem that it is a very efficient way of creating test scripts, however in reality it is not. You will make errors while recording (like misclicks or mistypings). Correcting those errors in recorded scripts is most often difficult or even impossible. Also if you limit yourself to just repeating recorded sequences, you will not benefit from all the good things that automation can offer compared with manual testing (like data-driven tests for example).
- Execute the scripts
Or maybe I should say try to execute the scripts. My experience shows that unless you are testing something that has a very simple UI, recording tools often fail to create working scripts out of the box, even if you have managed to ensure absolutely consistent environment for the test execution. Recording lacks the intelligence to understand what you are doing and what your actual intentions are in a complex UI.
- Analyze the results
In case you are executing recorded scripts, you will be hoping desperately that they pass successfully. In case something goes wrong (which will be most of the time), you will have extremely hard time trying to understand if it was a bug in the product, an issue with the test script or maybe a change in the application UI.
The time spent in analysis of test results is probably the greatest time-waster in test automation and you can optimize this only by having proper logging on a higher semantic level than UI interactions. Again this is something that I intend to write about.
- Update your scripts
Software changes and you will either have to adapt your tests or throw them away. With recording it is mostly about throwing away. It is true that most of the tools now provide some ways to cope with simple changes in the UI, but those are still very limited. For example if a caption of a button changes from 'New' to 'New...', in most cases you will be able to adjust your scripts. With the good tools you will have to add the '...' only in one place, no matter how many times you click the button in your test. However, imagine that this button is replaced by a context menu. Most tools will let you somehow change this in their recorded scripts, but you will have to do this change every single place where the button was clicked in your scenario. Most probably you will just want to re-record it from scratch.
If you plan to do any serious automation, do not base the choice of your tool on how well it handles record and playback. In my opinion writing tests using the scripting language (and doing it right) is the only way to make UI automation really work.
I find recording only useful when I am testing with a new UI technology or a new automation tool and I want to quickly find out how it recognizes a particular UI control. In such cases recording a few mouse clicks and then looking at the generated script can be useful.