Several months ago I started working on a new test automation project for a web site and one of the first things to do was to evaluate and choose a tool for the task. The two main contenders were Selenium and IBM Rational Functional Tester. I chose Selenium and since then never got to regret this choice. Here are some of the main reasons I lke this opensource tool much better than the expensive commerial alternatives for web automation:

Selenium acknowledges the fact that to create useful tests you will have to code them

Yes, it gives you a tool that can record sequences of user interactions, but as a whole Selenium is not built with the assumption that this will be the main approach you will use to create your tests. The recording tool (which is a very easy to install Firefox plugin) is useful as a learning aid and maybe for very simple tests. But what I like most about it is that it can covert your recorded sequence to code in any of the supported scripting languages.  And this brings me to the second point:

You get to choose the scripting language

Most commercial automation tools force you to use some Visual Basic dialect as a scripting language. If you are lucky you get something relatively functional like VBA (with TestPartner) or even VB.NET (with Rational Functional Tester). Otherwise you might be forced to code your tests in VBScript, which is really, really painful (the main reason why I don't like QuickTest Professional). Rational Functional Tester also lets you use Java, which makes it my favorite among the commercial tools.

Selenium is language-agnostic. You can use almost any programming language you want. C# and Java are popular and PHP is also an option if you are automating a PHP web app and want to stick to the same language for testing. Ruby is a great language that makes coding fun and the Selenium library for Ruby provides some domain-specific idioms for testing, similar to what Rails does for web development. If you are a Perl, Python or Groovy fan you can use them too. As an additional bonus you will be able to use any development environment you are comfortable with to write your scripts – be it Eclipse, Microsoft Visual Studio or even Vi or Emacs if you want to.

Selenium has a powerful approach for locating UI objects

Most automation tools are surprisingly limited in the way they identify the UI objects to interact with. They let you locate a control by some unique property (like id) or by a collection of properties which together identify it uniquely on the web page. In my experience this approach is sufficient in only about half of the real-world cases.

Imagine a web page containing 3 identical OK buttons. The buttons are exactly the same and have no id's or anything in their properties to distinguish them. Lets say that the first button is located in a form with id 'signup', the second is inside 'order' and the third is inside 'invoice'. Most automation tools do not let you easily identify a UI object in relation to other objects in the page, i.e. you can not say I want to click the button with caption 'OK', which is located inside the 'signup' form. Normally the most you get is I want to click the second button with caption 'OK' on the page. This will work, but only until you get a new version of your application with added additional OK button above the one you are interested in. The XPath expressions in Selenium can locate objects in the DOM document hierarchy (“//form[@id='signup']//input[@type='button' and @value='OK']”) and are very powerful – for instance you can specify that you want to click the checkbox, which is located inside a table row next to a cell with some specific text in it.

Your tests will run on any browser without changes (well, almost)

While you should not expect that automated tests will catch most browser-dependent bugs (like broken layout), the option to run your tests on all browsers that you support is definitely useful. As with programming languages, Selenium offers you a wide choice of browers to run your tests on – not only specific versions of IE and maybe Firefox (the choice you are likely to get from commercial test automation tools), but also Safari, Chrome and Opera. What is more important – about 95% of your tests will run without any changes on all supported browsers. I don't know other automation tool that does this.

It is surprisingly stable and robust for a 1.0 version

My experience with test automation is that if you want to run lots of long tests, you have to expect that something will go wrong at some point – random crashes are not that uncommon in this business. Selenium is at least as stable as anything else I have tried and we routinely run hour-long tests without any problems.

If you have ever had anything to do with UI test automation, you should know about recording.  It is an ability that most tools (commercial or open source) have - you start the recording session in the tool and then just happily click through the test scenario on the application you are testing.  Meanwhile the tool sits silently and records your actions, getting ready to replay them automatically when you want to. The ability to record the UI interactions is a marketing highlight for most automation tools. There are even some tools, where recording is the only way you can create test scripts.

On the surface it can really look like the best way to automate - you will be testing manually anyway,  so why don't just record your tests to replay them at a later stage? Unfortunately the view that test automation is something as simple as pressing the record button has spread widely, mostly fueled by the hype of the tool vendors.

If you do not have any hands-on experience with test automation, you might miss a very important fact about the whole thing - it is not only about creating the  scripts.  In fact the actual script creation rarely takes more than 20-30% of the time you invest in automation. Here is a rough outline of all the things you have to do:

• Prepare the test environment
  With automation you want to have a very consistent environment, with known data and configuration, where you execute your tests. Sometimes this is not possible  - maybe  you have to test in a live system, or you don't control every part of the system landscape.  In such cases you can succeed only if you create smart automated tests that can anticipate and cope with all the expected changes in the environment. I will write more about this topic in the future, but for now let's just say that recording is not smart at all.
• Create the test scripts
  This is the actual thing that recording addresses. It might seem that it is a very efficient way of creating test scripts, however in reality it is not. You will make errors while recording (like misclicks or mistypings). Correcting those errors in recorded scripts is most often difficult or even impossible.  Also if you limit yourself to just repeating recorded sequences, you will not benefit from all the good things that automation can offer compared with manual testing (like data-driven tests for example).
• Execute the scripts
  Or maybe I should say try to execute the scripts. My experience shows that unless you are testing something that has a very simple UI, recording tools often fail to create working scripts out of the box, even if you have managed to ensure absolutely consistent environment for the test execution. Recording lacks the intelligence to understand what you are doing and what your actual intentions are in a complex UI.
• Analyze the results
  In case you are executing recorded scripts, you will be hoping desperately that they pass successfully. In case something goes wrong (which will be most of the time), you will have extremely hard time trying to understand if it was a bug in the product, an issue with the test script or maybe a change in the application UI.
  The time spent in analysis of test results is probably the greatest time-waster in test automation and you can optimize this only by having proper logging on a higher semantic level than UI interactions. Again this is something that I intend to write about.
• Update your scripts
  Software changes and you will either have to adapt your tests or throw them away. With recording it is mostly about throwing away. It is true that most of the tools now provide some ways to cope with simple changes in the UI, but those are still very limited. For example if a caption of a button changes from 'New' to 'New...',  in most cases you will be able to adjust your scripts. With the good tools you will have to add the '...'  only in one place, no matter how many times you click the button in your test.  However,  imagine that this button is replaced by a context menu.  Most  tools will let you somehow change this in their recorded scripts, but you will have to do this change every single place where the button was clicked in your scenario. Most probably you will just want to re-record it from scratch.

If you plan to do any serious automation, do not base the choice of your tool on how well it handles record and playback. In my opinion writing tests using the scripting language (and doing it right) is the only way to make UI automation really work.
I find recording only useful when I am testing with a new UI technology or a new automation tool and I want to quickly find out how it recognizes  a particular UI control.  In such cases recording a few mouse clicks and then looking at the generated script can be useful.