Failed Automation Projects

May 13th, 2009 § 2 comments

"Do not automate buggy software."

I was thinking about the failed automation projects I’ve been involved in during the last few years and I seem to find a pattern. All those projects were very, very buggy. I realize now that you should only aim to automate functionality which is fairly stable, so that you do not expect to see a test failure in more than about 1 of 10 test runs. If you see that your tests are failing 50% of the time, then automation is not the cure for your problems. Try to find out why this is happening in the first place.

 

HP’s Center of Excellence

April 22nd, 2009 § no comments yet

Centralization leads to mediocrity at best.

center

Today I went to a seminar, organized by HP to promote their offerings in the quality management area. What struck me most was the 'Center of Excellence' notion.  Basically the idea is that (big) organizations should centralize their quality-related activities into one unit (boldly named Center of Excellence), in order to reduce costs.  HP is willing to support you in this endeavor with software and consulting. They even came up with their own maturity model which ranks organizations on one of several levels - the lowest called for some reason 'Manual Testing' and the highest  of course being 'Center of Excellence'.

Somehow I don’t buy into this.  I realize that cost saving measures are in vogue now, but my experience shows that centralization leads to mediocrity at best, not excellence.

The Forrester Wave™ – Functional Testing Solutions

April 11th, 2009 § no comments yet

Opensource might actually be better.

waveIn case you are in the process of evaluating test tools, this review by Forrester research might give you a good initial overview of the current commercial offerings - The Forrester Wave™: Functional Testing Solutions, Q3 2008

It looks at solutions that support the whole testing and quality management process and the conclusion is not surprising:

"In today's market, HP is the default choice"

I find the whole thing objective and if you read it carefully you will be able to find the key facts about the automation tools.
The evaluated vendors and the corresponding tools are HP (ex Mercury) with QuickTest Professional, IBM (ex Rational) with Rational Functional Tester, Compuware with TestPartner, Borland (ex Segue) with SilkTest plus also tools from Empirix and Seapine.

The review has its main emphasis on test management and not on test automation:

"Testing teams looking to purchase just a single product should focus on test management first, since mature practices in this area are a prerequisite for success with test automation."

While this sounds logical at first, I somehow can not agree with it entirely. But what follows is really interesting:

"There are exceptions, though: most notably, development teams looking to automate their own regression testing. Commercial test automation tools, however, tend to be less effective here than open source alternatives like FIT/FitNesse or Selenium."

and:

"Both open source testing tools and SOA testing tools are relatively small parts of the functional testing tools market, but they’re growing in importance. Why? Because iterative development methodologies demand that test automation start earlier in the life cycle, and the testing organizations that purchase commercial tools struggle to make this a reality."

It seems that the analysts that wrote this really have some insignt in the business. Overall it is a very interesting read. Did you know that the element Selenium is the antidote to Mercury poisoning?

Lesson 114 – Test tools are buggy

April 9th, 2009 § no comments yet

"Testers get indignant when they realize that the quality tools purchased at a serious price are themselves full of bugs. Indeed, test tools are often buggier than comparable (but cheaper) development tools. Plan to test your tools and spend time finding workarounds for bugs."

The quote comes from one of the best books on software quality I have read - Lessons Learned in Software Testingby Cem Kaner, James Bach and Bret Pettichord.  Be sure to read it if you haven't already.
I can not agree more with the authors. I spend a lot of my time in test automation trying to find workarounds for issues with the tool. Some of those issues can be considered 'functional idiosyncrasies', but most are just plain and simple bugs. During my work in the field of software quality I have noticed that users are normally quite tolerant of bugs, unless the bugs block them from doing their jobs. So for me there are two kinds of bugs in the test tools - those you can find a workaround for and those you can not.
If the tool is flexible enough and you know how to use it, most of the time you can make it do what you want - like for example using keyboard shortcuts and the clipboard to read the text from an unsupported GUI control. This is one advantage of tools that offer the flexibility of a real programming language for scripting.
There are also such things, for which you can not do much except waiting for the vendor to provide you a fix, and this can take long time. A few years ago I tried to upgrade to the latest TestPartner version (it was 5.2.3 back then). After the upgrade I found that all my automated scripts were causing random crashes in Internet Explorer. I just stayed with the old version until 5.6 came out, where this problem was fixed.

TestPartner 6 Review – Visual Tests Redux

April 6th, 2009 § no comments yet

Recording does not work, even with IE

 I realized that I was not fair in my previous test,  as I used the last version of Firefox, which unfortunately is not officially supported by TestPartner. To give it a second chance, I decided to repeat the test with Internet Explorer.  I re-recorded the same scenario, but this time in IE and tried to replay it.

 I got a better result, but only slightly. The recorded script was able to log-in, but could not click the button to create a new mail:

failed-in-ie

The playback failed at step 12, where the tool tries to ‘Attach’ to an HTML frame, identified with ‘Name=c36gxk9p2bo3a’. Attaching to a UI control in TestPartner means simply declaring that the subsequent actions in the script will be executed within this control.  c36gxk9p2bo3a is a generated name which changes each time when you enter GMail.  This was the reason the script could not find the frame, when trying to replay.

 If I had to code this script by hand, I would omit this step altogether. Usually it is sufficient to only attach to the browser window and not to specific frames within it. Also, I would not locate the other controls by ID and screen coordinates as the recording did in steps 13 and 14. This will likely fail too, even if the problematic step 12 is fixed.

TestPartner 6 Review, Part I – Visual Tests

April 3rd, 2009 § 1 comment

Yes, it is about capture/replay again.

spanersI decided to start my automation tool reviews with TestPartner 6  for two reasons - the first is that it is the UI automation tool that I am currently using on my project.  The second reason is because the 'Visual Tests'  (also called 'Visual Navigator') feature, introduced with the last major release of TestPartner, supposedly "allows application users—who are intimately familiar with the application under test, but often lack the technical expertise to use automated testing tools—to comfortably and confidently capture business transactions, verify the proper functionality is tested and assist in updating the test when inaccuracies are found or the application changes". The quoted text comes from the product review on the site of Compuware. With TestPartner 6 they are basically trying to catch up with the competition, because HP/Mercury QuickTest Professional (QTP) had this feature first - you record your UI actions and what is created is not code, but descriptive and supposedly less frightening to non-technical people list of entries in a table.

I do not actually use this feature and if you have read my previous post you know why, but I came up with this simple scenario to test how it works overal:

  1. Open Gmail
  2. Log-in
  3. Go to 'Compose Mail'

  4. Enter some text in the mail body
  5. Center the text, using the toolbar button
  6. Click ‘Save Now’ to save the email as draft
  7. Go to the 'Drafts' folder
  8. Click on the draft email you just created
  9. Use the features of the test tool to check that the contents
of the text in the email body are the same as entered
  10. Discard the draft message
  11. Log-out

I decided to use Gmail as a test application, because I wanted to challenge the tool a little by using some AJAX instead of static HTML and also because almost everyone is "intimately familiar" with Gmail.  The scenario is very short and I think that it is reasonable to expect that automation tools should be able to handle well this level of complexity.

I recorded the sequence with TestPartner.  In the screenshot below you will see how the recorded script looks like:

Recorded Script

(click image to enlarge)

It basically borrows from QTP a  lot – for each step of the recording you see a ‘Screen Preview’. The steps are shown as a sequence in the so called ‘Storyboard’ and the currently selected step is shown enlarged in the preview. This is not just a screenshot, but contains the metadata for the UI objects in the application at this particular moment.  It is a little like having the real application in front of you at each step and can be nice in case you want to change something little in the middle of a long test.

To the right of the preview you will see the table with the recorded test steps. The descriptions are verbose like Click the text ‘Sign In’.
Using the ‘Logic Toolbox’  to the right you can insert control flow constructs  like loops and If statements, using a series of wizards. You can also insert checks to verify things (like in step 9 in my sequence) and add additional UI interactions.

I am a little skeptical about the whole idea, because to me it seems that if you can handle this king of wizard-assisted programming, then you can also learn to write code and in the long run you will be happier in terms of your efficiency if you do so.

I wanted to replay the script without any changes in the test environment first.  I expected this to work, but afterwards wanted to see if the recorded script would manage to execute successfully if I have other draft emails in the folder (with other subjects and body texts). My intention was to check how well the tool chooses which properties of the UI elements to use for their identification.

Unfortunately TestPartner could not replay the recorded steps successfully, even without changes in the mailbox. It also failed very early - at the log-in phase:

playback_error

s

The reason for this is very simple.  If you look at the screenshot above, you will see that in step 9 the tool recorded that I clicked the text 'Sign in'. Unfortunately on the same screen there is another 'Sign in' text,  located above the button. When replaying it just clicks this text instead of the button:
two_texts
Normally the tool should have identified that you are clicking a button, not just any text in the page. If this action was recorded as a button click, the replay would not get confused there.  As a whole I was not impressed with the visual record & replay capability of TestPartner. If you really want something like this, you will probably be better off with QTP.
I think that the strengths of TestPartner lie elsewhere.  Read about this in the second part of my review.

Record/Playback Does Not Work…

March 16th, 2009 § 2 comments

Record/Playback does not work, but we all know this, don't we?


877158_old_tape

If you have ever had anything to do with UI test automation, you should know about recording.  It is an ability that most tools (commercial or open source) have - you start the recording session in the tool and then just happily click through the test scenario on the application you are testing.  Meanwhile the tool sits silently and records your actions, getting ready to replay them automatically when you want to. The ability to record the UI interactions is a marketing highlight for most automation tools. There are even some tools, where recording is the only way you can create test scripts.

On the surface it can really look like the best way to automate - you will be testing manually anyway,  so why don't just record your tests to replay them at a later stage? Unfortunately the view that test automation is something as simple as pressing the record button has spread widely, mostly fueled by the hype of the tool vendors.

If you do not have any hands-on experience with test automation, you might miss a very important fact about the whole thing - it is not only about creating the  scripts.  In fact the actual script creation rarely takes more than 20-30% of the time you invest in automation. Here is a rough outline of all the things you have to do:

  • Prepare the test environment
    With automation you want to have a very consistent environment, with known data and configuration, where you execute your tests. Sometimes this is not possible  - maybe  you have to test in a live system, or you don't control every part of the system landscape.  In such cases you can succeed only if you create smart automated tests that can anticipate and cope with all the expected changes in the environment. I will write more about this topic in the future, but for now let's just say that recording is not smart at all.
  • Create the test scripts
     This is the actual thing that recording addresses. It might seem that it is a very efficient way of creating test scripts, however in reality it is not. You will make errors while recording (like misclicks or mistypings). Correcting those errors in recorded scripts is most often difficult or even impossible.  Also if you limit yourself to just repeating recorded sequences, you will not benefit from all the good things that automation can offer compared with manual testing (like data-driven tests for example).
  • Execute the scripts
    Or maybe I should say try to execute the scripts. My experience shows that unless you are testing something that has a very simple UI, recording tools often fail to create working scripts out of the box, even if you have managed to ensure absolutely consistent environment for the test execution. Recording lacks the intelligence to understand what you are doing and what your actual intentions are in a complex UI.
  • Analyze the results
    In case you are executing recorded scripts, you will be hoping desperately that they pass successfully. In case something goes wrong (which will be most of the time), you will have extremely hard time trying to understand if it was a bug in the product, an issue with the test script or maybe a change in the application UI.
    The time spent in analysis of test results is probably the greatest time-waster in test automation and you can optimize this only by having proper logging on a higher semantic level than UI interactions. Again this is something that I intend to write about.
  • Update your scripts
    Software changes and you will either have to adapt your tests or throw them away. With recording it is mostly about throwing away. It is true that most of the tools now provide some ways to cope with simple changes in the UI, but those are still very limited. For example if a caption of a button changes from 'New' to 'New...',  in most cases you will be able to adjust your scripts. With the good tools you will have to add the '...'  only in one place, no matter how many times you click the button in your test.  However,  imagine that this button is replaced by a context menu.  Most  tools will let you somehow change this in their recorded scripts, but you will have to do this change every single place where the button was clicked in your scenario. Most probably you will just want to re-record it from scratch.

If you plan to do any serious automation, do not base the choice of your tool on how well it handles record and playback. In my opinion writing tests using the scripting language (and doing it right) is the only way to make UI automation really work.
I find recording only useful when I am testing with a new UI technology or a new automation tool and I want to quickly find out how it recognizes  a particular UI control.  In such cases recording a few mouse clicks and then looking at the generated script can be useful.

Newer Entries »