Posted by: Bernard Lelchuk | May 13, 2010

SoapUI: Breaking Validation using Load Testing Tools


Anyone in the SQA industry have been involved or at least heard a thing or two on Load Testing.
Such tools assist a SQA Engineer in his journey to find & analyze system bottlenecks, Performance, DB locks, system crashes & various related failures.
This is really nifty when you focus on Load Testing as we know it.

In this post, I’d like you to take a step backwards and think out-of-the-box for various other uses of such tools (and others).
Think not what such tools are meant to do (aka “The Result”) but rather think on the Actions they perform & how they relate & affect one another and off course the final result.

If we’ll break load testing tools to general action types we’ll get the following basic elements:

  • Running a sequence of scripted/recorded actions
    • In depended actions – E.g. Login, Browse, Send Mail, Delete File
    • Depended actions – E.g. CRUD – Create, read, update and delete
    • Triggering Actions – E.g. Add Item -> Creates new branch -> Triggers new sync action -> Triggers SMTP action, etc.
  • Simulate various load scenarios
    • Number of Virtual Users/Threads
    • Number of Actions per user
    • Load structure – Burst, Ramp-up/down, Pulse
    • Wait time between threads
    • Any combination of these parameters & actions

Once again focusing on the elements & actions above rather than the overall load testing concept might raise some interesting scenarios – which will lead you to hard-to-find failures.

  • Multiple Login actions
  • Multiple Logout actions
  • Multiple CRUD actions
  • Multiple Transaction (And Purchases) actions
  • Multiple Data validity actions

Simulating any of these set of actions alone will reveal some of your system’s failures.
However manipulating the load of the same actions may result in unexpected (and many times hidden) failures.
Now, failure does not necessarily means system crashes & performance issues – this could be even validation failure and I’ll explain bellow.
Let’s say the server is configured to receive a set of data and add a new record in the DB correspondingly.
On the GUI side there are validations of UNIQUE records that should deny any duplication/similar action.
Such validation also resides on the server (which is hidden from the end-user/tester).
Tester can simulate a set of actions which their result should end with record duplication.
This can be performed in two ways:

  1. Using the GUI interface
  2. Sending requests directly to the server (using direct links or web services)

If you’ll try to run any of these scenarios rapidly in search for duplicated records, you’re most likely to fail to do so.
There are 2 main reasons for that:

  1. You’re just not fast enough.
  2. You cannot repeat the EXACT steps over & over again manually.

The only cure for this is an automation tool that can assist here!
This is where you approach your available load test tool for assistance. Try repeating an action using different load profiles. You’ll be surprised by the diversity of the results upon various load profiles.

Here I’ll give you a real-life experience from years back with such a case.
In one of my tests I’ve noticed that one the unique records have been duplicated in the system. I’ve repeatedly tried to reproduce this behavior manually however could not break the server’s validation.
I then tried using an automated script to do so – still with no success.
It was only when I used SOAPUI’s Load Engine (you can use any similar tools per your preferences) for sending a request to the server that managed to break its validation and create a duplicate of that unique record. First phase was complete – I managed to reproduce it, however it was not reproducible every time, therefore I could not report a specific scenario for this failure.

I started tweaking the load engine’s parameters trying a wide spectrum of combination. As I was doing so I’ve noticed that I managed to break the validation more often, but yet could not reproduce it on a single run. I then figured that only at a specific load scenario the server fails to deal with my requests gracefully thus breaks its validation.
Actually there were two validations which broke here:

  • If Record is NULL then only UNIQUE NEW records should be created
  • Once a unique record is created – all similar record creation requests should be DENIDED.

After lowering a bit the wait response, increasing the number of thread/VU & adding several actions per VU I managed to find a specific load scenario that in a single run reproduce the validation break over & over again.
I then happily wrapped my project and filed a report with full explanation of my steps.

Try using load tools to stress your system and break its validation. Think on all those hard-to-find bugs which you could never reproduce however they kept showing here and then. There’s a large chance you’ll manage to do so with the correct actions & load profile.

Do you have similar experience? Share it here!

Cheers,
Bernard.

Advertisements

Responses

  1. @Bernard,
    Cool post. You are right about using different tools for different purpose even if the purpose of that tool is different. I have an example about how Performance Testing or Load Testing tool can behave as dDoS attack tool while I was chatting with Glory ( rdtx2005 – uTest forums username ) on uTest forums chat canvas.

    Thanks for sharing this post. I have to circulate this post to educate some of the testers who search with queries only like “Database testing tool” and etc. If they understand what they really require then they can do better searching.

    Thanks,
    Santhosh Shivanand Tuppad

    • Thanks Santhosh!
      I’d love to hear about your load testing experience as a dDoS attack tool.

      Cheers,
      Bernard.


What do YOU think about this post?

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Categories

%d bloggers like this: