Tag Archives: load

Load Testing: What Tool to Choose?

Classifying and evaluating load testing tools is not easy as they include different sets of functionality often crossing borders of whatever criteria are used. In most cases, any classification is either an oversimplification (which in some cases still may be useful) or a marketing trick to highlight advantages of specific tools. There are many criteria allowing to differentiate load testing tools and it is probably better to evaluate tools on each criterion separately.

First, there are three main approaches to workload generation and every tool may be evaluated on which of them it supports and how exactly.

Protocol-level recording and the list of supported protocols. Does the tool support protocol-level recording and, if it does, what protocols it supports. With quick Internet growth and popularity of browser-based clients, most products support HTTP only or a few Web-related protocols. According to my knowledge, only HP LoadRunner and Microfocus SilkPerformer try to keep up with support of all popular protocols. So if you need recording of a special protocol, you probably end up into looking at these two tools (unless you find a special niche tool supporting your specific protocol). That somewhat explains the popularity of LoadRunner at large corporations where you probably have almost all possible protocols used. The level of support of specific protocols differs significantly too. Some HTTP-based protocols are extremely difficult to correlate if there is no built-in support, so you may look for that kind of specific support. For example, Oracle Application Testing Suite may have better support of Oracle technologies.

UI-level recording. The option was available for a long time, but it is much more viable now. For example, there was a possibility to use Mercury/HP WinRunner or QuickTest Professional (QTP) scripts in load tests, but you needed a separate machine for each virtual user (or at least a separate terminal session). That limited the level of load you may achieve drastically. Other known options were, for example, Citrix and RDP (Remote Desktop Protocol) protocols in LoadRunner – which always were the last resort when nothing else was working, but were notoriously tricky to playback. New UI-level tools for browsers, such as Selenium, extended possibilities of the UI-level approach allowing to run multiple browser per machine (so scalability is limited by resources available to run browsers). Moreover, we got UI-less browsers, such as HtmlUnit, which require significantly less resources than real browsers. There are multiple tools supporting this approach now – such as PushToTest directly harnessing Selenium and HtmlUnit for load testing or LoadRunner TruClient protocol and SOASTA CloudTest using more proprietary solutions to achieve low-overhead playback. Still questions of supported technologies, scalability, and timing accuracy remain largely undocumented, so the approach requires evaluation in every specific non-trivial case.

Programming. There are cases when you can’t (or can, but it is more difficult) use recording at all. In such cases using API calls from the script may be an option. Other variations of this approach are web services scripting and using of unit testing scripts for load testing. And, of course, you may need to add some logic to your recorded script. You program the script using whatever way you have and use the tool to execute scripts, coordinate their executions, report and analyze results. To do this, the tool should have ability to add code to (or invoke code from) your script. And, of course, if tool’s language is different from the language of your API, you would need to figure out a way to plumb them. Tools, using standard languages such as C (e.g. LoadRunner) or Java (e.g. Oracle Application Testing Suite) may have an advantage here. However you should know all details of the communication between client and server that is often very challenging.

Other important criteria are related to the environment:

Deployment Model. There were a lot of discussions about different deployment models: lab vs. cloud vs. service. There are some advantages and disadvantage of each model. Depending on your goals and systems to test you may prefer one deployment model over another. But I still believe that for comprehensive performance testing you really need both lab testing (with reproducible results for performance optimization) and realistic outside testing from around the globe (to check real-life issues that you can’t simulate in the lab). Doing both would be expensive and makes sense when you really care about performance and have a global system – but it not rare and if you are not there yet, you can get there eventually. If there are such chances, it would be better to have a tool which supports different deployment models.

If it is lab or cloud, an important sub-question would be what kind of software / hardware / cloud the tool requires. Many tools use low-level system functionality, so is may be unpleasant surprises when the platform of your choice or your corporate browser standard is not supported.

Scaling. When you have a few users to simulate, it usually is not a problem. The more users you need to simulate, the more important it becomes. The tools differ drastically on how many resources they need per simulated user and how well they may handle large volumes of information. It may differ significantly even for specific tool depending on protocol used and specifics of your script. As soon as you get to thousands of users, it may become a major problem. For a very large number of users some automation, like automatic creation of a specified number of load generators across several clouds in SOASTA CloudTest, may be very handy.

Two other important sets of functionality are monitoring of the environment and result analysis. While theoretically it is possible to do it using other tools, it significantly degrades productivity and may require building some plumbing infrastructure. So while these two areas may look optional, integrated and powerful monitoring and result analysis are very important. And the more complex system and tests, the more important they are.

Of course, non-technical criteria are important too:

Cost. There are commercial tools (and license costs differ drastically) and free tools. And there are some choices in between: for example SOASTA has the CouldTest Light edition free up to 100 users. There are many free tools (some, as JMeter, are mature enough and well-known) and many inexpensive tools, but most of them are very limited in functionality.

Skills. Considering a large number of tools and a relatively small number of people working in the area, there is a kind of labor market only for the most popular tools. Even for the second-tier tools there are few people around and few positions available. So if you don’t choose the market leaders, you can’t count that you find people with this tool experience. Of course, an experienced performance engineer will learn any tool – but it may take some time until productivity will get to the expected level.

Support. Recording and load generation has a lot of sophistication in the background and issues may happen in every area. Availability of good support may significantly improve productivity.

This is, of course, not a comprehensive list of criteria – rather a few starting points. Unfortunately, in most cases you can’t just rank tools on the better – worse scale. It may be that a simple tool will work quite well in your case. If your business is built around a single web site, it doesn’t use sophisticated technologies, and load is not extremely high – almost every tool will work for you. The further you are from this state, the more challenging it would be to pick up the right tool. And it even may be that you need several tools.

And while you may evaluate tools with above mentioned criteria, it is not guaranteed that a specific tool will work with your specific product (unless it uses a well-known and straightforward technology). That actually means that if you have a few system to test, you need to evaluate the tools you consider using your systems and see if the tools can handle them. If you have many, choosing a tool supporting multiple load generation options is probably a good idea (and, of course, check it with at least the most important systems).

Source:

http://alexanderpodelko.com/blog/2012/05/10/load-testing-what-tool-to-chose/

Did you like this? Share it:

Smarter Than The Average Load Testing Bear

SmartBear Software has this week introduced loadUI Pro and upgraded its soapUI product to form a duo of testing software with the ability to monitor server side performance. Targeting testers working to refine API’s such as web services, loadUI Pro is a commercially charged version of loadUI, SmartBear’s open source load test solution for APIs.

 

 

In light of the mix of externally sourced web services and API protocols employed by today’s "complex composition applications", SmartBear suggests that its product is needed to perform "meticulous functional and load testing analysis" during both pre-deployment, throughout application lifecycle, and onward into regression testing post-deployment.

In operation, loadUI Pro generates scalable, high-volume, real-world load from a number of local and remote computers to help assess the impact felt on server load in terms of CPU and RAM usage. From its reports, the product should allow testers to channel information on database performance and resource utilization back to developers — it will also help identify the root causes of bottlenecks "anywhere" in the server stack.

Version 4.5 of soapUI has new test debugging functions to help streamline test creation. It also benefits from "improved assertion" to simplify the test validation process. Multi-environment support is available to enable quick changes between target test environments.

The new Test On Demand feature enables users to extend testing beyond the desktop and network by running tests from two external locations in SmartBear’s monitoring cloud for free. Additionally, soapUI 4.5 provides an improved experience for Mac users and addresses several Mac OS X issues.

"In the open source community, loadUI and soapUI have proven themselves as powerful, free tools helping developers to test and improve their code every step of the way. These releases not only make testing more streamlined and easy to administer but also provide a new level of insight so that roadblocks to quality become more apparent and easier to resolve pre-production," said Niclas Reimertz, GM of API testing solutions at SmartBear.

Source:

http://drdobbs.com/testing/232800240

Did you like this? Share it:

Heavy Load Testing

Visual Studio Team Agents 2010 consists of Test Load Agent, Test Load Controller and Lab Agent. Team Test Load Agent monitors a request for a new test from the controller by running a service locally. Team Test Load Controller is run by a service, and the service controls the test agents and reports the status and errors of the test. It can also transfer the sources into running test and data collection in logic, or has effect on the system running the testing environment again.

The main function of Test Load Agent is to cascade various machines to produce heavier load and higher stress when doing the load test. It is rarely seen that an average level PC can produce a load of 50 people, which still cannot compare with some great websites or systems with thousands or more people online. Thus, enough load and stress can be produced when we connect several server level machines, so that the results will be meaningful to us.

More than 3 computers are required when using Load Agent, and VS 2010 For Tester, Controller, Agent1, Agent2, etc. are supposed to be installed. If the number of computers is limited, you can install the Controller and Agent in the same computer, but the test will be limited, as the picture below shows.

clip_image002

As the picture above shows that, Team Agents include the Agent and Controller software, which are highly extendable and customizable, so that the testers enjoy great flexibility. Testers can do load testing directing at Web apps and organizations to improve service quality by measuring the performance of Web apps and servers under load more accurately.

Application & Practice: to modify and test x64 bits CLR program set with VS 2010

Visual Studio Team Test 2010 Load Agent testing platform provides host process for the test. You can enjoy the new feature that network emulation has been introduced for performance testing, and what’s more, Visual Studio 2010 has developed another feature, which is that it can test the program set in 64bits CLR. With Visual Studio 2008, the host process can only be tested in the 32bits mode, and the internal storage has been limited to 2GB. But with Visual Studio Team Test 2010, there is no such limit.

Here are the steps to modify and test x64 bits CLR program set with VS 2010.

1) Set your program set as opening your program set on “Any CPU” platform, and open the “Build” tab, then set your platform target as “Any CPU”.

clip_image004

2) Open test settings, and set the host process platform as “MSIL”. Open the “Local.testsettings” to set and then choose “Hosts” and at last choose the MSIL under the Host Process Platform options.

clip_image006

Tips:

If you have not installed Test Load Agent, you will find that only one CPU will be occupied by the local requests when doing performance test with VS 2010. It is because the process to generate stress is VSTESTHost.exe. Accordingly, if you want every CPU of a multinuclear CPU to produce stress, you have to use Visual Studio Team System Test Load Agent.

Did you like this? Share it:

Load Testing Tool: LoadComplete 2.0

SmartBear Software , a provider of software development, testing, and monitoring tools, has released LoadComplete 2.0, the latest version of SmartBear’s load-testing tool for web and rich Internet applications (RIAs). Following are highlights of the new features in LoadComplete 2.0:

  1. Support for load testing RIAs and associated protocols and formats. LoadComplete 2 supports load testing of web applications built with Adobe Flash, Flex, Ajax, and Silverlight, and supports the Action Message Format (AMF), XML (SOAP) and binary XML, and JSON web protocols and data formats. Additionally, LoadComplete can now decode data from server responses and encode values into further requests. Users can view, change, or parameterize traffic test data using the graphical tree view.
  2. Enhanced point-and-click automation for parameterizing test data in AMF, JSON, SOAP, XML, and Silverlight data formats. An improved Data Selectors feature now lets you insert data not only into URL parameters or into parameters of POST multipart/form-data or form-url encoded requests, but also into an arbitrary place in a body of any subsequent request.
  3. Higher load-generation capacity, enabling 25 percent more virtual users per node. LoadComplete Remote Agent now supports simulating up to 400 virtual users on a computer node.
  4. Enhanced reporting capabilities. The new "waterfall" graphs, Slow Pages and Slow Pages (Average) graphs in the Report Top 10 tab, now include detailed execution time of pages’ requests. You can use this information to analyze the order and duration of page requests and find performance bottlenecks. Additionally, you can now export test results to PDF or MHT (HTML) files or send them to the printer directly from the Report panel.
  5. Logging complete requests and responses. LoadComplete now saves the contents of all simulated requests and the contents of responses to these requests received from the server, displaying this information in two new test log panels: Request Body and Response Body.
  6. Enhanced support for ASP.NET applications. LoadComplete 2 improves built-in support for .axd file requests to eliminate the need for any manual data manipulation.

Source: http://www.devproconnections.com/blog/net-framework-blog-31/software-testing/load-test-loadcomplete-141578

Did you like this? Share it:

LaodUI – a free and open source laod testing tool for load testing

LoadUI is a free and open source load testing tool that allows you to do complex load tests and test the performance by simply dragging the different components around. LoadUI consist of building blocks called Components that you drag from the Component Toolbar to the Project you’re working on. Moreover, these components are connected by wires and what’s more there’s no limit for how many components that you can create and connect. Following are pro tips of loadUI:

1. Let a manager test

2. It’s interactive, use it

LoadUI is highly interactive and configurable in real-time, so take advantage of that.

3. Record your tests

Reproducing the tests is difficult as loadUI is too interactive. Hence, using a screen recorder to be able to replay your tests and see what caused the service to crash.

4. Work sturctured

LoadUI has support for test cases.

5. Don’t double-work

If you have created functional tests in soapUI, you can simply run these in loadUI with the soapUI component.

6. Write your own component

LoadUI is open-source, which means you can write your own components, in Groovy, if you wish.

Source: http://www.softwaretestinghelp.com/load-testing-using-loadui/

Did you like this? Share it:

Motherboard testing software

If a straightforward restart of your Computer doesn’t work out the kink, your dilemma might lay below the hood with an outdated or corrupt process driver Motherboard testing software. These Asus motherboard drivers are necessary to your laptop or computer running effectively at all times. Moreover, restrict the interruptions you may perhaps confront Motherboard testing software when doing work adequately. Motherboard testing software is really straightforward to see that if an Asus motherboard driver is out of whack. The most effective bang for your buck is to get driver update software program Motherboard testing software. It can be daily life saving for some people.

Source: http://www.djsingleagain.net/motherboard-testing-software.htm

Did you like this? Share it:

FAQ: virtual testing and development environments

Beyond all question, a virtual testing environment is the perfect place to test applications, operating systems and hardware before you place them into production. Virtual testing environments are segregated from production and often use free, simple virtualization tools. So it provides a safe, efficient and cost-effective way to test apps and other services. However, there are some frequently asked questions about virtualization test envirnments and their answers:

1. How can I use a virtual test lab for software testing?

You can make a virtual test lab as large or small as you want. To test applications, you can emulate different computers with different Oses on a single physical server. You can also test them on various platforms and using different configurations.

2. Should I use clustering in a virtual testing environment?

You can increase availability by creating a virtual server cluster in your virtualization test lab. But remember, building a server cluster can incur extra costs if you need more hardware, licenses or shared storage.

3. Can I run test servers and production servers on the same host?

You should never colocate virtual testing and production servers. Separating the virtual test lab makes it easier to manage test VMs and build clusters. Also, you can set rules to separate production and nonproduction VMs at the hypervisor level.

4. What are some challenges I’ll face with virtual testing?

When you virtualize a test environment, you need to make sure you have virtual testing management tools that integrate with your existing infrastructure. You may also have to upgrade hardware to work with a virtual testing environment.

5. What are some good tools for virtual testing environments?

Oracle VM VirtualBox 4, Vmware Lab Manager, and Vmware Workstation are all good tools for virtual testing environmetns.

6. How do I build a Vmware lab for virtual testing?

You can use hosted hypervisors to creat a Vmware lab, and you can run ESX or ESXi as guest VMs on a home computer while still using it for others tasks.

7. What is Vmware Labs, and how does it facilitate virtual testing?

Vmware Labs is the company’s online test and development site. Vcenter Mobile Access is a virtual appliance that provides a webpage-like interface for a mobile phone. And a fling called XVP Manager lets vCenter manage Microsoft Hyper-V hosts and VMs. Some of Vmware’s flings may turn into real products or features.

Source: http://searchservervirtualization.techtarget.com/tutorial/FAQ-Virtual-testing-and-development-environments

Did you like this? Share it:

Web testing with WAPT

While manual testers fail to test websites for performance, there are two reasons: they don’t have proper tools and they don’s have required skills for performance testing. Then how to succeed? The first thing you need to do is to ask for required tools and train your staff for necessary skills. The tool we will talk about here is called WAPT, which is short for Web Application Load, Stress and Performance Testing. It is a cost effective and easy to learn tool. It has some pros. It is easy to install, and easy to use with very short learning curve. Also you get run-time reports so that you can decide whether to continue the test or not. Meanwhile, it provides detailed test report with graphical representation, supports secure HTTPS protocol, and have 30 days free trial available.

Source: http://www.softwaretestinghelp.com/web-application-load-stress-and-performance-testing-using-wapt/

Did you like this? Share it:

Difference between Performance Testing, Load Testing and Stress Testing

Performance testing is to ascertain how the components of a system are performing in a particular situation. Load testing is to test the system by constantly and steadily increasing the load on the system till the time it reaches the threshold limit. And as for stress testing, various activities to overload the existing resources with excess jobs are carried out in an attempt to break the system down.

Performance testing mainly aims to estabilsh the benchmark behaviour of the system, while load testing are to expose the defects in application related to buffer overflow, memory leaks and mismanagement of memory and stress testing is to analyse post-crash reports to define the behaviour of application after failure.

Source: http://www.softwaretestinghelp.com/what-is-performance-testing-load-testing-stress-testing/

Did you like this? Share it:

Software Testing – Black Box Testing Strategy

Black Box Testing is not a type of testing; it instead is a testing strategy, which does not need any knowledge of internal design or code etc. As the name “black box” suggests, no knowledge of internal logic or code structure is required. The types of testing under this strategy are totally based/focused on the testing for requirements and functionality of the work product/software application. Black box testing is sometimes also called as “Opaque Testing”, “Functional/Behavioral Testing” and “Closed Box Testing”.
The base of the Black box testing strategy lies in the selection of appropriate data as per functionality and testing it against the functional specifications in order to check for normal and abnormal behavior of the system. Now a days, it is becoming common to route the Testing work to a third party as the developer of the system knows too much of the internal logic and coding of the system, which makes it unfit to test the application by the developer.
In order to implement Black Box Testing Strategy, the tester is needed to be thorough with the requirement specifications of the system and as a user, should know, how the system should behave in response to the particular action.
Various testing types that fall under the Black Box Testing strategy are: functional testing, stress testing, recovery testing, volume testing, User Acceptance Testing (also known as UAT), system testing, Sanity or Smoke testing, load testing, Usability testing, Exploratory testing, ad-hoc testing, alpha testing, beta testing etc.
These testing types are again divided in two groups: a) Testing in which user plays a role of tester and b) User is not required.

Testing method where user is not required:

Functional Testing:
In this type of testing, the software is tested for the functional requirements. The tests are written in order to check if the application behaves as expected.
Stress Testing:
The application is tested against heavy load such as complex numerical values, large number of inputs, large number of queries etc. which checks for the stress/load the applications can withstand.
Load Testing:
The application is tested against heavy loads or inputs such as testing of web sites in order to find out at what point the web-site/application fails or at what point its performance degrades.
Ad-hoc Testing:
This type of testing is done without any formal Test Plan or Test Case creation. Ad-hoc testing helps in deciding the scope and duration of the various other testing and it also helps testers in learning the application prior starting with any other testing.
Exploratory Testing:
This testing is similar to the ad-hoc testing and is done in order to learn/explore the application.
Usability Testing:
This testing is also called as ‘Testing for User-Friendliness’. This testing is done if User Interface of the application stands an important consideration and needs to be specific for the specific type of user.
Smoke Testing:
This type of testing is also called sanity testing and is done in order to check if the application is ready for further major testing and is working properly without failing up to least expected level.
Recovery Testing:
Recovery testing is basically done in order to check how fast and better the application can recover against any type of crash or hardware failure etc. Type or extent of recovery is specified in the requirement specifications.
Volume Testing:
Volume testing is done against the efficiency of the application. Huge amount of data is processed through the application (which is being tested) in order to check the extreme limitations of the system.

Testing where user plays a role/user is required:


User Acceptance Testing:
In this type of testing, the software is handed over to the user in order to find out if the software meets the user expectations and works as it is expected to.
Alpha Testing:
In this type of testing, the users are invited at the development center where they use the application and the developers note every particular input or action carried out by the user. Any type of abnormal behavior of the system is noted and rectified by the developers.
Beta Testing:
In this type of testing, the software is distributed as a beta version to the users and users test the application at their sites. As the users explore the software, in case if any exception/defect occurs that is reported to the developers.

source: http://www.buzzle.com/editorials/4-10-2005-68349.asp

Did you like this? Share it: