Tag Archives: hardware

Performance Testing vs Load Testing vs Stress Testing – Examples

Performance testing – It is performed to evaluate the performance of components of a particular system in a specific situation. It very wide term. It includes: Load Testing, Stress Testing, capacity testing, volume testing, endurance testing, spike testing, scalability testing and reliability testing etc. This type of testing generally does not give pass or fail. It is basically done to set the benchmark & standard of the application against Concurrency / Throughput, Server response time, Latency, Render response time etc. In other words, you can say it is technical & formal evaluation for responsiveness, speed, scalability and stability characteristics.

Performance Testing vs Load Testing vs Stress Testing - Examples

Load Testing is subset of performance testing. It is done by constantly increasing the load on the application under test till the time it reaches the threshold limit. The main goal of load testing is to identify the upper limit of the system in terms of database, hardware and network etc. The common goal of doing the load testing is to set the SLAs for the application. Example of load testing can be:

Running multiple applications on a computer simultaneously – starting with one application,Running multiple applications on a computer simultaneously – starting with one application, then start second application, then third and so on….Now see the performance of your computer.

Endurance test is also a part of load testing which used to calculate metrics like Mean Time Between Failure and Mean Time to Failure.
Load Testing helps to determine:

  • Throughput
  • Peak Production Load
  • Adequacy of H/W environment
  • Load balancing requirements
  • How many users application can handle with optimal performance results
  • How many users hardware can handle with optimal performance results

Stress testing – It is done to evaluate the application’s behaviour beyond normal or peak load conditions. It is basically testing the functionality of the application under high loads. Normally these are related to synchronization issues, memory leaks or race conditions etc. Some testing experts also call it as fatigue testing. Sometimes, it becomes difficult to set up a controlled environment before running the test. Example of Stress testing is:

A banking application can take a maximum user load of 20000 concurrent users. Increase the load to 21000 and do some transaction like deposit or withdraw. As soon as you did the transaction, banking application server database will sync with ATM database server. Now check with the user load of 21000 does this sync happened successfully. Now repeat the same test with 22000 thousand concurrent users and so on.

 

 

Spike test is also a part of stress testing which is performed when application is loaded with heavy loads repeatedly and increase beyond production operations for short duration.

Stress Testing helps to determine:

  • Errors in slowness & at peak user loads
  • Any security loop holes with over loads
  • How the hardware reacts with over loads
  • Data corruption issues at over loads

source: http://www.softwaretestingstuff.com/2011/09/performance-testing-vs-load-testing-vs.html

Did you like this? Share it:

The Shocking toll of Hardware and Software Fragmentation on Android Development

image

Android fragmentation is a huge problem. The fact that there are hundreds of different hardware devices running over half a dozen different versions of Google’s OS makes it annoying for users, but makes it an especially devastating issue for developers trying to make a business out of the Android ecosystem.

This was highlighted by the recent release of Temple Run on the Android platform. A previously (very) successful game on iOS, it was brought over to Android in order to take advantage of the huge number of devices that run the OS. And it has already hit 1 million downloads in just 3 days, good, even for a free app. But very quickly, the developers of the app discovered the pitfalls of fragmentation:

99.9% of support emails are complaining their device isn’t supported. We currently support 707 devices. Mindblowing.

— Natalia Luckyanova (@nattylux) March 27, 2012

Natalia Luckyanova of Imangi tweeted out that most of their 1200 emails over the past 12 hours of release had been devoted to the fact that the app just didn’t work on one of the hundreds of devices owned by those users.

But then Developer David Smith replied to her, saying that he had some 1443 unique devices on the logs for his Android app Audiobooks (also Free). We contacted Smith to chat with him a bit about whether the fragmentation of Android software and devices had affected the process of making software for the platform. What he shared with us was shocking.

Smith confirmed for us that his Audiobooks app has been run on 1443 different Android devices by its users. This makes it absolutely impossible to determine whether an app will run without problems for all of your customers. To drive home how ridiculously shattered the Android landscape is, check out this list of the most used single devices based on 1.3M downloads of his app:

  1. Droid X (7.8% of users)
  2. Samsung Galaxy S2 (4.3%)
  3. Droid (4%)
  4. HTC Desire HD (4%)
  5. HTC Evo 4G (3.7%)
  6. Droid incredible (2.3%)

This is insanity when you look at it from the standpoint of an iOS developer, who has to support only a handful of hardware varietals. Of course, Android’s very nature causes this. It is designed to work on an enormous array of hardware with all kinds of different components and feature sets. The way Android is designed to be used means that developers will likely always have to deal with this kind of fractured hardware landscape, it’s a fact of life on the platform.

testingdevices 520x365 The shocking toll of hardware and software fragmentation on Android development

Smith says that with just 8-10 iOS devices (the pic above is from last summer and shows a dozen), he can cover 100% of his users. There’s just no way to keep every version of hardware around for quality assurance on every permutation of Android. “I have a handful of Android devices, but since the range is so spread out trying to cover a large array of devices just isn’t practical.”

“And that is only half the picture,” Smith says. “The other side is Android version, where 50% of users are on 2.3.3 and then the rest are are on things ranging from 1.6 to 4.0.3.”

The contrast with developing for iOS is sharp. “With iOS, it’s usually safe to maintain compatibility about one year back,” Instapaper developer Marco Arment told us a couple of weeks ago. “So today, iOS 4 is a very safe minimum. Very soon, iOS 5 will be. Sometimes, an OS update offers so much for developers that it’s worth jumping the gun a bit.”

“Those two things combined make helping users with problems almost impossible,” Smith says, and support emails are a huge part of that, with the ‘vast majority’ related to some issue running the app on a hardware and Android version combination that has caused an issue. “The hardest part is that you can’t drop support for old things because such a high number of users are stuck, so it is often the newer devices that are the trickiest to support well.”

All of this leads to low reviews of the app, an enormous volume of support email dedicated to errors due to versioning or hardware flukes and lost revenue for the developer.

“It is a nightmare…the worst part for me is that ultimately it means that I have a lot of grumpy users that I just can’t realistically help. It just doesn’t make sense (economically) to run down every bug that is specialized to a particular device/OS combo.”

This is especially important because the Android platform is generally far less profitable than Apple’s iOS. People expect to pay less for Android apps and they’re vocal about it. Check out this thread on Reddit about the recently released (and incredibly beautiful) doubleTwist Alarm app for an example of the way that some Android users view paying even $1 for an arguably AAA title.tumblr lnx7zso4dv1qe4dqj 220x369 The shocking toll of hardware and software fragmentation on Android development

“I think most droid users have never heard of the android market, whereas most iOS users get their phone and then immediately go looking for apps. That is just anecdotal, but it seems like a lot of people just view it as the ‘free’ phone option at the cell phone store…sure there are people who are deliberately choosing it but I’d guess that is a minority.”

The chart on the right shows how Smith’s apps were performing on the App Store, Amazon and Google markets last July. Note that his apps are actually ranked higher on Android than they are on iOS. This is a systemic problem and one that makes the additional headache of supporting hundreds of devices running aging versions of Android even less worthwhile for developers. Especially those with limited resources.

In a report today, Flurry Analytics paints a slightly more optimistic picture of the Amazon App Store, which it says generates $0.89 for every $1 that the App Store makes, but the Google Play market still comes in a dismal third at $0.23 .

I asked Smith if he thought that there was any practical solution to the problem of having to support so many versions of an app.

“The only way it would get better realistically is if there were one or two phones that had runaway success and stayed that way for a long time…where their sales would dwarf the previous install base.”

Due to its approach, Apple’s ecosystem offers many benefits for developers. There are only two screen sizes in the entire pantheon of Apple’s mobile products and as many as 75% of users are already upgraded to iOS 5, allowing developers to support newer features and phase out older versions of iOS quicker.

In the case of hardware, Apple’s path has been one of restraint and focus. The business model of Google is an antithesis to this kind of thinking, which is fine. It’s great to give consumers more options and to have a strong alternative to iOS. Android fills a necessary role in many embedded systems and unique devices. And Ice Cream Sandwich is a bold step in a very good direction, not compromising the original vision of Android as a more customizable and flexible OS, while injecting it with more polish and logic.

But a platform does not exist on first party apps and a polished OS alone. If it did, then Windows Phone would be destroying Android in adoption and not just in users satisfaction surveys. A platform needs developers and it needs them to be able to please their customers. Apps are the key to any successful platform and if Google doesn’t work to improve the very real fragmentation issues, it will continue to be a less attractive option for the very people Android needs to survive.

Source:

http://thenextweb.com/mobile/2012/03/30/the-shocking-toll-of-hardware-and-software-fragmentation-on-android-development/

Did you like this? Share it: