Tag Archives: bug

Software Testing is not a commodity!

Stick in software testing long enough, and you will see enough ideas come and go to be able to sort out the ones that look promising to work, and the ones that you just hope will go away soon enough so that no manager will pay any of her attention to it. There have been quite a few in the history of software testing, and from my experience the worst things started to happen every time when someone tried to replace a skilled tester with some piece of automation – whether that particular automation was a tool-based approach or some sort of scripted testing approach.

Why do we test software?

If we were able to write software right the first time, there would be clearly no need to test it. Unfortunately we humans are way from perfect. Take for example the book I wrote mostly through 2011. 200 pages, lots of reviewing, production planning, and stuff happening in the end. And still, while reviewing the German translation, I spotted a problem in the book – clearly visible at face value. I had spend at least 2 weeks after work to go through the book once more, and get everything right. Yet, I failed to see this obvious problem.

The problem lies in our second-order ignorance: the things we don’t know that we don’t know them. These are the things of good hope, and prayers that it will work. Murphy’s Law also has a role to play here.

The very act of software testing then becomes to find out as much information about our unawareness as possible. This includes not only exercising the product, but also finding out new things about the product. Skilled testers learn more about the product and the product domain and the development team over the course of the whole product lifecycle.

Why do we repeat tests?

But how come we focus on regressions to often in our industry? It has to do with first-order ignorance. A regression problem is a bug that gets introduced a second time, although it already had been fixed in the meantime. Since we were already fully aware of the problem, the bug is no longer something that we don’t that we don’t know it. It has become something that we know now, but we don’t know whether we will know it still tomorrow. That’s why we introduce a regression check for tomorrow, so that it will remind us about the problem that we tried to avoid at this time.

Read that sentence again. Yes, it’s speculation. We speculate that we might break the software tomorrow again. With this speculation comes a whole lot of costs. We have opportunity costs for doing the test, for automating it, and with every run, we have the opportunity cost of analyzing the result (if we have to).

We wouldn’t need this if we were able to realize that a regression bug introduced in our software is an opportunity to learn what is not working in our current process that caused that bug to re-occur. Every regression bug discovered should be an invitation to start a root cause analysis and fix the underlying problem rather than deal with the symptoms.

Source: http://www.shino.de/2013/02/04/software-testing-is-not-a-commodity/

Did you like this? Share it:

8 Common Excuses in Software Testing

Excuses are common in the workplace. They seem to be more common in tech companies. If they weren’t, Dilbert would have been out of print a long time ago. But excuses inside tech companies who don’t test their software? In that case, they can be something of an epidemic.

And what are some of those excuses? Here are a few we’ve heard over the years:

1. “It’s working fine on staging.” – Applications always seem to work differently on staging than they do in production, don’t they? This leads many companies to only test before a major launch. What changes can happen in the time an app goes from staging to the real-world? Anything and everything! Users can access the app on different browsers and operating systems, or in the case of mobile apps, they are likely to use an app on a variety of devices, carriers and in disperse locations. In other words, a lot can change from staging to production, so there’s no excuse for not testing “in the wild.”

2. “We didn’t have enough time to test.” – This excuse is common within companies that tend to view software development as an assembly-line process, with testing being the final stage or “last line of defense.” The problem here is that when projects fall behind – which they almost always do – testing is done hastily at best, or worse, not at all. Ideally, the testing team is involved throughout the entire SDLC, but that’s a topic for another day. By the way, if you’re a tester, and you find yourself in this situation, this is actually a very valid excuse, but I digress…

3. “It’s okay, we’re a startup.” – Being lean and agile (and likely resource constrained) doesn’t give you the excuse to skip testing. If anything, startups should be more concerned about testing and quality, as they are making first impressions and/or trying to disrupt an entire industry. Poor quality will help them achieve neither. In our view, startup status should never be used as an excuse for not testing properly.

4. “It’s in beta, users will find the bugs.” – If that’s your excuse, rest easy knowing that users will indeed find the bugs. But will they report them to you in an easy to understand bug report? Will they effectively communicate the severity, frequency and steps to re-produce? The answer to that question is probably “no.” We see many companies use beta as an excuse for poor quality and as a substitute for professional testing – don’t be one of them.

5. “We don’t have enough money.” – Lack of budget is certainly an excuse for not doing lots of things in the tech world. But if your company has made it all the way from ideation to development to launch, then chances are there are enough funds kicking around for at least some formal testing.

6. “We haven’t made any major changes.” – Many companies do a fine job of testing for a major launch, but fail to regression test new versions. Truth is, any code change – no matter how insignificant it might appear to be – can have a major impact on an application.

7. “I didn’t think hackers would target us.” – Just because you’re not a major banking institution or a government agency doesn’t mean you shouldn’t be testing the security of your software. The motives of hackers are changing every day, and it’s only a matter of time before they find a reason to target YOU.

8. “They’re using it wrong.” – When in doubt, blame the users  As the saying goes, if a user can’t use it, then it doesn’t work. You might understand the application, but that’s not an excuse to forgo usability testing.

Source: http://blog.utest.com/8-common-excuses-in-software-testing/2013/01/

Did you like this? Share it:

Bug Hunter Hacks Chrome at CanSecWest; Earns Top Reward From Google

During Google’s Pwnium contest at the CanSecWest security conference in Vancouver on Wednesday, Russian bug hunter Sergey Glazunov demonstrated a Chrome exploit that completely defeats the browser’s much touted security sandbox.

Chrome is viewed as one of the most secure Web browsers by the security community, primarily because of its sandboxed architecture, which restricts how it interacts with the OS and significantly limits what attackers can do if they exploit a vulnerability.

A panel of security experts from Accuvant and Coverity, who analyzed the defensive capabilities of modern browsers in depth, said last week at the RSA security conference in San Francisco that Chrome’s sandbox prevents processes from doing much of anything on the system.

However, there is a consensus in the security community that while sandboxing is a strong anti-exploitation mechanism, it does not provide a perfect defense and a determined attacker can theoretically defeat it, although with a lot of work.

For this year’s CanSecWest conference, Google decided to run a contest called Pwnium in parallel with TippingPoint’s well known Pwn2Own contest, which rewards security researchers for finding and exploiting unpatched remote code execution (RCE) vulnerabilities in browsers.

Pwnium has a maximum prize pool of US$1 million and rewards various types of Chrome exploits. The largest prize is $60,000 and is awarded to researchers who demonstrate persistent RCE exploits that target only vulnerabilities in Google Chrome’s code.

Read More:

http://www.pcworld.com/businesscenter/article/251506/bug_hunter_

hacks_chrome_at_cansecwest_earns_top_reward_from_google.html

Did you like this? Share it:

The Differences between Performance Testing and Stress Testing

Performance testing is used to test the performance of software in a system. The performance testing can be done in every period of testing. Even in the element layer, the performance of a single module can also be evaluated by doing white-box testing. However, when the whole system is integrated, the actual performance of the system can be really tested.

Performance testing is often done with stress testing, and needs hardware and software testing equipments, which means, it is necessary to measure the use of resources in a demanding environment (for example, the processor cycle). The outward testing equipments can monitor the testing, when something happens (such as interruptions), it can record it. By testing the system, testers can find what reduces the efficiency and what causes the system bugs.

Stress testing is a kind of test which adds stress continuously to the system. By determining a system’s choke point or a failed performance point, the testing will enable you to know the maximum service level of the system. For example, you can measure when the system’s performance will decline or when it breaks down.

Performance test is a common used term when doing load test and force test alternately. Performance testing puts emphasis on the overall system. It has close relationship with the intensity, stress/load testing. Accordingly, stress and intensity testing should be done with performance testing.

The difference between stress testing and performance testing is their aim of the testing.

The aim of stress testing is to figure out the maximum load the system can support. The precondition is that the performance of the system can be tolerated. For example, it is usually agreed that the page should respond within 3 seconds. In a word, it tests the maximum load the system can bear on the condition that the performance can be tolerated.

Performance testing is for testing the response and the speed of the system and other performance indexes. Its precondition is that there is a certain load. For example, it will test the performance indexes when 100 users are online, and see if every user can operate normally. To sum up, it tests the performance of the system according to some indexes (such as the time of response) with different load (or with a certain load). If we say, some website performs badly, but to be exact, we should say, when how many people are online, the website performs badly.

In general, it is like a formula: overall performance=the load quantity*performance index. The overall performance is fixed, and stress testing is to test the biggest load when the performance indexes are the lowest, and performance test is to test the performance indexes when the load quantity is determined.

Did you like this? Share it:

Android Gets New Twitter Update

All you Tweeple out there, cheer up! Tweeting has now been made lot easier with your Android device. Twitter comes in an upgraded form on Android devices, and this time it has something exciting in it.

The Android devices become more popular as each day pass by. So the new upgrade seems very relevant. As you swipe through tweets, options to reply, re-tweet, favorite, or share instantly pop up, without carrying away any space from your time line. Now, this seems interesting, right?

The new upgrade is tailor-made for those who would spend a good share of time on Twitter. This is a custom oriented design, and actions such as reply could be made quicker than before. What more should one need to get lured into something like this?

Another fascinating addition in the new update is an amusing “find friends” update. It alerts users before importing your contacts details, email, phone number and address into the public domain. Earlier, there haven’t been any such confirmation messages showing up. And so, using Twitter is safer than before.

Twitter for Android is optimized to give maximum efficiency with Android 4.0 Ice Cream Sandwich, the Kindle Fire, and the Barnes & Noble Nook Color/ Tablet.

The update has released already, and could be obtained through Android market. Besides, the update packs some security improvements, new languages, and would fix many bugs. Hope you will enjoy the update.

Source: http://www.gizmocrave.com/11063-android-gets-new-twitter-update/

Did you like this? Share it:

Practical Combinatorial Testing (NIST)

Software implementation errors are one of the most significant contributors to information system security vulnerabilities, making software testing an essential part of system assurance. In 2003 NIST published a widely cited report which estimated that inadequate software testing costs the US economy $59.5 billion per year, even though 50% to 80% of development budgets go toward testing. Exhaustive testing – testing all possible combinations of inputs and execution paths – is impossible for real-world software, so high assurance software is tested using methods that require extensive staff time and thus have enormous cost. For less critical software, budget constraints often limit the amount of testing that can be accomplished, increasing the risk of residual errors that lead to system failures and security weaknesses.

Combinatorial testing is a method that can reduce cost and increase the effectiveness of software testing for many applications. The key insight underlying this form of testing is that not every parameter contributes to every failure and most failures are caused by interactions between relatively few parameters. Empirical data gathered by NIST and others suggest that software failures are triggered by only a few variables interacting (6 or fewer). This finding has important implications for testing because it suggests that testing combinations of parameters can provide highly effective fault detection. Pairwise (2-way combinations) testing is sometimes used to obtain reasonably good results at low cost, but pairwise testing may miss 10% to 40% or more of system bugs, and is thus not sufficient for mission-critical software. Combinatorial testing beyond 2-way has been limited, primarily due to a lack of good algorithms for higher interaction levels such as 4-way to 6-way testing. New algorithms, however, have made combinatorial testing beyond pairwise practical for industrial use.

Source: http://www.vietnamesetestingboard.org/zbxe/?mid=download&category=12632&document_srl=568740&listStyle=&cpage=

Did you like this? Share it:

Automation Testing Framework Approach

A framework approach to create an Automated Regression Test Bed has many benefits over the traditional simple capture/playback or custom scripts methodology. However evolving a keyword driven framework approach for automating the regression test bed of scripts, may take time and is generally not recommended if at all any major changes are expected to be met up in near or even near far future.
How to do Automation Testing :

• Reuse of Tests : When a certain set of test scenarios require frequent execution it makes more sense to automate them and reap the benefits of unattended automated execution of test scripts.

• Another example will be in cases where a single script needs to be executed with multiple data sets. In such cases the effort to automate a single script and running it with multiple data sets is far less than the manual execution effort for all those data sets.

• Time Save : Running unattended automated test scripts saves human time as well as machine time than executing scripts manually.

• Better use of people : While automated scripts are running unattended on machines, testers can do more useful tasks.

• Cost Saving : On test engagements requiring a lot of regression testing, usage of automated testing reduces the people count and time requirement to complete the engagement and helps reduce the costs.

• Machines are more reliable than humans : Great confidence will be gained when a system is released, when we use automated testing . With some approach instead fo just record and is playback , it becomes very easy easy to maintain and update the scripts with newer releases or changes done to the application. Also the degree of scripting gets minimized as the resources working on the automation scripting mature to successfully understand and develop some set of library for their reusability.

• Selective testing : There should be an easy way to only test the breadth of the application without doing a lot of depth testing. This is to be used for stability testing of an application • There should be an easy way to select test scenarios at a more granular level in order set up more targeted regression.

• Recovery : If test script execution is stopped at any moment for any reason, the test suite should be able to resume testing from that point at any time later. We should not need to re-test functionality that had passed earlier.

• Scalability : It should be possible to execute tests from different machines at the same time in order to gain time efficiencies. 

• Inter-environment portability : The scripts should run on any given test environment without any code change (technical intervention).

• Usability : With minimal amount of training, anyone on the team must be able to execute the automated suite . The script must run without any manual intervention.

Source: http://www.vietnamesetestingboard.org/zbxe/?mid=download&category=12632&document_srl=539680&listStyle=&cpage=

Did you like this? Share it:

5 common solutions to software development problems

  1. solid requirements – clear, complete, detailed, cohesive, attainable, testable requirements that are agreed to by all players. Use prototypes to help nail down requirements.
  2. realistic schedules – allow adequate time for planning, design, testing, bug fixing, re-testing, changes, and documentation; personnel should be able to complete the project without burning out.
  3. adequate testing – start testing early on, re-test after fixes or changes, plan for adequate time for testing and bug-fixing.
  4. stick to initial requirements as much as possible – be prepared to defend against changes and additions once development has begun, and be prepared to explain consequences. If changes are necessary, they should be adequately reflected in related schedule changes.If possible, use rapid prototyping during the design phase so that customers can see what to expect.This will provide them a higher comfort level with their requirements decisions and minimize changes later on.
  5. communication – require walkthroughs and inspections when appropriate; make extensive use of group communication tools – e-mail, groupware, networked bug-tracking tools and change management tools,     intranet capabilities, etc.; insure that documentation is available and up-to-date – preferably electronic, not paper; promote teamwork and cooperation; use protoypes early on so that customers’expectations are clarified.

Source: http://www.vietnamesetestingboard.org/zbxe/?mid=download&category=12646&document_srl=432592&listStyle=&cpage=

Did you like this? Share it:

Top 20 practical software testing tips

Here are some of the best testing practices I learned by experience:

1) Learn to analyze your test results thoroughly. Do not ignore the test result. The final test result may be ‘pass’ or ‘fail’ but troubleshooting the root cause of ‘fail’ will lead you to the solution of the problem. Testers will be respected if they not only log the bugs but also provide solutions.

2) Learn to maximize the test coverage every time you test any application. Though 100 percent test coverage might not be possible still you can always try to reach near it.

3) To ensure maximum test coverage break your application under test (AUT) into smaller functional modules. Write test cases on such individual unit modules. Also if possible break these modules into smaller parts.

4) While writing test coverage, write test cases for intended functionality first i.e. for valid conditions according to requirements. Then write test cases for invalid conditions. This will cover expected as well unexpected behavior of application under test.

5) Think positive. Start testing the application by intend of finding bugs/errors. Don’t think beforehand that there will not be any bugs in the application. If you test the application by intention of finding bugs you will definitely succeed to find those subtitle bugs also.

6) Write your test cases in requirement analysis and design phase itself. This way you can ensure all the requirements are testable.

7) Make your test cases available to developers prior to coding. Don’t keep your test cases with you waiting to get final application release for testing, thinking that you can log more bugs. Let developers analyze your test cases thoroughly to develop quality application. This will also save the re-work time.

8 ) If possible identify and group your test cases for regression testing. This will ensure quick and effective manual regression testing.

9) Applications requiring critical response time should be thoroughly tested for performance. Performance testing is the critical part of many applications.

10) Programmers should not test their own code.

11) Go beyond requirement testing. Test application for what it is not supposed to do.

12) While doing regression testing use previous bug graph (Bug graph – number of bugs found against time for different modules).

13) Note down the new terms, concepts you learn while testing.

14) Many times testers or developers make changes in code base for application under test.

15) Keep developers away from test environment.

16) It’s a good practice to involve testers right from software requirement and design phase.

17) Testing teams should share best testing practices, experience with other teams in their organization.

18) Increase your conversation with developers to know more about the product.

19) Don’t run out of time to do high priority testing tasks.

20) Write clear, descriptive, unambiguous bug report.

Source: http://www.vietnamesetestingboard.org/zbxe/?document_srl=565689

Did you like this? Share it:

Software Bug Handling Process And Tools

QQ截图20111226094638 Software bug handling is a significant area of quality assurance activities.

At the time of software testing the developers who correct identified software bugs are generally not the identical as software testers who monitored and reported the errors in the first place. The exclusion is unit testing, which is often performed simultaneously to coding by the same individual.

In the majority of companies software bugs handling is implicitly assumed to be segment of the project management activities.

A formalized software bug handling process distinguishes weighty activities and rules, principles, parties involved and their obligations as well.

It is generally defined by the various states associated with individual bug status and transitions among these states due to status modifications.

At the time of software testing of big software systems in IBM there are 2 bug tracking tools:

  • IDSS (an IBM internal tool, were used for defect tracking)
  • CMVC (IBM product for configuration management and version control)

In the majority of companies different software project management tools are also used for version control and bug tracking. Such tools as Bugzilla and Issuezilla are generally employed to handling bugs for medium and large open source projects.

Source: http://www.vietnamesetestingboard.org/zbxe/?mid=download&category=76358&document_srl=602440&listStyle=&cpage=

Did you like this? Share it: