Friday, January 30, 2009

Some bugs missed

In testing we are trying to avoid bugs to be shipped to production environment. As tool we use test cases. The good part of using test cases you can control the depth of testing. If a tester is less experience you can write detailed test scripts. If the tester is very experienced and skilled you can do without some details. Using these test scripts some nice metrics can be created like: how many of what in which time period and is it according plan, schedule and expectations.

A bad thing is that we often instruct the lesser experienced testers to test only those things which are written down. And we give the same instruction also to the experienced testers as the time schedule must not slip.

What I saw is that for small parts this is not yet an issue as test cases are defined based on using test techniques and this gives us the ability of measuring some coverage. In large tests like chain tests a risk occurs that defects are missed as the focus is defined on high level to perform certain actions. Obviously those defects don’t hurt the system and process immediately and therefore accepted as intended functionality.

If the testers are also the key-users there is a chance that this "wrong" behavior is explained to other users and also documented in work instructions and manuals. People will continue working with the process on production ignoring these errors. Sometimes it you will see that unintentional these actions will become part of a workaround without knowing it is a workaround. It can also happen that those errors keep unnoticed.

Since testers try to re-use testware the next time testers will perform the same actions. During their usage on production their knowledge of the system is increased.
In the past I saw situations that this gained knowledge lead to finding the errors during testing which were "previously" not there.

This will lead to a discussion if the functionality is as expected as in the past it worked that way, so why should it be now an error? Should it be registered as an error in the "new" functionality or should it be logged as a production error?

Logging it as an error in the test environment it will have impact on the advice for release. If it is a production error it should be decided when to solve it. A situation is created if it is allowed to disturb the project or keep disturbing the production.

What will you decide and for what reason?

No comments:

Post a Comment