Despite unit testing has formal metrics that can prove at least minimal level of thoroughness; many organizations struggle to uncover measurement that can help telling how much of unit testing is enough. The minimal level of testing can be measured in terms of code coverage. Code coverage can be as weak as statement to as strong as du-paths. But this is not my goal to explain each of those. My goal is to help those who try finding an alternative measure of unit testing quality.
One of ideas is measuring it post-factum. Well it may be too late to know the truth. Nonetheless this knowledge can be used in the following project to reduce number of defects of a specific sort. I mean the feedback that you receive from testing.
Software is being developed in several stages. Even in Agile, there are stages; but they are shorter and tightly intertwined. Defects injected at a given stage can be found at that stage or at the following ones. Depending on how many defects injected at the stage escaped quality control, we can draw our conclusion about testing quality.
For example, we have created a unit and performed unit tests. In result of testing we have found and fixed 16 issues. But later when a unit comes to Integration and System testing, we discover additional 4 issues attributed to unit. This means that defect removal efficiency of unit testing for that module was:
DRE = (1-4/(16+4))*100%=80%
To enable this measurement you need to analyze issues found at later stages of development. If you have a defect tracking system (and I hope you do) then you can simply introduce the field that will denote to what of development stages a defect can be attributed. The biggest problem here is having defects found at unit testing stage filed somehow. Many developers see it as an additional unneeded paperwork. So, if you are going to implement this metric be prepared to the resistance. You may decrease the pain by claiming only brief information on unit testing defects to be filed (summary only). There are tools that can be integrated into development IDE to soothe this pain for developers.
Other testing stages' effectiveness can be measured the same way. All you need is to collect this information continuously. Two or three projects later you will have plenty of interesting information to think of:
- Why is this module's unit testing is lower in effectiveness than others?
- Why developer A usually produces higher effectiveness than others? Can the latter share the experience to the team?
- Who is responsible for most of the issues down the road? What can you do about it?
Hope this helps! :)
Friday, September 18, 2009
Wednesday, September 9, 2009
Management means no friendship?
Not so long ago I partook in a long discussion related to whether or not a manager can have friends among his subordinates. As I remember, the vast majority people in management role believe there is no problem having friendly relationships with your teammates.
But I remember another very successful manager (a top manager) who kept saying "you should not be a friend to your people", "you should not go to lunch with them". Since then I keep replaying this through my mind and all the time it responds with "It is wrong", so I cannot adopt this thinking.
To me, building unnatural barriers is counterproductive. Open communication is a key to effective professional relationships. People are smart enough to pick up the signs of too artificial behavior. They will mind a reason behind it and will start doing the same. So they will never be open with you.
The same top manager once told me "why you cannot speak to me frankly?" How could I if I did not trust him because of his ways of managing people?! That was kind of paradox that we could not solve. I guess he had same problems with others. I heard it many times that people preferred to talk to him what he wanted to hear, not what they thought. No need to say that he was not able to manage the team effectively. So, despite of his experience I would say that he was wrong. Friendly open relationships help managers doing the job. It does not free people up of the responsibility; it rather imposes the responsibility on them. Subordinates tend to feel badly when they did not do things right, when they couldn't keep the promises not only to a manager, but to a friend as well.
Of course, this way of management works not for all of people you will encounter in your managerial career. As I always liked to say "In management there are always several ways to do the same. The way that you like is not the only one. So, keep learning."
It may also depend on the culture. If you have anything to say on this matter I would gladly hear that.
What do you all think? Long to seeing your comments…
But I remember another very successful manager (a top manager) who kept saying "you should not be a friend to your people", "you should not go to lunch with them". Since then I keep replaying this through my mind and all the time it responds with "It is wrong", so I cannot adopt this thinking.
To me, building unnatural barriers is counterproductive. Open communication is a key to effective professional relationships. People are smart enough to pick up the signs of too artificial behavior. They will mind a reason behind it and will start doing the same. So they will never be open with you.
The same top manager once told me "why you cannot speak to me frankly?" How could I if I did not trust him because of his ways of managing people?! That was kind of paradox that we could not solve. I guess he had same problems with others. I heard it many times that people preferred to talk to him what he wanted to hear, not what they thought. No need to say that he was not able to manage the team effectively. So, despite of his experience I would say that he was wrong. Friendly open relationships help managers doing the job. It does not free people up of the responsibility; it rather imposes the responsibility on them. Subordinates tend to feel badly when they did not do things right, when they couldn't keep the promises not only to a manager, but to a friend as well.
Of course, this way of management works not for all of people you will encounter in your managerial career. As I always liked to say "In management there are always several ways to do the same. The way that you like is not the only one. So, keep learning."
It may also depend on the culture. If you have anything to say on this matter I would gladly hear that.
What do you all think? Long to seeing your comments…
Wednesday, September 2, 2009
What makes a Good tester?
Everyone can easily say a good tester from a not as good one. A good tester finds more quality defects. But what makes one to be good tester? What is the difference that helps identifying important sound bugs faster?
First is technique. A good tester knows what the weak points about the application are. If it accepts a text for the input and produces a tagged text as the output, then most likely the issues will start popping up if you feed it a large volume of text. If it ate it successfully then change the text to something developer would not expect for the input. This is a sort of destructive thinking that helped my son breaking seemingly unbreakable toys when he was from 3 to 5. Thinking in categories "what else people usually omit?" helps finding more defects faster.
Second difference is the feeling of urgency. A good tester will explore entire functionality of the application rather than focusing on a small part of it. In result, the number of defects and the severity of defects is usually higher than quantity and quality of defects produced by testers bogged down into details of a specific dialogue. Likewise, a tester who focused on functionality will produce more quality issues than a person who is too deep into UI testing. Small issues reported early in the testing when application is full or crashes may make developers and project manager go mad. You are very likely to hear: "What are you thinking about?" So, better start from beginning and focus on how system works. Of course they may be exceptions. For example, it may be not reasonable to test a part of functionality and get back to it later. Anyway, if it does not cost too much I would rather recommend focusing on what matters now.
Third difference is level of expertise. A good tester can quickly imagine in which way a new functionality can affect the existing one. Knowledge about system functional and structural dependencies can help finding integration issues. Such issues usually have heavy consequence, so they are rated as critical.
Forth difference is seeing it all as a whole. A good tester always finds out how the functionality is supposed to be used and tries it on realistic tasks. There is no need to remind what use case or end-to-end testing is. This is all natural for a highly skilled testing professional to think of what real user is supposed to do with a system. It also helps finding usability issues. Recommendations on how to make system better are always welcome!
I wish all testers around would be good ones. Hopefully this article will help some of the readers to become one :)
First is technique. A good tester knows what the weak points about the application are. If it accepts a text for the input and produces a tagged text as the output, then most likely the issues will start popping up if you feed it a large volume of text. If it ate it successfully then change the text to something developer would not expect for the input. This is a sort of destructive thinking that helped my son breaking seemingly unbreakable toys when he was from 3 to 5. Thinking in categories "what else people usually omit?" helps finding more defects faster.
Second difference is the feeling of urgency. A good tester will explore entire functionality of the application rather than focusing on a small part of it. In result, the number of defects and the severity of defects is usually higher than quantity and quality of defects produced by testers bogged down into details of a specific dialogue. Likewise, a tester who focused on functionality will produce more quality issues than a person who is too deep into UI testing. Small issues reported early in the testing when application is full or crashes may make developers and project manager go mad. You are very likely to hear: "What are you thinking about?" So, better start from beginning and focus on how system works. Of course they may be exceptions. For example, it may be not reasonable to test a part of functionality and get back to it later. Anyway, if it does not cost too much I would rather recommend focusing on what matters now.
Third difference is level of expertise. A good tester can quickly imagine in which way a new functionality can affect the existing one. Knowledge about system functional and structural dependencies can help finding integration issues. Such issues usually have heavy consequence, so they are rated as critical.
Forth difference is seeing it all as a whole. A good tester always finds out how the functionality is supposed to be used and tries it on realistic tasks. There is no need to remind what use case or end-to-end testing is. This is all natural for a highly skilled testing professional to think of what real user is supposed to do with a system. It also helps finding usability issues. Recommendations on how to make system better are always welcome!
I wish all testers around would be good ones. Hopefully this article will help some of the readers to become one :)
Friday, August 28, 2009
Test design: do it right the first time!
If you are in software testing field you probably noted how difficult it is to grasp the idea behind the test cases when all you see is a collection of steps presented for review. This is too easy to get bogged down into step details. You may have even got this impression "Wait a minute! I wanted to review what is supposed to be tested instead of reading individual steps trying to compile them into logical test sequences!" If you did – then read on. Some advises may prove to be useful for you. If you never experienced such issue read on anyway. The fact you never noted the problem does not mean it cannot not hit you ;) Remember, this is not what you know will kill you.
The solution to the problem is of the same sort as we used to apply in other types of activities - early validation. So, you need the means to early identify problems with your design, until it gets "hardcoded" into written form, with steps and test data. The question remains – how to present testing ideas in a short and convenient for review form? There are a couple of ideas. Both proved to be quite successful in my experience.
The first idea is to generate a list of test conditions or test requirements before starting to generate test cases. The list can be presented in form of a tree:
- file types:
---- doc
---- txt
---- pdf
---- xml
- file content:
---- text
---- pictures
---- text + pictures
- file size:
---- large
---- empty
The time needed to generate such a list is negligible. You don't even need to bother about test data at this point. The whole idea you put behind the testing may be wrong, so time spent for creating complex test files will be a total waste.
Tree forms allows for easy review. It helps providing your thoughts in a structured way and keeps reader's attention at one context at a time, thus improving the quality of analysis. Seeing such a list one may easily find out that there are no tests for specific conditions (text in UT8 format for example). The only problem with this approach is that you do not tell how you are going to validate those conditions.
Another way is providing an overview along with your test design. It works as well to better review process, but it does not help against rework. If you have chosen wrong way in your design, you will need to re-develop your test cases.
Overview is written for the reviewer. It shall answer the following questions:
- What is supposed to be tested?
- How is it supposed to be tested?
- What conditions are to be tried out?
- How the results are going to verified?
- Whether or not automated testing will be used?
- Risks and limitations of selected testing strategy.
Having the answers on these questions will allow a reviewer to easily grasp the ideas behind the testing and to validate them.
Test design review is a hard task because there is a lot of a detail. The main goal of review is not to make sure details are correct. We are much more interested to see if testing design is adequate to get confidence that, once it pass, we have a quality product. There is no need to provide obscuring details to review. The early you present your work to review and the easiest it will be to understand it - the best quality and productivity you can achieve.
The solution to the problem is of the same sort as we used to apply in other types of activities - early validation. So, you need the means to early identify problems with your design, until it gets "hardcoded" into written form, with steps and test data. The question remains – how to present testing ideas in a short and convenient for review form? There are a couple of ideas. Both proved to be quite successful in my experience.
The first idea is to generate a list of test conditions or test requirements before starting to generate test cases. The list can be presented in form of a tree:
- file types:
---- doc
---- txt
---- xml
- file content:
---- text
---- pictures
---- text + pictures
- file size:
---- large
---- empty
The time needed to generate such a list is negligible. You don't even need to bother about test data at this point. The whole idea you put behind the testing may be wrong, so time spent for creating complex test files will be a total waste.
Tree forms allows for easy review. It helps providing your thoughts in a structured way and keeps reader's attention at one context at a time, thus improving the quality of analysis. Seeing such a list one may easily find out that there are no tests for specific conditions (text in UT8 format for example). The only problem with this approach is that you do not tell how you are going to validate those conditions.
Another way is providing an overview along with your test design. It works as well to better review process, but it does not help against rework. If you have chosen wrong way in your design, you will need to re-develop your test cases.
Overview is written for the reviewer. It shall answer the following questions:
- What is supposed to be tested?
- How is it supposed to be tested?
- What conditions are to be tried out?
- How the results are going to verified?
- Whether or not automated testing will be used?
- Risks and limitations of selected testing strategy.
Having the answers on these questions will allow a reviewer to easily grasp the ideas behind the testing and to validate them.
Test design review is a hard task because there is a lot of a detail. The main goal of review is not to make sure details are correct. We are much more interested to see if testing design is adequate to get confidence that, once it pass, we have a quality product. There is no need to provide obscuring details to review. The early you present your work to review and the easiest it will be to understand it - the best quality and productivity you can achieve.
Wednesday, August 19, 2009
Resistance to process changes
They believe this is in the nature of people to resist changes, especially when everything is ok. Things go well, so why bothering with changes. The truth is that those who simply do things as usually never achieve great goals. If a company gets complacent it may even die, overruled by concurrent who took on the challenges.
Similar rule works for quality assurance, defect prevention, and process improvements in the organization. I always face resistance when I speak of the need in code design, review, and unit testing. Even most persuading words may break down upon the blind "prove me why I should be doing it". This is very discouraging when people nodding heads in agreement on the meeting, who look as if to buy the idea, allow it silently die a month after.
This is not reasonable to believe that another preaching session in a while will change things for better. Repetition is good but it's not enough to foster right attitude toward the case. People who are responsible for using the change in the process must believe the changes are needed. And what is more important they need to believe that they need it personally to do their job better. If you fail convincing them in it, there is little change your ideas will get due support. Another necessary thing is management contribution. Management shall keep insisting on following best practices. They need to patiently explain what the purpose of a process is and how it is going to help the team to reach the goal.
Only buy-in from performers and consistent message from the management will do the job. If one of these is absent the initiative will die a silent death.
Similar rule works for quality assurance, defect prevention, and process improvements in the organization. I always face resistance when I speak of the need in code design, review, and unit testing. Even most persuading words may break down upon the blind "prove me why I should be doing it". This is very discouraging when people nodding heads in agreement on the meeting, who look as if to buy the idea, allow it silently die a month after.
This is not reasonable to believe that another preaching session in a while will change things for better. Repetition is good but it's not enough to foster right attitude toward the case. People who are responsible for using the change in the process must believe the changes are needed. And what is more important they need to believe that they need it personally to do their job better. If you fail convincing them in it, there is little change your ideas will get due support. Another necessary thing is management contribution. Management shall keep insisting on following best practices. They need to patiently explain what the purpose of a process is and how it is going to help the team to reach the goal.
Only buy-in from performers and consistent message from the management will do the job. If one of these is absent the initiative will die a silent death.
Tuesday, August 11, 2009
Interviewing candidates
I have done it many times. A lot of lessons were learned. However, even now, from the heights of experience, I realize that I still can fail. The reason is that people are very creative if it comes to making impression. The more intelligent a person who sits against you in the interview, the more difficult it is to probe the true qualities of that person. Being smart is just one of qualities that you look in candidates. It defines aptitude to some degree. However, this is far not all that you may need to find out.
If a person is smart enough he or she may tell you rather what you want to hear, not what he or she is thinking. So, in result, you may get a wrong image of a candidate. Such a hire, if made, may prove to be wrong not only for you, but for a hired person as well. So, it makes sense to put it clear right before interview that this is not only the examination of person's qualities but also a test if a person fits the position. The later will be evident in a while. So, you both better not to waste time and to be honest. A candidate must be honest about achievements and experience as well as you must be honest in your responses (overtime work, salary increases, etc.). Lie leads to another lie. Don't start your relationships in the wrong way.
Now back to the interview… I used to think of candidates in terms of three A (Attitude, Aptitude, and Achievements). Fist two “A” are most important for newbie candidates. The last one is extremely important for a senior position.
Attitude is how a person sees his or her future in your company. This is a measure of desire to work in your team, the level of loyalty. And this is what candidates are dishonest about most of the time. So, do not believe just words. Look at the reaction of a person to specific questions. If there is lie you should guess it from the looks and gestures with easy.
Aptitude is how candidate's skills cover requirements of a position. Some things can be caught up at easy. Some can't. The difference is crucial. If there is anything in a candidate that makes you believe he or she will not be 100% efficient in a new role and if you are not sure you can fix that - never hire such a candidate.
Achievement is demonstration of the skills at the tasks similar to one you want a candidate to perform in the future. If the tasks are different and he or she will have to do incomparably different things then you better consider him or her to be a newbie.
Hiring newbie is more complicated because you simply do not know what to ask. Usually we ask people to describe what they did in the past and drive their talking with questions. A newbie do not have achievements to talk about, so you need to use standard cliché questions like "why you have chosen this domain?", "what have you read on this?", "what do you want to achieve in 5 years from now?", "what are you ambitions?" and so on. But those questions will reveal just a bit of a candidate. To learn more, I used to give them short tests, which allow me judge upon professional qualities. I use these tests for more than 10 years and have never regret about it.
What a test may be? Well, for a software tester this is a quick test to produce as many as possible tests for a simple function of a well known and broadly used software product. Calculator or Notepad will do just perfect. The answer not only indicates candidate’s natural ability to create test cases, it also provides you some feedback on how good a person is at using the software at all. You may guess it by the depth of the testing a candidate is generating. One may come up with tests for Undo operation, another may have it omitted.
Such tests can be developed for any position. For a developer it can be a test task to design a simple system using OOP, or to write a piece of code on a paper. For a technical writer it may be a quick description of some software function. System architect shall be able to come up with a solution to the architectural problem that you faced in the past. Or it may be just a sketchy design of a system described right at the meeting (chat machine of online magazine, for example).
Yes, such tests will require your time and time of a candidate. But you will hardly be able to gauge person’s abilities without it. Only practical task show how good candidate is at work that you want him or her to do in the future.
Of course, aside of above, there are still a lot of things to look at. How a person will be able to meet deadlines? Is the level of responsibility is high enough? Can this person learn new things quickly? Is there any problem with communication? And so on.
Sorry for this a bit clumsy but hopefully useful piece of information. I will get to it back when I have time to polish it.
If a person is smart enough he or she may tell you rather what you want to hear, not what he or she is thinking. So, in result, you may get a wrong image of a candidate. Such a hire, if made, may prove to be wrong not only for you, but for a hired person as well. So, it makes sense to put it clear right before interview that this is not only the examination of person's qualities but also a test if a person fits the position. The later will be evident in a while. So, you both better not to waste time and to be honest. A candidate must be honest about achievements and experience as well as you must be honest in your responses (overtime work, salary increases, etc.). Lie leads to another lie. Don't start your relationships in the wrong way.
Now back to the interview… I used to think of candidates in terms of three A (Attitude, Aptitude, and Achievements). Fist two “A” are most important for newbie candidates. The last one is extremely important for a senior position.
Attitude is how a person sees his or her future in your company. This is a measure of desire to work in your team, the level of loyalty. And this is what candidates are dishonest about most of the time. So, do not believe just words. Look at the reaction of a person to specific questions. If there is lie you should guess it from the looks and gestures with easy.
Aptitude is how candidate's skills cover requirements of a position. Some things can be caught up at easy. Some can't. The difference is crucial. If there is anything in a candidate that makes you believe he or she will not be 100% efficient in a new role and if you are not sure you can fix that - never hire such a candidate.
Achievement is demonstration of the skills at the tasks similar to one you want a candidate to perform in the future. If the tasks are different and he or she will have to do incomparably different things then you better consider him or her to be a newbie.
Hiring newbie is more complicated because you simply do not know what to ask. Usually we ask people to describe what they did in the past and drive their talking with questions. A newbie do not have achievements to talk about, so you need to use standard cliché questions like "why you have chosen this domain?", "what have you read on this?", "what do you want to achieve in 5 years from now?", "what are you ambitions?" and so on. But those questions will reveal just a bit of a candidate. To learn more, I used to give them short tests, which allow me judge upon professional qualities. I use these tests for more than 10 years and have never regret about it.
What a test may be? Well, for a software tester this is a quick test to produce as many as possible tests for a simple function of a well known and broadly used software product. Calculator or Notepad will do just perfect. The answer not only indicates candidate’s natural ability to create test cases, it also provides you some feedback on how good a person is at using the software at all. You may guess it by the depth of the testing a candidate is generating. One may come up with tests for Undo operation, another may have it omitted.
Such tests can be developed for any position. For a developer it can be a test task to design a simple system using OOP, or to write a piece of code on a paper. For a technical writer it may be a quick description of some software function. System architect shall be able to come up with a solution to the architectural problem that you faced in the past. Or it may be just a sketchy design of a system described right at the meeting (chat machine of online magazine, for example).
Yes, such tests will require your time and time of a candidate. But you will hardly be able to gauge person’s abilities without it. Only practical task show how good candidate is at work that you want him or her to do in the future.
Of course, aside of above, there are still a lot of things to look at. How a person will be able to meet deadlines? Is the level of responsibility is high enough? Can this person learn new things quickly? Is there any problem with communication? And so on.
Sorry for this a bit clumsy but hopefully useful piece of information. I will get to it back when I have time to polish it.
Walking problem
What if one of your team members appears to be a problem? He or she may not fit into the team and having such a person among the others may prove to be too risky. People are getting infected with examples of bad behavior way too fast. (I wish they did as well mimicking something good, *sigh*)
The first advice is not to lay it off for too long. Yes. This is harsh business but it has to be done. Otherwise what kind of a manager you are? I usually start fro, talking to a person explaining him or her: what is wrong with the behavior and how it affects the team. Once a person is aware of the problem he or she can do something about it. In this respect, this is a service to a person that you do.
It may not be as easy though. Some people take critics personally. It may make your efforts all in vain. It helps to explain that the reason you are talking to your team mate now is not to "put in place" but to help, help to grow, help to get better.
Mostly, things are looking too good from the inside. We all fool ourselves that we work as well as it can only be. The reality is a little bit different. The extent in which it differs depends on our inner qualities. Some assess their achievements with due criticism. Some think that they do much better than others having no real reason to think so.
Your goal is to provide objective feedback on the achievements. Do not forget to mention good ones. It will make your team mate more open to discussion. And it's always better to start meeting from talking about positive achievements. Remember, you goal is a constructive discussion. Having someone in bad resistive mood will make things much more complicated.
Once you delivered your viewpoint explain the consequences. I usually make it clear if I am going to let a person go in case there is no progress on the improvement plan. This is tough but fair. Do not try to make corners flat. Just put it clear.
Then sit together and define strategy of how to make it all better for a person and for the team. Once you have a plan, define deadlines and metrics that will help making sure you have achieved the goals.
Following these simple recommendations may save you a lot of time and nerves.
The first advice is not to lay it off for too long. Yes. This is harsh business but it has to be done. Otherwise what kind of a manager you are? I usually start fro, talking to a person explaining him or her: what is wrong with the behavior and how it affects the team. Once a person is aware of the problem he or she can do something about it. In this respect, this is a service to a person that you do.
It may not be as easy though. Some people take critics personally. It may make your efforts all in vain. It helps to explain that the reason you are talking to your team mate now is not to "put in place" but to help, help to grow, help to get better.
Mostly, things are looking too good from the inside. We all fool ourselves that we work as well as it can only be. The reality is a little bit different. The extent in which it differs depends on our inner qualities. Some assess their achievements with due criticism. Some think that they do much better than others having no real reason to think so.
Your goal is to provide objective feedback on the achievements. Do not forget to mention good ones. It will make your team mate more open to discussion. And it's always better to start meeting from talking about positive achievements. Remember, you goal is a constructive discussion. Having someone in bad resistive mood will make things much more complicated.
Once you delivered your viewpoint explain the consequences. I usually make it clear if I am going to let a person go in case there is no progress on the improvement plan. This is tough but fair. Do not try to make corners flat. Just put it clear.
Then sit together and define strategy of how to make it all better for a person and for the team. Once you have a plan, define deadlines and metrics that will help making sure you have achieved the goals.
Following these simple recommendations may save you a lot of time and nerves.
Subscribe to:
Posts (Atom)