Unit testing is what you need

Preface

"Whatever you do today will get you one step closer or one step further from the intended goals."

How can this statement be reflected on the idea of Unit testing? Unit testing is usually seen by people as the additional cost related to development. If you do not write additional lines of code we save time. We could spend this time writing production code, for example.

Well this is only partially true. In many cases we [mistakenly] assume that writing a line of unit the same as writing a line of production code. This leads us to incorrect understanding of unit testing benefits.

Testing code is as simple as it takes 10 to 100 times less time than writing a piece of production code of the same size. So, do not even listen to voices of people who claim that development with unit testing takes 1.5 to 2 times more time based on code line calculation.

Unit tests are trivial, as simple as code below. Just think of how many such tests you can write in the course of an hour...

public void testAddNegative () {
assertEqual(MyClass.Add(2,-1) , 1);
}

Supporting unit tests after change is as trivial as writing them. You can even afford rewriting them from scratch with easy.

Why to unit test?

If you can write code without error you would probably not need to test it. If you cannot write code without error them how do you check that it does what it should do? Using debugging is a tiresome tedious manual activity, not in the way how developers are meant to do it. Not to say this is hardly possible to run all required tests in debugger after changes. I know no developer who would do this stupid thing.

So, having produced a piece of code with undefined quality developer simple gives the responsibility for testing it away to someone else down the line. Let's see if there are right people to do it. Hm...

Integration engineers are supposed to see your modules as black boxes. So, they do not care about your inner functionality. All they work with is external interfaces, input and outputs of your module. If there is a branch in the code that was never tested it's very probably remain such after Integration testing.

The next possible candidates to catch your defects are system testers (or QA as they called sometimes). These guys care about code structure even less than integration and deployment engineers. Testers focus on business logic. Their job is to make sure the ideas you have implemented is exactly what customers wanted. Testers care about neither code structure nor architecture. If they catch your intrinsic code error it is only by chance.

The next in the food chain is the end user. Unfortunately those guys are extremely lucky at finding the issues. If you leave defects in the release system be sure they will stumble upon it %)

There is no one down the line who can catch and correct your code errors. Only you, as a developer, can check if all branches through the code are examined carefully. Only you are responsible for quality of code that you check into the code base. So, find time to test.

How do you know you are done?

How developers get to know that code piece is done? If they have automated unit tests then this is not a problem to run them and see the results. But what is there are no tests? Do such developers have habit to provide to someone code of unknown quality? Just imagine what if cars were made using the same attitude.

Having your code run under debugger with 1 typical data set is not enough. There are boundary conditions, zero and incorrect input, big volumes of data, stressful conditions that also can lead to failures. Who of developers do such testing under debugger? I guess no one.

So, there is no other way to make sure you finished the code but Unit testing. If you know a reasonable alternative to Unit testing just drop me a line and I'll fix this statement and will eat my hat ;)

The most effective unit testing

Unit testing has proved itself as a powerful tool against software regressions (issues that appear after changes or defect fixes). So, the more changes you do to the project keeping tests untouched the more return of investment you have.

Who does unit testing?

To me the answer is simple - developers should do it because:

* They know the code and there is no need to explain it to someone else.
* They think of testing during code design.
* They need something to demonstrate that the code is finished.

The crude world

If unit testing was so easy it would not be so difficult to start. Yes, there are problem. But there is none that was not faced and overcame by someone else. If you faced a problem just look around, ask your peers, search Internet but don't give up! You need unit testing anyway.

While talking about unit testing with developers I was often asked for recommendations on testing:

* Databases
* JavaScript
* UI

No problem. Just try Google for any of these to see how many possible solution you have at hand.

Wrap up

All you need to do unit testing is to start it. And do not lay it off until tomorrow, start today. As the final reason I will tell you that developers who do unit testing prove to be more successful than those who test under debugger. If you do not start today you waste critical experience and fall behind your peers. Do not be surprised if someone of them will suddenly appear one step higher in the company hierarchy ;)

The silver bullet of test automation

Note: This topic is about system test automation only. Developers (unit) testing is out of scope of this article.

I have purposely chosen such a sound name for this topic. I myself am a big believer in system test automation. But, as any other kind of tool, automation likes those who know what to do with it.

This is way too easy to be misguided by marketing stuff promising you to become more effective using their new generation super-duper automation tool. Beware, 90% of what advertisement says is a lie. The solution how it is presented may work on a very limited number of projects. What vendors don't say may hit you hard or even kill your test automation endeavor. Read on in order to get ready to concealed challenges.

Test automation is development

First thing to remember is that test automation is creating a program that tests another program. As any other kind of code it is governed by same lows. Unnecessary complexity, unclearness, duplicates, high coupling, hard-coding strings and constants have the same bad effect for test code as they do for production code. So, make sure that you know how to write code efficiently before jumping into automation boat.

Record&playback does not work

Do not believe anyone who tells you that test automation is as easy as record&playback. This is a pure lie. Following this model is undermining the biggest automation benefit - having reliable regression suite that requires little changes after changes in application. Tests generated by action recording tools are non-maintainable and you will need to re-record them afterwards. Test execution in this case will be more expensive than running those tests manually. Now think of how often you make changes to your system and try calculating whether you are going to have benefit from automation with record&playback.

Test automation is the investment

Be ready to invest. Automation is not a tool that starts working right away. There is always some leading cost involved. Yet the benefit you get down the line worth it all.

Implementation of a good automated test suite requires skilled staff, resources and time. Our suite took 4 engineers for 4 years. But the result was outstanding. It worked so well that we almost stopped testing for regressions manually. It helped us finding the issues we could never find any other way. So, our investment into automation stared paying back after about 2 years since the project start.

I concluded that automation starts working when you have at least all critical tests automation but it shines when the number of automated tests is over about 40% of your test backlog.

Automated test is strict

Automated test is a program and it can help you finding only those issues for which it is programmed. Manual testers look around, note different strange things that automation tests just pass by, on his mission to click this and read that.

Make sure you have a human being tested the feature before it is handed over to test automation.

Automation tools are as easy

I would never start automation project having no skilled professionals involved. Experience is the key. Else you are in position to pay for automation as much as 300% of its actual cost. Rephrasing famous Russian advertisement: Have too much money? So, we are going after you ;)

Tests have to be selected carefully

Ideally, you need to think of testing as a whole in the very beginning of the project. You have to get a clear picture of which tests you will have automated and which are better to be left manual. Test design should be oriented at contributing test automation using data-driven testing technique. Due test design may make your automation efforts 10 times more effective.

In the real world, we sometime face the situation when tests are already designed. Even in that case, if you have the possibility, look at test design from automation perspective. Change it the way as to get most benefit of your automation tool. Sometimes it makes sense to add tests in places where we had to compromise coverage for execution cost.

As far as implementing automated tests is an additional cost it can only be saved by execution. The best candidates for automation are those, which are executed often. Also there can be tests that are easy to automate. Doing them manually may be not reasonable.

Automated testing economics

I mentioned that automated tests pay for their creation during execution. There is a simple math example that demonstrates this postulate:
• 10 manual tests require 2 hours for execution.
• Implementation of these 10 tests would take 16 man hours.
• Execution of 10 automated tests takes 0 (zero) man hours.
• So in order to get the benefit we have to execute tests at least 16/2=8 times.
Follow this principle in assessing automation feasibility and you will have positive return on investment (ROI) at your test automation projects.

Note: Sometimes we fool ourselves wrongly assuming "perceived" frequency of test usage for the real one. The problem is that once we have tests implemented it's very easy to run them. So we run even those tests, which we would never execute manually. If you want to know the real benefit of the automation, try to get rid of desire to calculate this "perceived" frequency. However, reporting the “perceived” benefit can be good for you career ;)

Conclusion

Automation is a silver bullet but only in case when it is done by experiences professionals, on a lengthy projects (2 years and more), following strict best development practices and disciplines.

To test or not to test?

Having some experience in software development outsourcing company, I faced the problem with how our customers think of testing. Some of customers believe that testing is an added cost that can simply be cut. They believe that testing is simple and they can do it on their own. Others just keep insisting that developers write code without defects, so testers won't be needed.

Let's start from the first problem. Customers who come to us asking for development only and testing product themselves soon find themselves drowned in the information regarding quality. Not to say they rarely know what to test in the system because they simply do not know what boundary conditions or security testing means. Such projects usually result in mutual frustration. Customer is frustrated with quality of development work and quality of the resulting product (under-tested). Developers are frustrated with the project failure.

In order to avoid such problem, we have to warn customer of possible bad scenarios that may be caused by decision to buy only development services and to cut testing off.

The second bad scenario is when customer believes that testing is not needed, that developers can produce software without defects. Yes, they can. Developers can check they produced correct code but they have trouble making sure they implemented right thing. Developers are too biased toward inner code structure and design, so they have trouble seeing a big picture. Independent testers play role of end user advocate. They pretend they are users during the testing. I doubt developers can do it effectively because they have to think of inner design all the time.

So, having independent system testing is crucial for project success. The question in the title of this post has the meaning similar to the famous question posed by Shakespear. And it has the same crucial importance to the project as it was for Hamlet.

Quality Vampires

Before writing materials on technical aspects of testing, QA and management I decided to provide my vision of what can make any quality initiative bound to fail.

Vampire 1: Quality costs.

First of vampires is the notion that quality costs. Yes it does, but not in the way you may think. Just look at the best examples of the contemporary automobile industry. There are players on the market who provide a very different level of quality for the same money. Those who produce cars of better quality feel themselves more secure on the market. They also show better result financially, having sold more cars and having fewer warranty cases. Now ask yourself whether or not quality costs and in which way?

Right! This is not the quality that may cost you reputation, lost sales, and frustrated customers. This is the lack of quality solution that may make your business sinking.

So, before even thinking of saving a coin today by skipping important quality practices like design, review, unit testing, and the like, ask yourself whether or not it may result in heavier losses down the production line.

I tell you from my own experience. All the compromises we make today will hit us tomorrow. Insignificant time we won cutting design or test review may cost complete module redesign or missed defects.

In the famous quality/resources/feature triangle quality is the most important! You may change the other two as you need to fit your business goals.

Vampire 2: This is not possible to produce software without defects.

The phrase itself makes sense. The problem is how people tend to be using it. Some developers use it for the excuse and take making defects in code too lightly. I am a human, I make mistake. So, who cares? This is a bad-bad habit in the software industry that is easy to accommodate but hard to get rid of.

Developers like any other kind of workers make mistakes. This is true - It’s ok until they learn from those mistakes. We learn it from the childhood that making mistakes is bad. Those of us who make more mistakes than other are valued less. They have fewer career opportunities and earn less money than those who produce high quality results. But both are humans. The difference is how they feel about making mistakes.

One may say that it's ok, because I am a human. Another will ask himself: “what can I do differently in order to avoid making these types of mistake in the future?” This is a completely different story, thinking that way.

So, the phrase itself is true, however do not let it misguide you. Always analyze your mistakes and learn from them.

Vampire 3: Everyone can do testing; it's easy.

This is the strangest misconception I ever met! Who decides? Usually I hear it from people who never practiced software testing. So, they cannot provide arguments if asked a simple question: why?

Testing is a big knowledge area. It takes several years to grow into software testing professional. Being professional in testing and development is beyond human possibilities. Testers are not looser-developers. Just look around. These people have chosen testing deliberately because they like it. Of many testers with whom I worked, only few were wannabe developers.

So, let's stop talking flame about who is more important for the success and get back to job!

Welcome!

At last... After so many years of pondering this idea away, I decided to start my own blog. I feel like I have enough to say and to share. I hope that experience I gained for 11 years in software testing and quality assurance area and the ideas that I have will help others being more successful in such a challenging, tricky and enjoyable bsuiness like quality engineering!

I also have a decent knowledge in management and some thoughts that you may find interesting.