What’s an integration tests? Really?!
Submitted by Ainars Galvans on Mon, 15/01/2007 - 10:05.
context-driven testing | functional testing | test techniques
Continue talking how a testing concept that are simple in text-books are so complicated in real life within a context. Integration testing is typically described as testing either two components or two systems interacting with each other by channels designed for this purpose. Both types of tests are aimed to check if intended interaction works as designed/specified. But how about unintended interactions? My vision of testing to be done by testers are to test unintended misbehavior of the software. Developer unit tests are typically enough to test the intended behavior. Reveal such type of integration defects are quite complicated task. I found how to solve it in my project, but have no idea how it could be solved in general case.
Example
We have a server type application written in JAVA. It’s functionality is basically this: during startup it reads configuration stored in DB and then start monitoring DB and perform some activities asynchronously depending on DB status (e.g. trigger some event when a new row appears and then deletes it). The new feature recently implemented is JMS support for it. There are jconsole.exe in java bin directory, you could read about it more un sun, but what id does basically - connects to JVM, monitor it’s resource usage and allow to interact with certain classes given they have implemented some interfaces. So our server has implemented start and stop methods for managing the server. Guess what? The stop/start sequence does not result in configuration update (does not re-read configuration from DB). Is it a bug? Yes it is – what is the most common reason you restart a server, except if it is hanging all the time :) ?
Integration tests? Integration bug?
So is it an integration bug? Yes it is! The feature “reading configuration” and “stopping/starting“ server does not have integration designed. You simply can’t write the integration design for any two out of hundreds of features. If you have hundred features then you have ten thousands integration pairs (well, OK only 4950, but it is sill a lot of them).
You can’t also test all of them. When I found this defect – was I doing integration tests? It was scenario testing so some degree as I tried to imagine use cases for which stop/start may be used. On the other hand I tested it with an integration in mind – integration of two features was what the test was aimed at.
Three type of not-designed integration issues
So suppose a new Feature (lets’ call it feature A) is added and unit tested and it accord to specification. I test it and report numerous issues. Guess why? Because it does not work correctly together with another features B, K and M. There is no integration between features A and B by design – they should not interact with each other. Feature K has universal integration with other features, but it appears that this universal integration does not suite feature A quite well. Feature M always had a defect never manifested itself until feature A triggered some specific sequence of events.
Those ore the most typical defects in large software projects where different features are created in different time, by different developers (or even different developer groups), addressing different goals (e.g. different quality goals). There are rumors that huge amount of defects of such type was the primary reason for Oracle version 9 failure (and further rework in version 10).
So how to test that? I don’t really know
I’m sorry not to be ready to share any more examples as components I’m testing are quite a low level, complicated and requires context to understand. If I will run into any defect that is simple to describe I will add it here.
To find such a bugs you should know about both features involved, so you need a tester who knows each feature of the product to test it, don’t you? I do report such a bugs quite frequently. I’m test team lead for my project. The team is not too big and I am in the project from it’s very beginning (1.5 years already) so I’m lucky to know all the features of the product (it’s also supported by my skills). Well for each single feature they may be another tester who knows it better than me, but it is not required for finding integration issues. I don’t know how to solve this in a projects so large or long that you can’t have a single person who knows all the features (as in case of Oracle 9). If anyone knows, please share.
Example
We have a server type application written in JAVA. It’s functionality is basically this: during startup it reads configuration stored in DB and then start monitoring DB and perform some activities asynchronously depending on DB status (e.g. trigger some event when a new row appears and then deletes it). The new feature recently implemented is JMS support for it. There are jconsole.exe in java bin directory, you could read about it more un sun, but what id does basically - connects to JVM, monitor it’s resource usage and allow to interact with certain classes given they have implemented some interfaces. So our server has implemented start and stop methods for managing the server. Guess what? The stop/start sequence does not result in configuration update (does not re-read configuration from DB). Is it a bug? Yes it is – what is the most common reason you restart a server, except if it is hanging all the time :) ?
Integration tests? Integration bug?
So is it an integration bug? Yes it is! The feature “reading configuration” and “stopping/starting“ server does not have integration designed. You simply can’t write the integration design for any two out of hundreds of features. If you have hundred features then you have ten thousands integration pairs (well, OK only 4950, but it is sill a lot of them).
You can’t also test all of them. When I found this defect – was I doing integration tests? It was scenario testing so some degree as I tried to imagine use cases for which stop/start may be used. On the other hand I tested it with an integration in mind – integration of two features was what the test was aimed at.
Three type of not-designed integration issues
So suppose a new Feature (lets’ call it feature A) is added and unit tested and it accord to specification. I test it and report numerous issues. Guess why? Because it does not work correctly together with another features B, K and M. There is no integration between features A and B by design – they should not interact with each other. Feature K has universal integration with other features, but it appears that this universal integration does not suite feature A quite well. Feature M always had a defect never manifested itself until feature A triggered some specific sequence of events.
Those ore the most typical defects in large software projects where different features are created in different time, by different developers (or even different developer groups), addressing different goals (e.g. different quality goals). There are rumors that huge amount of defects of such type was the primary reason for Oracle version 9 failure (and further rework in version 10).
So how to test that? I don’t really know
I’m sorry not to be ready to share any more examples as components I’m testing are quite a low level, complicated and requires context to understand. If I will run into any defect that is simple to describe I will add it here.
To find such a bugs you should know about both features involved, so you need a tester who knows each feature of the product to test it, don’t you? I do report such a bugs quite frequently. I’m test team lead for my project. The team is not too big and I am in the project from it’s very beginning (1.5 years already) so I’m lucky to know all the features of the product (it’s also supported by my skills). Well for each single feature they may be another tester who knows it better than me, but it is not required for finding integration issues. I don’t know how to solve this in a projects so large or long that you can’t have a single person who knows all the features (as in case of Oracle 9). If anyone knows, please share.