Test data is very important for testing, but why? and what is test data?
ISTQB - Standard Glossary defines test data as follows
Test data is data that exists (for example, in a database) before a test is executed, and that affects or is affected by the component or system under test.
Types and forms of test Data
I would categorize test data in 3 different types of test data.
Input Data
Test data which is used in a confirmatory way, typically to verify that a given set of input to a given function produces some expected result. Other data may be used in order to challenge the ability of the program to respond to unusual, extreme, exceptional, or unexpected input. This data is usually bound to an individual test case.
Baseline Data
Test data which forms the baseline for a test to be executed (e.g. data records in a database so that you can tests whether they can be deleted). Baseline test data typically refers to data in a database but not explicitly (e.g. existing files on a drive to test file operations or configuration files which brings your application into a desired state). This data is not necessarily bound to a specific test case but founds the basis (pre-condition) on which some test cases can be performed.
Output Data
Test data which is produced as an outcome of a test. This could be possibly used as input data for subsequent tests or as documented evidence.
All these test data comes in various forms like
- Parameters or variables
- Files
- Data lists (Excel files, CSV files, …)
- Data files (to be fed to the SUT via an specified interface and using a particular protocol ASTM, HL7)
- Configuration files (XML, master files, …)
- Database
This point of view of mine can be discussed but I think if you keep in mind that you will probably handle the each of them slightly different (keyword “Test Data Management”) the definition is not all too bad.
Parameters / Variables
This form of test data is most common for unit tests, where you test particular code like a method or function of a class. It is embedded in the code and therefore stored and maintained in the source control.
public void testSumPositiveNumbers() {
Adder adder = new AdderImpl();
assert(adder.add(1, 7) == 8);
}
The numbers 1 and 7 used to passed to the function add are test data. Also the result returned from adder.add(1,7) is test data. This data is usually provided directly in the code and not fed by an external source like a data list. Sure, you can also use parametrized tests, here as an example with JUnit
@RunWith(Parameterized.class)
public class SimpleConsoleParseTests {
@Parameters
public static Collection<Object[]> data() {
return Arrays.asList(new Object[][] {
{ 1, 2, 3},
{ 2, 2, 4},
{ -1, -1, -2},
{ -1, 1, 0}
});
}
private int iNbr1;
private int iNbr1;
private int iResult;
public SimpleConsoleParseTests(int iNbr1, int iNbr2, int iResult){
iNbr1= iNbr1;
iNbr2= iNbr2;
iResult = iResult;
}
@Test
public void testSumPositiveNumbers() {
Adder adder = new AdderImpl();
assert(adder.add(iNbr1, iNbr2) == iResult);
}
}
In this example you have 4 different sets of test data each of it provides 2 numbers to be added and a results to be verified.
Files
Data list
Data lists are usually test data which is represented in a table (Excel, CSV) where each line within the file represents a test data record and are a common input to run one an the same test with different data i.e. data-driven testing.
Username,Password,Login
testuser1,test1234,No
testuser2,test1234,Yes
testuser3,test1234,Yes
In the example above we could imagine that a test runs 3 times, trying to login with given username and password and verifies that the user either can login or not. The test framework would read each line of the data list and pass it to a function or interface which has to be tested. So in most cases the data in files are also only variables. However this form of data is treated slightly different as variables which are embedded in the source code; the data is separated from the source code and not necessarily in the source control (but it would be still a good idea to do so).
Data files
Data files usually represent a more complex type of test data and are more common for integration or even system testing. A single file - in contrary data lists - usually represents a single test record. In my work as test engineer in a medical device company I had to deal a lot with such data files. Here an example:
H|\^&|||Department^Company^Instrument^V7.00^1^127^10.124.69.120||||||M|P|1394-97|20100101040404
P|1||PTN040||PTNF040^PTNL430|||U||||||||0.0^cm|0.0^kg
O|1||^86839^^^^Syringe||||||||||||Blood^Venous^A. femoralis l.
R|1|^^^pH^^^M^1|7.327||7.350^7.450^reference\7.200^7.600^critical|L||F||COMPANY||20100101040404
R|2|^^^PO2^^^M^3|3.19|mmHg|10.67^13.33^reference\8.00^106.67^critical|LL||F
R|3|^^^PCO2^^^M^4|7.96|mmHg|4.67^6.00^reference\2.67^8.00^critical|H||F
R|4|^^^Hct^^^M^5|30.3|%|35.0^50.0^reference\25.0^65.0^critical|L||F
R|5|^^^Na^^^M^6|140.7|mmol/l|135.0^148.0^reference\125.0^160.0^critical|N||F
R|6|^^^K^^^M^7|3.05|mmol/l|3.50^4.50^reference\2.80^6.00^critical|L||F
R|7|^^^Ca^^^M^8|0.940|mmol/l|1.120^1.320^reference\1.050^1.500^critical|LL||F
R|8|^^^Cl^^^M^9|101.2|mmol/l|98.0^107.0^reference\80.0^115.0^critical|N||F
R|9|^^^tHb^^^M^10|8.9|g/dL|11.5^17.4^reference\8.0^23.0^critical|L||F
R|10|^^^SO2^^^M^11|39.6|%|75.0^99.0^reference\60.0^100.0^critical|LL||F
R|11|^^^O2Hb^^^M^12|38.4|%|95.0^99.0^reference\80.0^100.0^critical|LL||F
R|12|^^^COHb^^^M^13|2.3|%|0.5^2.5^reference\0.0^10.0^critical|N||F
R|13|^^^MetHb^^^M^14|0.7|%|0.4^1.5^reference\0.0^5.0^critical|N||F
R|14|^^^HHb^^^M^16|58.6|%|1.0^5.0^reference\0.0^20.0^critical|HH||F
R|15|^^^Bili^^^M^17||µmol/L|51^855^reference\51^855^critical|A||F
R|16|^^^Glu^^^M^18|4.9|mmol/l|3.3^6.1^reference\2.8^7.8^critical|N||F
R|17|^^^Lac^^^M^19|1.2|mmol/l|0.4^2.2^reference\0.2^5.0^critical|N||F
R|18|^^^Urea^^^M^24|2.6|mmol/l|2.5^6.4^reference\0.5^30.0^critical|N||F
L|1|N
Another example could be a CSV file - similar to the example above - but where the whole file is used as an import for e.g. testing and import function of your SUT.
Configuration files
I also consider any configuration file as test data. It may not be necessarily explicit for a particular test case but such files configure your SUT as desired for your testing.
Databases
A database used for testing is also test data. Yes also a database is mainly a data file but is serves usually as a baseline for further tests and due it’s possible big size and format it may also not be stored in a source control. Whenever possible, I would create script files which can re-create the data structure and data in the database. This also simplifies correction and adjustments to the test data as well as the versioning using source control.
Data Sources
Data can come from different sources and depending on it data acquisition and data conditioning looks different
- production / customer data
- from scratch
- existing test data
Something I will discuss more thoroughly in another article.
Test Data Management
Test data management is all about handling of test data (analyzing, creating, versioning, …) and is very crucial in the test engineering as test data is strongly coupled to a particular test case version and likewise the test case version itself is strongly coupled to a particular version of the test object. I always illustrate this with a simple example:
So it shall be clear that test data shall be managed to ensure that
- data is correct and adequate to the test goal and the test object
- data belongs to the correct version of the test case
- data which is referenced and deployed was not modified (i.e. test data matches with test case)
- data can be easily extracted and stored
However this is a big topic and therefore I will discuss this in another article.