Most Java developers are familiar with, either by practice or at least by reputation, JUnit. JUnit is an open source unit testing framework for Java often used in test-driven development environments. However, there is another option that is less know but equally, and some might say more, powerful than JUnit and that is TestNG. Both JUnit and TestNG are both considered Java implementations of Kent Beck and Eric Gamma's Smalltalk SUnit (commonly referred to as xUnit frameworks).
Both are so close in implementation, that Spring's Test Context Framework allows either to be used (to some degree interchangeably) with the same effectiveness.
Getting your Java Unit Testing Framework
JUnit is available at http://www.junit.org/ and the current release is 4.10. TestNG is available at http://testng.org. .As of this writing, the latest release is 6.3.1.
Comparison
At the core, you'll find JUnit and TestNG very similar in API and functionality. Both TestNG and JUnit are largely annotation-based frameworks. Both offer tests that can be executed through Eclipse (via plugin in one case) and other IDEs like IntelliJ by IDEA (http://www.jetbrains.com/idea/features/junit_testng.html).
There are a few good online comparisons of JUnit and TestNG (links provided below) and so I won't provide a complete comparison here.
There are plenty of examples of small syntactical differences between the two frameworks, but these are usually more stylistic than substantive. Take for example the annotation to ignore a test. In JUnit, a special @Ignore annotation defines a test method that should be skipped. TestNG accomplishes the same objective with a parameter on @Test annotation (the enabled parameter).
1: // TestNG skipped test
2: @Test(enabled=false)
3: public void testSaveContactList() {
4: for (Contact contact : list.getContacts()) {
5: service.addContact(contact);
6: System.out.println(contact + " was added to the Contact DB");
7: }
8: }
1: // JUnit skipped test
2: @Test
3: @Ignore
4: public void testSaveContactList() {
5: for (Contact contact : list.getContacts()) {
6: service.addContact(contact);
7: System.out.println(contact + " was added to the Contact DB");
8: }
9: }
Another example of a small syntactical difference can be found in the before and after test initialization method annotations. JUnit uses @Before and @After. TestNG uses @BeforeMethod and @AfterMethod.
1: // TestNG test initialization
2: @BeforeMethod
3: public void doSetup(){
4: System.out.println("initializing");
5: }
1: // JUnit test initalization
2: @Before
3: public void doSetup(){
4: System.out.println("initializing");
5: }
Where the two differ is that TestNG offers a bit more functionality around initialization and tear down procedures regarding test suites/groups. @BeforeSuite, @AfterSuite, @BeforeGroups, and @AfterGroups are TestNG annotations for conducting setup/tear down work before or after a suite or group of tests. JUnit and TestNG offer capability to do initialization and clean up before and after each test or before or after all tests in a class, but these unique TestNG annotations allow for initialization and clean up before and after suites and groups of tests. JUnit does not offer similar capability.
Organizationally, the diagram below indicates the order in which these initialization/tear down methods are invoked with regard to TestNG tests.
TestNG also allows for dependencies between tests. Using the dependsOnMethods parameter of the @Test annotated methods, you can create methods that execute only upon the successful execution of other test methods. In the example below, the testDeleteContact method (deleting a Contact with ID = 1) only executes if the testGetContact method is able to retrieve a Contact with ID = 1.
1: @Test
2: public void testGetContact(){
3: System.out.print("Getting contact 1");
4: service.getContact(1L);
5: }
6:
7:
8: @Test(dependsOnMethods = { "testGetContact" })
9: public void testDeleteContact(){
10: System.out.println("Deleting contact 1");
11: service.removeContact(1L);
12: }
JUnit also has a construct foreign to TestNG. The JUnit Runner allows you to tell JUnit how a test should run and then executes the tests. The Runner notifies a RunNotifier of events. This allows the Runner to notify JUnit of the progress running tests. Runners and RunNotifiers allow for a lot of customization in how JUnit executes tests and displays results.
There are also capabilities that both frameworks offer that are implemented in such different ways that some may really prefer the syntax and organization of the function better in one framework over the other. TestNG groups versus JUnit categories comes to mind. Both allow test methods to be organized into more discrete partitions of tests. This allows you to include or exclude certain tests and/or devise more flexible test plan organizations. I find TestNG's groups a little easier to work with as JUnit's categories require setting up special marker interfaces, but both groups and categories get the job done.
1: // TestNG groups. In this case two groups are defined: foos and bars
2: @Test(groups = { "foos" })
3: public void testMethodFoo() {
4:
5: }
6:
7: @Test(groups = { "bars" })
8: public void testMethodBar() {
9:
10: }
11:
12: @Test(groups = { "foos", "bars" })
13: public void testMethodFarBar() {
14: }
1: // JUnit categories. In this case two categories are defined: foos and bars
2: public interface Foos { }
3: public interface Bars { }
4:
5: public class A {
6: @Category(Foos.class)
7: @Test public void testMethodFoo() {}
8: }
9:
10: @Category(Bards.class})
11: public class B {
12: @Test public void testMethodBar() {}
13: }
14:
15: @RunWith(Categories.class)
16: @IncludeCategory(Bars.class)
17: @ExcludeCategory(Foos.class)
18: @SuiteClasses({ A.class, B.class })
19: public class MyTestSuite {}
Note, categories were added to JUnit 4.8. Therefore it is a pretty recent addition to JUnit and not available in most JUnit environments.
Another example of somewhat similar functionality but rather different implementations comes in parameter passing between tests (something JUnit did not allow by design in order to keep individual tests isolated and separate from on another until relatively recent).
1: // TestNG - one of many ways to pass parameters to a test method
2: @DataProvider(name = "firstName")
3: public Object[][] createData() {
4: return new Object[][] { { "Jim" }, { "Joe" }, };
5: }
6:
7: @Test(dataProvider = "firstName")
8: public void testPassParameter(String firstName) {
9: System.out.println("Invoked test with String " + firstName);
10: }
1: // JUnit - one way to use parameters in methods
2: @RunWith(Parameterized.class)
3: public class MyTest {
4:
5: private String firstName;
6:
7: public MyTest(String firstName){
8: this.firstName = firstName;
9: }
10:
11: @Parameters
12: public static Collection<Object[]> generateData() {
13: return Arrays.asList(new Object[][] { { "Jim" }, { "Joe" } });
14: }
15:
16: @Test
17: public void testPassParameter() {
18: System.out.println("Invoked test with String " + firstName);
19: }
20: }
IDE Support
By and large, you will find the major IDE's support either JUnit or TestNG. The difference is whether the IDE comes with built-in support or does it require a plug-in to support the unit test framework. JUnit support is already built into Eclipse. If you create a JUnit test or test suite, you can immediately run the class as a JUnit Test. Results of the test are displayed in a built in a JUnit Eclipse view.
TestNG requires a plugin be installed to execute TestNG tests. Instructions for installing the plugin can be found on the Intertech site here: TestNG Plugin Document. Once the plug-in is installed, a TestNG view displays the test results similar to the JUnit view.
IntelliJ supports both JUnit and TestNG (see here). NetBeans supports JUnit (see here), but also requires a plugin like Eclipse for TestNG (see here).
Performance (Unscientific)
I conducted a little test to see how well both JUnit and TestNG performed. In particular, I had a ContactDAO class. I performed 1000 each of the following test operations: get all contact objects, get a contact object by id, update a contact object, and add a new contact object. So each unit test performed 4000 DAO method calls. After each unit test, I emptied the database and reset the environment back to its original state. The results are shown in the table below.
| times in milliseconds |
TestNG |
JUnit |
| Test Run #1 |
6803 |
6076 |
| Test Run #2 |
7317 |
6108 |
| Test Run #3 |
9020 |
6045 |
| Test Run #4 |
6850 |
5872 |
| Test Run #5 |
8004 |
5952 |
| Average Run Time |
7599 |
6011 |
| Standard Deviation |
851 |
87 |
Based on my test, you can see that JUnit did marginally better, but perhaps more importantly, JUnit performed more consistently than did TestNG. Again, this study should not be considered scientific and you should perform your own analysis to be sure the performance of the unit test framework you choose meets your needs.
Spring Integration
Spring has acknowledge the importance of both frameworks in that the Spring Test Context framework allows either to be used. Today, the TestContext Framework works with JUnit (version 4 or better), and TestNG (version 5 or better) unit testing tools out of the box. JUnit 3.8 is supported but considered deprecated. The reasons for using the TestContext Framework (versus just using a unit test framework directly) are many.
- Using the Spring TestContext Framework allows unit tests and unit test code to be decoupled from the actual testing framework. This allows you to switch between TestNG and JUnit as your unit test framework of choice with relative ease.
- Spring provides transaction management that better supports unit testing.
- Transaction management support allows changes to the database to be appropriately rolled back after conducting tests.
- Spring dependency injection can be used to construct more/better integrated component tests - facilitating tests at the unit and integration level.
Except for the unit test framework specific annotations that can be used, the only real difference in using JUnit versus TestNG with Spring is that when using JUnit4, the TestContext class can be annotated with the @RunWith annotation that specifies the unit test runner used by the framework for executing tests. When using TestNG, the TestContext class must extend one of Spring's TestContext support classes (like AbstractTestNGSpringContextTests).
Wrap Up
Beyond JUnit and TestNG, you'll find plenty of other open-source Java Testing Tools at java-source.net/open-source/testing-tools. Before thinking Java quality assurance and test driven development is all about JUnit, check out TestNG and do a little research into some of the additional tooling.
While you are at it, check out Intertech's JUnit class. Find more information at Express JUnit. While it is only a small part of our Spring class, you'll also find JUnit and TestNG integration is covered in our Spring class (see Complete Spring 3 Framework).