Example of Managing Auto Failure of Tests

0

If a unit test is not able to finish its run then there is an indication that probably it is hanging which in turn may break the whole testing. Its important for the developer to know that which test is consuming more time. In this article, we will show you how to enforces a test case to run for only an allotted time so that we can prevent them from hanging and instead exit the JVM.

Idea is to run a separate thread which can monitor the unit test’s processing time. If the method is taking more time than the allowed processing time and test has not finished its run, the thread will dump all the threads in stack and terminate the currently running Java Virtual Machine. If the test has managed to finish in time, a flag will be set indicating the success and the thread monitoring the cases will be interrupted. The interrupted thread will then verify whether the test’s run was successful. If the unit test has run OK, the thread will die and a new monitoring thread will start for the next unit test.

In setUp(), we start the auto-failure thread to monitor the processing time. The thread started will simply sleep for the allowed run time. There are two possibilities, either the unit test completes its run within allowed time or the thread exceeds the allowed run time. If it is completes its run, it will interrupt the monitoring thread from its sleep. Once awake, the thread will simply exit and a new monitoring thread will start when next test is ready to run.
If the thread exceed the allowed time, the monitoring thread will automatically come off its sleep and then it will decide whether to exit the JVM. If the test has run by that time, it will simply exit else its going to dump all the threads and shutdown the JVM.

In dumpAllThreads(), we will get all the stack frames. We will then print each thread and its stack trace elements.

In tearDown(), we will mark the test as successful and interrupt the monitoring thread.

AutoFailTest:

package com.javarticles.test;

import java.util.Map;
import java.util.Map.Entry;
import java.util.concurrent.atomic.AtomicBoolean;

import junit.framework.TestCase;

public class AutoFailTest extends TestCase {
    public static final int EXIT_ERROR = 1;
    private long maxTestTime = 1 * 60 * 1000;
    private Thread autoFailThread;
    private AtomicBoolean isTestSuccess;

    protected void setUp() throws Exception {
        startAutoFailThread();
        super.setUp();
    }
    
    protected void tearDown() throws Exception {
        super.tearDown();

        markTestAsSuccesfulAndStopAutoFailThread();
    }

    public void startAutoFailThread() {
        isTestSuccess = new AtomicBoolean(false);
        autoFailThread = new Thread(new Runnable() {
            public void run() {
                try {
                    // Wait for test to finish succesfully
                    Thread.sleep(getMaxTestTime());
                } catch (InterruptedException e) {
                    // This usually means the test was successful
                } finally {
                    // Check if the test was able to tear down succesfully,
                    // which usually means, it has finished its run.
                    if (!isTestSuccess.get()) {
                        System.out.println("Test case has exceeded the maximum allotted time to run of: " + getMaxTestTime() + " ms.");
                        dumpAllThreads(getName());
                        System.exit(EXIT_ERROR);
                    }
                }
            }
        }, "AutoFailThread");

        System.out.println("Starting auto fail thread...");
        autoFailThread.start();
    }

    public void setMaxTestTime(long val) {
        this.maxTestTime = val;
    }

    public long getMaxTestTime() {
        return this.maxTestTime;
    }
    
    
    public static void dumpAllThreads(String prefix) {
        Map<Thread, StackTraceElement[]> stacks = Thread.getAllStackTraces();
        for (Entry<Thread, StackTraceElement[]> stackEntry : stacks.entrySet()) {
            System.err.println(prefix + " " + stackEntry.getKey());
            for(StackTraceElement element : stackEntry.getValue()) {
                System.err.println("     " + element);
            }
        }
    }
    
    public void markTestAsSuccesfulAndStopAutoFailThread() {
        if (autoFailThread != null && autoFailThread.isAlive()) {
            isTestSuccess.set(true);
            System.out.println("Stopping auto fail thread...");
            autoFailThread.interrupt();
        }
    }
}

In order to test the failure scenario where the test is going to take more time than allowed, we will make test case testA(), sleep a millisecond more than the allowed time, testB() completes its run within time.

FailTest:

package com.javarticles.test;


public class FailTest extends AutoFailTest {

    public void testA() {
        System.out.println("test A");
        try {
            Thread.sleep(getMaxTestTime() + 1);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }
    
    public void testB() {
        System.out.println("test B");
    }   
}

You can see from the output, the monitoring thread exists the system.

Output:

Starting auto fail thread...
test B
Stopping auto fail thread...
Starting auto fail thread...
test A
Test case has exceeded the maximum allotted time to run of: 60000 ms.
Stopping auto fail thread...
testA Thread[Reference Handler,10,system]
     java.lang.Object.wait(Native Method)
     java.lang.Object.wait(Object.java:502)
     java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
testA Thread[ReaderThread,5,main]
     java.net.SocketInputStream.socketRead0(Native Method)
     java.net.SocketInputStream.read(SocketInputStream.java:150)
     java.net.SocketInputStream.read(SocketInputStream.java:121)
     sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
     sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
     sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
     java.io.InputStreamReader.read(InputStreamReader.java:184)
     java.io.BufferedReader.fill(BufferedReader.java:161)
     java.io.BufferedReader.readLine(BufferedReader.java:324)
     java.io.BufferedReader.readLine(BufferedReader.java:389)
     org.eclipse.jdt.internal.junit.runner.RemoteTestRunner$ReaderThread.run(RemoteTestRunner.java:135)
testA Thread[AutoFailThread,5,main]
     java.lang.Thread.dumpThreads(Native Method)
     java.lang.Thread.getAllStackTraces(Thread.java:1603)
     com.javarticles.test.AutoFailTest.dumpAllThreads(AutoFailTest.java:61)
     com.javarticles.test.AutoFailTest$1.run(AutoFailTest.java:40)
     java.lang.Thread.run(Thread.java:745)
testA Thread[Signal Dispatcher,9,system]
testA Thread[main,5,main]
     java.lang.Thread.isAlive(Native Method)
     com.javarticles.test.AutoFailTest.markTestAsSuccesfulAndStopAutoFailThread(AutoFailTest.java:71)
     com.javarticles.test.AutoFailTest.tearDown(AutoFailTest.java:23)
     junit.framework.TestCase.runBare(TestCase.java:140)
     junit.framework.TestResult$1.protect(TestResult.java:110)
     junit.framework.TestResult.runProtected(TestResult.java:128)
     junit.framework.TestResult.run(TestResult.java:113)
     junit.framework.TestCase.run(TestCase.java:124)
     junit.framework.TestSuite.runTest(TestSuite.java:243)
     junit.framework.TestSuite.run(TestSuite.java:238)
     org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
     org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
     org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
     org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
     org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
     org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
     org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)
testA Thread[Finalizer,8,system]
     java.lang.Object.wait(Native Method)
     java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:142)
     java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:158)
     java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
testA Thread[Attach Listener,5,system]

In the success scenario, we have two test cases which simply prints some message.

SuccessTest:

package com.javarticles.test;


public class SuccessTest extends AutoFailTest {

    public void testA() {
        System.out.println("test A");       
    }
    
    public void testB() {
        System.out.println("test B");
    }    
}

Output:

Starting auto fail thread...
test B
Stopping auto fail thread...
Starting auto fail thread...
test A
Stopping auto fail thread...

Download the source code

This was an example about how to manage auto failure of tests.

You can download the source code here: javaAutoFailTestExample.zip
Share.

Comments are closed.