Testing speed of code execution with Java/Eclipse

Question

I want to test which of the two methods executes faster, however depending on which one I run first, that method seems to always run slower. I suspect there is some caching mechanism Eclipse is using that speeds up the execution of the second method.

private static int method1(int n) {
    int result = 0;
    for (int i = 0; i < n + 1; ++i) {
        result += i;
    }
    return result;
}

private static int method2(int n) {
    int result = 0;
    int i = 0;
    while (i < n + 1) {
        result += i++;
    }
    return result;
}

This is the main function I am using to test the difference in times.

long start, end;

start = new Date().getTime();
for (int i = 0; i < 100000; ++i) {
    method1(i);
}
end = new Date().getTime();
System.out.println(end - start); // 47

start = new Date().getTime();
for (int i = 0; i < 100000; ++i) {
    method2(i);
}
end = new Date().getTime();
System.out.println(end - start); // 32

Answer 1

Using jmh, I get the following results (for various values of n), score is in nanoseconds per method call (smaller = better):

Benchmark                          (n)  Mode  Samples          Score         Error  Units
c.a.p.SO31495089.method1             1  avgt       10          2.149        0.027  ns/op
c.a.p.SO31495089.method1        100000  avgt       10      34626.763      441.915  ns/op
c.a.p.SO31495089.method1    1000000000  avgt       10  322506247.405  6774340.047  ns/op
c.a.p.SO31495089.method2             1  avgt       10          2.159        0.028  ns/op
c.a.p.SO31495089.method2        100000  avgt       10      34581.273      571.416  ns/op
c.a.p.SO31495089.method2    1000000000  avgt       10  320011679.005  4049907.844  ns/op

Second run:

Benchmark                          (n)  Mode  Samples          Score         Error  Units
c.a.p.SO31495089.method1             1  avgt       10          2.164        0.029  ns/op
c.a.p.SO31495089.method1        100000  avgt       10      34706.194      365.189  ns/op
c.a.p.SO31495089.method1    1000000000  avgt       10  320269697.300  1696038.683  ns/op
c.a.p.SO31495089.method2             1  avgt       10          2.160        0.040  ns/op
c.a.p.SO31495089.method2        100000  avgt       10      34627.163      325.981  ns/op
c.a.p.SO31495089.method2    1000000000  avgt       10  320698252.840  2332275.430  ns/op

Bottom line: the two methods perform similarly, as expected (the difference in execution time is smaller than the measurement error).

Answer 2

Using new Date().getTime() gives the wall clock time. But it is really not what we as developers want (most of the time, until doing some benchmarking where enterprise level benchmarking are used) because wall clock time is effected by many back ground processes, so to counter that Java provides more sophisticated API for measuring the time.

To exclude the effects of other system activity, you need to measure application "User time" instead.

• "User time" is the time spent running your application's own code.
• "CPU time" is user time plus system time. It's the total time spent using a CPU for your application.

Below example to demonstrate CPU and User time calculation using ManagementFactory.getThreadMXBean() API.

    Thread thread = new Thread(){
            public void run() {
                for (int i = 0; i < 100000; ++i) {
                    int result = 0;
                    for (int j = 0; j < i + 1; ++j) {
                        result += j;
                    }
                }
                System.out.println("FOR approach: ThreadCpuTime = " + ManagementFactory.getThreadMXBean().getCurrentThreadCpuTime()/1000000000d);
                System.out.println("FOR approach: UserTime = " + ManagementFactory.getThreadMXBean().getCurrentThreadUserTime()/1000000000d);
            };
        };
        thread.start();

        Thread thread2 = new Thread(){
            public void run() {
                for (int i = 0; i < 100000; ++i) {
                    int result = 0;
                    int j = 0;
                    while (j < i + 1) {
                        result += j++;
                    }
                }
                System.out.println("WHILE approach: ThreadCpuTime = " + ManagementFactory.getThreadMXBean().getCurrentThreadCpuTime()/1000000000d);
                System.out.println("WHILE approach: UserTime = " + ManagementFactory.getThreadMXBean().getCurrentThreadUserTime()/1000000000d);
            };
        };
        thread2.start();

Having said, I am really not sure why you are getting unexpected behavior, I ran your code both in Eclipse and IntelliJ IDE, and I always got FOR loop approach as faster than WHILE loop.

Probably try to restart you Eclipse and have lesser number of background processes running OR do not run it Eclipse but run the tests from Java command line so that you can sure of results.

As can be seen from below Byte Code Analysis that WHILE and FOR loop approach has same bytes codes generated, which means there would be same assemble code and hence same time CPU will take to execute instructions.

But practically when we run in your IDE or other wise, then is being effect by background processes, so different times are observed. But in this particular case - WHILE v/s FOR, it is more appropriate to do byte code analysis and conclude that WHILE and FOR loop approaches will take same time.

---

Byte code of FOR loop:

{
  public Test2();
    flags: ACC_PUBLIC
    Code:
      stack=1, locals=1, args_size=1
         0: aload_0
         1: invokespecial #1                  // Method java/lang/Object."<init>":()V
         4: return
      LineNumberTable:
        line 1: 0

  public static void main(java.lang.String[]);
    flags: ACC_PUBLIC, ACC_STATIC
    Code:
      stack=2, locals=2, args_size=1
         0: iconst_0
         1: istore_1
         2: iload_1
         3: bipush        10
         5: if_icmpge     21
         8: getstatic     #2                  // Field java/lang/System.out:Ljava/io/PrintStream;
        11: iload_1
        12: invokevirtual #3                  // Method java/io/PrintStream.println:(I)V
        15: iinc          1, 1
        18: goto          2
        21: return
      LineNumberTable:
        line 3: 0
        line 4: 8
        line 3: 15
        line 6: 21
      StackMapTable: number_of_entries = 2
           frame_type = 252 /* append */
             offset_delta = 2
        locals = [ int ]
           frame_type = 250 /* chop */
          offset_delta = 18

}

Byte code of WHILE loop:

{
  public Test();
    flags: ACC_PUBLIC
    Code:
      stack=1, locals=1, args_size=1
         0: aload_0
         1: invokespecial #1                  // Method java/lang/Object."<init>":()V
         4: return
      LineNumberTable:
        line 1: 0

  public static void main(java.lang.String[]);
    flags: ACC_PUBLIC, ACC_STATIC
    Code:
      stack=2, locals=2, args_size=1
         0: iconst_0
         1: istore_1
         2: iload_1
         3: bipush        10
         5: if_icmpge     21
         8: getstatic     #2                  // Field java/lang/System.out:Ljava/io/PrintStream;
        11: iload_1
        12: invokevirtual #3                  // Method java/io/PrintStream.println:(I)V
        15: iinc          1, 1
        18: goto          2
        21: return
      LineNumberTable:
        line 3: 0
        line 4: 2
        line 5: 8
        line 6: 15
        line 8: 21
      StackMapTable: number_of_entries = 2
           frame_type = 252 /* append */
             offset_delta = 2
        locals = [ int ]
           frame_type = 18 /* same */

}

Further reading:

• http://nadeausoftware.com/articles/2008/03/java_tip_how_get_cpu_and_user_time_benchmarking

Answer 3

While and For Loop have a similar performance.

Maybe this similar post would help you Java For Loop Vs While Loop Performance Difference