Unpredictable value when converting byte array to int | Eclipse Temurin-17.0.10+11

[ameys2907]

Please provide a brief summary of the bug

Problem Description:

I am encountering an issue when running the following multi-threaded program. The program spawns a large number of threads (10,000) that process the same byte array value. The issue becomes more frequent under high CPU usage and when memory constraints are applied (e.g., setting -Xms32m -Xmx32m).

The method byteArrayToInt is supposed to convert a specific segment of the byte array into an integer. However, intermittently, it produces incorrect values. The expected value for liVersion is 1, but random values are being observed instead. This issue occurs even when the program is executed with sufficient memory and less number of threads (even single thread) but becomes significantly less frequent.

The integer conversion of 2nd to 4th value of pTestData which is [1, 0] should be 1, but it intermittently gives random values in the output.

Environment:

Operating system: Windows 10

Java version: Eclipse Temurin-17.0.10+11

Code snippet:

import java.util.Arrays;

public class TestMain
{
	private static int miParallelThreadCount = 10000;

        private static long mlExecutionIntervalInMillis=100;
	    
	public static void main(String[] args)
	{
		System.out.println("Startin tornado at interval: " + mlExecutionIntervalInMillis + " with parallel threads: " + miParallelThreadCount);
		while(true)
		{
                        //Runs the test code to reproduce the issue
			runTornado();
			try
			{
				Thread.sleep(mlExecutionIntervalInMillis);
			} catch (Throwable lThrowable)
			{
				System.out.println("Task failed due to: " + lThrowable.getLocalizedMessage());
    	    	lThrowable.printStackTrace();
			}
		}
	}

    private static void runTornado() 
    { 	
        //Test byte array
    	byte[] data = new byte[]
    			{ 65, 50, 1, 0, 1, 0, 36, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 16, 0, 53, -76, -66, 125, -115, 69, -111, 80, -81, -33, -62, 84, -59, -64, -114, 113, -92, -59, 37, 106, 85, -111, -100, -19, 114, 99, 18, -9, -62, -123, -75, -22, -40, 70, -70, -46, 119, 11, -17, -71, -93, -126, -28, 116, 104, -4, -45, -58, 121, 4, 50, 76, 46, 13, -91, -76, 37, -103, 101, 57, 54, 75, 83, -124 };
    	
    	for(int i=0; i<miParallelThreadCount; ++i)
        {
                //This snippet executes the group of compareInner methods in new threads
    		new Thread(() -> 
            {
            	try
            	{
            		compareInner(data, 0, data.length);
            		compareInner(data, 0, data.length);
            		compareInner(data, 0, data.length);
            		compareInner(data, 0, data.length);
            		compareInner(data, 0, data.length);
            		compareInner(data, 0, data.length);
            		compareInner(data, 0, data.length);
            		compareInner(data, 0, data.length);
            		compareInner(data, 0, data.length);
            		compareInner(data, 0, data.length);
            		compareInner(data, 0, data.length);
            		compareInner(data, 0, data.length);
            		compareInner(data, 0, data.length);
            		compareInner(data, 0, data.length);
            		compareInner(data, 0, data.length);
        	    } 
        	    catch (Throwable lThrowable) 
        	    {
        	    	System.out.println("Something went wrong: " + lThrowable.getLocalizedMessage());
        	    	lThrowable.printStackTrace();
        	    }
            }).start();
        }
    	System.out.println("Ran tornado");
    }

	private static byte[] compareInner(byte[] pTestData, int pStartIndex, int pLength)
	{
                //Copies the pTestData into new array as it is
		byte[] lbarrData = Arrays.copyOfRange(pTestData, 0, 0 + pTestData.length);
               //copies only 2ndand 3rd element from the array which is [1, 0]
		byte[] lbarrVer = Arrays.copyOfRange(lbarrData, 2, 4);

               //Converts the array [1, 0] into integer, which should always return 1
		int liVersion = byteArrayToInt(lbarrVer);

		if(liVersion != 1)
		{
			System.out.println("Invalid Version : " + liVersion + " data array: " + Arrays.toString(lbarrData) +  " header array: " + Arrays.toString(lbarrVer));
		}
		
		return lbarrData;
	}
	
	public static final int byteArrayToInt(byte[] pSource)
	{
		int lValue = 0;
		for (int i = 0; i < pSource.length; i++)
		{
			lValue += (pSource[i] & 0xff) << (8 * i);
		}
		return lValue;
	}
}

Further analysis:

• I tried running this code with 32m xms and xmx, issue is reproducible.
• Issue is reproducible even when xms and xmx is 1024 but less frequent.
• Issue is seen in single thread (miParallelThreadCount = 1) but is less frequent.
• Issue has not been observed when taken a debug on the code.
• Issue is not observed when a "compareInner(enc, 0, enc.length);" method is called only once per thread execution.

Did you test with the latest update version?

• Yes

Please provide steps to reproduce where possible

No response

Expected Results

Startin tornado at interval: 100 with parallel threads: 10000
(Nothing will be in output except the starting line since the liVersion is 1)

Actual Results

Startin tornado at interval: 100 with parallel threads: 10000
Invalid Version : 50 data array: [65, 50, 1, 0, 1, 0, 36, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 16, 0, 53, -76, -66, 125, -115, 69, -111, 80, -81, -33, -62, 84, -59, -64, -114, 113, -92, -59, 37, 106, 85, -111, -100, -19, 114, 99, 18, -9, -62, -123, -75, -22, -40, 70, -70, -46, 119, 11, -17, -71, -93, -126, -28, 116, 104, -4, -45, -58, 121, 4, 50, 76, 46, 13, -91, -76, 37, -103, 101, 57, 54, 75, 83, -124] header array: [1, 0]

What Java Version are you using?

openjdk 17.0.10 2024-01-16 OpenJDK Runtime Environment Temurin-17.0.10+7 (build 17.0.10+7) OpenJDK 64-Bit Server VM Temurin-17.0.10+7 (build 17.0.10+7, mixed mode, sharing)

What is your operating system and platform?

Windows 10, 64-bit operating system, x64-based processor.

How did you install Java?

Manually placed the java zip and added JAVA_HOME and path in system variable.

Did it work before?

No response

Did you test with other Java versions?

Yes, with AdoptOpenJDK 11.0.11 and the issue has not been observed yet.

Relevant log output

Startin tornado at interval: 100 with parallel threads: 10000
Invalid Version : 50 data array: [65, 50, 1, 0, 1, 0, 36, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 16, 0, 53, -76, -66, 125, -115, 69, -111, 80, -81, -33, -62, 84, -59, -64, -114, 113, -92, -59, 37, 106, 85, -111, -100, -19, 114, 99, 18, -9, -62, -123, -75, -22, -40, 70, -70, -46, 119, 11, -17, -71, -93, -126, -28, 116, 104, -4, -45, -58, 121, 4, 50, 76, 46, 13, -91, -76, 37, -103, 101, 57, 54, 75, 83, -124] header array: [1, 0]

[karianna]

please try 17.0.13 and report back

[ameys2907]

Yes, i’m able to reproduce it on 17.0.13 as well.

[karianna]

can you please post the full log for 17.0.13 (pathways might be different).

[ameys2907]

can you please post the full log for 17.0.13 (pathways might be different).

Hi please find the below image with the java version and logs of the code snippet:

[karianna]

So I think you're running across a limit of what the JVBM supports with this programming model:

The runTornado method creates a massive number of threads (miParallelThreadCount), each running tasks that are CPU-intensive. With miParallelThreadCount set to 10000, this can overwhelm the thread scheduler, leading to:
• High context-switching overhead.
• Memory exhaustion (each thread consumes stack space).
• Potential JVM crashes or degraded performance.

Recommend that you use a thread pool (e.g., ExecutorService) instead of creating new threads in each iteration.

If you still want to report this as a bug, then the upstream OpenJDK project is the place to do so: https://mail.openjdk.org/mailman/listinfo/concurrency-discuss

[ameys2907]

Thank you for your detailed explanation. I agree that the current programming model may not be optimal, and using a thread pool instead of creating a massive number of threads is a better approach.

However, my concern is about the specific behavior of Eclipse Temurin 17. Unlike versions 11 and 21 LTS, this issue has only been observed in version 17 so far. Additionally, even when I set miParallelThreadCount to 1, the issue still occurs intermittently after some time of execution.

This suggests that the problem may not solely be related to thread exhaustion but could also involve how the JVM handles specific operations in version 17.

Ideally, the JVM should throw an error or exception to indicate system limits or other underlying issues, instead of silently returning incorrect results. I wanted to highlight this as something worth investigating.

Also, If I run the code on multiple Docker containers (e.g., 5 containers with high CPU and thread usage, and 1 container with a single thread and minimal operations) on the same host machine, where all containers share the same kernel, can the high usage in the other containers impact the low-usage container?

[karianna]

Hi @ameys2907 Are you able to try with Amazon Corretto 17 or another vendor and see if you have the same issue?

[ameys2907]

Hi @ameys2907 Are you able to try with Amazon Corretto 17 or another vendor and see if you have the same issue?

Yes, It is reproducible on Amazon Corretto 17 as well. Please find the attached screenshot of the output:

[karianna]

OK, so that's a general OpenJDK issue then. Please report this to the upstream OpenJDK project. I think the best place to start is https://mail.openjdk.org/mailman/listinfo/concurrency-discuss

[AlexanderSchuetz97]

I saw this issue by accident and was intrigued by it.
Since it also occurs with just one other thread one doesnt have to look at the 10.000 thread use case.

My opinion for why the problem occures is as follows.
You are missing a memory fence I think.

T1 Allocates memory but defers writing the memory to ram until after the thread was started.
T2 Reads the memory without any synchronization. You are likely reading unitialized memory there.

If this is truely whats going on then I am not suprised you are seeing different behavior with different jvms. I wouldnt even be suprised if different jvm start parameters make a difference here.

Try adding VarHandle.fullFence() before your for loop. Other than that a lock/synchronized block (Not around the computations only around the thread starts and within the preamble of your child thread) should also create a memory barrier between the threads.