CartPole Java

This example demonstrates using the classic CartPole example in Java. It assumes the default inkling when the user creates a CartPole example in the Bonsai dashboard.

The focus of the example is the connection between a model in Java and the Bonsai platform.

Structure

The main entry point is App.java.

References

This example makes us of the Bonsai client SDK for Java, available .

Download the client SDK project, then run:

mvn install 

in the root directory where the pom.xml file exists. This installs in your local Maven repository and makes it available for the CartPole example to use.

References are managed in the pom.xml file:

<!-- bonsai reference -->
<dependency>
    <groupId>com.microsoft.bonsai.simulatorapi</groupId>
    <artifactId>bonsai-sdk</artifactId>
    <version>0.1.0</version>
    <scope>compile</scope>
</dependency>

App.java

The main interaction between the model and Bonsai happens in App.java.

Imports

The following imports need to be included:

import java.util.LinkedHashMap;

import com.microsoft.bonsai.client.*;
import com.microsoft.bonsai.generated.Sessions;
import com.microsoft.bonsai.generated.models.*;

import com.fasterxml.jackson.databind.*;

Client configuration

All initial sequence IDs are set to 1. Subsequent sequence IDs are generated by the service.

It should be noted that this example is using access keys to connect to the platform.

BonsaiClientConfig bcConfig = new BonsaiClientConfig(getWorkspace(),getAccessKey());
BonsaiClient client = new BonsaiClient(bcConfig);

Be sure to edit the getWorkspace and getAccessKey to put your respective workspace and access key.

Registering

The following steps are required for registering with Bonsai:

Sessions sessions = client.sessions();
                
SimulatorInterface sim_interface = new SimulatorInterface();
sim_interface.withName("Cartpole-Java");
sim_interface.withTimeout(60.0);
sim_interface.withCapabilities(null);

// minimum required
sim_interface.withSimulatorContext(bcConfig.simulatorContext);

//create only returns an object, so we need to check what type of object
Object registrationResponse = sessions.create(workspaceName, sim_interface);

// if we get an error during registration
if(registrationResponse.getClass() == ProblemDetails.class)
{
    ProblemDetails details = (ProblemDetails)registrationResponse;
    System.out.println(java.time.LocalDateTime.now() + " - ProblemDetails - " +  details.title());
}
// successful registration
else if(registrationResponse.getClass() == SimulatorSessionResponse.class)
{
    registered = registrationResponse;
    SimulatorSessionResponse sessionResponse = (SimulatorSessionResponse)registrationResponse;

    //this is required
    sessionId = sessionResponse.sessionId();
}

System.out.println(java.time.LocalDateTime.now() + " - registered session " + sessionId);

Control loop

After registering an receiving a session ID, the following can be used to check the events that are received from the service in the control loop:

// build the SimulatorState object
SimulatorState simState = new SimulatorState();
simState.withSequenceId(sequenceId); // required
simState.withState(model.getState()); // required
simState.withHalted(model.halted()); // required

// advance only returns an object, so we need to check what type of object
Object response = client.sessions().advance(workspaceName, sessionId, simState);

// if we get an error during advance
if (response.getClass() == ProblemDetails.class) {
    ProblemDetails details = (ProblemDetails) response;

    System.out.println(java.time.LocalDateTime.now() + " - ProblemDetails - " + details.title());
}
// succesful advance
else if (response.getClass() == Event.class) {

    Event event = (Event) response;
    System.out.println(java.time.LocalDateTime.now() + " - received event: " + event.type());
    sequenceId = event.sequenceId(); // get the sequence from the result

    // now check the type of event and handle accordingly

    if (event.type() == EventType.EPISODE_START) {
        // ignored event in Cartpole
        Config config = new Config();

        // If you have config values, pass them to the start event on your model:
        //event.episodeStart().config()
        model.start(config);

    } else if (event.type() == EventType.EPISODE_STEP) {
        Action action = new Action();

        // action() returns an Object with a class value of LinkedHashMap
        LinkedHashMap map = (LinkedHashMap) event.episodeStep().action();

        // get the value of the action
        Object theCommand = map.get("command");

        // sometimes that value is an integer -- somtimes its a double. in either case,
        // Action.command is a double
        if (theCommand.getClass() == Integer.class)
            action.command = ((Integer) theCommand).doubleValue();
        else if (theCommand.getClass() == Double.class)
            action.command = ((Double) theCommand).doubleValue();

        // move the model forward
        model.step(action);
    } else if (event.type() == EventType.EPISODE_FINISH) {
        System.out.println("Episode Finish");
    } else if (event.type() == EventType.IDLE) {
        Thread.sleep(event.idle().callbackTime().longValue() * 1000);
    } else if (event.type() == EventType.UNREGISTER) {
        client.sessions().delete(workspaceName, sessionId);
    }
}

Package

You can run mvn package from the command prompt in the directory where the pom.xml file exists. This will create the target\cartpole-1.0.jar and target\cartpole-1.0-jar-with-dependencies.jar files. The cartpole-1.0-jar-with-dependencies.jar file is a single jar file that contains all of the needed references to run.

Run Local

After packaging the app, run:

java -jar target\cartpole-1.0-jar-with-dependencies.jar

If you want to run in prediction, run:

java -jar target\cartpole-1.0-jar-with-dependencies.jar predict <URL>

where is your http endpoint for your exported brain.

Containerize

This example also includes steps to containerize and scale the training of the simulator.

From the command prompt in the docker folder, run:

build-docker.bat <docker_image_name> <azure_acr_name>

This will copy the JAR file over, build a docker container, prompt for a login to the Azure Container Registry, tag, and push your container to it.

You can now register the simulator with Bonsai.

Be sure to update the package statement in your inkling code.