Add support for Amazon EC2 spot instances #133

cimi · 2013-01-29T22:07:20Z

http://aws.amazon.com/ec2/spot-instances/

Related issue #52

buildhive · 2013-01-29T22:16:23Z

Axemblr.com » axemblr-provisionr #33 SUCCESS
This pull request looks good
(what's this?)

buildhive · 2013-01-30T16:28:28Z

Axemblr.com » axemblr-provisionr #34 SUCCESS
This pull request looks good
(what's this?)

andreisavu · 2013-01-31T11:17:08Z

providers/amazon/src/main/java/com/axemblr/provisionr/amazon/activities/RunSpotInstances.java

+    	    } catch (Exception e) {
+    	        // Do nothing because it woke up early.
+    	    }
+    	} while (anyOpen);


This loop needs to be implemented as an activity similar to:

https://github.com/axemblr/axemblr-provisionr/blob/master/providers/amazon/src/main/java/com/axemblr/provisionr/amazon/activities/AllInstancesMatchPredicate.java

I've changed this in the latest commit, but I still need to figure out what the exact responsibilities of the BPM are to see if I can also skip the polling of "open" requests.

buildhive · 2013-01-31T14:26:56Z

Axemblr.com » axemblr-provisionr #35 SUCCESS
This pull request looks good
(what's this?)

buildhive · 2013-01-31T14:31:49Z

Axemblr.com » axemblr-provisionr #36 SUCCESS
This pull request looks good
(what's this?)

buildhive · 2013-01-31T16:55:41Z

Axemblr.com » axemblr-provisionr #37 SUCCESS
This pull request looks good
(what's this?)

buildhive · 2013-01-31T23:03:12Z

Axemblr.com » axemblr-provisionr #38 SUCCESS
This pull request looks good
(what's this?)

andreisavu · 2013-02-01T13:17:16Z

...on/src/main/java/com/axemblr/provisionr/amazon/activities/AllSpotRequestsMatchPredicate.java

+            }
+	    } catch (AmazonServiceException exception) {				
+			// TODO: what happens here when the request id isn't found? 
+			// The error code needs to be checked, I couldn't find anything in the docs yet.


Unless we know for sure you can always propagate the exception. The processEngine will automatically retry the activity as expected.

buildhive · 2013-02-05T08:59:55Z

Axemblr.com » axemblr-provisionr #41 SUCCESS
This pull request looks good
(what's this?)

buildhive · 2013-02-05T09:26:39Z

Axemblr.com » axemblr-provisionr #42 SUCCESS
This pull request looks good
(what's this?)

buildhive · 2013-02-05T15:12:19Z

Axemblr.com » axemblr-provisionr #43 SUCCESS
This pull request looks good
(what's this?)

andreisavu · 2013-02-06T16:04:00Z

As you make progress on this you should probably squash commits as needed and update the branch using --force.

buildhive · 2013-02-07T16:11:44Z

Axemblr.com » axemblr-provisionr #44 ABORTED

(what's this?)

…cess test for this case.

andreisavu · 2013-02-07T16:24:58Z

providers/amazon/src/main/java/com/axemblr/provisionr/amazon/activities/AmazonActivity.java

+
+    protected List<Instance> collectInstancesFromReservations(List<Reservation> reservation) {
+        /* Make a copy as an ArrayList to force lazy collection evaluation */
+        List<List<Instance>> allInstances = Lists.newArrayList(Lists.transform(reservation, new Function<Reservation, List<Instance>>() {


A defensive copy is not needed here

buildhive · 2013-02-07T18:17:33Z

Axemblr.com » axemblr-provisionr #45 ABORTED

(what's this?)

andreisavu · 2013-02-07T18:47:25Z

karaf/commands/src/main/java/com/axemblr/provisionr/commands/CreatePoolCommand.java

+
+    @Option(name = "-b", aliases = "--bid", description = "Bid for Amazon Spot Instance. If specified, requests" +
+        "spot instances, otherwise defaults to on demand instances.")
+    private Float spotBid;


We need to implement this as a generic option. E.g. provisionr:create --provider-option bid=0.04

andreisavu · 2013-02-07T19:06:05Z

providers/amazon/src/main/java/com/axemblr/provisionr/amazon/activities/RunSpotInstances.java

+                execution.getVariable(ProcessVariables.SPOT_INSTANCE_REQUEST_IDS));
+        if (alreadySent.isPresent()) {
+            try {
+                Thread.sleep(60 * 1000);


I would implement this as a loop that calls describeSpotInstanceRequests every 10 seconds for 1 minute or more. Check Stopwatch from Guava.

andreisavu · 2013-02-07T19:08:44Z

Looks good overall! Great job!

- added generic option for provider-specific properties with tests - fixed formatting and comments - better collection handling with Guava - used polling & stopwatches instead of just sleeping when waiting for spot requests

buildhive · 2013-02-08T13:25:42Z

Axemblr.com » axemblr-provisionr #48 FAILURE
Looks like there's a problem with this pull request
(what's this?)

cimi · 2013-02-08T13:40:13Z

The build error and the merge conflicts seem strange to me:

stdout: Auto-merging providers/amazon-tests/src/test/java/com/axemblr/provisionr/amazon/AmazonProvisionrLiveTest.java
Auto-merging karaf/commands/src/test/java/com/axemblr/provisionr/commands/CreatePoolCommandTest.java
CONFLICT (content): Merge conflict in karaf/commands/src/test/java/com/axemblr/provisionr/commands/CreatePoolCommandTest.java
Automatic merge failed; fix conflicts and then commit the result.

However, if I do:

$ git fetch origin master
$ git rebase origin/master
$ git status
# On branch issue-52
nothing to commit (working directory clean)

andreisavu · 2013-02-08T13:41:46Z

You have to rebase. I have pushed new code to master.

cimi · 2013-02-08T13:45:18Z

I think I'm doing it right, but probably I'm wrong. :) Here's my full output, I ran it now:

$ git remote -v
cimi    https://github.com/cimi/axemblr-provisionr.git (fetch)
cimi    https://github.com/cimi/axemblr-provisionr.git (push)
origin  https://github.com/axemblr/axemblr-provisionr.git (fetch)
origin  https://github.com/axemblr/axemblr-provisionr.git (push)
$ git fetch origin master
From https://github.com/axemblr/axemblr-provisionr
 * branch            master     -> FETCH_HEAD
$ git rebase origin/master
Current branch issue-52 is up to date.
$ git push cimi issue-52
Everything up-to-date

Doesn't fetch get me the latest commits from origin/master?

andreisavu · 2013-02-08T13:53:06Z

Yes that should work. Check the log and you should probably close this pull request and create a new one attached to issue #52 (using hub pull-request -i 52)

cimi · 2013-02-08T14:22:43Z

I've managed to get the latest code through git pull, I don't understand why fetch didn't get it from the remote.

I managed to merge it, but now I can't rebuild my Eclipse projects. I get the following error:

[INFO] Axemblr Provisionr :: Karaf Features .............. FAILURE [0.379s]
[...]
[ERROR] Failed to execute goal org.ops4j.pax.exam:maven-paxexam-plugin:1.2.4:generate-config (generate-config) on project provisionr-features: Unable to create dependencies file: java.lang.NullPointerException -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.ops4j.pax.exam:maven-paxexam-plugin:1.2.4:generate-config (generate-config) on project provisionr-features: Unable to create dependencies file: java.lang.NullPointerException
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:217)

I will create a new pull request attached to #52.

buildhive · 2013-02-08T14:26:31Z

Axemblr.com » axemblr-provisionr #49 UNSTABLE
Looks like there's a problem with this pull request
(what's this?)

cimi added 3 commits January 29, 2013 15:02

added spot instances stubs

29b1b8d

creating a spot instance request

fae8220

checking if spot instances are open or not

6fba680

andreisavu reviewed Jan 31, 2013
View reviewed changes

andreisavu reviewed Feb 1, 2013
View reviewed changes

cimi added 2 commits February 7, 2013 18:13

Initial draft for spot instance requests from the Amazon tutorial.

073465c

Moved the logic for spot requests in separate activities. Working pro…

73c90a0

…cess test for this case.

ghost assigned cimi Feb 7, 2013

andreisavu reviewed Feb 7, 2013
View reviewed changes

Fixed review issues

7747169

- added generic option for provider-specific properties with tests - fixed formatting and comments - better collection handling with Guava - used polling & stopwatches instead of just sleeping when waiting for spot requests

merged latest changes from master

4526fd3

cimi closed this Feb 8, 2013

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for Amazon EC2 spot instances #133

Add support for Amazon EC2 spot instances #133

cimi commented Jan 29, 2013

buildhive commented Jan 29, 2013

buildhive commented Jan 30, 2013

andreisavu Jan 31, 2013

cimi Jan 31, 2013

buildhive commented Jan 31, 2013

buildhive commented Jan 31, 2013

buildhive commented Jan 31, 2013

buildhive commented Jan 31, 2013

andreisavu Feb 1, 2013

buildhive commented Feb 5, 2013

buildhive commented Feb 5, 2013

buildhive commented Feb 5, 2013

andreisavu commented Feb 6, 2013

buildhive commented Feb 7, 2013

andreisavu Feb 7, 2013

buildhive commented Feb 7, 2013

andreisavu Feb 7, 2013

andreisavu Feb 7, 2013

andreisavu commented Feb 7, 2013

buildhive commented Feb 8, 2013

cimi commented Feb 8, 2013

andreisavu commented Feb 8, 2013

cimi commented Feb 8, 2013

andreisavu commented Feb 8, 2013

cimi commented Feb 8, 2013

buildhive commented Feb 8, 2013

Add support for Amazon EC2 spot instances #133

Add support for Amazon EC2 spot instances #133

Conversation

cimi commented Jan 29, 2013

buildhive commented Jan 29, 2013

buildhive commented Jan 30, 2013

andreisavu Jan 31, 2013

Choose a reason for hiding this comment

cimi Jan 31, 2013

Choose a reason for hiding this comment

buildhive commented Jan 31, 2013

buildhive commented Jan 31, 2013

buildhive commented Jan 31, 2013

buildhive commented Jan 31, 2013

andreisavu Feb 1, 2013

Choose a reason for hiding this comment

buildhive commented Feb 5, 2013

buildhive commented Feb 5, 2013

buildhive commented Feb 5, 2013

andreisavu commented Feb 6, 2013

buildhive commented Feb 7, 2013

andreisavu Feb 7, 2013

Choose a reason for hiding this comment

buildhive commented Feb 7, 2013

andreisavu Feb 7, 2013

Choose a reason for hiding this comment

andreisavu Feb 7, 2013

Choose a reason for hiding this comment

andreisavu commented Feb 7, 2013

buildhive commented Feb 8, 2013

cimi commented Feb 8, 2013

andreisavu commented Feb 8, 2013

cimi commented Feb 8, 2013

andreisavu commented Feb 8, 2013

cimi commented Feb 8, 2013

buildhive commented Feb 8, 2013