On-Demand and Spot Price Retrieval Filters w/ a Cache

[bwagner5]

Issue #, if available:
#71

Description of changes:
This update implements support for querying pricing data from the AWS Pricing API and the Amazon EC2 spot pricing history API. Users can now query for on-demand prices or avg spot price over the past 30 days. Pricing data can be displayed in the table output using --output table-wide. If no pricing criteria is specified as a filter, then pricing lookups are not performed since there is a pretty significant slow down.

• TODO: mac1.metal pricing is not being retrieved

Examples:

Usage:

$  build/ec2-instance-selector --help
ec2-instance-selector is a CLI tool to filter EC2 instance types based on resource criteria.
Filtering allows you to select all the instance types that match your application requirements.
Full docs can be found at github.com/aws/amazon-ec2-instance-selector

Usage:
  ec2-instance-selector [flags]

Examples:
ec2-instance-selector --vcpus 4 --region us-east-2 --availability-zones us-east-2b
ec2-instance-selector --memory-min 4 --memory-max 8 --vcpus-min 4 --vcpus-max 8 --region us-east-2

Filter Flags:
      --allow-list string                 List of allowed instance types to select from w/ regex syntax (Example: m[3-5]\.*)
  -z, --availability-zones strings        Availability zones or zone ids to check EC2 capacity offered in specific AZs
      --baremetal                         Bare Metal instance types (.metal instances)
  -b, --burst-support                     Burstable instance types
  -a, --cpu-architecture string           CPU architecture [x86_64/amd64, i386, or arm64]
      --current-generation                Current generation instance types (explicitly set this to false to not return current generation instance types)
      --deny-list string                  List of instance types which should be excluded w/ regex syntax (Example: m[1-2]\.*)
  -e, --ena-support                       Instance types where ENA is supported or required
  -f, --fpga-support                      FPGA instance types
      --gpu-memory-total string           Number of GPUs' total memory (Example: 4 GiB) (sets --gpu-memory-total-min and -max to the same value)
      --gpu-memory-total-max string       Maximum Number of GPUs' total memory (Example: 4 GiB) If --gpu-memory-total-min is not specified, the lower bound will be 0
      --gpu-memory-total-min string       Minimum Number of GPUs' total memory (Example: 4 GiB) If --gpu-memory-total-max is not specified, the upper bound will be infinity
  -g, --gpus int                          Total Number of GPUs (Example: 4) (sets --gpus-min and -max to the same value)
      --gpus-max int                      Maximum Total Number of GPUs (Example: 4) If --gpus-min is not specified, the lower bound will be 0
      --gpus-min int                      Minimum Total Number of GPUs (Example: 4) If --gpus-max is not specified, the upper bound will be infinity
      --hibernation-support               Hibernation supported
      --hypervisor string                 Hypervisor: [xen or nitro]
  -m, --memory string                     Amount of Memory available (Example: 4 GiB) (sets --memory-min and -max to the same value)
      --memory-max string                 Maximum Amount of Memory available (Example: 4 GiB) If --memory-min is not specified, the lower bound will be 0
      --memory-min string                 Minimum Amount of Memory available (Example: 4 GiB) If --memory-max is not specified, the upper bound will be infinity
      --network-interfaces int            Number of network interfaces (ENIs) that can be attached to the instance (sets --network-interfaces-min and -max to the same value)
      --network-interfaces-max int        Maximum Number of network interfaces (ENIs) that can be attached to the instance If --network-interfaces-min is not specified, the lower bound will be 0
      --network-interfaces-min int        Minimum Number of network interfaces (ENIs) that can be attached to the instance If --network-interfaces-max is not specified, the upper bound will be infinity
      --network-performance int           Bandwidth in Gib/s of network performance (Example: 100) (sets --network-performance-min and -max to the same value)
      --network-performance-max int       Maximum Bandwidth in Gib/s of network performance (Example: 100) If --network-performance-min is not specified, the lower bound will be 0
      --network-performance-min int       Minimum Bandwidth in Gib/s of network performance (Example: 100) If --network-performance-max is not specified, the upper bound will be infinity
      --placement-group-strategy string   Placement group strategy: [cluster, partition, spread]
      --price-per-hour float              Price/hour in USD (Example: 0.09) (sets --price-per-hour-min and -max to the same value)
      --price-per-hour-max float          Maximum Price/hour in USD (Example: 0.09) If --price-per-hour-min is not specified, the lower bound will be 0
      --price-per-hour-min float          Minimum Price/hour in USD (Example: 0.09) If --price-per-hour-max is not specified, the upper bound will be infinity
      --root-device-type string           Supported root device types: [ebs or instance-store]
  -u, --usage-class string                Usage class: [spot or on-demand]
  -c, --vcpus int                         Number of vcpus available to the instance type. (sets --vcpus-min and -max to the same value)
      --vcpus-max int                     Maximum Number of vcpus available to the instance type. If --vcpus-min is not specified, the lower bound will be 0
      --vcpus-min int                     Minimum Number of vcpus available to the instance type. If --vcpus-max is not specified, the upper bound will be infinity
      --vcpus-to-memory-ratio string      The ratio of vcpus to GiBs of memory. (Example: 1:2)
      --virtualization-type string        Virtualization Type supported: [hvm or pv]


Suite Flags:
      --base-instance-type string   Instance Type used to retrieve similarly spec'd instance types
      --flexible                    Retrieves a group of instance types spanning multiple generations based on opinionated defaults and user overridden resource filters
      --service string              Filter instance types based on service support (Example: eks, eks-20201211, or emr-5.20.0)


Global Flags:
  -h, --help              Help
      --max-results int   The maximum number of instance types that match your criteria to return (default 20)
  -o, --output string     Specify the output format (table, table-wide, one-line)
      --profile string    AWS CLI profile to use for credentials and config
  -r, --region string     AWS Region to use for API requests (NOTE: if not passed in, uses AWS SDK default precedence)
  -v, --verbose           Verbose - will print out full instance specs
      --version           Prints CLI version

Filter On-Demand Price:

$ build/ec2-instance-selector --price-per-hour-min 0.88 -o table-wide
Instance Type  VCPUs   Mem (GiB)  Hypervisor  Current Gen  Hibernation Support  CPU Arch  Network Performance  ENIs    GPUs    GPU Mem (GiB)  GPU Info  On-Demand Price/Hr
-------------  -----   ---------  ----------  -----------  -------------------  --------  -------------------  ----    ----    -------------  --------  ------------------
c3.8xlarge     32      60         xen         false        true                 x86_64    10 Gigabit           8       0       0                        $1.68
c4.8xlarge     36      60         xen         true         true                 x86_64    10 Gigabit           8       0       0                        $1.591
c5.12xlarge    48      96         nitro       true         true                 x86_64    12 Gigabit           8       0       0                        $2.04
c5.18xlarge    72      144        nitro       true         true                 x86_64    25 Gigabit           15      0       0                        $3.06
c5.24xlarge    96      192        nitro       true         false                x86_64    25 Gigabit           15      0       0                        $4.08
c5.9xlarge     36      72         nitro       true         true                 x86_64    10 Gigabit           8       0       0                        $1.53
c5.metal       96      192        none        true         false                x86_64    25 Gigabit           15      0       0                        $4.08
c5a.12xlarge   48      96         nitro       true         false                x86_64    12 Gigabit           8       0       0                        $1.848
c5a.16xlarge   64      128        nitro       true         false                x86_64    20 Gigabit           15      0       0                        $2.464
c5a.24xlarge   96      192        nitro       true         false                x86_64    20 Gigabit           15      0       0                        $3.696
c5a.8xlarge    32      64         nitro       true         false                x86_64    10 Gigabit           8       0       0                        $1.232
c5ad.12xlarge  48      96         nitro       true         false                x86_64    12 Gigabit           8       0       0                        $2.064
c5ad.16xlarge  64      128        nitro       true         false                x86_64    20 Gigabit           15      0       0                        $2.752
c5ad.24xlarge  96      192        nitro       true         false                x86_64    20 Gigabit           15      0       0                        $4.128
c5ad.8xlarge   32      64         nitro       true         false                x86_64    10 Gigabit           8       0       0                        $1.376
c5d.12xlarge   48      96         nitro       true         false                x86_64    12 Gigabit           8       0       0                        $2.304
c5d.18xlarge   72      144        nitro       true         false                x86_64    25 Gigabit           15      0       0                        $3.456
c5d.24xlarge   96      192        nitro       true         false                x86_64    25 Gigabit           15      0       0                        $4.608
c5d.9xlarge    36      72         nitro       true         false                x86_64    10 Gigabit           8       0       0                        $1.728
c5d.metal      96      192        none        true         false                x86_64    25 Gigabit           15      0       0                        $4.608
NOTE: 177 entries were truncated, increase --max-results to see more

Filter Spot Price:

$ build/ec2-instance-selector --price-per-hour-min 0.88 --usage-class spot -o table-wide
Instance Type  VCPUs   Mem (GiB)  Hypervisor  Current Gen  Hibernation Support  CPU Arch  Network Performance  ENIs    GPUs    GPU Mem (GiB)  GPU Info  Spot Price/Hr (30 days)
-------------  -----   ---------  ----------  -----------  -------------------  --------  -------------------  ----    ----    -------------  --------  -----------------------
c5.18xlarge    72      144        nitro       true         true                 x86_64    25 Gigabit           15      0       0                        $1.19131
c5.24xlarge    96      192        nitro       true         false                x86_64    25 Gigabit           15      0       0                        $1.5546
c5.metal       96      192        none        true         false                x86_64    25 Gigabit           15      0       0                        $1.5546
c5a.16xlarge   64      128        nitro       true         false                x86_64    20 Gigabit           15      0       0                        $1.05403
c5a.24xlarge   96      192        nitro       true         false                x86_64    20 Gigabit           15      0       0                        $1.57089
c5ad.16xlarge  64      128        nitro       true         false                x86_64    20 Gigabit           15      0       0                        $1.06093
c5ad.24xlarge  96      192        nitro       true         false                x86_64    20 Gigabit           15      0       0                        $1.5546
c5d.12xlarge   48      96         nitro       true         false                x86_64    12 Gigabit           8       0       0                        $1.07622
c5d.18xlarge   72      144        nitro       true         false                x86_64    25 Gigabit           15      0       0                        $1.26112
c5d.24xlarge   96      192        nitro       true         false                x86_64    25 Gigabit           15      0       0                        $1.55781
c5d.metal      96      192        none        true         false                x86_64    25 Gigabit           15      0       0                        $1.55769
c5n.18xlarge   72      192        nitro       true         false                x86_64    100 Gigabit          15      0       0                        $1.25587
c5n.metal      72      192        none        true         false                x86_64    100 Gigabit          15      0       0                        $1.1659
c6g.16xlarge   64      128        nitro       true         false                arm64     25 Gigabit           15      0       0                        $1.0882
c6g.metal      64      128        none        true         false                arm64     25 Gigabit           15      0       0                        $1.0882
c6gd.16xlarge  64      128        nitro       true         false                arm64     25 Gigabit           15      0       0                        $1.08925
c6gd.metal     64      128        none        true         false                arm64     25 Gigabit           15      0       0                        $1.0882
c6gn.16xlarge  64      128        nitro       true         false                arm64     100 Gigabit          15      0       0                        $1.0882
d2.8xlarge     36      244        xen         true         false                x86_64    10 Gigabit           8       0       0                        $1.65611
d3.8xlarge     32      256        nitro       true         false                x86_64    25 Gigabit           3       0       0                        $1.1987
NOTE: 92 entries were truncated, increase --max-results to see more

No Price Filter Specified w/ Table Output Wide:

build/ec2-instance-selector -o table-wide
Instance Type  VCPUs   Mem (GiB)  Hypervisor  Current Gen  Hibernation Support  CPU Arch      Network Performance  ENIs    GPUs    GPU Mem (GiB)  GPU Info  On-Demand Price/Hr
-------------  -----   ---------  ----------  -----------  -------------------  --------      -------------------  ----    ----    -------------  --------  ------------------
a1.2xlarge     8       16         nitro       false        false                arm64         Up to 10 Gigabit     4       0       0                        -No Price Filter Specified-
a1.4xlarge     16      32         nitro       false        false                arm64         Up to 10 Gigabit     8       0       0                        -No Price Filter Specified-
a1.large       2       4          nitro       false        false                arm64         Up to 10 Gigabit     3       0       0                        -No Price Filter Specified-
a1.medium      1       2          nitro       false        false                arm64         Up to 10 Gigabit     2       0       0                        -No Price Filter Specified-
a1.metal       16      32         none        false        false                arm64         Up to 10 Gigabit     8       0       0                        -No Price Filter Specified-
a1.xlarge      4       8          nitro       false        false                arm64         Up to 10 Gigabit     4       0       0                        -No Price Filter Specified-
c1.medium      2       1.69922    xen         false        false                i386, x86_64  Moderate             2       0       0                        -No Price Filter Specified-
c1.xlarge      8       7          xen         false        false                x86_64        High                 4       0       0                        -No Price Filter Specified-
c3.2xlarge     8       15         xen         false        true                 x86_64        High                 4       0       0                        -No Price Filter Specified-
c3.4xlarge     16      30         xen         false        true                 x86_64        High                 8       0       0                        -No Price Filter Specified-
c3.8xlarge     32      60         xen         false        true                 x86_64        10 Gigabit           8       0       0                        -No Price Filter Specified-
c3.large       2       3.75       xen         false        true                 i386, x86_64  Moderate             3       0       0                        -No Price Filter Specified-
c3.xlarge      4       7.5        xen         false        true                 x86_64        Moderate             4       0       0                        -No Price Filter Specified-
c4.2xlarge     8       15         xen         true         true                 x86_64        High                 4       0       0                        -No Price Filter Specified-
c4.4xlarge     16      30         xen         true         true                 x86_64        High                 8       0       0                        -No Price Filter Specified-
c4.8xlarge     36      60         xen         true         true                 x86_64        10 Gigabit           8       0       0                        -No Price Filter Specified-
c4.large       2       3.75       xen         true         true                 x86_64        Moderate             3       0       0                        -No Price Filter Specified-
c4.xlarge      4       7.5        xen         true         true                 x86_64        High                 4       0       0                        -No Price Filter Specified-
c5.12xlarge    48      96         nitro       true         true                 x86_64        12 Gigabit           8       0       0                        -No Price Filter Specified-
c5.18xlarge    72      144        nitro       true         true                 x86_64        25 Gigabit           15      0       0                        -No Price Filter Specified-
NOTE: 363 entries were truncated, increase --max-results to see more

We don't show the price when pricing filter is not specified because of the latency in retrieving pricing data. It would be nice to show it on a list operation, but the filtering engine does not know what type of output the user selected, it just aggregates, filters, and then returns the data. We might be able to make it a little smarter later.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[brycahta]

love the new feature-- looks great on a first pass!

[brycahta]

Awesome feature add 🔥 🚀

[brycahta]

Looks good! 🔥