Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Event parser can parse new format of target #160

Merged
2 changes: 1 addition & 1 deletion app/models/manageiq/providers/amazon/builder.rb
Original file line number Diff line number Diff line change
@@ -28,7 +28,7 @@ def build_inventory(ems, target)
ManageIQ::Providers::Amazon::Inventory::Persister::StorageManager::S3,
[ManageIQ::Providers::Amazon::Inventory::Parser::StorageManager::S3]
)
when ManageIQ::Providers::Amazon::TargetCollection
when ManagerRefresh::TargetCollection
inventory(
ems,
target,
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
class ManageIQ::Providers::Amazon::CloudManager::EventTargetParser
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 for this class

Now extract another class, e.g. EventTarget which holds add_reference and the available_targets data. Then you have fully encapsulated that data you want to transport into a class and moved away from hashes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done, ManagerRefresh::Target and ManagerRefresh::TargetCollection are being introduced as a dependency

attr_reader :ems_event

# @param ems_event [EmsEvent] EmsEvent object
def initialize(ems_event)
@ems_event = ems_event
end

# Parses all targets that are present in the EmsEvent given in the initializer
#
# @return [Array] Array of ManagerRefresh::Target objects
def parse
parse_ems_event_targets(ems_event)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that you add ems_event as a state (instance variable), I think you can also create target_collection as an instance var and let collect_cloudwatch_api_references operate on that.

end

private

# Parses list of ManagerRefresh::Target out of the given EmsEvent
#
# @param event [EmsEvent] EmsEvent object
# @return [Array] Array of ManagerRefresh::Target objects
def parse_ems_event_targets(event)
target_collection = ManagerRefresh::TargetCollection.new(:manager => event.ext_management_system, :event => event)

case event.full_data["event_source"]
when :cloud_watch_api
collect_cloudwatch_api_references!(target_collection,
event.full_data.fetch_path("detail", "requestParameters") || {})
collect_cloudwatch_api_references!(target_collection,
event.full_data.fetch_path("detail", "responseElements") || {})
when :cloud_watch_ec2
collect_cloudwatch_ec2_references!(target_collection, event.full_data)
when :config
collect_config_references!(target_collection, event.full_data)
end

target_collection.targets
end

def parsed_targets(target_collection = {})
target_collection.select { |_target_class, references| references[:manager_ref].present? }
end

def add_target(target_collection, association, ref)
target_collection.add_target(:association => association, :manager_ref => {:ems_ref => ref})
end

def collect_cloudwatch_ec2_references!(target_collection, event_data)
instance_id = event_data.fetch_path("detail", "instance-id")
add_target(target_collection, :vms, instance_id) if instance_id
end

def collect_config_references!(target_collection, event_data)
resource_type = event_data.fetch_path("configurationItem", "resourceType")
resource_id = event_data.fetch_path("configurationItem", "resourceId")
target_class = case resource_type
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would it make sense to do a hash lookup here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

possibly, but maybe the case is enough? Should be still pretty quick

when "AWS::EC2::Instance"
:vms
when "AWS::EC2::SecurityGroup"
:security_groups
when "AWS::EC2::Volume"
:cloud_volumes
when "AWS::EC2::NetworkInterface"
:network_ports
when "AWS::EC2::VPC"
:cloud_networks
when "AWS::EC2::Subnet"
:cloud_subnets
when "AWS::EC2::EIP"
:floating_ips
end

add_target(target_collection, target_class, resource_id) if target_class && resource_id
end

def collect_cloudwatch_api_references!(target_collection, event_data, depth = 0)
# Check a reasonable depth, so this can't fail with max stack size
raise "Depth 20 reached when scanning EmsEvent for Targets" if depth > 20

# Cloud
add_target(target_collection, :vms, event_data["instanceId"]) if event_data["instanceId"]
add_target(target_collection, :miq_templates, event_data["imageId"]) if event_data["imageId"]
add_target(target_collection, :key_pairs, event_data["keyName"]) if event_data["keyName"]
add_target(target_collection, :orchestration_stacks, event_data["stackId"]) if event_data["stackId"]
add_target(target_collection, :orchestration_stacks, event_data["stackName"]) if event_data["stackName"]
# Network
add_target(target_collection, :cloud_networks, event_data["vpcId"]) if event_data["vpcId"]
add_target(target_collection, :cloud_subnets, event_data["subnetId"]) if event_data["subnetId"]
add_target(target_collection, :network_ports, event_data["networkInterfaceId"]) if event_data["networkInterfaceId"]
add_target(target_collection, :security_groups, event_data["groupId"]) if event_data["groupId"]
add_target(target_collection, :floating_ips, event_data["allocationId"]) if event_data["allocationId"]
add_target(target_collection, :load_balancers, event_data["loadBalancerName"]) if event_data["loadBalancerName"]
# Block Storage
add_target(target_collection, :cloud_volumes, event_data["volumeId"]) if event_data["volumeId"]
add_target(target_collection, :cloud_volume_snapshots, event_data["snapshotId"]) if event_data["snapshotId"]

# TODO(lsmola) how to handle tagging? Tagging affects e.g. a name of any resource, but contains only a generic
# resourceID
# "requestParameters"=>
# {"resourcesSet"=>{"items"=>[{"resourceId"=>"vol-07ad036724e3175a5"}]},
# "tagSet"=>{"items"=>[{"key"=>"Name", "value"=>"ladas_volue_2"}]}},
# I think we can parse the resource id, so guess where it belongs.
# TODO(lsmola) RegisterImage, by creating image from a volume snapshot, should we track the block device
# mapping and refresh also snapshot?

# Collect nested references
collect_cloudwatch_api_references!(target_collection, event_data["networkInterface"], depth + 1) if event_data["networkInterface"]

(event_data.fetch_path("groupSet", "items") || []).each do |x|
collect_cloudwatch_api_references!(target_collection, x, depth + 1)
end
(event_data.fetch_path("instancesSet", "items") || []).each do |x|
collect_cloudwatch_api_references!(target_collection, x, depth + 1)
end
(event_data.fetch_path("instances") || []).each do |x|
collect_cloudwatch_api_references!(target_collection, x, depth + 1)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would it make sense to expand this api? collect_cloud_watch_api_referenceses!

of course, naming is hard. Then again, do you ever collect_cloudwatch_api_references with a single array?

collect_cloud_watch_api_referenceses!(target_collection, event_data.fetch_path("instances") || [], depth + 1)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no usually the cloudwatch events have a nested arrays that contain references

end
(event_data.fetch_path("networkInterfaceSet", "items") || []).each do |x|
collect_cloudwatch_api_references!(target_collection, x, depth + 1)
end
end
end
Original file line number Diff line number Diff line change
@@ -46,9 +46,10 @@ def preprocess_targets
all_targets, sub_ems_targets = targets.partition { |x| x.kind_of?(ExtManagementSystem) }

unless sub_ems_targets.blank?
ems_event_collection = ManageIQ::Providers::Amazon::TargetCollection.new(sub_ems_targets)
if refresher_options.try(:[], :event_targeted_refresh)
# We can disable targeted refresh with a setting, then we will just do full ems refresh on any event
ems_event_collection = ManagerRefresh::TargetCollection.new(:targets => sub_ems_targets,
:manager_id => ems_id)
all_targets << ems_event_collection
else
all_targets << @ems_by_ems_id[ems_id]
60 changes: 16 additions & 44 deletions app/models/manageiq/providers/amazon/inventory/collector.rb
Original file line number Diff line number Diff line change
@@ -3,22 +3,22 @@ class ManageIQ::Providers::Amazon::Inventory::Collector < ManagerRefresh::Invent
require_nested :NetworkManager
require_nested :TargetCollection

attr_reader :instances, :instances_refs, :instances_deleted
attr_reader :flavors, :flavors_refs, :flavors_deleted
attr_reader :availability_zones, :availability_zones_refs, :availability_zones_deleted
attr_reader :key_pairs, :key_pairs_refs, :key_pairs_deleted
attr_reader :private_images, :private_images_refs, :private_images_deleted
attr_reader :shared_images, :shared_images_refs, :shared_images_deleted
attr_reader :public_images, :public_images_refs, :public_images_deleted
attr_reader :cloud_networks, :cloud_networks_refs, :cloud_networks_deleted
attr_reader :cloud_subnets, :cloud_subnets_refs, :cloud_subnets_deleted
attr_reader :security_groups, :security_groups_refs, :security_groups_deleted
attr_reader :floating_ips, :floating_ips_refs, :floating_ips_deleted
attr_reader :network_ports, :network_ports_refs, :network_ports_deleted
attr_reader :load_balancers, :load_balancers_refs, :load_balancers_deleted
attr_reader :stacks, :stacks_refs, :stacks_deleted
attr_reader :cloud_volumes, :cloud_volumes_refs
attr_reader :cloud_volume_snapshots, :cloud_volume_snapshots_refs
attr_reader :instances
attr_reader :flavors
attr_reader :availability_zones
attr_reader :key_pairs
attr_reader :private_images
attr_reader :shared_images
attr_reader :public_images
attr_reader :cloud_networks
attr_reader :cloud_subnets
attr_reader :security_groups
attr_reader :floating_ips
attr_reader :network_ports
attr_reader :load_balancers
attr_reader :stacks
attr_reader :cloud_volumes
attr_reader :cloud_volume_snapshots
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👏 for less attr_readers

attr_reader :cloud_objects_store_containers
attr_reader :cloud_objects_store_objects

@@ -30,49 +30,21 @@ def initialize(_manager, _target)

def initialize_inventory_sources
@instances = []
@instances_refs = Set.new
@instances_deleted = []
@flavors = []
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yay.

I'm assuming this doesn't add any O(N) lookups

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the refs were moved to ManagerRefresh::TargetCollection

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ooh, then that is very cool

@flavors_refs = Set.new
@flavors_deleted = []
@availability_zones = []
@availability_zones_refs = Set.new
@availability_zones_deleted = []
@key_pairs = []
@key_pairs_refs = Set.new
@key_pairs_deleted = []
@private_images = []
@private_images_refs = Set.new
@private_images_deleted = []
@shared_images = []
@shared_images_refs = Set.new
@shared_images_deleted = []
@public_images = []
@public_images_refs = Set.new
@public_images_deleted = []
@cloud_networks = []
@cloud_networks_refs = Set.new
@cloud_networks_deleted = []
@cloud_subnets = []
@cloud_subnets_refs = Set.new
@cloud_subnets_deleted = []
@security_groups = []
@security_groups_refs = Set.new
@security_groups_deleted = []
@floating_ips = []
@floating_ips_refs = Set.new
@floating_ips_deleted = []
@network_ports = []
@network_ports_refs = Set.new
@network_ports_deleted = []
@load_balancers = []
@load_balancers_refs = Set.new
@stacks = []
@stacks_refs = Set.new
@cloud_volumes = []
@cloud_volumes_refs = Set.new
@cloud_volume_snapshots = []
@cloud_volume_snapshots_refs = Set.new
# Nested resources
@stack_resources = {}
@stack_resources_refs = {}
Loading