Add kafka as an option for prototype.queue_type #19984

agrare · 2020-03-20T14:37:07Z

Currently we have two queue_types, miq_queue and artemis with the latter allowing the use of ManageIQ::Messaging client.

ManageIQ::Messaging also has support for kafka which we can also allow to be used here.

$ bin/kafka-console-consumer.sh --bootstrap-server 'localhost:9092' --topic events --from-beginning
{"event_type":"VmPoweredOnEvent",
"chain_id":"651818",
"is_task":false,
"source":"VC",
"message":"cfme-5.11 on  dell-r430-13.cloudforms.lab.eng.rdu2.redhat.com in dev-vc67-DC is powered on",
"timestamp":"2020-03-23T20:51:56.366773Z",
"full_data":{"key":"651820",
  "chainId":"651818",
  "createdTime":"2020-03-23T20:51:56.366773Z",
  "userName":"VSPHERE.LOCAL\\Administrator",
  "datacenter":{"name":"dev-vc67-DC", "datacenter":"datacenter-104"},
  "computeResource":{"name":"dev-vc67-cluster","computeResource":"domain-c109"},
  "host":{"name":"dell-r430-13.cloudforms.lab.eng.rdu2.redhat.com","host":"host-114"},
  "vm":{"name":"cfme-5.11","vm":"vm-685","path":"[NFS Share] cfme-5.11/cfme-5.11.vmx","uuid":"423dbb7a-bc76-a738-2740-4de960bd33da"},
  "fullFormattedMessage":"cfme-5.11 on  dell-r430-13.cloudforms.lab.eng.rdu2.redhat.com in dev-vc67-DC is powered on",
  "changeTag":"", 
  "template":"false", 
  "eventType":"VmPoweredOnEvent"},
"ems_id":"2",
"username":"VSPHERE.LOCAL\\Administrator",
"vm_ems_ref":"vm-685",
"vm_name":"cfme-5.11",
"vm_location":"[NFS Share] cfme-5.11/cfme-5.11.vmx",
"vm_uid_ems":"423dbb7a-bc76-a738-2740-4de960bd33da",
"host_name":"dell-r430-13.cloudforms.lab.eng.rdu2.redhat.com",
"host_ems_ref":"host-114"}

app/models/miq_queue.rb

Fryguy · 2020-03-20T21:31:25Z

cc @carbonin

Fryguy · 2020-03-20T21:35:13Z

app/models/miq_queue.rb

+      :Kafka
+    end
+  end
+  private_class_method :messaging_client_protocol


As discussed, I feel like this should be pushed down into manageiq-messaging... feels like something the caller shouldn't have to know about.

Yeah I agree

app/models/miq_queue.rb

agrare · 2020-03-23T19:42:46Z

#20000

app/models/miq_queue.rb

agrare · 2020-03-23T20:57:50Z

So when I have this use YAML, manageiq-messaging fails on YAML.safe_load:

INFO -- : MIQ(MiqQueue.raw_publish) Published to topic(manageiq.events), msg({:event_type=>"VmPoweredOnEvent", :chain_id=>"651785", :is_task=>false, :source=>"VC", :message=>"cfm)
ERROR -- : MIQ(MiqQueue.process_topic_message) Event processing error: Tried to load unspecified class: Symbol
ERROR -- : MIQ(MiqQueue.process_topic_message) /usr/lib/ruby/2.5.0/psych/class_loader.rb:97:in `find'
 /usr/lib/ruby/2.5.0/psych/class_loader.rb:28:in `load'
 /usr/lib/ruby/2.5.0/psych/class_loader.rb:39:in `block (2 levels) in <class:ClassLoader>'
 ...
 /usr/lib/ruby/2.5.0/psych.rb:325:in `safe_load'

And when I use the JSON encoding the payload that ManageIQ::Messaging yields to the block is nil

ERROR -- : MIQ(MiqQueue.process_topic_message) Event processing error: undefined method `[]' for nil:NilClass
ERROR -- : MIQ(MiqQueue.process_topic_message) /home/grare/adam/src/manageiq/manageiq/app/models/miq_event_handler/runner.rb:16:in `block in do_before_work_loop'
 /home/grare/adam/.gem/gems/manageiq-messaging-0.1.5/lib/manageiq/messaging/kafka/common.rb:104:in `process_topic_message'
 /home/grare/adam/.gem/gems/manageiq-messaging-0.1.5/lib/manageiq/messaging/kafka/topic.rb:20:in `block in subscribe_topic_impl'

Going to mark this WIP until this is figured out, or we could yield the JSON messages to the topic and publish to MiqQueue which is what we're planning on for metrics and allow kafka as an MIQ Queue replacement in a future PR @Fryguy

Fryguy · 2020-03-23T21:57:36Z

Yeah the symbol keys were what I was concerned about.

agrare · 2020-03-24T12:49:10Z

Okay instead of a queue replacement this will now publish events to kafka/artemis if available in a json format in addition to the normal event handling over MiqQueue.

This should make it easier for other services to consume events (json format not yaml w/ symbols) and when we want to move to kafka as a queue replacement we can update the event handler to deal with json payloads

jrafanie · 2020-03-27T15:50:25Z

So when I have this use YAML, manageiq-messaging fails on YAML.safe_load:

INFO -- : MIQ(MiqQueue.raw_publish) Published to topic(manageiq.events), msg({:event_type=>"VmPoweredOnEvent", :chain_id=>"651785", :is_task=>false, :source=>"VC", :message=>"cfm)
ERROR -- : MIQ(MiqQueue.process_topic_message) Event processing error: Tried to load unspecified class: Symbol
ERROR -- : MIQ(MiqQueue.process_topic_message) /usr/lib/ruby/2.5.0/psych/class_loader.rb:97:in `find'
 /usr/lib/ruby/2.5.0/psych/class_loader.rb:28:in `load'
 /usr/lib/ruby/2.5.0/psych/class_loader.rb:39:in `block (2 levels) in <class:ClassLoader>'
 ...
 /usr/lib/ruby/2.5.0/psych.rb:325:in `safe_load'

FYI, in case this was a surprise or worked previously... it could be due to #19701 because we weren't honoring safe_load until that PR "fixed" it. The fix could then cause these Psych::DisallowedClass where it worked previously.

I agree, using string keys or json is probably best.

Currently we have two queue_types, miq_queue and artemis with the later allowing the use of ManageIQ::Messaging client. ManageIQ::Messaging also has support for kafka which we can also allow to be used here.

miq-bot · 2020-03-30T14:40:34Z

Checked commits agrare/manageiq@f724ebe~...cf48c7e with ruby 2.5.7, rubocop 0.69.0, haml-lint 0.28.0, and yamllint
4 files checked, 0 offenses detected
Everything looks fine. 🍪

Add kafka as an option for prototype.queue_type (cherry picked from commit beea070)

simaishi · 2020-04-08T12:54:25Z

Jansa backport details:

$ git log -1
commit c19e513f8fa559fb0df7780007ffc8351475562c
Author: Jason Frey <[email protected]>
Date:   Mon Mar 30 15:46:50 2020 -0400

    Merge pull request #19984 from agrare/add_support_for_kafka

    Add kafka as an option for prototype.queue_type

    (cherry picked from commit beea070a44ee6e52654df333cb2128f8e0f4981b)

agrare requested review from Fryguy, gtanzillo, jrafanie and kbrock as code owners March 20, 2020 14:37

miq-bot added the wip label Mar 20, 2020

Fryguy reviewed Mar 20, 2020

View reviewed changes

app/models/miq_queue.rb Outdated Show resolved Hide resolved

agrare force-pushed the add_support_for_kafka branch 3 times, most recently from 54a7579 to b7b0fa8 Compare March 20, 2020 18:10

agrare mentioned this pull request Mar 20, 2020

Syndicate metrics to Kafka #19987

Merged

Fryguy reviewed Mar 20, 2020

View reviewed changes

Fryguy added the enhancement label Mar 20, 2020

Fryguy mentioned this pull request Mar 21, 2020

[Providers] Syndicate events to Kafka #19583

Closed

2 tasks

chessbyte added this to the Jansa milestone Mar 21, 2020

carbonin reviewed Mar 23, 2020

View reviewed changes

app/models/miq_queue.rb Outdated Show resolved Hide resolved

agrare changed the title ~~[WIP] Add kafka as an option for prototype.queue_type~~ Add kafka as an option for prototype.queue_type Mar 23, 2020

agrare added core/queue and removed wip labels Mar 23, 2020

agrare commented Mar 23, 2020

View reviewed changes

app/models/miq_queue.rb Show resolved Hide resolved

agrare force-pushed the add_support_for_kafka branch 4 times, most recently from a7db1ee to 951dfcd Compare March 23, 2020 20:37

agrare changed the title ~~Add kafka as an option for prototype.queue_type~~ [WIP] Add kafka as an option for prototype.queue_type Mar 23, 2020

miq-bot added the wip label Mar 23, 2020

agrare force-pushed the add_support_for_kafka branch from 23c694c to d895578 Compare March 24, 2020 12:46

agrare changed the title ~~[WIP] Add kafka as an option for prototype.queue_type~~ Add kafka as an option for prototype.queue_type Mar 24, 2020

miq-bot removed the wip label Mar 24, 2020

agrare force-pushed the add_support_for_kafka branch from 57dfefb to 18bd565 Compare March 24, 2020 13:43

agrare linked an issue Mar 24, 2020 that may be closed by this pull request

[Providers] Syndicate events to Kafka #19583

Closed

2 tasks

agrare assigned Fryguy Mar 25, 2020

agrare added 5 commits March 30, 2020 10:38

Add kafka as an option for prototype.queue_type

f724ebe

Currently we have two queue_types, miq_queue and artemis with the later allowing the use of ManageIQ::Messaging client. ManageIQ::Messaging also has support for kafka which we can also allow to be used here.

Add an ENV var override for queue_type

682e7b7

Rename QUEUE_* to MESSAGING_*

862fb60

Encode in json and always send events to miq_queue

83b7517

Publish to ems-events topic

cf48c7e

agrare force-pushed the add_support_for_kafka branch from 18bd565 to cf48c7e Compare March 30, 2020 14:38

Fryguy merged commit beea070 into ManageIQ:master Mar 30, 2020

Fryguy added the jansa/yes? label Mar 30, 2020

agrare deleted the add_support_for_kafka branch March 30, 2020 19:50

chessbyte added jansa/yes and removed jansa/yes? labels Apr 7, 2020

simaishi pushed a commit that referenced this pull request Apr 8, 2020

Merge pull request #19984 from agrare/add_support_for_kafka

c19e513

Add kafka as an option for prototype.queue_type (cherry picked from commit beea070)

simaishi added jansa/backported and removed jansa/yes labels Apr 8, 2020

agrare mentioned this pull request May 15, 2020

Add ems_uid and ems_type to event payload #20179

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add kafka as an option for prototype.queue_type #19984

Add kafka as an option for prototype.queue_type #19984

agrare commented Mar 20, 2020 •

edited

Loading

Fryguy commented Mar 20, 2020

Fryguy Mar 20, 2020

agrare Mar 20, 2020

agrare commented Mar 23, 2020

agrare commented Mar 23, 2020

Fryguy commented Mar 23, 2020

agrare commented Mar 24, 2020

jrafanie commented Mar 27, 2020

miq-bot commented Mar 30, 2020

simaishi commented Apr 8, 2020

Add kafka as an option for prototype.queue_type #19984

Add kafka as an option for prototype.queue_type #19984

Conversation

agrare commented Mar 20, 2020 • edited Loading

Fryguy commented Mar 20, 2020

Fryguy Mar 20, 2020

Choose a reason for hiding this comment

agrare Mar 20, 2020

Choose a reason for hiding this comment

agrare commented Mar 23, 2020

agrare commented Mar 23, 2020

Fryguy commented Mar 23, 2020

agrare commented Mar 24, 2020

jrafanie commented Mar 27, 2020

miq-bot commented Mar 30, 2020

simaishi commented Apr 8, 2020

agrare commented Mar 20, 2020 •

edited

Loading