Introduce FirmwareRegistry and FirmwareBinary (with FirmwareTargets) #18774

miha-plesko · 2019-05-16T13:41:52Z

One can update physical server's firmware, but the firmware has to be accessible somewhere, usually on a HTTP/FTP/SAMBA link. Furthermore, there is usually some metadata associated with the firmware to tell what kind of hardware (manufacturer, model type) is it meant for.

With this commit we introduce a new model, FirmwareRegistry, that supports various kinds of metadata parsing by leveraging STI. User is supposed to create FirmwareRegistry instance via UI feeding it url+authentication and then click "Refresh Relationships" on it. ManageIQ will then invoke

registry.sync_fw_binaries_queue

which in turn will connect to the metadata repository to list available firmwares and store them as FirmwareBinary (no binary data is really stored, only HTTP/FTP/SAMBA link and some metadata).

In followup PRs we will then be able to list available firmware binaries to update physical server's firmware.

Depends on:

Introduce :firmware_registry and :firmware_binary tables manageiq-schema#371

@miq-bot add_label enhancement
@miq-bot assign @agrare

/cc @tadeboro @matejart

Example usage:

registry = FirmwareRegistrySimple.create(:name => 'Firmware Registry A')
registry.authentication = Authentication.create(:userid => 'user', :password => 'pass')
registry.endpoint = Endpoint.create(:url => 'http://localhost:5000/images/')

# registry.sync_fw_binaries then does something like:
binary = FirmwareBinary.find_or_create_by(:firmware_registry => registry, :external_id => 'ID')
binary.endpoints = [Endpoint.create(:url => 'URL01'), Endpoint.create(:url => 'URL02')]
binary.firmware_targets = [
  FirmwareTarget.find_or_create_by(:manufacturer => 'MANU', :model => 'MODEL_S'),
  FirmwareTarget.find_or_create_by(:manufacturer => 'MANU', :model => 'MODEL_T')
]

# then when we need to list appropriate binaries for given server:
server = PhysicalServer.find_by(:name => 'My Server')
compatible_binaries = FirmwareTarget.find_by(
  :manufacturer => server.manufacturer,
  :model => server.model
).firmware_binaries

miha-plesko · 2019-05-16T13:56:18Z

Hey @agrare not sure if you're aware what we're doing with these firmware thing, so let me recap our current plan: we have a requirement to support operation "update firmware" on physical server (Redfish). We'll be doing it via a (yet not existing) internal state machine.

But MIQ currently has no model for "installable firmware software" that user could pick from a drop-down to update server with. So we need to introduce the new model first.

Looking forward to your comments.

miha-plesko · 2019-05-21T15:14:39Z

Provided a bunch of unit tests, VCR ones as well.

miha-plesko · 2019-05-22T06:13:07Z

/cc @gtanzillo

app/models/firmware_registry.rb

miha-plesko · 2019-06-05T08:30:43Z

Sorry for a bit of delay here @agrare I think we're green now. This PR seems huge, but it's mostly due to all the FactoryBot definitions and specs we are providing. In essence we're only delivering the firmware inventory, as simple as it can get... Would you prefer if I try to split the PR in smaller parts?

miha-plesko · 2019-06-05T12:59:26Z

Kicking travis because rspec only failed for ruby 2.4.6 and I can't reproduce locally on same ruby version

agrare · 2019-06-06T12:36:45Z

app/models/firmware_binary.rb

+  end
+
+  def urls
+    endpoints.map(&:url)


.pluck(:url) is slightly more efficient

agrare · 2019-06-06T12:39:09Z

app/models/firmware_registry_simple.rb

@@ -0,0 +1,59 @@
+class FirmwareRegistrySimple < FirmwareRegistry


Hm wonder if we could come up with something more descriptive than Simple...maybe Http or HttpFileServer?

That is essentially all this registry type is right? Just an http file server with authentication?

Yeah, kind of: it's http server that servers file metadata; the actual files may be stored separately (on some external ftp or samba or something)

Okay, I ended up with a small refactoring. Now we have

firmware_registry |- rest_api_depot.rb # actual STI-driven implementation firmware_registry.rb # base class

I think

class FirmwareRegistry::RestApiDepot

is better than HttpFileServer because we emphasise the metadata part. Thanks for kicking my ass, I didn't like Simple either because it tells literally nothing.

agrare · 2019-06-06T12:40:25Z

app/models/firmware_registry_simple.rb

+class FirmwareRegistrySimple < FirmwareRegistry
+  def sync_fw_binaries_raw
+    remote_binaries.each do |binary_hash|
+      binary = FirmwareBinary.find_or_create_by(:firmware_registry => self, :external_ref => binary_hash['id'])


This feels like something we could use graph refresh for treating the registry as the manager

We can do that in a follow up though I won't hold this up for that

I've asked @Ladas for some guides how to employ graph refresh on non-ems classes, but it should really be done as a followup IMAO. Because quite some files will have to be added (at least collector, parser, persister, definitions etc.).

agrare · 2019-06-06T12:44:09Z

app/models/firmware_registry_simple.rb

+
+      unless binary.urls.sort == binary_hash['urls'].sort
+        _log.info("Updating FirmwareBinary [#{binary.id} | #{binary.name}] endpoints...")
+        binary.endpoints.destroy_all


This could be a bit better if we consider what needs to be deleted and what needs to be added,

endpoints_by_url = binary.endpoints.index_by(&:url) urls_to_delete = binary.urls - binary_hash['urls'] urls_to_delete.each { |url| endpoints_by_url[url].destroy } urls_to_create = binary_hash['urls'] - binary.urls urls_to_create.each { |url| binary.endpoints.create!(:url => url) }

Sweet, thanks, integrated. I assumed that binary endpoints are so rarely updated that we don't care performance on such event, but it can't hurt to do it properly.

agrare · 2019-06-06T12:48:04Z

app/models/firmware_registry_simple.rb

+
+      currents = binary.firmware_targets.map(&:to_hash)
+      remotes = binary_hash['compatible_server_models'].map { |r| r.symbolize_keys.slice(*FirmwareTarget::MATCH_ATTRIBUTES) }
+      unless currents.map(&:to_a).sort == remotes.map(&:to_a).sort


Do you have an example of what the data here would look like? Having a hard time picturing what is going on with the .map(&:to_hash) above then the .map(&:to_a) here

I do https://github.com/ManageIQ/manageiq/pull/18774/files?file-filters%5B%5D=.json&owned-by%5B%5D=#diff-98d768062372d6463bc80189df295c18

agrare · 2019-06-06T12:50:36Z

app/models/firmware_target.rb

+  end
+
+  def to_hash
+    attributes.symbolize_keys.slice(*MATCH_ATTRIBUTES)


Would it make sense to not store attributes that aren't needed instead of overriding to_hash?

We mainly get rid of created/updated timestamps here to get a "pure" hash with relevant attributes only (:manufacturer and :model). Perhaps I should just rename the function to something else?

miha-plesko · 2019-06-10T13:23:02Z

I couldn't figure why Travis won't use VCR for examples marked with :vcr so I'm falling back to the explicit VCR.use_cassette approach..

One can update physical server's firmware, but the firmware has to be accessible somewhere, usually on some HTTP/FTP/SAMBA link. Furthermore, there is usually some metadata associated with the firmware to tell what kind of hardware (manufacturer, model type) is it meant for. With this commit we introduce a new model, FirmwareRegistry, that supports various kinds of metadata parsing by leveraging STI. User is supposed to create FirmwareRegistry instance via UI feeding it url+authentication and then click "Refresh Relationships" on it. ManageIQ will then invoke ``` registry.sync_fw_binaries_queue ``` which in turn will connect to the metadata repository to list available firmwares and store them as FirmwareBinary (no binary data is really stored, only HTTP/FTP/SAMBA link and some metadata). In followup PRs we will then be able to list available firmware binaries to update physical server's firmware. Signed-off-by: Miha Pleško <[email protected]>

miq-bot · 2019-06-11T09:11:12Z

Checked commit xlab-si@481b3dc with ruby 2.3.3, rubocop 0.69.0, haml-lint 0.20.0, and yamllint 1.10.0
19 files checked, 3 offenses detected

spec/vcr_cassettes/firmware_registry/rest_api_depot_when-200.yml

💣 💥 🔥 🚒 - Line 90, Col 18 - trailing-spaces - trailing spaces

spec/vcr_cassettes/firmware_registry/rest_api_depot_when-bad-credentials.yml

💣 💥 🔥 🚒 - Line 39, Col 18 - trailing-spaces - trailing spaces

spec/vcr_cassettes/firmware_registry/rest_api_depot_when-bad-json.yml

💣 💥 🔥 🚒 - Line 37, Col 18 - trailing-spaces - trailing spaces

miha-plesko · 2019-06-11T09:32:29Z

@agrare this is now green. Had quite some issues with VCR being turned off, not sure why or who does it, so I ended up forcibly turning them on

agrare · 2019-06-13T17:46:39Z

spec/models/firmware_registry/rest_api_depot_spec.rb

+    end
+    # Is seems some other test is turning VCR off in some random cases.
+    # We're best off manually turning it on.
+    VCR.turn_on!


Hm this is strange that it'd be needed, I'm okay with this for now but we should look at why it is needed

miha-plesko changed the title ~~Introduce FirmwareRegistry and FirmwareBinary~~ [WIP] Introduce FirmwareRegistry and FirmwareBinary May 16, 2019

miq-bot added the enhancement label May 16, 2019

miq-bot assigned agrare May 16, 2019

miq-bot added the wip label May 16, 2019

miha-plesko mentioned this pull request May 16, 2019

Introduce :firmware_registry and :firmware_binary tables ManageIQ/manageiq-schema#371

Merged

miha-plesko force-pushed the firmware-registry branch 2 times, most recently from 1d51b95 to 8ea4d1a Compare May 21, 2019 15:08

miha-plesko changed the title ~~[WIP] Introduce FirmwareRegistry and FirmwareBinary~~ Introduce FirmwareRegistry and FirmwareBinary (with FirmwareConstraints) May 21, 2019

miq-bot removed the wip label May 21, 2019

miha-plesko force-pushed the firmware-registry branch from 8ea4d1a to c439b59 Compare May 23, 2019 11:30

gtanzillo reviewed May 23, 2019

View reviewed changes

app/models/firmware_registry.rb Show resolved Hide resolved

miha-plesko changed the title ~~Introduce FirmwareRegistry and FirmwareBinary (with FirmwareConstraints)~~ [WIP] Introduce FirmwareRegistry and FirmwareBinary (with FirmwareConstraints) May 24, 2019

miq-bot added the wip label May 24, 2019

miha-plesko changed the title ~~[WIP] Introduce FirmwareRegistry and FirmwareBinary (with FirmwareConstraints)~~ [WIP] Introduce FirmwareRegistry and FirmwareBinary (with FirmwareTargets) May 24, 2019

miha-plesko force-pushed the firmware-registry branch from c439b59 to efa0f4c Compare May 24, 2019 09:10

miha-plesko closed this May 24, 2019

miha-plesko reopened this May 24, 2019

miha-plesko force-pushed the firmware-registry branch from efa0f4c to 754206a Compare May 30, 2019 13:05

miha-plesko mentioned this pull request May 31, 2019

Internal state machine implementation for firmware update ManageIQ/manageiq-providers-redfish#69

Merged

8 tasks

miha-plesko force-pushed the firmware-registry branch from 754206a to 8df2bd0 Compare May 31, 2019 13:33

miha-plesko closed this Jun 4, 2019

miha-plesko reopened this Jun 4, 2019

miha-plesko force-pushed the firmware-registry branch from 8df2bd0 to 84f8249 Compare June 5, 2019 08:24

miha-plesko changed the title ~~[WIP] Introduce FirmwareRegistry and FirmwareBinary (with FirmwareTargets)~~ Introduce FirmwareRegistry and FirmwareBinary (with FirmwareTargets) Jun 5, 2019

miq-bot removed the wip label Jun 5, 2019

miha-plesko closed this Jun 5, 2019

miha-plesko reopened this Jun 5, 2019

miha-plesko force-pushed the firmware-registry branch from 84f8249 to 5c63bdf Compare June 6, 2019 09:43

agrare added providers/physical providers/physical/redfish labels Jun 6, 2019

agrare reviewed Jun 6, 2019

View reviewed changes

miha-plesko force-pushed the firmware-registry branch 2 times, most recently from 9b6b8c6 to 3535018 Compare June 10, 2019 13:21

miha-plesko force-pushed the firmware-registry branch from 3535018 to 37decaf Compare June 11, 2019 08:04

miq-bot added the dependencies label Jun 11, 2019

miha-plesko force-pushed the firmware-registry branch from 37decaf to b06a7e1 Compare June 11, 2019 08:46

miha-plesko force-pushed the firmware-registry branch from b06a7e1 to 481b3dc Compare June 11, 2019 09:09

agrare approved these changes Jun 13, 2019

View reviewed changes

agrare merged commit fa99bd9 into ManageIQ:master Jun 13, 2019

agrare added this to the Sprint 114 Ending Jun 24, 2019 milestone Jun 13, 2019

miha-plesko mentioned this pull request Jun 17, 2019

Introduce Request and Task for firmware update #18801

Merged

tadeboro deleted the firmware-registry branch June 19, 2019 15:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce FirmwareRegistry and FirmwareBinary (with FirmwareTargets) #18774

Introduce FirmwareRegistry and FirmwareBinary (with FirmwareTargets) #18774

miha-plesko commented May 16, 2019 •

edited

Loading

miha-plesko commented May 16, 2019 •

edited

Loading

miha-plesko commented May 21, 2019

miha-plesko commented May 22, 2019

miha-plesko commented Jun 5, 2019

miha-plesko commented Jun 5, 2019

agrare Jun 6, 2019

agrare Jun 6, 2019

agrare Jun 6, 2019

miha-plesko Jun 6, 2019

miha-plesko Jun 10, 2019

agrare Jun 6, 2019

agrare Jun 6, 2019

miha-plesko Jun 10, 2019

agrare Jun 6, 2019

miha-plesko Jun 10, 2019

agrare Jun 6, 2019

miha-plesko Jun 6, 2019

agrare Jun 6, 2019

miha-plesko Jun 6, 2019

miha-plesko commented Jun 10, 2019

miq-bot commented Jun 11, 2019

miha-plesko commented Jun 11, 2019

agrare Jun 13, 2019

		@@ -0,0 +1,59 @@
		class FirmwareRegistrySimple < FirmwareRegistry

Introduce FirmwareRegistry and FirmwareBinary (with FirmwareTargets) #18774

Introduce FirmwareRegistry and FirmwareBinary (with FirmwareTargets) #18774

Conversation

miha-plesko commented May 16, 2019 • edited Loading

miha-plesko commented May 16, 2019 • edited Loading

miha-plesko commented May 21, 2019

miha-plesko commented May 22, 2019

miha-plesko commented Jun 5, 2019

miha-plesko commented Jun 5, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

miha-plesko commented Jun 10, 2019

miq-bot commented Jun 11, 2019

miha-plesko commented Jun 11, 2019

Choose a reason for hiding this comment

miha-plesko commented May 16, 2019 •

edited

Loading

miha-plesko commented May 16, 2019 •

edited

Loading