Fog::JSON::EncodeError when using to_json on servers collection

[imriz]

Fog versions:
fog-core (2.1.0)
fog-google (1.3.3)
fog-json (1.1.0)

Code I used to reproduce:

connection = Fog::Compute.new({:provider => 'google'})
connection.servers.all.to_json

I get the following error:

Fog::JSON::EncodeError: "\xE3" from ASCII-8BIT to UTF-8
	from /home/imriz/.gem/ruby/gems/multi_json-1.13.1/lib/multi_json/adapters/json_common.rb:19:in `encode'
	from /home/imriz/.gem/ruby/gems/multi_json-1.13.1/lib/multi_json/adapters/json_common.rb:19:in `to_json'
	from /home/imriz/.gem/ruby/gems/multi_json-1.13.1/lib/multi_json/adapters/json_common.rb:19:in `dump'
	from /home/imriz/.gem/ruby/gems/multi_json-1.13.1/lib/multi_json/adapter.rb:25:in `dump'
	from /home/imriz/.gem/ruby/gems/multi_json-1.13.1/lib/multi_json.rb:139:in `dump'
	from /home/imriz/.gem/ruby/gems/fog-json-1.1.0/lib/fog/json.rb:34:in `encode'
	from /home/imriz/.gem/ruby/gems/fog-core-2.1.0/lib/fog/core/collection.rb:106:in `to_json'
	from (irb):4
	from /usr/bin/irb:11:in `<main>'

Any ideas what I am doing wrong?

[imriz]

These chars are coming from the metadata "fingerprint" item, btw

[Temikus]

@imriz - acknowledged. I’m doing some work on SQL tomorrow so I’ll take a look.

[Temikus]

Was able to repro this:

pry(main)> connection = Fog::Compute.new(:provider => "Google")
pry(main)> connection.servers.first.to_json
Fog::JSON::EncodeError: "\xE3" from ASCII-8BIT to UTF-8
from /Users/temikus/.rbenv/versions/2.4.3/lib/ruby/gems/2.4.0/gems/multi_json-1.13.1/lib/multi_json/adapters/json_common.rb:19:in `encode'

Fingerprint is:

"fingerprint": "iI4Ao2hYH-4="

Model field is actually:

label_fingerprint="\xE3e\xA6J\x90|\xAD#"

Hash returned by Google API Lib has an ASCII encoding...

[9] pry(#<Fog::Compute::Google::Servers>)> data[0][:label_fingerprint]
=> "\xE3e\xA6J\x90|\xAD#"
[11] pry(#<Fog::Compute::Google::Servers>)> data[0][:label_fingerprint].encoding
=> #<Encoding:ASCII-8BIT>

I'm not that savvy with encoding issues, @icco @plribeiro3000 - any idea on a non-hacky way to force model fields to utf-8? Or is this something we need to file upstream to Google API client?

[icco]

After reading through this @Temikus, I'm like 70% sure It's either a bug with the api-client, how api-client is using multi-json or multi-json itself.

Everything should utf-8, so I'm confused why we're ever in the state of ASCII-8BIT. I wonder if there is somewhere we can be explicit to the API and say "never give us anything besides utf-8".

[Temikus]

Found the problem:
https://github.com/google/google-api-ruby-client/blob/master/MIGRATING.md

Migrating from version0.10 to 0.11

Unicode normalization

The client no longer normalizes unicode strings in path parameters. This may affect some applications using multibyte strings that were previously normalized.: To restore the previous behavior, set the following option prior to creating a service.

ClientOptions.default.normalize_unicode = true

I'll need to pass a client option in, I'll get to it by EOW next week (probably sooner).

[icco]

Nice find!

[Temikus]

Hmmm, just tried those options - that's not it. It seems that the behaviour is correct from Google API client point of view - discovery doc labels the field as bytes, so it returns byte strings.

Hmmm, I need to think about it...

[Temikus]

So, in short - this is not an entirely easy problem.

The thing here is - label_fingerprint property is a byte string as per the DiscoveryDoc that the Google API client generates classes from: https://www.googleapis.com/discovery/v1/apis/compute/v1/rest

   "properties": {
    "labelFingerprint": {
     "type": "string",
     "description": "Fingerprint of the previous set of labels for this resource, used to prevent conflicts. Provide the latest fingerprint value when making a request to add or change labels.",
     "format": "byte"
    },

JSON as a format doesn't support byte strings.

There are some possible workarounds like encoding options but due to us using MultiJson in Fog::JSON we cannot rely on a particular adapter to be present.

So the question here is - is anyone going to use this to marshall objects? If so - we'll need to extend Fog::JSON .decode and .encode to properly Base64.encode64 and Base64.decode64 byte strings.

If this is only for dumping the information to display it somewhere - I can probably override the .to_json for Fog::Google models and encode it one way.

@icco @plribeiro3000 - do you have any thoughts about this? You guys are more experienced so maybe I'm missing something.

@imriz - Can you let me know about your use case a bit more? E.g. are you using json to just display data or... ?

[icco]

Uhm, hmm. TIL about json not supporting it. I'd be supportive of fixing fog:json if it's something useful, but that depends on if it's a bug and not just something people want for testing.

[imriz]

@imriz - Can you let me know about your use case a bit more? E.g. are you using json to just display data or... ?

Our use case is local caching of huge lists of instances/disks/snapshots.

[Temikus]

@imriz Understood. Thank you! Do you decode(unmarshal) them back to a Fog::Collection object afterwards?

I'm asking since making a backwards-incompatible json dump (changing json string to a set of UTF8 characters) is relatively easy. Modifying Fog::JSON.decode to produce identical objects from it is a bit more complicated, however.

[github-actions]

This issue has been marked inactive and will be closed if no further activity occurs.