PubsubMessage data comes in plaintext instead of Base64

[tumido]

We are accessing PubSub service via fog and I've noticed that fog does Base64 decoding on the message payload (:data). This does not reflect real world experience, since the data comes in plaintext for me.

Calling :pull method on a subscription follows Google API specification and does a Base64 decoding of the payload. This is consistent with Google API documentation for PubsubMessage. However in real world, this data comes in plain text so the Base64 decoding makes it all gibberish.

Our current workaround is to pull the subscription directly from the service, like the pull method would do here, so we have control over the data processing.

It would be really nice to have this working as an user would expect, however I know that you follow Google API specification already, so there might be nothing you can do about it...

[Temikus]

I think this is the same issue as in #345. I’ll bump this in my priority list. Should have some time to code this week.

[Temikus]

The workaround in the meantime is to lock the google client dep to 0.10

[Temikus]

Hmmm, I was mistaken, sorry. I'll see what we can do about this.

[Temikus]

@tumido So PubsubMessage object represents the data as a sequence of bytes:
https://cloud.google.com/pubsub/docs/reference/rest/v1/PubsubMessage

Google API client uses the Discovery Document (https://www.googleapis.com/discovery/v1/apis/pubsub/v1/rest) to generate classes so this is not something we can change.

However, I am thinking of better representation of attributes in regards to #345, so depending on your use-case we may be able to produce better output.

Are you looking for .inspect producing better data or... ? Is it possible to get a short code snippet reproducing what you're looking for?

[tumido]

Ok, I'll try to explain you our usecase in as much detail as I can. We use PubSub to receive events from GCE. We assume user has an log export set:

• In Google cloud web console menu -> Stackdriver -> Logging -> Logs -> Create Export
• Set the Sink service as Google Pub/Sub and have a topic
• For reference we use resource.type="gce_instance" logName="projects/cfme-eng/logs/compute.googleapis.com%2Factivity_log" as a filter and manageiq-activity-log as a sink destination\topic

Then getting the messages via fog should work like this:

> pubsub = ::Fog::Google::Pubsub.new(...)
> subscription = pubsub.subscriptions.get(subscription_name)
> msgs = subscription.pull()
[
   [0]   <Fog::Google::Pubsub::ReceivedMessage
    ack_id="QBJMNgVESVMrQwsqWBFOBCEhPjA-RVNEUAYWLF1GSFE3GQhoUQ5PXiM_NSAoRRULUxNRXHQCRRBpVFkaB1ENGXJ8Z3Y-XkUIVBAHe1VbEQ16bVxXOFAMGHR9YHxrWBcFC0RTfXfvsP612Z5RZho9XBJLLD5-PTBFQV4",
    message={:attributes=>{:"logging.googleapis.com/timestamp"=>"2018-06-06T12:02:23.009752Z"}, :data=>"\x8A{\x1E\xAE\xD2\x1D\xD7G)\xD3\xBE\xA0\xD6\x99\xEE\xB5\xB9\xA3\xB2\x89\xCFk)hi\xD6\x9C\xB6\x8A\xEE\xB1\xEA\xEDr\x8B\x9FjZ\xDEv\x16\xADr\x89\x9E\xBD\xE9\xED\xB2\xE6\xED\xCA\x97\x9C\xA2jn\xB5\xE8\xA7\xB2\xD6\xA7q\xEB\"\x9E\xC7\xAB\xB5\xEB\xDE\x9E\xDBb\x99\xEB-jjn\xB3^v\xF3o:\xE7\x8D\xF4\xD3\xDE\xF9\xD9\xEB\xDE\x9E\xDBr\xA5\xE1\x82\x10\x03\xC8\b\x02\xCB\x8A\x96\x9Dv\xB7\xAC\xB2\x8A^\xAD\xABb\xA2x\x9D\xE7]\xB5\xE3n\xB5\xDF]:\xF3o9\xF3\x9E\xE7jg\xA8\xA5\xEA\xDA\xB6*'\xD7\x9D\xBC\xDB\xCE\xB9\xE3^\xBA\xD3\x9E\x9D\x7F\xB7\xF4\xE9\xED\xDF\xEB]8\x7F\xAE\x9Do\xCF6k_9o\xCBr\xA5\xEA)z\xB6\xAD\x8A\x89\xF3\xA2w\xAE\xB1\xE6\xAC\xB7V\xEBz\xAB\x9E\xB2\xD6\xE8w)\xEE\x96[\xAB\x96\em\xA6\xCF\xFF\xC3\f \xA2\x88%y\xAAb\xB1\xCA&\xFD\xCA&\xA6\xEB^\xFD\xB7\xADk\xFAk\xA27\x9C\xB6\xCF\xDC~g\x9E\x9E\x0F\xF3\xA2w\xAC\xFE\xEB\x1Ej\xCBuo\xF8\xA7\xB2\xD6\xA7q\xEB${", :message_id=>"111601912449103", :publish_time=>"2018-06-06T12:02:23.580Z"}
  >
]
> msgs.map { |e| JSON.parse(e.message[:data]) }
JSON::ParserError: 765: unexpected token at '�...

This is because the messages which are coming from GCE have :data in plaintext and not Base64 encoded (as it should be by the documentation). So pulling messages like this and skipping the decoding part, makes it readible again:

> pubsub = ::Fog::Google::Pubsub.new(...)
> subscription = pubsub.subscriptions.get(subscription_name)
> msgs = pubsub.pull_subscription(subscription_name)
#<Google::Apis::PubsubV1::PullResponse:0x000055bea51a6d78 @received_messages=[#<Google::Apis::PubsubV1::ReceivedMessage:0x000055bea51a5608 @ack_id="QBJMNgVESVMrQwsqWBFOBCEhPjA-RVNEUAYWLF1GSFE3GQhoUQ5PXiM_NSAoRRULUxNRXHQCRRBpVFkaB1ENGXJ8Z3Y-XkUIVBAHe1VbEQ16bVxXOFAMGHR9YHxrWBcFC0RTfXfvsP612Z5RZho9XBJLLD5-PTBFQV4", @message=#<Google::Apis::PubsubV1::Message:0x000055bea51a4a28 @attributes={"logging.googleapis.com/timestamp"=>"2018-06-06T12:02:23.009752Z"}, @data="{\"insertId\":\"10cp076g1pnutbm\",\"jsonPayload\":{\"actor\":{\"user\":\"XXXXXX\"},\"event_subtype\":\"compute.instances.insert\",\"event_timestamp_us\":\"1528286543009752\",\"event_type\":\"GCE_API_CALL\",\"ip_address\":\"\",\"operation\":{\"id\":\"5121426131068285857\",\"name\":\"operation-1528286541660-56df7f06e3f61-04f66db8-82a185b8\",\"type\":\"operation\",\"zone\":\"us-east1-b\"},\"request\":{\"body\":\"null\",\"url\":\"https://www.googleapis.com/compute/beta/projects/cfme-eng/zones/us-east1-b/instances?key=AIzaSyCgxnNGJFOtw9DF3eySks0uRxQ3SiQW2jA\"},\"resource\":{\"id\":\"6142937983063061410\",\"name\":\"instance-1\",\"type\":\"instance\",\"zone\":\"us-east1-b\"},\"trace_id\":\"operation-1528286541660-56df7f06e3f61-04f66db8-82a185b8\",\"user_agent\":\"Pantheon Google-API-Java-Client Google-HTTP-Java-Client/1.23.0-SNAPSHOT (gzip)\",\"version\":\"1.2\"},\"labels\":{\"compute.googleapis.com/resource_id\":\"6142937983063061410\",\"compute.googleapis.com/resource_name\":\"instance-1\",\"compute.googleapis.com/resource_type\":\"instance\",\"compute.googleapis.com/resource_zone\":\"us-east1-b\"},\"logName\":\"projects/cfme-eng/logs/compute.googleapis.com%2Factivity_log\",\"receiveTimestamp\":\"2018-06-06T12:02:23.117912873Z\",\"resource\":{\"labels\":{\"instance_id\":\"6142937983063061410\",\"project_id\":\"cfme-eng\",\"zone\":\"us-east1-b\"},\"type\":\"gce_instance\"},\"severity\":\"INFO\",\"timestamp\":\"2018-06-06T12:02:23.009752Z\"}", @message_id="111601912449103", @publish_time="2018-06-06T12:02:23.580Z">>]>
> msgs.to_h[:received_messages].map { |e| JSON.parse(e[:message][:data]) }
[
    [0] {
                "insertId" => "kjcfkafrc4d7i",
             "jsonPayload" => {
                         "actor" => {
                "user" => "XXXXXX"
            },
                 "event_subtype" => "compute.instances.insert",
...
    }
]

(I've skipped the part where we ACK the messages, to keep the example short)

I know that fog-google follows the documentation for PubSub. But our observation of reality does not meet the expectation according to documentation. In other words they say that the data should be Base64 encoded, so you do the decode. But we're getting plaintext data and when you do such decode on it, it makes it gibberish.

That's not fog's fault. It just doesn't work as expected / as it should 😜 .

[Temikus]

@tumido - ahhhhh, got it, thanks! If I’m reading this right this should be fixeable. I’ll take a look by EOW.

[github-actions]

This issue has been marked inactive and will be closed if no further activity occurs.