collect_many_bds empty data

[rafaelelter]

Hi Matthew!

The way the collector was set, raised an error when the query returned no data.
This way it returns the empty data it receives.
For my applications, this fix made things easier.

This is my first pull request ever, so I apologize in advance for any problems with it.

Cheers

[matthewgilbert]

Thanks @rafaelelter , do you have an example query with output you can show running before and after the fix? It would also be helpful for me to see the types of Bloomberg messages this fix enables dealing with

[rafaelelter]

Hi @matthewgilbert.
What motivated me to do this pull request was the following exception, which was raised duo to the fact that one of these securities has never paid any dividends.

>>> from blp import blp
>>> conn = blp.BlpQuery().start()
>>> securities = [
...     "ITUB4 BZ Equity",
...     "BBAS3 BZ Equity",
...     "TRAD3 BZ Equity"
... ]
>>> DIVIDENDS_FIELD = "DVD_HIST_ALL"
>>> rd = blp.create_reference_query(
...     securities=securities,
...     fields=[DIVIDENDS_FIELD]
... )
>>> res = conn.query(
...     request_data=rd,
...     collector=conn.collect_many_to_bds
... )
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\miniconda3\envs\blp-teste\lib\site-packages\blp\blp.py", line 646, in query
    res = collector(res)
  File "C:\miniconda3\envs\blp-teste\lib\site-packages\blp\blp.py", line 890, in collect_many_to_bds
    rows.extend(data)
TypeError: 'NoneType' object is not iterable

For the security without dividends, the bloomberg response is as follows:

[{'security': 'TRAD3 BZ Equity', 'fields': ['DVD_HIST'], 'data': {'DVD_HIST': None}}]

Additionally, when I try to call the bds method, the following exception is raised.

>>> conn.bds(
...     "TRAD3 BZ Equity",
...     DIVIDENDS_FIELD
... )
Traceback (most recent call last):
  File "C:\miniconda3\envs\blp-test\lib\site-packages\blp\blp.py", line 871, in collect_to_bds
    rows.extend(data)
TypeError: 'NoneType' object is not iterable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\miniconda3\envs\blp-test\lib\site-packages\blp\blp.py", line 856, in bds
    return self.query(query, self.parser, self.collect_to_bds)
  File "C:\miniconda3\envs\blp-test\lib\site-packages\blp\blp.py", line 646, in query
    res = collector(res)
  File "C:\miniconda3\envs\blp-test\lib\site-packages\blp\blp.py", line 873, in collect_to_bds
    raise TypeError(f"response data must be bulk reference data, received {response['data']}")
TypeError: response data must be bulk reference data, received {'DVD_HIST_ALL': None}

I think that both collectors shouldn't raise exceptions for the presented case. Instead, they should return empty pandas.DataFrames, None or an empty dictionary.
My fix makes the collect_many_to_bds return an empty dictionaty for empty fields, as the response from the first example follows:

{
   "ITUB4 BZ Equity": {"DVD_HIST_ALL": pd.DataFrame}, 
   "BBAS3 BZ Equity": {"DVD_HIST_ALL": pd.DataFrame}, 
   'TRAD3 BZ Equity': {}
}

I didn't make any changes to the collect_to_bds collector, but I think I should.
Hope this clarifies what I tried to fix.

[matthewgilbert]

Could you post the output from running the following the query without parsing / a collector, i.e.

res = conn.query(
     request_data=rd,
    parse=False,
    collector=list
)
print(json.dumps(res, indent=2, default=str))

[rafaelelter]

Here it is:

[
  {
    "eventType": 5,
    "eventTypeName": "blpapi.Event.RESPONSE",
    "messageNumber": 0,
    "message": {
      "fragmentType": 0,
      "correlationIds": [
        147
      ],
      "messageType": "ReferenceDataResponse",
      "timeReceived": null,
      "element": {
        "ReferenceDataResponse": [
          {
            "securityData": {
              "security": "TRAD3 BZ Equity",
              "eidData": [],
              "fieldExceptions": [
                {
                  "fieldExceptions": {
                    "fieldId": "DVD_HIST_ALL",
                    "errorInfo": {
                      "errorInfo": {
                        "source": "19465:rsfrdsvc-slow2",
                        "code": 9,
                        "category": "BAD_FLD",
                        "message": "Field not applicable to security",
                        "subcategory": "NOT_APPLICABLE_TO_REF_DATA"
                      }
                    }
                  }
                }
              ],
              "sequenceNumber": 0,
              "fieldData": {
                "fieldData": {}
              }
            }
          }
        ]
      }
    }
  }
]

[matthewgilbert]

Thanks for the fix @rafaelelter!