in query empty start_cursor option throws error

[rzylius]

hi,

I try to make a query:

next_cursor = None
results = notion.databases.query(
        **{
            "database_id": DB_GMB_ID,
            "start_cursor": next_cursor
        }
    )

I receive the error:

APIResponseError: body failed validation: body.start_cursor should be a string or `undefined`, instead was `null`.

"" produces the same result.

What value should be used to satisfy undefined start_cursor?

[ramnes]

Hello @rzylius, that's quite a funny one and I've no idea what Notion is expecting from us. I'll investigate eventually, please tell me if you find anything new in the meantime!

[ramnes]

Removing that pick thing and using proper arguments would probably help.

[nicobako]

I don't think this is an issue with notion_client. In the notion API documentation it states that start_cursor must be a string.

From https://developers.notion.com/reference/post-database-query

start_cursor
string
When supplied, returns a page of results starting after the cursor provided. If not supplied, this endpoint will return the first page of results.

It seems that start_cursor must either be specified as a string, or not specified at all. null (or None, in Python), is not a valid value of start_cursor.

[rzylius]

It seems that start_cursor must either be specified as a string, or not specified at all. null (or None, in Python), is not a valid > value of start_cursor.

The error thrown "APIResponseError: body failed validation: body.start_cursor should be a string or undefined" says, that the value can be 'undefined'. This makes sense in javascript, but how 'undefined' works in Python?

[nicobako]

The error thrown "APIResponseError: body failed validation: body.start_cursor should be a string or undefined" says, that the value can be 'undefined'. This makes sense in javascript, but how 'undefined' works in Python?

There is no undefined in Python. Python has None, which is equivalent to null.

In Python, we have the concept of an optional value, where something can either be a value or None. For example, x = 1 or x = None. But, in both cases, a variable named x exists (i.e. x is not undefined).

[nicobako]

I see two possible fixes here:

1. Remove usage start_cursor=None from your code
2. Modify pick(...) so that it will remove start_cursor=None for you

I'm not sure which one of these is the best solution... Perhaps others have an opinion.

[rzylius]

@nicobako , option 1 is clearly a workaround, as then you need to differentiate if this is a first run of the query or the next page.

Can you elaborate on option 2?

[ramnes]

In the meantime, I would do:

next_cursor = None
...
query = {"database_id": DB_GMB_ID}
if next_cursor:
    query["start_cursor"] = next_cursor
results = notion.databases.query(**query)
...

[nicobako]

@rzylius ,

Option 2 is about changing the behaviour of notion_client.helpers.pick. This function is used throughout the code to help pick only the keyword-value pairs that are of interest. See the source code here

For example, the endpoint databases.query accepts the following keywords: filter, sorts, start_cursor, page_size. You can see how the pick function is used to extract only the pertinent keyword-value pairs for this API endpoint here

[nicobako]

@ramnes , would a good solution to this problem be to change the implementation of pick a little? We can change it so that, if any keyword-value pair has a value that is None, it could be removed:

def pick(base: Dict[Any, Any], *keys: str,) -> Dict[Any, Any]:
    """Return a dict composed of key value pairs for keys passed as args."""
    return {key: base[key] for key in keys if key in base and base[key] is not None}

Here's an example of its usage:

>>> base = {"start_cursor":None}
>>> pick(base, "start_cursor")
{}
>>> base = {"start_cursor": "start"}
>>> pick(base, "start_cursor")       
{'start_cursor': 'start'}

Not sure if this would always work with the Notion API. This assumes that if a value in None, then we can just remove it, but maybe sometimes we have to keep values that are None.

O maybe you have an idea for a better solution.

[thomashirtz]

I stumbled on this issue today as well, it is so annoying when fetching a whole database:
I had to write:

    results = []
    query = notion.databases.query(database_id=database_id, page_size=page_size)
    results.extend(query['results'])
    while query['next_cursor']:
        query = notion.databases.query(database_id=database_id, start_cursor=query['next_cursor'], page_size=page_size)
        results.extend(query['results'])

Instead of something a tad more elegant, such as:

    results = []
    query = {'next_cursor': None}
    while query['next_cursor'] or query['next_cursor'] is None:
        query = notion.databases.query(database_id=database_id, start_cursor=query['next_cursor'], page_size=page_size)
        results.extend(query['results'])

notion_client.errors.APIResponseError: body failed validation: body.start_cursor should be a string or `undefined`, instead was `null`.

or

    results = []
    query = {}
    while not query or query['next_cursor']:
        query = notion.databases.query(database_id=database_id, start_cursor=query.get('next_cursor', None), page_size=page_size) # doesn't work because the api error when None is passed
        results.extend(query['results'])

notion_client.errors.APIResponseError: body failed validation: body.start_cursor should be a string or `undefined`, instead was `null`.

or

    results = []
    query = {'next_cursor': 'undefined'}
    while query['next_cursor']:
        query = notion.databases.query(database_id=database_id, start_cursor=query['next_cursor'], page_size=page_size) # doesn't work because the api error when None is passed
        results.extend(query['results'])

notion_client.errors.APIResponseError: body failed validation: body.start_cursor should be a valid uuid or `undefined`, instead was `"undefined"`.

Not sure how to code that elegantly now

I think it is needed to remove the argument start_cursor internally if None is passed, because I don't see any situation where it is wanted to get this error response

[ramnes]

Should be fixed now. :)

[thomashirtz]

Maybe it would be nice to have a new release ? So it is possible to have the right version of notion-sdk-py as requirement for tools that need this change ? :)

[nicobako]

Really good point @thomashirtz ,

In the meanwhile, you can pip install directly from GitHub. Here is a link to the pip documentation on vcs support:

https://pip.pypa.io/en/stable/topics/vcs-support/

Here's another blog post which might be easier to read and extract useful information:

https://adamj.eu/tech/2019/03/11/pip-install-from-a-git-repository/

[ramnes]

I just released 0.9.0. Thanks!

[jwasp]

how I can write this in javascript? I stuck on start_cursor