Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

search_phase_execution_exception thrown when parent and child have different id types #576

Open
dealertouch opened this issue Sep 15, 2024 · 0 comments

Comments

@dealertouch
Copy link

PGSync version: main branch

Postgres version: latest

Elasticsearch/OpenSearch version: latest

Redis version: latest

Python version: 3.12.5

Problem Description:
Assume we have three tables, inventory (primary key is of type UUID), feature_map and features(Primary key is of type Long)
when we try the update/insert following exception is thrown

opensearchpy.exceptions.RequestError: RequestError(400, 'search_phase_execution_exception', 'failed to create query: For input string: "6cc0eb22-2703-4fce-bf24-eafe6900f059"')

This is due to _meta.features.id being of type long in elasticsearch but pgsync trying to create a terms query with value "6cc0eb22-2703-4fce-bf24-eafe6900f059"

After doing some investigation, the reason behind this failure is this peace of code in symc.py:_root_foreign_key_resolver() function

        foreign_values: list = [
            payload.new.get(key) for key in foreign_keys[node.name]
        ]

this code will generate a terms query like [1, '6cc0eb22-2703-4fce-bf24-eafe6900f059'] which is invalid (two different types)
there are two issues with this code, first is the one I pointed out in this ticket with different id types, the other one I believe is since it's adding both foreign key values in feature_map table, even if they are the same type it's looking for the wrong id since it's inserting id for inventory too

I was able to fix this with replacing this with the following code (Column is of sqlalchemy type)

        foreign_values: list = []
        for column in node.columns:
            if isinstance(column, Column):
                for foreign_key in column.foreign_keys:
                    referenced_column = foreign_key.column
                    base_table = referenced_column.table
                    if base_table.name == node.parent.table:
                        foreign_values.append(payload.new.get(column.name))

also, in general, when there is an exception is thrown, the pgsync completely stops processing any data instead of moving to the next data

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant