Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

different pks on base table of materialized view #565

Open
Cbudano opened this issue Aug 13, 2024 · 0 comments
Open

different pks on base table of materialized view #565

Cbudano opened this issue Aug 13, 2024 · 0 comments

Comments

@Cbudano
Copy link

Cbudano commented Aug 13, 2024

PGSync version: 3.2

Postgres version: 14.5

Elasticsearch/OpenSearch version: OpenSearch 2.13

Redis version: 6.0.16

Python version: 3.10

Problem Description:
using a materialized view as source, with 3 base tables with diferent pks, the index gets created on opensearch, but any update on any of the base tables gives the following error:

O:main:2024-08-13 16:24:41.987:DEBUG:pgsync.sync: on_publish len 1

INFO:main:2024-08-13 16:24:41.988:ERROR:pgsync.sync: Primary keys ['sd_secid', 'secid', 'sm_secid'] not subset of payload data dict_keys(['sm_secid']) for table master_ref_data.v_security_stats_mv2

INFO:main:NoneType: None

INFO:main:2024-08-13 16:24:41.988:ERROR:pgsync.search_client: Exception No active exception to reraise

INFO:main:Traceback (most recent call last):

INFO:main:File "/usr/local/lib/python3.12/site-packages/pgsync/search_client.py", line 133, in bulk

INFO:main:self._bulk(

INFO:main:File "/usr/local/lib/python3.12/site-packages/pgsync/search_client.py", line 171, in _bulk

INFO:main:for _ in self.streaming_bulk(

INFO:main:^^^^^^^^^^^^^^^^^^^^

INFO:main:File "/usr/local/lib/python3.12/site-packages/opensearchpy/helpers/actions.py", line 328, in streaming_bulk

INFO:main:for bulk_data, bulk_actions in _chunk_actions(

INFO:main:^^^^^^^^^^^^^^^

INFO:main:File "/usr/local/lib/python3.12/site-packages/opensearchpy/helpers/actions.py", line 168, in _chunk_actions

INFO:main:for action, data in actions:

This is my schema.json:

[
{
"database": "client_facing",
"index": "v_security_stats_mv2",
"nodes": {
"table": "v_security_stats_mv2",
"base_tables": ["fi_security", "sec_master", "sec_detl"],
"schema": "master_ref_data",
"columns": [
"isfixedincome",
"status",
"secid",
"sd_secid",
"sm_secid",
"comp_id",
"cusip",
"company_name",
"symbol"
],
"primary_key": ["sd_secid","secid","sm_secid"]
}
}
]

sd_secid is the PK of sec_detl, secid is the PK of fi_security and sm_secid is the pk of sec_master

Error Message (if any):



Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant