Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

i133 Ensure all data updates if an item has already been harvested #22

Merged
merged 7 commits into from
Aug 18, 2022

Conversation

bkiahstroud
Copy link
Contributor

@bkiahstroud bkiahstroud commented Aug 9, 2022

Ref https://github.com/harvard-lts/CURIOSity/issues/133

find_or_create_by won't execute its do block if a record is found. This was skipping the update of data for existing records. Additionally, newly added fields were not persisting on records that already existed. This PR fixes both issues

@bkiahstroud bkiahstroud marked this pull request as ready for review August 10, 2022 16:13
@dl-maura dl-maura self-requested a review August 11, 2022 23:19
@bkiahstroud bkiahstroud changed the title Ensure all data updates if an item has already been harvested i133 Ensure all data updates if an item has already been harvested Aug 12, 2022
@bkiahstroud bkiahstroud marked this pull request as draft August 12, 2022 18:52
@bkiahstroud
Copy link
Contributor Author

Converting to draft: I discovered this introduces a bug where only a small set of data is harvested on the first harvest

Benjamin Kiah Stroud added 3 commits August 15, 2022 10:41
This fixes a bug where, the first time a set is harvested, none of the
resource metadata gets indexed in solr. Harvesting the same set again
would populate the data in solr.

My best guess as to what was happening is that #find_or_create_by was
creating an empty sidecar doc, which then for some reason wasn't getting
updated when calling #save_and_index on its resource.

Instead, I've opted to set up all the data before committing anything to
the db/solr. Now, the first harvest of a set brings in the metadata as
expected.
@bkiahstroud
Copy link
Contributor Author

Converting to draft: I discovered this introduces a bug where only a small set of data is harvested on the first harvest

See this commit for the resolution & explanation

@bkiahstroud bkiahstroud marked this pull request as ready for review August 16, 2022 18:25
@bkiahstroud bkiahstroud mentioned this pull request Aug 16, 2022
Copy link
Contributor

@dl-maura dl-maura left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worked with Kiah to demo this

@dl-maura dl-maura merged commit fd2a5ac into main Aug 18, 2022
@dl-maura dl-maura deleted the i133-existing-items-dont-update-on-reharvest branch August 18, 2022 18:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants