Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove fk_rec_file_id from t_transactions #600

Merged
merged 8 commits into from
Mar 16, 2020
Merged

Remove fk_rec_file_id from t_transactions #600

merged 8 commits into from
Mar 16, 2020

Conversation

apeksharma
Copy link
Contributor

Detailed description:

  • fk_rec_file_id in t_transactions was deprecated and its value is always 0. Removing the column.
  • No need of 2-part insert using f_file_create/complete.
    Just checking for presense on file start and inserting row on file complete is sufficient.
  • Earlier, fk_rec_file_id was foreign key, so many tests for transactions needed to added
    a row to t_record_files. That foreign key was dropped recently, so update tests to remove chaff.
  • Changes in REST code are limited to tests

BRD uses t_record_files table to simulate blocks internally. So we can't just remove it.
We have two possible options going forward:

  1. Remove t_record_files tables. We have no regular need of finding which file a txn belongs to.
    If needed for debugging, one can use txn's epoch datetime and do prefix ls on S3.
    Will need to understand how BRD uses t_record_files and if it is just as easy to simulate blocks
    using t_transactions.
  2. If we keep the table around, to support lookup of all txns associated with a file, we can add
    two columns filled by consensus timestamps of first and last transaction in the file.

Signed-off-by: Apekshit Sharma [email protected]

Which issue(s) this PR fixes:
Followup of #566

Special notes for your reviewer:
Followup from #585 (comment)

Checklist

  • Documentation added
  • Tests updated

@apeksharma apeksharma added P3 parser Area: File parsing database Area: Database labels Mar 13, 2020
@apeksharma apeksharma self-assigned this Mar 13, 2020
@steven-sheehy steven-sheehy changed the title Remove fild_id from t_record_files Remove file_id from t_record_files Mar 13, 2020
@steven-sheehy steven-sheehy added this to the Mirror 0.8.0 milestone Mar 13, 2020
- fk_rec_file_id in t_transactions was deprecated and its value is always 0. Removing the column.
- No need of 2-part insert using f_file_create/complete.
  Just checking for presense on file start and inserting row on file complete is sufficient.
- Earlier, fk_rec_file_id was foreign key, so many tests for transactions needed to added
  a row to t_record_files. That foreign key was dropped recently, so update tests to remove chaff.
- Changes in REST code are limited to tests

BRD uses t_record_files table to simulate blocks internally. So we can't just remove it.
We have two possible options going forward:
1. Remove t_record_files tables. We have no regular need of finding which file a txn belongs to.
If needed for debugging, one can use txn's epoch datetime and do prefix ls on S3.
Will need to understand how BRD uses t_record_files and if it is just as easy to simulate blocks
using t_transactions.
2. If we keep the table around, to support lookup of all txns associated with a file, we can add
two columns filled by consensus timestamps of first and last transaction in the file.

Signed-off-by: Apekshit Sharma <[email protected]>
Signed-off-by: Apekshit Sharma <[email protected]>
Copy link
Member

@steven-sheehy steven-sheehy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

f_file_complete and f_file_create are no longer used and should be removed in a db migration.

Signed-off-by: Apekshit Sharma <[email protected]>
Signed-off-by: Apekshit Sharma <[email protected]>
@codecov
Copy link

codecov bot commented Mar 13, 2020

Codecov Report

Merging #600 into master will decrease coverage by 0.15%.
The diff coverage is 66.66%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master     #600      +/-   ##
============================================
- Coverage     64.79%   64.64%   -0.16%     
  Complexity      171      171              
============================================
  Files           101      102       +1     
  Lines          3159     3142      -17     
  Branches        362      360       -2     
============================================
- Hits           2047     2031      -16     
  Misses          959      959              
+ Partials        153      152       -1
Impacted Files Coverage Δ Complexity Δ
...com/hedera/mirror/importer/domain/Transaction.java 81.81% <ø> (-0.8%) 0 <0> (ø)
...era/mirror/importer/exception/ParserException.java 33.33% <ø> (ø) 0 <0> (ø) ⬇️
...ror/importer/exception/DuplicateFileException.java 33.33% <33.33%> (ø) 0 <0> (?)
...irror/importer/parser/record/RecordFileParser.java 76.31% <80%> (-0.96%) 0 <0> (ø)
...record/PostgresWritingRecordParsedItemHandler.java 84.48% <85.71%> (+1.49%) 0 <0> (ø) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9294d42...a124ccd. Read the comment docs.

Signed-off-by: Apekshit Sharma <[email protected]>
Signed-off-by: Apekshit Sharma <[email protected]>
steven-sheehy
steven-sheehy previously approved these changes Mar 16, 2020
Copy link
Member

@steven-sheehy steven-sheehy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Will reapply once you've merged in Nana's REST changes, which might be a bit tricky

@Nana-EC
Copy link
Collaborator

Nana-EC commented Mar 16, 2020

LGTM. Will reapply once you've merged in Nana's REST changes, which might be a bit tricky

@apeksharma just pull in my changes, over yours for the rest side. I already took out the record logic after your last comment. Let me know if you're uncertain about any line

@apeksharma
Copy link
Contributor Author

updated.
Those transient VM errors 🤦‍♂ .
Can reapply approvals, i'll merge in when build_maven succeeds.

Copy link
Collaborator

@Nana-EC Nana-EC left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@apeksharma apeksharma merged commit c7ab9ce into master Mar 16, 2020
@apeksharma apeksharma deleted the parser_133 branch March 16, 2020 21:21
@apeksharma apeksharma changed the title Remove file_id from t_record_files Remove fk_rec_file_id from t_transactions Mar 16, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
database Area: Database P3 parser Area: File parsing
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants