Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: migration("re-indexing"), backfilling and diasgnostics tooling for the ChainIndexer #12450

Merged
merged 70 commits into from
Oct 14, 2024
Merged

feat: migration("re-indexing"), backfilling and diasgnostics tooling for the ChainIndexer #12450

merged 70 commits into from
Oct 14, 2024

Commits on Sep 16, 2024

  1. fix conflicts with chain indexer

    @aarshkshah1992
    aarshkshah1992 committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    0f4c627 View commit details
    Browse the repository at this point in the history

Commits on Sep 17, 2024

  1. feat: chain indexer todos [skip changelog] (#12462) 

    * feat: finish todos of validation api

* feat: add indexed data verification with chain store

* feat: address comments and finish TODO

* fix: build issue

* address comments

* fix: ci issue
    @akaladarshi
    akaladarshi authored Sep 17, 2024
    Configuration menu
    Copy the full SHA
    1b5fed3 View commit details
    Browse the repository at this point in the history

  2. Apply suggestions from code review 

    Co-authored-by: Rod Vagg <[email protected]>
    @aarshkshah1992 @rvagg
    aarshkshah1992 and rvagg authored Sep 17, 2024
    Configuration menu
    Copy the full SHA
    260cde6 View commit details
    Browse the repository at this point in the history

  3. changes to Index Validation API based on Rodds first review

    @aarshkshah1992
    aarshkshah1992 committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    127b144 View commit details
    Browse the repository at this point in the history

  4. build chain indexer API

    @aarshkshah1992
    aarshkshah1992 committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    05b84ec View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2024

  1. improve error handling

    @aarshkshah1992
    aarshkshah1992 committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    11712bf View commit details
    Browse the repository at this point in the history

  2. feat: lotus-shed tooling for chain indexer (#12474) 

    * feat: add lotus-shed command for backfilling chain indexer

* feat: add lotus-shed command for inspecting the chain indexer

* feat: use single lotus-shed command to inspect and backfill

* fix: remove the unused queries

* small changes

* add change log
    @akaladarshi
    akaladarshi authored Sep 18, 2024
    Configuration menu
    Copy the full SHA
    e7506cc View commit details
    Browse the repository at this point in the history

  3. backfilling improvements and fixes

    @aarshkshah1992
    aarshkshah1992 committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    efbf51f View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2024

  1. finish chain index validation and backfill tooling

    @aarshkshah1992
    aarshkshah1992 committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    c18998d View commit details
    Browse the repository at this point in the history

  2. Merge branch 'feat/msg-eth-tx-index' into feat/implement-index-valida… 

    …tion-api
    @aarshkshah1992
    aarshkshah1992 authored Sep 19, 2024
    Configuration menu
    Copy the full SHA
    f30585d View commit details
    Browse the repository at this point in the history

Commits on Sep 23, 2024

  1. user documentation for the

    @aarshkshah1992
    aarshkshah1992 committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    67f81a9 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'feat/implement-index-validation-api' of github.com:file… 

    …coin-project/lotus into feat/implement-index-validation-api
    @aarshkshah1992
    aarshkshah1992 committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    b5d4fd5 View commit details
    Browse the repository at this point in the history

  3. validate from epoch

    @aarshkshah1992
    aarshkshah1992 committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    a84b38e View commit details
    Browse the repository at this point in the history

Commits on Sep 24, 2024

  1. Apply suggestions from code review 

    Suggestions from Steve's read of the user doc.

Co-authored-by: Steve Loeppky <[email protected]>
    @aarshkshah1992 @BigLep
    aarshkshah1992 and BigLep authored Sep 24, 2024
    Configuration menu
    Copy the full SHA
    24527bd View commit details
    Browse the repository at this point in the history

  2. changes to user doc as per review

    @aarshkshah1992
    aarshkshah1992 committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    ec8da8a View commit details
    Browse the repository at this point in the history

Commits on Sep 25, 2024

  1. Apply suggestions from code review 

    Co-authored-by: Steve Loeppky <[email protected]>
    @aarshkshah1992 @BigLep
    aarshkshah1992 and BigLep authored Sep 25, 2024
    Configuration menu
    Copy the full SHA
    651e7b9 View commit details
    Browse the repository at this point in the history

  2. changes to user doc as per review

    @aarshkshah1992
    aarshkshah1992 committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    149b944 View commit details
    Browse the repository at this point in the history

  3. Apply suggestions from code review 

    Co-authored-by: Steve Loeppky <[email protected]>
    @aarshkshah1992 @BigLep
    aarshkshah1992 and BigLep authored Sep 25, 2024
    Configuration menu
    Copy the full SHA
    ee6a5a1 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'feat/implement-index-validation-api' of github.com:file… 

    …coin-project/lotus into feat/implement-index-validation-api
    @aarshkshah1992
    aarshkshah1992 committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    8825d9a View commit details
    Browse the repository at this point in the history

  5. changes as per review

    @aarshkshah1992
    aarshkshah1992 committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    c07ca44 View commit details
    Browse the repository at this point in the history

  6. feat: add event entries count in validation API (#12506) 

    * feat: add event entry count in validation API

* address comments
    @akaladarshi
    akaladarshi authored Sep 25, 2024
    Configuration menu
    Copy the full SHA
    4c94b28 View commit details
    Browse the repository at this point in the history

  7. use sqllite defaults (#12504)

    @aarshkshah1992
    aarshkshah1992 authored Sep 25, 2024
    Configuration menu
    Copy the full SHA
    8190b4d View commit details
    Browse the repository at this point in the history

Commits on Sep 30, 2024

  1. Apply suggestions from code review 

    Co-authored-by: Steve Loeppky <[email protected]>
    @aarshkshah1992 @BigLep
    aarshkshah1992 and BigLep authored Sep 30, 2024
    Configuration menu
    Copy the full SHA
    4e3aa88 View commit details
    Browse the repository at this point in the history

  2. write chain index to a different dir

    @aarshkshah1992
    aarshkshah1992 committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    9a13d59 View commit details
    Browse the repository at this point in the history

Commits on Oct 1, 2024

  1. Apply suggestions from code review 

    Co-authored-by: Steve Loeppky <[email protected]>
    @aarshkshah1992 @BigLep
    aarshkshah1992 and BigLep authored Oct 1, 2024
    Configuration menu
    Copy the full SHA
    bca9ed0 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/feat/msg-eth-tx-index' into feat… 

    …/implement-index-validation-api
    @aarshkshah1992
    aarshkshah1992 committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    9f84a27 View commit details
    Browse the repository at this point in the history

  3. fix conflicts

    @aarshkshah1992
    aarshkshah1992 committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    35ccc7d View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2024

  1. Merge remote-tracking branch 'origin/feat/msg-eth-tx-index' into feat… 

    …/implement-index-validation-api
    @aarshkshah1992
    aarshkshah1992 committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    fdf5c6e View commit details
    Browse the repository at this point in the history

  2. UX improvements to backfilling

    @aarshkshah1992
    aarshkshah1992 committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    0207fa2 View commit details
    Browse the repository at this point in the history

Commits on Oct 4, 2024

  1. feat: tests for the chain indexer (#12521) 

    * ddl tests

* tests for the chain indexer

* finish unit tests for chain indexer

* fix formatting
    @aarshkshah1992
    aarshkshah1992 authored Oct 4, 2024
    Configuration menu
    Copy the full SHA
    3c48644 View commit details
    Browse the repository at this point in the history

  2. cleanup reverted tipsets to avoid db bloat

    @aarshkshah1992
    aarshkshah1992 committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    5e6ae08 View commit details
    Browse the repository at this point in the history

  3. fix logging

    @aarshkshah1992
    aarshkshah1992 committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    9511c46 View commit details
    Browse the repository at this point in the history

  4. test for filter by address

    @aarshkshah1992
    aarshkshah1992 committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    2194dcb View commit details
    Browse the repository at this point in the history

Commits on Oct 7, 2024

  1. test gc cascade delete

    @aarshkshah1992
    aarshkshah1992 committed Oct 7, 2024
    Configuration menu
    Copy the full SHA
    62f59d3 View commit details
    Browse the repository at this point in the history

  2. fix db locked error during backfilling

    @aarshkshah1992
    aarshkshah1992 committed Oct 7, 2024
    Configuration menu
    Copy the full SHA
    688ab72 View commit details
    Browse the repository at this point in the history

  3. fix var name

    @aarshkshah1992
    aarshkshah1992 committed Oct 7, 2024
    Configuration menu
    Copy the full SHA
    9a64de7 View commit details
    Browse the repository at this point in the history

  4. increase db locked timeout

    @aarshkshah1992
    aarshkshah1992 committed Oct 7, 2024
    Configuration menu
    Copy the full SHA
    1d7af84 View commit details
    Browse the repository at this point in the history

  5. fix db locked issue

    @aarshkshah1992
    aarshkshah1992 committed Oct 7, 2024
    Configuration menu
    Copy the full SHA
    d4606e7 View commit details
    Browse the repository at this point in the history

  6. reduce db lock timeout

    @aarshkshah1992
    aarshkshah1992 committed Oct 7, 2024
    Configuration menu
    Copy the full SHA
    5f541b9 View commit details
    Browse the repository at this point in the history

Commits on Oct 8, 2024

  1. no lock in gc

    @aarshkshah1992
    aarshkshah1992 committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    c223802 View commit details
    Browse the repository at this point in the history

  2. reconcile does not need lock

    @aarshkshah1992
    aarshkshah1992 committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    fcbb3e5 View commit details
    Browse the repository at this point in the history

  3. improved error handling

    @aarshkshah1992
    aarshkshah1992 committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    47e421f View commit details
    Browse the repository at this point in the history

  4. Update chain-indexing-overview-for-rpc-providers.md 

    Doc updates based on @jennijuju feedack.
    @BigLep
    BigLep authored Oct 8, 2024
    Configuration menu
    Copy the full SHA
    33c1ca1 View commit details
    Browse the repository at this point in the history

  5. Update chain-indexing-overview-for-rpc-providers.MD 

    Fixes after reviewing 33c1ca1
    @BigLep
    BigLep authored Oct 8, 2024
    Configuration menu
    Copy the full SHA
    926852a View commit details
    Browse the repository at this point in the history

Commits on Oct 9, 2024

  1. better metrics for backfilling

    @aarshkshah1992
    aarshkshah1992 committed Oct 9, 2024
    Configuration menu
    Copy the full SHA
    23741cf View commit details
    Browse the repository at this point in the history

  2. Merge branch 'feat/implement-index-validation-api' of github.com:file… 

    …coin-project/lotus into feat/implement-index-validation-api
    @aarshkshah1992
    aarshkshah1992 committed Oct 9, 2024
    Configuration menu
    Copy the full SHA
    f7bc885 View commit details
    Browse the repository at this point in the history

  3. Update chain/index/chain-indexing-overview-for-rpc-providers.MD 

    Co-authored-by: Rod Vagg <[email protected]>
    @BigLep @rvagg
    BigLep and rvagg authored Oct 9, 2024
    Configuration menu
    Copy the full SHA
    5eb4be4 View commit details
    Browse the repository at this point in the history

  4. Update chain/index/chain-indexing-overview-for-rpc-providers.MD 

    Co-authored-by: Rod Vagg <[email protected]>
    @BigLep @rvagg
    BigLep and rvagg authored Oct 9, 2024
    Configuration menu
    Copy the full SHA
    bc88f8c View commit details
    Browse the repository at this point in the history

  5. Update chain/index/chain-indexing-overview-for-rpc-providers.MD 

    Co-authored-by: Rod Vagg <[email protected]>
    @BigLep @rvagg
    BigLep and rvagg authored Oct 9, 2024
    Configuration menu
    Copy the full SHA
    097b90d View commit details
    Browse the repository at this point in the history

  6. Update chain/index/chain-indexing-overview-for-rpc-providers.MD 

    Co-authored-by: Rod Vagg <[email protected]>
    @BigLep @rvagg
    BigLep and rvagg authored Oct 9, 2024
    Configuration menu
    Copy the full SHA
    5170f22 View commit details
    Browse the repository at this point in the history

  7. Update chain/index/chain-indexing-overview-for-rpc-providers.MD 

    Co-authored-by: Rod Vagg <[email protected]>
    @BigLep @rvagg
    BigLep and rvagg authored Oct 9, 2024
    Configuration menu
    Copy the full SHA
    feefed9 View commit details
    Browse the repository at this point in the history

  8. Update chain/index/chain-indexing-overview-for-rpc-providers.MD 

    Co-authored-by: Rod Vagg <[email protected]>
    @BigLep @rvagg
    BigLep and rvagg authored Oct 9, 2024
    Configuration menu
    Copy the full SHA
    2fb608f View commit details
    Browse the repository at this point in the history

  9. Update chain/index/chain-indexing-overview-for-rpc-providers.MD 

    Co-authored-by: Rod Vagg <[email protected]>
    @BigLep @rvagg
    BigLep and rvagg authored Oct 9, 2024
    Configuration menu
    Copy the full SHA
    637e6a2 View commit details
    Browse the repository at this point in the history

Commits on Oct 11, 2024

  1. Merge remote-tracking branch 'origin/feat/msg-eth-tx-index' into feat… 

    …/implement-index-validation-api
    @aarshkshah1992
    aarshkshah1992 committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    7a6233f View commit details
    Browse the repository at this point in the history

  2. tests for changes to event addressing

    @aarshkshah1992
    aarshkshah1992 committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    6badc36 View commit details
    Browse the repository at this point in the history

Commits on Oct 14, 2024

  1. Apply suggestions from code review 

    Co-authored-by: Rod Vagg <[email protected]>
    @aarshkshah1992 @rvagg
    aarshkshah1992 and rvagg authored Oct 14, 2024
    Configuration menu
    Copy the full SHA
    c0a6471 View commit details
    Browse the repository at this point in the history

  2. changes as per review -> round 1

    @aarshkshah1992
    aarshkshah1992 committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    a017e71 View commit details
    Browse the repository at this point in the history

  3. Apply suggestions from code review 

    Co-authored-by: Rod Vagg <[email protected]>
    @aarshkshah1992 @rvagg
    aarshkshah1992 and rvagg authored Oct 14, 2024
    Configuration menu
    Copy the full SHA
    1645f76 View commit details
    Browse the repository at this point in the history

  4. Apply suggestions from code review 

    Co-authored-by: Rod Vagg <[email protected]>
    @aarshkshah1992 @rvagg
    aarshkshah1992 and rvagg authored Oct 14, 2024
    Configuration menu
    Copy the full SHA
    00a4ea2 View commit details
    Browse the repository at this point in the history

  5. log tipset key cid

    @aarshkshah1992
    aarshkshah1992 committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    584b5c5 View commit details
    Browse the repository at this point in the history

  6. Apply suggestions from code review 

    Co-authored-by: Rod Vagg <[email protected]>
    @aarshkshah1992 @rvagg
    aarshkshah1992 and rvagg authored Oct 14, 2024
    Configuration menu
    Copy the full SHA
    c5329b1 View commit details
    Browse the repository at this point in the history

  7. Apply suggestions from code review 

    Co-authored-by: Rod Vagg <[email protected]>
    @aarshkshah1992 @rvagg
    aarshkshah1992 and rvagg authored Oct 14, 2024
    Configuration menu
    Copy the full SHA
    00612c8 View commit details
    Browse the repository at this point in the history

  8. fix docs

    @aarshkshah1992
    aarshkshah1992 committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    185ed0f View commit details
    Browse the repository at this point in the history

  9. Apply suggestions from code review 

    Co-authored-by: Rod Vagg <[email protected]>
    @aarshkshah1992 @rvagg
    aarshkshah1992 and rvagg authored Oct 14, 2024
    Configuration menu
    Copy the full SHA
    4f26511 View commit details
    Browse the repository at this point in the history

  10. fix tests

    @aarshkshah1992
    aarshkshah1992 committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    cafec26 View commit details
    Browse the repository at this point in the history

  11. Merge branch 'feat/implement-index-validation-api' of github.com:file… 

    …coin-project/lotus into feat/implement-index-validation-api
    @aarshkshah1992
    aarshkshah1992 committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    a3d3543 View commit details
    Browse the repository at this point in the history

  12. fix tests

    @aarshkshah1992
    aarshkshah1992 committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    8701f55 View commit details
    Browse the repository at this point in the history

  13. make jen

    @aarshkshah1992
    aarshkshah1992 committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    77a0b60 View commit details
    Browse the repository at this point in the history

  14. Merge remote-tracking branch 'origin/feat/msg-eth-tx-index' into feat… 

    …/implement-index-validation-api
    @aarshkshah1992
    aarshkshah1992 committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    8bdb105 View commit details
    Browse the repository at this point in the history

  15. fix conflicts

    @aarshkshah1992
    aarshkshah1992 committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    9cd86fa View commit details
    Browse the repository at this point in the history