Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[improvement] improvement for light weight schema change #10860

Merged
merged 10 commits into from
Jul 18, 2022

Conversation

yiguolei
Copy link
Contributor

Proposed changes

  1. not use schema as lru cache key any more.
  2. load segment just use the rowset's original schema not the current read schema.
  3. generate column reader and column iterator using the original schema, using the read schema if it is a new column.
  4. using column unique id as key instead of column ordinals.

Problem Summary:

Describe the overview of changes.

Checklist(Required)

  1. Does it affect the original behavior: (Yes/No/I Don't know)
  2. Has unit tests been added: (Yes/No/No Need)
  3. Has document been added or modified: (Yes/No/No Need)
  4. Does it need to update dependencies: (Yes/No)
  5. Are there any changes that cannot be rolled back: (Yes/No)

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@Lchangliang
Copy link
Contributor

LGTM

@yiguolei yiguolei force-pushed the dynamic_schema branch 3 times, most recently from 6a77cdf to 7d0f17e Compare July 18, 2022 00:56
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman added kind/improvement area/schema-change Issues or PRs related to schema change labels Jul 18, 2022
@yiguolei yiguolei merged commit a2ed4b5 into apache:master Jul 18, 2022
miswujian pushed a commit to miswujian/doris that referenced this pull request Jul 28, 2022
* improvement for dynamic schema
not use schema as lru cache key any more.
load segment just use the rowset's original schema not the current read schema.
generate column reader and column iterator using the original schema, using the read schema if it is a new column.
using column unique id as key instead of column ordinals.
Co-authored-by: yiguolei <[email protected]>
whutpencil pushed a commit to whutpencil/incubator-doris that referenced this pull request Jul 29, 2022
* improvement for dynamic schema
not use schema as lru cache key any more.
load segment just use the rowset's original schema not the current read schema.
generate column reader and column iterator using the original schema, using the read schema if it is a new column.
using column unique id as key instead of column ordinals.
Co-authored-by: yiguolei <[email protected]>
eldenmoon pushed a commit to eldenmoon/incubator-doris that referenced this pull request Aug 1, 2022
* improvement for dynamic schema
not use schema as lru cache key any more.
load segment just use the rowset's original schema not the current read schema.
generate column reader and column iterator using the original schema, using the read schema if it is a new column.
using column unique id as key instead of column ordinals.
Co-authored-by: yiguolei <[email protected]>
@yiguolei
Copy link
Contributor Author

yiguolei commented Aug 3, 2022

part of #10136

@yiguolei yiguolei deleted the dynamic_schema branch March 30, 2023 10:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/schema-change Issues or PRs related to schema change kind/improvement
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants