Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature Request]: Reparse a file shall reuse existing chunks if possible #3793

Closed
1 task done
yuzhichang opened this issue Dec 2, 2024 · 0 comments · Fixed by #3983
Closed
1 task done

[Feature Request]: Reparse a file shall reuse existing chunks if possible #3793

yuzhichang opened this issue Dec 2, 2024 · 0 comments · Fixed by #3983
Assignees
@yuzhichang
Labels
Feature

Comments

@yuzhichang
Copy link
Member

Is there an existing issue for the same feature request?

  • I have checked the existing issues.

Is your feature request related to a problem?

#2851 [Bug]: Failure of raptor cost too much?
Parsing a big file into many chunks, and a few failed for some reason. Reparsing discards all existing chunks of this file and parse again. This waste money of the previous one.

Describe the feature you'd like

Reparse a file shall reuse existing chunks if possible

Describe implementation you've considered

If two parsing share the same config (embedding model, strategy, page range and etc.), the existing chunk can be reused.

Documentation, adoption, use case

No response

Additional information

No response
This was referenced Dec 2, 2024
Closed
Closed
@yuzhichang yuzhichang self-assigned this Dec 11, 2024
yuzhichang added a commit to yuzhichang/ragflow that referenced this issue Dec 11, 2024
@yuzhichang
Try to reuse existing chunks. Close infiniflow#3793
a0ed06d
@yuzhichang yuzhichang mentioned this issue Dec 11, 2024
Merged
1 task
yuzhichang added a commit to yuzhichang/ragflow that referenced this issue Dec 11, 2024
@yuzhichang
Try to reuse existing chunks. Close infiniflow#3793
f6d2246
yuzhichang added a commit to yuzhichang/ragflow that referenced this issue Dec 11, 2024
@yuzhichang
Try to reuse existing chunks. Close infiniflow#3793
829c389
yuzhichang added a commit to yuzhichang/ragflow that referenced this issue Dec 11, 2024
@yuzhichang
Try to reuse existing chunks. Close infiniflow#3793
5aeef10
yuzhichang added a commit to yuzhichang/ragflow that referenced this issue Dec 12, 2024
@yuzhichang
Try to reuse existing chunks. Close infiniflow#3793
22e82ba
yuzhichang added a commit to yuzhichang/ragflow that referenced this issue Dec 12, 2024
@yuzhichang
Try to reuse existing chunks. Close infiniflow#3793
b94bd8d
yuzhichang added a commit that referenced this issue Dec 12, 2024
@yuzhichang
Try to reuse existing chunks (#3983)
301f958 
### What problem does this PR solve?

Try to reuse existing chunks. Close #3793
### Type of change

- [x] New Feature (non-breaking change which adds functionality)
cike8899 added a commit to cike8899/ragflow that referenced this issue Dec 13, 2024
@cike8899
Feat: Reparse a file shall reuse existing chunks if possible infinifl…
a649963 
…ow#3793
@cike8899 cike8899 mentioned this issue Dec 13, 2024
Merged
1 task
KevinHuSh pushed a commit that referenced this issue Dec 13, 2024
@cike8899
Feat: Reparse a file shall reuse existing chunks if possible #3793 (#…
9363567 
…4021)

### What problem does this PR solve?

Feat: Reparse a file shall reuse existing chunks if possible #3793

### Type of change


- [x] New Feature (non-breaking change which adds functionality)
isthaison pushed a commit to isthaison/ragflow that referenced this issue Dec 13, 2024
@yuzhichang @isthaison
Try to reuse existing chunks (infiniflow#3983)
9c2a484 
### What problem does this PR solve?

Try to reuse existing chunks. Close infiniflow#3793
### Type of change

- [x] New Feature (non-breaking change which adds functionality)
isthaison pushed a commit to isthaison/ragflow that referenced this issue Dec 13, 2024
@yuzhichang @isthaison
Try to reuse existing chunks (infiniflow#3983)
6953019 
### What problem does this PR solve?

Try to reuse existing chunks. Close infiniflow#3793
### Type of change

- [x] New Feature (non-breaking change which adds functionality)
isthaison added a commit to isthaison/ragflow that referenced this issue Dec 13, 2024
@isthaison
Squashed commit of the following:
0f0eb35 
commit 9b2ef62
Author: Zhichang Yu <[email protected]>
Date:   Fri Dec 13 17:31:15 2024 +0800

    Fix xinfo_groups returns unexpected result (infiniflow#4026)

    ### What problem does this PR solve?

    Fix xinfo_groups returns unexpected result. Close infiniflow#3545

    ### Type of change

    - [x] Bug Fix (non-breaking change which fixes an issue)

commit 86507af
Author: Zhichang Yu <[email protected]>
Date:   Fri Dec 13 17:15:08 2024 +0800

    Set task progress on exception (infiniflow#4025)

    ### What problem does this PR solve?

    Set task progress on exception

    ### Type of change

    - [x] Bug Fix (non-breaking change which fixes an issue)

commit 9363567
Author: balibabu <[email protected]>
Date:   Fri Dec 13 16:55:13 2024 +0800

    Feat: Reparse a file shall reuse existing chunks if possible infiniflow#3793 (infiniflow#4021)

    ### What problem does this PR solve?

    Feat: Reparse a file shall reuse existing chunks if possible infiniflow#3793

    ### Type of change

    - [x] New Feature (non-breaking change which adds functionality)

commit 1defe0b
Author: balibabu <[email protected]>
Date:   Fri Dec 13 14:43:24 2024 +0800

    Feat: Supports to debug single component in Agent. infiniflow#3993 (infiniflow#4007)

    ### What problem does this PR solve?

    Feat: Supports to debug single component in Agent. infiniflow#3993
    Fix: The github button on the login page is displayed incorrectly  infiniflow#4002

    ### Type of change

    - [x] New Feature (non-breaking change which adds functionality)

commit 0bca46a
Author: Zhichang Yu <[email protected]>
Date:   Fri Dec 13 13:43:56 2024 +0800

    Migrate infinity at startup (infiniflow#3858)

    ### What problem does this PR solve?

    Migrate infinity at startup

    infiniflow#3809
    infiniflow/infinity#2321

    ### Type of change

    - [x] New Feature (non-breaking change which adds functionality)

commit 1ecb687
Author: liuhua <[email protected]>
Date:   Fri Dec 13 10:25:52 2024 +0800

    Fix bugs in agent api and update api document (infiniflow#3996)

    ### What problem does this PR solve?

    Fix bugs in agent api and update api document

    ### Type of change

    - [x] Bug Fix (non-breaking change which fixes an issue)
    - [x] New Feature (non-breaking change which adds functionality)

    ---------

    Co-authored-by: liuhua <[email protected]>

commit 68d46b2
Author: Luo Pan <[email protected]>
Date:   Fri Dec 13 08:50:58 2024 +0800

    Fix bug in hierarchical_merge function (infiniflow#4006)

    ### What problem does this PR solve?

    Fix hierarchical_merge function. From idx vs. actual value to actual
    value vs. actual value.
    Related issue infiniflow#4003

    ### Type of change

    - [x] Bug Fix (non-breaking change which fixes an issue)

    Co-authored-by: luopan <[email protected]>

commit 7559bbd
Author: Kevin Hu <[email protected]>
Date:   Fri Dec 13 08:50:32 2024 +0800

    Component debugging funcionality. (infiniflow#4012)

    ### What problem does this PR solve?

    infiniflow#3993
    ### Type of change

    - [x] New Feature (non-breaking change which adds functionality)

commit 275b5d1
Author: Jin Hai <[email protected]>
Date:   Thu Dec 12 20:34:46 2024 +0800

    Fix json file parse (infiniflow#4004)

    ### What problem does this PR solve?

    Fix json file parsing

    ### Type of change

    - [x] Bug Fix (non-breaking change which fixes an issue)

    ---------

    Signed-off-by: jinhai <[email protected]>

commit 9ae81b4
Author: writinwaters <[email protected]>
Date:   Thu Dec 12 19:46:53 2024 +0800

    Updated UI (infiniflow#4011)

    ### What problem does this PR solve?

    ### Type of change

    - [x] Documentation Update

commit d6c74ff
Author: so95 <[email protected]>
Date:   Thu Dec 12 18:26:44 2024 +0700

    Add mssql support (infiniflow#3985)

    some thing
    -  execsql  add connection mssql
    - fix bug duckduckgo-search rate limit
    - update typo vi res

    ---------

    Co-authored-by: Kevin Hu <[email protected]>

commit e8d7410
Author: Roccat <[email protected]>
Date:   Thu Dec 12 19:00:34 2024 +0800

    Fix: Completion AttributeError: 'list' object has no attribute 'get' (infiniflow#3999)

    ### What problem does this PR solve?

    ### Type of change

    - [x] Bug Fix (non-breaking change which fixes an issue)

    Co-authored-by: [email protected] <[email protected]>

commit c8b1a56
Author: Zhichang Yu <[email protected]>
Date:   Thu Dec 12 17:47:39 2024 +0800

    Replaced md5 with xxhash64 for chunk id (infiniflow#4009)

    ### What problem does this PR solve?

    Replaced md5 with xxhash64 for chunk id

    ### Type of change

    - [x] Refactoring

commit 301f958
Author: Zhichang Yu <[email protected]>
Date:   Thu Dec 12 16:38:03 2024 +0800

    Try to reuse existing chunks (infiniflow#3983)

    ### What problem does this PR solve?

    Try to reuse existing chunks. Close infiniflow#3793
    ### Type of change

    - [x] New Feature (non-breaking change which adds functionality)

commit 835fd7a
Author: writinwaters <[email protected]>
Date:   Thu Dec 12 11:45:59 2024 +0800

    Updated RAGFlow edition descriptions (infiniflow#4001)

    ### What problem does this PR solve?

    ### Type of change

    - [x] Documentation Update

commit bb8f97c
Author: writinwaters <[email protected]>
Date:   Thu Dec 12 09:57:52 2024 +0800

    UI updates + RAGFlow image description (infiniflow#3995)

    ### What problem does this PR solve?

    ### Type of change

    - [x] Documentation Update

commit 6d19294
Author: Kevin Hu <[email protected]>
Date:   Wed Dec 11 19:23:59 2024 +0800

    Support debug components. (infiniflow#3994)

    ### What problem does this PR solve?

    infiniflow#3993

    ### Type of change

    - [x] Bug Fix (non-breaking change which fixes an issue)

commit f61c276
Author: Jin Hai <[email protected]>
Date:   Wed Dec 11 18:39:09 2024 +0800

    Update comment (infiniflow#3981)

    ### What problem does this PR solve?

    _Briefly describe what this PR aims to solve. Include background context
    that will help reviewers understand the purpose of the PR._

    ### Type of change

    - [x] Documentation Update

    ---------

    Signed-off-by: Jin Hai <[email protected]>
    Co-authored-by: writinwaters <[email protected]>

commit 409acf0
Author: balibabu <[email protected]>
Date:   Wed Dec 11 16:29:17 2024 +0800

    Fix:  Fixed the issue where two consecutive indexes were displayed incorrectly infiniflow#3839 (infiniflow#3988)

    ### What problem does this PR solve?

    Fix: Fixed the issue where two consecutive indexes were displayed
    incorrectly infiniflow#3839

    ### Type of change

    - [x] Bug Fix (non-breaking change which fixes an issue)

commit 74c6b21
Author: liuhua <[email protected]>
Date:   Wed Dec 11 12:38:57 2024 +0800

    Update api documents (infiniflow#3979)

    ### What problem does this PR solve?

    Update api documents

    ### Type of change

    - [x] Documentation Update

    Co-authored-by: liuhua <[email protected]>

commit beeacd3
Author: Kevin Hu <[email protected]>
Date:   Wed Dec 11 11:44:59 2024 +0800

    Fix exec sql exception issue. (infiniflow#3982)

    ### What problem does this PR solve?
    infiniflow#3978

    ### Type of change

    - [x] Bug Fix (non-breaking change which fixes an issue)

commit 95259af
Author: so95 <[email protected]>
Date:   Wed Dec 11 10:12:57 2024 +0700

    update typo vietnamese (infiniflow#3973)

    update typo vietnamese

    ### Type of change
    - [x] Refactoring

    ---------

    Co-authored-by: Yingfeng <[email protected]>
    Co-authored-by: Jin Hai <[email protected]>
    Co-authored-by: bill <[email protected]>

commit 8554550
Author: Scott Davidson <[email protected]>
Date:   Wed Dec 11 03:10:33 2024 +0000

    Disable SQL DB binlog in Helm chart (infiniflow#3976)

    ### What problem does this PR solve?

    The initial Helm chart implementation added in infiniflow#3815 suffers from an
    issue where the 5GB data volume for the SQL DB is filled up with
    [binlog](https://dev.mysql.com/doc/refman/8.4/en/binary-log.html) files
    after just a few days. Since the app uses a non-replicated SQL DB config
    I think it makes sense to disable the binlog in the SQL DB container.
    This is achieved by simply adding the required argument to the container
    startup command.

    ### Type of change

    - [X] Bug Fix (non-breaking change which fixes an issue)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yuzhichang yuzhichang

Labels
Feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Try to reuse existing chunks yuzhichang/ragflow
1 participant
@yuzhichang