🐛 fix: improve delete orphan chunks when delete files

[arvinxx]

💻 变更类型 | Change Type

• ✨ feat
• 🐛 fix
• ♻️ refactor
• 💄 style
• 👷 build
• ⚡️ perf
• 📝 docs
• 🔨 chore

🔀 变更说明 | Description of Change

📝 补充信息 | Additional Information

在 LobeChat Cloud 实际生产中遇到一个奇怪的 bug，然后有两个用户提来反馈：

具体抛错如下：

RangeError: Maximum call stack size exceeded
    at /var/task/.next/server/chunks/59261.js:22:22558
    ... 8 lines matching cause stack trace ...
    at Object.startActiveSpan (/var/task/.next/server/chunks/59261.js:22:28365) {
  code: 'INTERNAL_SERVER_ERROR',
  name: 'TRPCError',
  [cause]: RangeError: Maximum call stack size exceeded
      at /var/task/.next/server/chunks/59261.js:22:22558
      at m.buildQueryFromSourceParams (/var/task/.next/server/chunks/59261.js:22:22661)
      at /var/task/.next/server/chunks/59261.js:22:23020
      at Array.map (<anonymous>)
      at m.buildQueryFromSourceParams (/var/task/.next/server/chunks/59261.js:22:22664)
      at /var/task/.next/server/chunks/59261.js:22:23020
      at Array.map (<anonymous>)
      at m.buildQueryFromSourceParams (/var/task/.next/server/chunks/59261.js:22:22664)
      at /var/task/.next/server/chunks/59261.js:22:22086
      at Object.startActiveSpan (/var/task/.next/server/chunks/59261.js:22:28365)
}

首先在 src 下搜了下不存在 buildQueryFromSourceParams 这个调用，因此抛错处不是源码部分。然后拿这个搜了下发现是 Drizzle 的调用实现，也有一个一模一样的 issue:

• [BUG]: Callstack exceeded for large inserts drizzle-team/drizzle-orm#1702
• [BUG]: Stack overflow when trying to insert many values at once drizzle-team/drizzle-orm#2628

找抛错的位置找到了这一部分：

deleteOrphanChunks = async () => {
    const orphanedChunks = await serverDB
      .select({ chunkId: chunks.id })
      .from(chunks)
      .leftJoin(fileChunks, eq(chunks.id, fileChunks.chunkId))
      .where(isNull(fileChunks.fileId));
    const ids = orphanedChunks.map((chunk) => chunk.chunkId);
    if (ids.length === 0) return;

    await serverDB.delete(chunks).where(inArray(chunks.id, ids));
  };

它的问题就出在如果 chunkIds 的总数过多，那么 inArray(chunks.id, chunkIds) 这样的写法就会出现堆栈溢出的情况。

而在 Neon 控制台写了段 SQL 查询，发现 orphanedChunks 居然有 161k 之多… 这才导致线上删文件时无法正常删除。

因此尝试将 inArrary 的实现优化为平行分批删除，期望优化该效果能解决过多的问题

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated (UTC)
lobe-chat-preview	✅ Ready (Inspect)	Visit Preview	Sep 27, 2024 5:20pm

[lobehubbot]

👍 @arvinxx

Thank you for raising your pull request and contributing to our Community
Please make sure you have followed our contributing guidelines. We will review it as soon as possible.
If you encounter any problems, please feel free to connect with us.
非常感谢您提出拉取请求并为我们的社区做出贡献，请确保您已经遵循了我们的贡献指南，我们会尽快审查它。
如果您遇到任何问题，请随时与我们联系。

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 92.51%. Comparing base (b0c3abc) to head (81626cf).
Report is 3 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #4179   +/-   ##
=======================================
  Coverage   92.50%   92.51%           
=======================================
  Files         484      484           
  Lines       34748    34757    +9     
  Branches     2143     2145    +2     
=======================================
+ Hits        32145    32154    +9     
  Misses       2603     2603           

Flag	Coverage Δ
app	92.51% <100.00%> (+<0.01%)	⬆️
server	97.37% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[lobehubbot]

❤️ Great PR @arvinxx ❤️

The growth of project is inseparable from user feedback and contribution, thanks for your contribution! If you are interesting with the lobehub developer community, please join our discord and then dm @arvinxx or @canisminor1990. They will invite you to our private developer channel. We are talking about the lobe-chat development or sharing ai newsletter around the world.
项目的成长离不开用户反馈和贡献，感谢您的贡献! 如果您对 LobeHub 开发者社区感兴趣，请加入我们的 discord，然后私信 @arvinxx 或 @canisminor1990。他们会邀请您加入我们的私密开发者频道。我们将会讨论关于 Lobe Chat 的开发，分享和讨论全球范围内的 AI 消息。

[lobehubbot]

🎉 This PR is included in version 1.20.3 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀