Skip to content
This repository has been archived by the owner on Sep 22, 2023. It is now read-only.

Handle limit exceeding git queries #324

Closed
Chalarangelo opened this issue Apr 13, 2021 · 1 comment · Fixed by #345
Closed

Handle limit exceeding git queries #324

Chalarangelo opened this issue Apr 13, 2021 · 1 comment · Fixed by #345
Assignees
Labels
backend Related to backend code bug Something isn't working maintenance Maintenance task
Milestone

Comments

@Chalarangelo
Copy link
Owner

As soon as we add enough content, we will hit a soft limit, where we'll start getting weird errors such as:

/bin/sh: fork: Resource temporarily unavailable

This stems from the following code (src/blocks/parsers/text/index.js:30):

if (withMetadata) {
  promises.push(
    new Promise(rsl =>
      exec(
        `cd ${dirPath}; git log --diff-filter=A --pretty=format:%at -- ${fileName} | head -1`,
        (error, stdout) => rsl(stdout.toString().replace('\n', ''))
      )
    )
  );
  promises.push(
    new Promise(rsl =>
      exec(
        `cd ${dirPath}; git log -n 1 --pretty=format:%at -- ${fileName} | head -1`,
        (error, stdout) => rsl(stdout.toString().replace('\n', ''))
      )
    )
  );
}

We can put a bandaid on it for the time being, as we need the data, but we'll have to tackle this at some point. The upcoming changes will introduce a far more efficient version of this system, using a single git command:

if (withMetadata) {
  promises.push(
    new Promise(rsl =>
      exec(
        `cd ${dirPath}; git log --pretty=format:%at -- ${fileName}`,
        (error, stdout) => {
          const dates = stdout.toString().split('\n');
          rsl([dates[0], dates.slice(-1)]);
        }
      )
    )
  );
}

The gain from this is that we'll have to more or less double our content volume to hit this limit (which we are fast approaching with the current version - hit it locally when testing the git repo addition), but we still might have to address this with a more resilient solution.


A proposal is to manually add both dates in the snippet metadata. We can create a quick mod to update all existing content and speed up parsing by quite a lot using that. The benefit of this is that scheduling content is going to be significantly easier (no git trickery involved) and we'll have more fine-tuned control of snippet update dates (typos should not count for example). The obvious issue is that we'll have to manually update anything we edit, which can get a bit tiresome over time, so we might have to invent a utility to update those dates for us if we need to do that every once in a while.

@Chalarangelo Chalarangelo added bug Something isn't working backend Related to backend code maintenance Maintenance task Size: L/4 labels Apr 13, 2021
@Chalarangelo Chalarangelo self-assigned this Apr 13, 2021
Chalarangelo added a commit that referenced this issue Apr 14, 2021
Chalarangelo added a commit that referenced this issue Apr 14, 2021
@Chalarangelo Chalarangelo added this to the 2021/05 milestone Apr 26, 2021
@Chalarangelo Chalarangelo removed this from the 2021/05 milestone May 25, 2021
@Chalarangelo Chalarangelo added this to the 2021/06 milestone Jun 13, 2021
@Chalarangelo
Copy link
Owner Author

We're starting work on this issue as part of #345. We'll probably create and run a codemod for all repos, then update guidelines etc before testing out. Individual branches will be merged alongside this repo's PR.

Chalarangelo added a commit that referenced this issue Jun 13, 2021
Chalarangelo added a commit that referenced this issue Jun 17, 2021
Resolves #344
Resolves #337
Resolves #339
Resolves #331
Resolves #330
Resolves #251
Resolves #343
Resolves #346
Resolves #324
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
backend Related to backend code bug Something isn't working maintenance Maintenance task
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant