-
-
Notifications
You must be signed in to change notification settings - Fork 432
premature proc.kill() on sapper export
child process - some .json
files not getting built as a result of race condition
#893
Comments
possibly related? #869 IO backpressure from @khrome83 a reader also pointed out that all posts from "Netlify Year One" on down on https://www.swyx.io/writing/ are the ones missing the .json. a huge clue! though I have no idea what to do with this information. |
got it. overeager proc.kill: Line 238 in 910d28e
commenting out this one line fixes it. how can we avoid killing that child process while it still has work left to do? EDIT: proof: i injected a line where yootils.queue.close is called, which is supposed to block execution so that we dont proceed to the deadly proc.kill.. but it doesn't do its job. look at all the json files being written thereafter > Crawling http://localhost:3000/
2.87 kB index.html
4.42 kB data/writing___ssg___writing_index.json
2.71 kB about/index.html
9.91 kB data/writing___ssg___svelte-static.json
3.56 kB data/talks___ssg___talks_index.json
13.7 kB writing/index.html
25.2 kB writing/svelte-static/index.html
11.7 kB talks/index.html
23.8 kB data/writing___ssg___netlify-nextjs-ssg.json
9.27 kB writing/keyboard-first/index.html
5.13 kB writing/compilers-copout/index.html
5.13 kB writing/gatsby-remark-essential-plugins/index.html
7.11 kB data/writing___ssg___graphql-variables-why.json
19.3 kB writing/graphql-variables-why/index.html
34.2 kB writing/netlify-worlds-greatest-demo-v0/index.html
59.1 kB writing/netlify-nextjs-ssg/index.html
11.5 kB writing/book-poll-2019/index.html
9.98 kB writing/netlify-year-one-360-review/index.html
12.4 kB writing/typescript-generics/index.html
4.6 kB writing/react-not-bad/index.html
17.4 kB writing/svelte-metaphysics/index.html
59.6 kB writing/netlify-jamstack-indiehackers/index.html
42.8 kB writing/react-native-web-singularity/index.html
5.24 kB writing/netlify-env-vars/index.html
63.5 kB data/writing___ssg___netlify-gatsby-jamstack-apps-authenticated-serverless.json
2.59 kB data/writing___ssg___keyboard-first.json
26.4 kB writing/10000-loose-fans/index.html
6.36 kB writing/index/index.html
155 kB writing/netlify-gatsby-jamstack-apps-authenticated-serverless/index.html
51.2 kB writing/netlify-python-ssg/index.html
44.3 kB writing/slowmad-qna/index.html
12.3 kB data/writing___ssg___netlify-worlds-greatest-demo-v0.json
844 B data/writing___ssg___compilers-copout.json
919 B data/writing___ssg___gatsby-remark-essential-plugins.json
105 kB writing/types-already/index.html
56.8 kB writing/netlify-redirects-i18n/index.html
4.87 kB writing/types-or-tests/index.html
69.4 kB writing/netlify-year-one/index.html
28.4 kB writing/betwixt-reason-and-results/index.html
23.7 kB writing/learning-gears/index.html
25.6 kB writing/post-graduation-mindmap/index.html
5.17 kB writing/star-apps/index.html
26.1 kB data/writing___ssg___netlify-jamstack-indiehackers.json
3.46 kB data/writing___ssg___book-poll-2019.json
2.96 kB data/writing___ssg___netlify-year-one-360-review.json
51.4 kB writing/netlify-dev-cors/index.html
145 kB writing/naked-promises/index.html
15.6 kB writing/cuttlefish/index.html
14.3 kB writing/h1b1/index.html
13.3 kB writing/specialize-in-the-new/index.html
12.7 kB writing/clone-open-source-apps/index.html
11.2 kB writing/know-your-tools/index.html
17.7 kB data/writing___ssg___react-native-web-singularity.json
69.5 kB writing/how-to-name-things/index.html
3.65 kB data/writing___ssg___typescript-generics.json
58.3 kB writing/rise-of-immer/index.html
17.9 kB writing/learn-in-public/index.html
6.61 kB talks/ducktapes-podcast/index.html
7.25 kB talks/reactroundup-podcast/index.html
6.32 kB talks/programming-typescript/index.html
6.6 kB talks/kentcdodds-podcast/index.html
5.79 kB talks/jamstack-victory/index.html
26.4 kB talks/learn-in-public/index.html
43.8 kB data/writing___ssg___types-already.json
8.41 kB talks/netlify-functions-lunchnlearn/index.html
6.09 kB talks/learn-in-public-singapore/index.html
6.47 kB data/writing___ssg___svelte-metaphysics.json
517 B data/writing___ssg___react-not-bad.json
188 kB writing/getting-closure-on-hooks/index.html
25.8 kB talks/jamstack-jumpstart/index.html
5.69 kB talks/serverless-graphql-netlify/index.html
30.2 kB talks/intent-based-cli/index.html
16.3 kB talks/serverless-graphql/index.html
19.7 kB talks/freecodecamp-podcast/index.html
21.3 kB data/writing___ssg___netlify-python-ssg.json
6.04 kB talks/jamstack-victory-jsla/index.html
6.32 kB talks/react-hooks-lightning/index.html
5.06 kB talks/react-podcast/index.html
201 kB writing/gatsby-static-dynamic/index.html
6.56 kB talks/react-typescript/index.html
7.9 kB talks/reacts-new-defaults/index.html
14.2 kB talks/contributing-to-open-source/index.html
155 kB talks/intro-to-typescript/index.html
25.3 kB talks/babel-macros/index.html
23.3 kB data/writing___ssg___netlify-redirects-i18n.json
8.61 kB talks/babel-graphql/index.html
5.24 kB talks/infinite-building/index.html
5.28 kB talks/never-bundle-react-again/index.html
8.13 kB talks/creating-cra/index.html
5.09 kB talks/react-suspense/index.html
19.7 kB data/writing___ssg___slowmad-qna.json
4.45 kB talks/fullstack-heaps/index.html
5.12 kB talks/contributing-to-react/index.html
10.8 kB data/writing___ssg___10000-loose-fans.json
101 kB talks/immutability-is-changing/index.html
55.9 kB talks/react-not-reactive/index.html
5.3 kB talks/fullstack-crossbones/index.html
5.29 kB talks/fullstack-react-trip-planner/index.html
215 kB talks/react-hooks/index.html
58.9 kB data/writing___ssg___naked-promises.json
848 B data/writing___ssg___netlify-env-vars.json
31 kB data/writing___ssg___netlify-year-one.json
1.46 kB data/writing___ssg___index.json
721 B data/writing___ssg___types-or-tests.json
11.9 kB data/writing___ssg___betwixt-reason-and-results.json
2.63 kB service-worker-index.html
swyx: close called { pending: 0 }
> Finished in 8.4s. Type npx serve __sapper__/export to run the app.
20.8 kB data/writing___ssg___netlify-dev-cors.json
29.4 kB data/writing___ssg___how-to-name-things.json
9.37 kB data/writing___ssg___learning-gears.json
76.8 kB data/writing___ssg___getting-closure-on-hooks.json
8.68 kB data/writing___ssg___post-graduation-mindmap.json
810 B data/writing___ssg___star-apps.json
5.64 kB data/writing___ssg___cuttlefish.json
81.9 kB data/writing___ssg___gatsby-static-dynamic.json
23.8 kB data/writing___ssg___rise-of-immer.json
5.1 kB data/writing___ssg___h1b1.json
4.84 kB data/writing___ssg___specialize-in-the-new.json
4.44 kB data/writing___ssg___clone-open-source-apps.json
3.83 kB data/writing___ssg___know-your-tools.json
6.79 kB data/writing___ssg___learn-in-public.json
62.2 kB data/talks___ssg___intro-to-typescript.json
1.03 kB data/talks___ssg___ducktapes-podcast.json
1.58 kB data/talks___ssg___reactroundup-podcast.json
677 B data/talks___ssg___programming-typescript.json
1.2 kB data/talks___ssg___kentcdodds-podcast.json
9.15 kB data/talks___ssg___learn-in-public.json
865 B data/talks___ssg___jamstack-victory.json
1.57 kB data/talks___ssg___netlify-functions-lunchnlearn.json
705 B data/talks___ssg___learn-in-public-singapore.json
86.6 kB data/talks___ssg___react-hooks.json
9.88 kB data/talks___ssg___jamstack-jumpstart.json
10.5 kB data/talks___ssg___intent-based-cli.json
890 B data/talks___ssg___serverless-graphql-netlify.json
6.57 kB data/talks___ssg___freecodecamp-podcast.json
4.92 kB data/talks___ssg___serverless-graphql.json
985 B data/talks___ssg___react-hooks-lightning.json
971 B data/talks___ssg___jamstack-victory-jsla.json
575 B data/talks___ssg___react-podcast.json
8.42 kB data/talks___ssg___babel-macros.json
39.3 kB data/talks___ssg___immutability-is-changing.json
1.01 kB data/talks___ssg___react-typescript.json
1.61 kB data/talks___ssg___reacts-new-defaults.json
4.23 kB data/talks___ssg___contributing-to-open-source.json
1.78 kB data/talks___ssg___babel-graphql.json
23.3 kB data/talks___ssg___react-not-reactive.json
625 B data/talks___ssg___infinite-building.json
1.24 kB data/talks___ssg___creating-cra.json
451 B data/talks___ssg___never-bundle-react-again.json
397 B data/talks___ssg___react-suspense.json
475 B data/talks___ssg___contributing-to-react.json
339 B data/talks___ssg___fullstack-heaps.json
510 B data/talks___ssg___fullstack-crossbones.json
461 B data/talks___ssg___fullstack-react-trip-planner.json |
sapper export
child process - some .json
files not getting built as a resultsapper export
child process - some .json
files not getting built as a result
You’re welcome! |
Ok I am officially stumped. i know -why- this is happening: yootils.queue is made to deal with async queues of work... except:
but i dont know what the solution is here. hoping @Rich-Harris see this and has an idea. |
A definite improvement we can make, that doesn't solve this but helps, is to make the I/O non-blocking. rewriting this writeFileSync code to promisified async helps more data to be output, but we are still killing the child process prematurely. |
@sw-yx I have a disappearing json file on Netlify, and no where near as many file exports as thou... |
repro? |
i thought that merging #869 would fix this issue, but after checking futher it doesn't. :( still unsolved. |
sapper export
child process - some .json
files not getting built as a resultsapper export
child process - some .json
files not getting built as a result of race condition
update: i think i have a janky fix... here's what needs to happen:
To fix, either spin out the file writing to yet another child process (wasteful) or... // swyx: solve race condition where saving in child process is prematurely stopped by proc.kill
// https://github.com/sveltejs/sapper/issues/893#issuecomment-538811020
let latestUrl = null;
proc.on('message', message => {
if (!message.__sapper__ || message.event !== 'file') return;
latestUrl = message.url;
const saveresult = save(message.url, message.status, message.type, message.body);
if (saveresult) saveresult.then(() => checkWritingDone(message.url));
});
// swyx: introduced to solve race condition
// https://github.com/sveltejs/sapper/issues/893#issuecomment-538811020
function checkWritingDone(currentUrl) {
if (latestUrl != currentUrl || queue.getLength() > 0) ; else {
proc.kill();
}
} and comment out the bottom proc.kill. i'm implementing this in my sapper fork. let me know if you want this in the main repo. |
just swimming by to confirm that @nealalpert has fixed this once and for all by including the i really think this belongs in core sapper. others will run into what we ran into. |
Hitting the same problem. Any chance of getting #869 merged soon? |
yes, he updated it after i reported my issues. |
Fixed in 0.27.14, with the long-awaited merge of 0.27.14. |
This issue still seems to be unsolved with Build output with default setup (everything gets exported): Testing with a delay of 5 seconds on the blog data: Example code: https://github.com/bummzack/sapper-wtf/commit/b0d9f8bd6ff6f537b91933e83adfd7868fcc8a82 Running |
Nevermind, increasing the timeout via |
i've spent so much time trying to figure out why sapper export often misses exporting the [slug].json for certain pages. in b0b3b83 i assumed the issue was (somehow) using hyphens in slugs so switched to using underscores but this started happening again. i found the followng issues which all are "closed" or "merged" but this is _definitely_ still a bug. ended up looking at options for the sapper export command and saw i can turn off concurrent requests, which seems to have fixed the issue. - sveltejs/sapper#1078 - sveltejs/sapper#893 - sveltejs/sapper#869 hoping to submit a repro of the issue to sapper soon
Currently in I commented out Another log showing 2 files missed: It seems like how things work now there's no way to know we've waited long enough for all |
I can confirm that this bug still exists... So everybody who still has this problem should try
|
@ingomc i'm glad that comment on my commit came in handy! yeah, my first step for an exported sapper project is now to add |
The bug still exists on the latest version(0.29.1), the issue should be open again.
|
@darkyzhou we're in the process of deprecating Sapper in favour of Svelte Kit. We don't think this issue will exist on Svelte Kit since it's operation is entirely different to the way Sapper works in this respect. The |
Same here, using --concurrent 1 and high timeout helps but doesn't solve. After some testing it seems the issue is related to file size and happen for HTML files too. |
Describe the bug
as above, i'm seeing dropped messages in
sapper export
- some.json
files not getting built as a result. I don't see a pattern yet among what gets dropped, but it is somewhat consistent (i.e. i run it several times and the same .json files are missing) EDIT: I found the cause, see below, but still dont know what the solution should beLogs
I placed tracker logs on outbound messages in the server middleware and I can see the correct outbound message:
swyx sendmessage: /data/writing___ssg___rise-of-immer.json
I then also place tracker logs on received messages on the
sapper export
cli. I expect to see a corresponding inbound messageswyx trackmessage: /data/writing___ssg___rise-of-immer.json
. However, i don't.As you can see the message is always sent, but not always received.
To Reproduce
repro is going to be difficult for this one since its almost certainly a scale problem. my work is here: https://github.com/sw-yx/swyxdotio/tree/moveToSapper and you will note the logs above what the result of the code execution is.
Expected behavior
as explained above in the logs - messages that are sent ought to be received.
Information about your Sapper Installation:
Severity
very severe. can't trust sapper export if it doesnt reliably work.
The text was updated successfully, but these errors were encountered: