Skip to content
This repository has been archived by the owner on Jan 11, 2023. It is now read-only.

premature proc.kill() on sapper export child process - some .json files not getting built as a result of race condition #893

Closed
swyxio opened this issue Sep 14, 2019 · 23 comments

Comments

@swyxio
Copy link

swyxio commented Sep 14, 2019

Describe the bug

as above, i'm seeing dropped messages in sapper export - some .json files not getting built as a result. I don't see a pattern yet among what gets dropped, but it is somewhat consistent (i.e. i run it several times and the same .json files are missing) EDIT: I found the cause, see below, but still dont know what the solution should be

Logs

I placed tracker logs on outbound messages in the server middleware and I can see the correct outbound message: swyx sendmessage: /data/writing___ssg___rise-of-immer.json

I then also place tracker logs on received messages on the sapper export cli. I expect to see a corresponding inbound message swyx trackmessage: /data/writing___ssg___rise-of-immer.json. However, i don't.

swyx trackmessage: undefined
   2.87 kB   index.html
   2.71 kB   about/index.html
swyx sendmessage: /data/writing___ssg___writing_index.json
swyx sendmessage: /data/talks___ssg___talks_index.json
swyx sendmessage: /data/writing___ssg___svelte-static.json
swyx trackmessage: /data/writing___ssg___writing_index.json
   4.44 kB   data/writing___ssg___writing_index.json
swyx trackmessage: /data/talks___ssg___talks_index.json
   3.56 kB   data/talks___ssg___talks_index.json
swyx trackmessage: /data/writing___ssg___svelte-static.json
   9.91 kB   data/writing___ssg___svelte-static.json
   13.7 kB   writing/index.html
   11.7 kB   talks/index.html
   25.2 kB   writing/svelte-static/index.html
swyx sendmessage: /data/writing___ssg___netlify-gatsby-jamstack-apps-authenticated-serverless.json
swyx sendmessage: /data/writing___ssg___netlify-nextjs-ssg.json
swyx sendmessage: /data/writing___ssg___graphql-variables-why.json
swyx sendmessage: /data/writing___ssg___keyboard-first.json
swyx sendmessage: /data/writing___ssg___compilers-copout.json
swyx sendmessage: /data/writing___ssg___netlify-worlds-greatest-demo-v0.json
swyx sendmessage: /data/writing___ssg___gatsby-remark-essential-plugins.json
swyx sendmessage: /data/writing___ssg___netlify-jamstack-indiehackers.json
swyx sendmessage: /data/writing___ssg___book-poll-2019.json
swyx sendmessage: /data/writing___ssg___netlify-year-one-360-review.json
swyx sendmessage: /data/writing___ssg___react-native-web-singularity.json
swyx sendmessage: /data/writing___ssg___typescript-generics.json
swyx sendmessage: /data/writing___ssg___types-already.json
   9.27 kB   writing/keyboard-first/index.html
   5.13 kB   writing/compilers-copout/index.html
   19.3 kB   writing/graphql-variables-why/index.html
   34.3 kB   writing/netlify-worlds-greatest-demo-v0/index.html
   5.13 kB   writing/gatsby-remark-essential-plugins/index.html
swyx trackmessage: /data/writing___ssg___netlify-gatsby-jamstack-apps-authenticated-serverless.json
   63.5 kB   data/writing___ssg___netlify-gatsby-jamstack-apps-authenticated-serverless.json
swyx sendmessage: /data/writing___ssg___svelte-metaphysics.json
swyx sendmessage: /data/writing___ssg___react-not-bad.json
swyx sendmessage: /data/writing___ssg___netlify-python-ssg.json
swyx sendmessage: /data/writing___ssg___netlify-redirects-i18n.json
swyx sendmessage: /data/writing___ssg___slowmad-qna.json
swyx sendmessage: /data/writing___ssg___10000-loose-fans.json
swyx sendmessage: /data/writing___ssg___naked-promises.json
   11.5 kB   writing/book-poll-2019/index.html
   9.98 kB   writing/netlify-year-one-360-review/index.html
   12.4 kB   writing/typescript-generics/index.html
swyx sendmessage: /data/writing___ssg___netlify-env-vars.json
swyx trackmessage: /data/writing___ssg___netlify-nextjs-ssg.json
   23.8 kB   data/writing___ssg___netlify-nextjs-ssg.json
   59.1 kB   writing/netlify-nextjs-ssg/index.html
   42.8 kB   writing/react-native-web-singularity/index.html
swyx trackmessage: /data/writing___ssg___graphql-variables-why.json
   7.11 kB   data/writing___ssg___graphql-variables-why.json
swyx trackmessage: /data/writing___ssg___keyboard-first.json
   2.59 kB   data/writing___ssg___keyboard-first.json
swyx trackmessage: /data/writing___ssg___compilers-copout.json
     844 B   data/writing___ssg___compilers-copout.json
   59.6 kB   writing/netlify-jamstack-indiehackers/index.html
swyx sendmessage: /data/writing___ssg___netlify-year-one.json
swyx sendmessage: /data/writing___ssg___index.json
swyx sendmessage: /data/writing___ssg___types-or-tests.json
swyx sendmessage: /data/writing___ssg___betwixt-reason-and-results.json
swyx sendmessage: /data/writing___ssg___netlify-dev-cors.json
swyx sendmessage: /data/writing___ssg___how-to-name-things.json
swyx sendmessage: /data/writing___ssg___learning-gears.json
    4.6 kB   writing/react-not-bad/index.html
   5.24 kB   writing/netlify-env-vars/index.html
swyx trackmessage: /data/writing___ssg___netlify-worlds-greatest-demo-v0.json
   12.4 kB   data/writing___ssg___netlify-worlds-greatest-demo-v0.json
swyx trackmessage: /data/writing___ssg___gatsby-remark-essential-plugins.json
     919 B   data/writing___ssg___gatsby-remark-essential-plugins.json
swyx sendmessage: /data/writing___ssg___getting-closure-on-hooks.json
   17.5 kB   writing/svelte-metaphysics/index.html
   26.4 kB   writing/10000-loose-fans/index.html
   44.3 kB   writing/slowmad-qna/index.html
    105 kB   writing/types-already/index.html
   51.2 kB   writing/netlify-python-ssg/index.html
   56.9 kB   writing/netlify-redirects-i18n/index.html
swyx sendmessage: /data/writing___ssg___post-graduation-mindmap.json
   4.87 kB   writing/types-or-tests/index.html
swyx sendmessage: /data/writing___ssg___star-apps.json
swyx sendmessage: /data/writing___ssg___cuttlefish.json
swyx sendmessage: /data/writing___ssg___gatsby-static-dynamic.json
swyx sendmessage: /data/writing___ssg___rise-of-immer.json
swyx sendmessage: /data/writing___ssg___h1b1.json
   6.36 kB   writing/index/index.html
swyx trackmessage: /data/writing___ssg___netlify-jamstack-indiehackers.json
   26.1 kB   data/writing___ssg___netlify-jamstack-indiehackers.json
swyx trackmessage: /data/writing___ssg___book-poll-2019.json
   3.46 kB   data/writing___ssg___book-poll-2019.json
swyx sendmessage: /data/writing___ssg___specialize-in-the-new.json
   23.7 kB   writing/learning-gears/index.html
   28.4 kB   writing/betwixt-reason-and-results/index.html
swyx trackmessage: /data/writing___ssg___netlify-year-one-360-review.json
   2.96 kB   data/writing___ssg___netlify-year-one-360-review.json
swyx sendmessage: /data/writing___ssg___clone-open-source-apps.json
    155 kB   writing/netlify-gatsby-jamstack-apps-authenticated-serverless/index.html
   51.4 kB   writing/netlify-dev-cors/index.html
swyx trackmessage: /data/writing___ssg___react-native-web-singularity.json
   17.7 kB   data/writing___ssg___react-native-web-singularity.json
swyx sendmessage: /data/writing___ssg___learn-in-public.json
swyx sendmessage: /data/writing___ssg___know-your-tools.json
swyx sendmessage: /data/talks___ssg___intro-to-typescript.json
swyx sendmessage: /data/talks___ssg___ducktapes-podcast.json
swyx sendmessage: /data/talks___ssg___reactroundup-podcast.json
   14.3 kB   writing/h1b1/index.html
   5.17 kB   writing/star-apps/index.html
swyx trackmessage: /data/writing___ssg___typescript-generics.json
   3.65 kB   data/writing___ssg___typescript-generics.json
swyx sendmessage: /data/talks___ssg___programming-typescript.json
    145 kB   writing/naked-promises/index.html
   69.4 kB   writing/netlify-year-one/index.html
   69.5 kB   writing/how-to-name-things/index.html
   25.6 kB   writing/post-graduation-mindmap/index.html
   15.6 kB   writing/cuttlefish/index.html
   13.3 kB   writing/specialize-in-the-new/index.html
swyx sendmessage: /data/talks___ssg___kentcdodds-podcast.json
   12.7 kB   writing/clone-open-source-apps/index.html
swyx sendmessage: /data/talks___ssg___learn-in-public.json
   58.3 kB   writing/rise-of-immer/index.html
swyx sendmessage: /data/talks___ssg___netlify-functions-lunchnlearn.json
swyx sendmessage: /data/talks___ssg___jamstack-victory.json
swyx sendmessage: /data/talks___ssg___learn-in-public-singapore.json
swyx sendmessage: /data/talks___ssg___react-hooks.json
   11.2 kB   writing/know-your-tools/index.html
   6.61 kB   talks/ducktapes-podcast/index.html
   7.25 kB   talks/reactroundup-podcast/index.html
swyx sendmessage: /data/talks___ssg___jamstack-jumpstart.json
   17.9 kB   writing/learn-in-public/index.html
   6.32 kB   talks/programming-typescript/index.html
swyx trackmessage: /data/writing___ssg___types-already.json
   43.8 kB   data/writing___ssg___types-already.json
swyx sendmessage: /data/talks___ssg___intent-based-cli.json
swyx sendmessage: /data/talks___ssg___serverless-graphql-netlify.json
    6.6 kB   talks/kentcdodds-podcast/index.html
swyx trackmessage: /data/writing___ssg___svelte-metaphysics.json
   6.47 kB   data/writing___ssg___svelte-metaphysics.json
swyx trackmessage: /data/writing___ssg___react-not-bad.json
     517 B   data/writing___ssg___react-not-bad.json
swyx sendmessage: /data/talks___ssg___freecodecamp-podcast.json
swyx sendmessage: /data/talks___ssg___serverless-graphql.json
swyx sendmessage: /data/talks___ssg___react-hooks-lightning.json
swyx sendmessage: /data/talks___ssg___jamstack-victory-jsla.json
   26.4 kB   talks/learn-in-public/index.html
   8.41 kB   talks/netlify-functions-lunchnlearn/index.html
   5.79 kB   talks/jamstack-victory/index.html
   6.09 kB   talks/learn-in-public-singapore/index.html
swyx sendmessage: /data/talks___ssg___react-podcast.json
    188 kB   writing/getting-closure-on-hooks/index.html
swyx trackmessage: /data/writing___ssg___netlify-python-ssg.json
   21.3 kB   data/writing___ssg___netlify-python-ssg.json
swyx sendmessage: /data/talks___ssg___babel-macros.json
   25.8 kB   talks/jamstack-jumpstart/index.html
   5.69 kB   talks/serverless-graphql-netlify/index.html
swyx sendmessage: /data/talks___ssg___immutability-is-changing.json
swyx sendmessage: /data/talks___ssg___react-typescript.json
   30.2 kB   talks/intent-based-cli/index.html
swyx sendmessage: /data/talks___ssg___reacts-new-defaults.json
swyx sendmessage: /data/talks___ssg___contributing-to-open-source.json
swyx sendmessage: /data/talks___ssg___babel-graphql.json
swyx sendmessage: /data/talks___ssg___react-not-reactive.json
   6.32 kB   talks/react-hooks-lightning/index.html
   6.04 kB   talks/jamstack-victory-jsla/index.html
swyx sendmessage: /data/talks___ssg___infinite-building.json
    201 kB   writing/gatsby-static-dynamic/index.html
   19.7 kB   talks/freecodecamp-podcast/index.html
   16.3 kB   talks/serverless-graphql/index.html
   5.06 kB   talks/react-podcast/index.html
swyx trackmessage: /data/writing___ssg___netlify-redirects-i18n.json
   23.3 kB   data/writing___ssg___netlify-redirects-i18n.json
swyx sendmessage: /data/talks___ssg___creating-cra.json
    155 kB   talks/intro-to-typescript/index.html
swyx sendmessage: /data/talks___ssg___react-suspense.json
swyx sendmessage: /data/talks___ssg___never-bundle-react-again.json
   25.3 kB   talks/babel-macros/index.html
   6.56 kB   talks/react-typescript/index.html
swyx trackmessage: /data/writing___ssg___slowmad-qna.json
   19.7 kB   data/writing___ssg___slowmad-qna.json
   8.61 kB   talks/babel-graphql/index.html
    7.9 kB   talks/reacts-new-defaults/index.html
   14.2 kB   talks/contributing-to-open-source/index.html
swyx trackmessage: /data/writing___ssg___10000-loose-fans.json
   10.8 kB   data/writing___ssg___10000-loose-fans.json
swyx sendmessage: /data/talks___ssg___fullstack-heaps.json
swyx sendmessage: /data/talks___ssg___fullstack-crossbones.json
swyx sendmessage: /data/talks___ssg___fullstack-react-trip-planner.json
swyx sendmessage: /data/talks___ssg___contributing-to-react.json
   5.24 kB   talks/infinite-building/index.html
   8.13 kB   talks/creating-cra/index.html
   55.9 kB   talks/react-not-reactive/index.html
   5.09 kB   talks/react-suspense/index.html
   5.28 kB   talks/never-bundle-react-again/index.html
    101 kB   talks/immutability-is-changing/index.html
    215 kB   talks/react-hooks/index.html
   5.29 kB   talks/fullstack-react-trip-planner/index.html
   4.45 kB   talks/fullstack-heaps/index.html
   5.12 kB   talks/contributing-to-react/index.html
    5.3 kB   talks/fullstack-crossbones/index.html
swyx trackmessage: /data/writing___ssg___naked-promises.json
   58.9 kB   data/writing___ssg___naked-promises.json
swyx trackmessage: /data/writing___ssg___netlify-env-vars.json
     848 B   data/writing___ssg___netlify-env-vars.json
swyx trackmessage: /data/writing___ssg___netlify-year-one.json
     31 kB   data/writing___ssg___netlify-year-one.json
swyx trackmessage: /data/writing___ssg___index.json
   1.46 kB   data/writing___ssg___index.json
swyx trackmessage: /data/writing___ssg___types-or-tests.json
     721 B   data/writing___ssg___types-or-tests.json
swyx trackmessage: /data/writing___ssg___betwixt-reason-and-results.json
   11.9 kB   data/writing___ssg___betwixt-reason-and-results.json
   2.63 kB   service-worker-index.html

As you can see the message is always sent, but not always received.

To Reproduce

repro is going to be difficult for this one since its almost certainly a scale problem. my work is here: https://github.com/sw-yx/swyxdotio/tree/moveToSapper and you will note the logs above what the result of the code execution is.

Expected behavior

as explained above in the logs - messages that are sent ought to be received.

Information about your Sapper Installation:

  • Your browser and the version: n/a
  • Your operating system: osx
  • Your hosting environment: n/a
  • Sapper version: 0.27.8
  • Svelte version: n/a
  • If it is an exported (npm run export) or dynamic application: exported
  • Whether your application uses Webpack or Rollup: rollup

Severity

very severe. can't trust sapper export if it doesnt reliably work.

@swyxio
Copy link
Author

swyxio commented Sep 14, 2019

possibly related? #869 IO backpressure from @khrome83

a reader also pointed out that all posts from "Netlify Year One" on down on https://www.swyx.io/writing/ are the ones missing the .json. a huge clue! though I have no idea what to do with this information.

@swyxio
Copy link
Author

swyxio commented Sep 14, 2019

got it. overeager proc.kill:

proc.kill()

commenting out this one line fixes it.

how can we avoid killing that child process while it still has work left to do?

EDIT: proof: i injected a line where yootils.queue.close is called, which is supposed to block execution so that we dont proceed to the deadly proc.kill.. but it doesn't do its job. look at all the json files being written thereafter

> Crawling http://localhost:3000/
   2.87 kB   index.html
   4.42 kB   data/writing___ssg___writing_index.json
   2.71 kB   about/index.html
   9.91 kB   data/writing___ssg___svelte-static.json
   3.56 kB   data/talks___ssg___talks_index.json
   13.7 kB   writing/index.html
   25.2 kB   writing/svelte-static/index.html
   11.7 kB   talks/index.html
   23.8 kB   data/writing___ssg___netlify-nextjs-ssg.json
   9.27 kB   writing/keyboard-first/index.html
   5.13 kB   writing/compilers-copout/index.html
   5.13 kB   writing/gatsby-remark-essential-plugins/index.html
   7.11 kB   data/writing___ssg___graphql-variables-why.json
   19.3 kB   writing/graphql-variables-why/index.html
   34.2 kB   writing/netlify-worlds-greatest-demo-v0/index.html
   59.1 kB   writing/netlify-nextjs-ssg/index.html
   11.5 kB   writing/book-poll-2019/index.html
   9.98 kB   writing/netlify-year-one-360-review/index.html
   12.4 kB   writing/typescript-generics/index.html
    4.6 kB   writing/react-not-bad/index.html
   17.4 kB   writing/svelte-metaphysics/index.html
   59.6 kB   writing/netlify-jamstack-indiehackers/index.html
   42.8 kB   writing/react-native-web-singularity/index.html
   5.24 kB   writing/netlify-env-vars/index.html
   63.5 kB   data/writing___ssg___netlify-gatsby-jamstack-apps-authenticated-serverless.json
   2.59 kB   data/writing___ssg___keyboard-first.json
   26.4 kB   writing/10000-loose-fans/index.html
   6.36 kB   writing/index/index.html
    155 kB   writing/netlify-gatsby-jamstack-apps-authenticated-serverless/index.html
   51.2 kB   writing/netlify-python-ssg/index.html
   44.3 kB   writing/slowmad-qna/index.html
   12.3 kB   data/writing___ssg___netlify-worlds-greatest-demo-v0.json
     844 B   data/writing___ssg___compilers-copout.json
     919 B   data/writing___ssg___gatsby-remark-essential-plugins.json
    105 kB   writing/types-already/index.html
   56.8 kB   writing/netlify-redirects-i18n/index.html
   4.87 kB   writing/types-or-tests/index.html
   69.4 kB   writing/netlify-year-one/index.html
   28.4 kB   writing/betwixt-reason-and-results/index.html
   23.7 kB   writing/learning-gears/index.html
   25.6 kB   writing/post-graduation-mindmap/index.html
   5.17 kB   writing/star-apps/index.html
   26.1 kB   data/writing___ssg___netlify-jamstack-indiehackers.json
   3.46 kB   data/writing___ssg___book-poll-2019.json
   2.96 kB   data/writing___ssg___netlify-year-one-360-review.json
   51.4 kB   writing/netlify-dev-cors/index.html
    145 kB   writing/naked-promises/index.html
   15.6 kB   writing/cuttlefish/index.html
   14.3 kB   writing/h1b1/index.html
   13.3 kB   writing/specialize-in-the-new/index.html
   12.7 kB   writing/clone-open-source-apps/index.html
   11.2 kB   writing/know-your-tools/index.html
   17.7 kB   data/writing___ssg___react-native-web-singularity.json
   69.5 kB   writing/how-to-name-things/index.html
   3.65 kB   data/writing___ssg___typescript-generics.json
   58.3 kB   writing/rise-of-immer/index.html
   17.9 kB   writing/learn-in-public/index.html
   6.61 kB   talks/ducktapes-podcast/index.html
   7.25 kB   talks/reactroundup-podcast/index.html
   6.32 kB   talks/programming-typescript/index.html
    6.6 kB   talks/kentcdodds-podcast/index.html
   5.79 kB   talks/jamstack-victory/index.html
   26.4 kB   talks/learn-in-public/index.html
   43.8 kB   data/writing___ssg___types-already.json
   8.41 kB   talks/netlify-functions-lunchnlearn/index.html
   6.09 kB   talks/learn-in-public-singapore/index.html
   6.47 kB   data/writing___ssg___svelte-metaphysics.json
     517 B   data/writing___ssg___react-not-bad.json
    188 kB   writing/getting-closure-on-hooks/index.html
   25.8 kB   talks/jamstack-jumpstart/index.html
   5.69 kB   talks/serverless-graphql-netlify/index.html
   30.2 kB   talks/intent-based-cli/index.html
   16.3 kB   talks/serverless-graphql/index.html
   19.7 kB   talks/freecodecamp-podcast/index.html
   21.3 kB   data/writing___ssg___netlify-python-ssg.json
   6.04 kB   talks/jamstack-victory-jsla/index.html
   6.32 kB   talks/react-hooks-lightning/index.html
   5.06 kB   talks/react-podcast/index.html
    201 kB   writing/gatsby-static-dynamic/index.html
   6.56 kB   talks/react-typescript/index.html
    7.9 kB   talks/reacts-new-defaults/index.html
   14.2 kB   talks/contributing-to-open-source/index.html
    155 kB   talks/intro-to-typescript/index.html
   25.3 kB   talks/babel-macros/index.html
   23.3 kB   data/writing___ssg___netlify-redirects-i18n.json
   8.61 kB   talks/babel-graphql/index.html
   5.24 kB   talks/infinite-building/index.html
   5.28 kB   talks/never-bundle-react-again/index.html
   8.13 kB   talks/creating-cra/index.html
   5.09 kB   talks/react-suspense/index.html
   19.7 kB   data/writing___ssg___slowmad-qna.json
   4.45 kB   talks/fullstack-heaps/index.html
   5.12 kB   talks/contributing-to-react/index.html
   10.8 kB   data/writing___ssg___10000-loose-fans.json
    101 kB   talks/immutability-is-changing/index.html
   55.9 kB   talks/react-not-reactive/index.html
    5.3 kB   talks/fullstack-crossbones/index.html
   5.29 kB   talks/fullstack-react-trip-planner/index.html
    215 kB   talks/react-hooks/index.html
   58.9 kB   data/writing___ssg___naked-promises.json
     848 B   data/writing___ssg___netlify-env-vars.json
     31 kB   data/writing___ssg___netlify-year-one.json
   1.46 kB   data/writing___ssg___index.json
     721 B   data/writing___ssg___types-or-tests.json
   11.9 kB   data/writing___ssg___betwixt-reason-and-results.json
   2.63 kB   service-worker-index.html
   
swyx: close called { pending: 0 }

> Finished in 8.4s. Type npx serve __sapper__/export to run the app.
   20.8 kB   data/writing___ssg___netlify-dev-cors.json
   29.4 kB   data/writing___ssg___how-to-name-things.json
   9.37 kB   data/writing___ssg___learning-gears.json
   76.8 kB   data/writing___ssg___getting-closure-on-hooks.json
   8.68 kB   data/writing___ssg___post-graduation-mindmap.json
     810 B   data/writing___ssg___star-apps.json
   5.64 kB   data/writing___ssg___cuttlefish.json
   81.9 kB   data/writing___ssg___gatsby-static-dynamic.json
   23.8 kB   data/writing___ssg___rise-of-immer.json
    5.1 kB   data/writing___ssg___h1b1.json
   4.84 kB   data/writing___ssg___specialize-in-the-new.json
   4.44 kB   data/writing___ssg___clone-open-source-apps.json
   3.83 kB   data/writing___ssg___know-your-tools.json
   6.79 kB   data/writing___ssg___learn-in-public.json
   62.2 kB   data/talks___ssg___intro-to-typescript.json
   1.03 kB   data/talks___ssg___ducktapes-podcast.json
   1.58 kB   data/talks___ssg___reactroundup-podcast.json
     677 B   data/talks___ssg___programming-typescript.json
    1.2 kB   data/talks___ssg___kentcdodds-podcast.json
   9.15 kB   data/talks___ssg___learn-in-public.json
     865 B   data/talks___ssg___jamstack-victory.json
   1.57 kB   data/talks___ssg___netlify-functions-lunchnlearn.json
     705 B   data/talks___ssg___learn-in-public-singapore.json
   86.6 kB   data/talks___ssg___react-hooks.json
   9.88 kB   data/talks___ssg___jamstack-jumpstart.json
   10.5 kB   data/talks___ssg___intent-based-cli.json
     890 B   data/talks___ssg___serverless-graphql-netlify.json
   6.57 kB   data/talks___ssg___freecodecamp-podcast.json
   4.92 kB   data/talks___ssg___serverless-graphql.json
     985 B   data/talks___ssg___react-hooks-lightning.json
     971 B   data/talks___ssg___jamstack-victory-jsla.json
     575 B   data/talks___ssg___react-podcast.json
   8.42 kB   data/talks___ssg___babel-macros.json
   39.3 kB   data/talks___ssg___immutability-is-changing.json
   1.01 kB   data/talks___ssg___react-typescript.json
   1.61 kB   data/talks___ssg___reacts-new-defaults.json
   4.23 kB   data/talks___ssg___contributing-to-open-source.json
   1.78 kB   data/talks___ssg___babel-graphql.json
   23.3 kB   data/talks___ssg___react-not-reactive.json
     625 B   data/talks___ssg___infinite-building.json
   1.24 kB   data/talks___ssg___creating-cra.json
     451 B   data/talks___ssg___never-bundle-react-again.json
     397 B   data/talks___ssg___react-suspense.json
     475 B   data/talks___ssg___contributing-to-react.json
     339 B   data/talks___ssg___fullstack-heaps.json
     510 B   data/talks___ssg___fullstack-crossbones.json
     461 B   data/talks___ssg___fullstack-react-trip-planner.json

@swyxio swyxio changed the title dropped messages in sapper export child process - some .json files not getting built as a result premature proc.kill() on sapper export child process - some .json files not getting built as a result Sep 14, 2019
@rheinardkorf
Copy link

a reader also pointed out

You’re welcome!

@swyxio
Copy link
Author

swyxio commented Sep 14, 2019

Ok I am officially stumped. i know -why- this is happening: yootils.queue is made to deal with async queues of work... except:

but i dont know what the solution is here. hoping @Rich-Harris see this and has an idea.

@swyxio
Copy link
Author

swyxio commented Sep 14, 2019

A definite improvement we can make, that doesn't solve this but helps, is to make the I/O non-blocking. rewriting this writeFileSync code to promisified async helps more data to be output, but we are still killing the child process prematurely.

@arxpoetica
Copy link
Member

@sw-yx I have a disappearing json file on Netlify, and no where near as many file exports as thou...

@swyxio
Copy link
Author

swyxio commented Sep 19, 2019

repro?

@arxpoetica
Copy link
Member

@sw-yx I figured out that my bug wasn't related at all to json file generation (that's working fine), but static export crawling when using something other than this.fetch, so my issue is unrelated. See #908

Sorry for the noise on this ticket.

@swyxio
Copy link
Author

swyxio commented Oct 3, 2019

i thought that merging #869 would fix this issue, but after checking futher it doesn't. :( still unsolved.

@swyxio swyxio changed the title premature proc.kill() on sapper export child process - some .json files not getting built as a result premature proc.kill() on sapper export child process - some .json files not getting built as a result of race condition Oct 7, 2019
@swyxio
Copy link
Author

swyxio commented Oct 7, 2019

update: i think i have a janky fix... here's what needs to happen:

To fix, either spin out the file writing to yet another child process (wasteful) or...

	// swyx: solve race condition where saving in child process is prematurely stopped by proc.kill
	// https://github.com/sveltejs/sapper/issues/893#issuecomment-538811020
	let latestUrl = null; 
	proc.on('message', message => {
		if (!message.__sapper__ || message.event !== 'file') return;
		latestUrl = message.url;
		const saveresult = save(message.url, message.status, message.type, message.body);
		if (saveresult) saveresult.then(() => checkWritingDone(message.url));
	});

	// swyx: introduced to solve race condition
	// https://github.com/sveltejs/sapper/issues/893#issuecomment-538811020
	function checkWritingDone(currentUrl) {
		if (latestUrl != currentUrl || queue.getLength() > 0) ; else {
			proc.kill();
		}
	}

and comment out the bottom proc.kill.

i'm implementing this in my sapper fork. let me know if you want this in the main repo.

@swyxio
Copy link
Author

swyxio commented Oct 8, 2019

just swimming by to confirm that @nealalpert has fixed this once and for all by including the save() method in the export queue: nealalpert@3247d50

i really think this belongs in core sapper. others will run into what we ran into.

@bummzack
Copy link

Hitting the same problem. Any chance of getting #869 merged soon?

@antony
Copy link
Member

antony commented Feb 20, 2020

@sw-yx I'm assuming that #869 now fixes this issue, based on your previous comment about addSave?

@swyxio
Copy link
Author

swyxio commented Feb 20, 2020

yes, he updated it after i reported my issues.

@Conduitry
Copy link
Member

Fixed in 0.27.14, with the long-awaited merge of 0.27.14.

@bummzack
Copy link

This issue still seems to be unsolved with 0.27.16.
Creating a fresh sapper project and making the src/blog/index.json.js waiting for data does not export all pages.

Build output with default setup (everything gets exported):
Screenshot 2020-07-13 at 12 06 20

Testing with a delay of 5 seconds on the blog data:
Screenshot 2020-07-13 at 12 06 46

Example code: https://github.com/bummzack/sapper-wtf/commit/b0d9f8bd6ff6f537b91933e83adfd7868fcc8a82

Running sapper export --legacy --no-timeout does not generate any pages at all.

@bummzack
Copy link

bummzack commented Jul 13, 2020

Nevermind, increasing the timeout via --timeout=10000 does the trick. Although I still get some weird problems with some json files not being exported. I'll try to reproduce it in a simple test-case.

zhammer added a commit to zhammer/writing that referenced this issue Oct 3, 2020
i've spent so much time trying to figure out why sapper export often
misses exporting the [slug].json for certain pages. in b0b3b83 i
assumed the issue was (somehow) using hyphens in slugs so switched
to using underscores but this started happening again.

i found the followng issues which all are "closed" or "merged" but
this is _definitely_ still a bug. ended up looking at options for the
sapper export command and saw i can turn off concurrent requests,
which seems to have fixed the issue.
- sveltejs/sapper#1078
- sveltejs/sapper#893
- sveltejs/sapper#869

hoping to submit a repro of the issue to sapper soon
@DylanVann
Copy link

DylanVann commented Oct 5, 2020

Currently in export.ts onDone of the queue is sometimes called before the callback in proc.on('message', cb).

I commented out proc.kill() in the onDone callback to get some logs:

image

Another log showing 2 files missed:

image

It seems like how things work now there's no way to know we've waited long enough for all process.send calls to be done. One simple solution is to add a delay of half a second in handleFetch, this slows things down though and is not gaurunteed to work. A better solution would be to have some way to know we're not waiting on process.send, maybe the server could always use process.send to signal that it's done.

@ingomc
Copy link

ingomc commented Nov 9, 2020

I can confirm that this bug still exists...
i have a huge _posts.js with a lot of data in it, and Sapper will not export all json files!
the hint from @zhammer really solved it for me until the real thing is solved

So everybody who still has this problem should try

"export": "sapper export --legacy --concurrent 1",

@zhammer
Copy link

zhammer commented Nov 9, 2020

@ingomc i'm glad that comment on my commit came in handy! yeah, my first step for an exported sapper project is now to add --concurrent 1 to the export package.json command

@darkyzhou
Copy link

The bug still exists on the latest version(0.29.1), the issue should be open again.

--concurrent 1 can help prevent the bug from happening.

@antony
Copy link
Member

antony commented Mar 17, 2021

@darkyzhou we're in the process of deprecating Sapper in favour of Svelte Kit. We don't think this issue will exist on Svelte Kit since it's operation is entirely different to the way Sapper works in this respect.

The --concurrent 1 solution you mentioned is an appropriate workaround for existing Sapper users.

@iceye
Copy link

iceye commented Mar 30, 2021

Same here, using --concurrent 1 and high timeout helps but doesn't solve. After some testing it seems the issue is related to file size and happen for HTML files too.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests