Caching errors are back a la 5729 #5962

bjornharvold · 2021-06-10T17:22:16Z

Current Behavior

I do a clean build of my 1000+ module monorepo which means I remove node_modules/.cache/nx and dist folders. A full build is almost 4 hours but now it runs through it in less than 15 minutes with barely any modules created.

My command is:
node --max_old_space_size=16384 ./node_modules/.bin/nx run-many --target=build --projects=administration,booking-engine,corporate,developer,extranet,hotel-booking-engine,maintenance,seller,single-sign-on,travel-agent,web-components --configuration=production --with-deps --parallel --max-parallel=8

Expected Behavior

All SPA apps should be built at the end of it

Steps to Reproduce

As with #5729, it's very hard to tell you why it suddenly stopped working. After #5729 was resolved, the build worked flawlessly. In the meantime, Angular got updated and it stopped working again.

Failure Logs

There are no error logs. It just completes with a success.

Environment

@nrwl/angular : 12.3.6
@nrwl/cli : 12.3.6
@nrwl/cypress : 12.3.6
@nrwl/devkit : 12.3.6
@nrwl/eslint-plugin-nx : 12.3.6
@nrwl/express : Not Found
@nrwl/jest : 12.3.6
@nrwl/linter : 12.3.6
@nrwl/nest : Not Found
@nrwl/next : Not Found
@nrwl/node : Not Found
@nrwl/react : Not Found
@nrwl/schematics : Not Found
@nrwl/tao : 12.3.6
@nrwl/web : Not Found
@nrwl/workspace : 12.3.6
@nrwl/storybook : 12.3.6
@nrwl/gatsby : Not Found
typescript : 4.2.4
Angular : 12.0.4
Angular CLI : 12.0.3
@angular-devkit/build-angular : 12.0.3
ng-packagr 12.0.5

The text was updated successfully, but these errors were encountered:

FrozenPandaz · 2021-06-10T22:25:49Z

Can you provide more information? What builders are being used? Can you provide a repo to help reproduce the issue?

The issue was resolved in 12.3.5 and is back with 12.3.6?

bjornharvold · 2021-06-11T02:11:31Z

Hi @FrozenPandaz

Updated the ticket with build-angular version. I cannot provide the private repo and I cannot yet provide a test repo because I don't yet know what causes the issue. I will be downgrading libraries incrementally today to try find out which which version of which library caused this regression.

bjornharvold · 2021-06-11T07:24:12Z

Reporting back...

I tried creating a small repo to show the bug. I have not been able to do so.
I've incrementally downgraded [one by one] nx.dev / angular / angular CLI / Angular builder libraries down to 12.3.5 / 12.0.1 / 12.0.1 / 12.0.0 respectively without being able to fix the issue
I'm getting this build error on Ubuntu and Mac.

Yet, my repo is consistently only building 4 out of 11 SPAs without giving any error or trying to compile the libraries. The build takes approx 10 minutes to build from scratch when it should be taking 4 hours.

Here's my demo repo with the same dependencies as my main repo: https://github.com/bjornharvold/nx-dev-5962

The issue I am seeing looks exactly like the previous issue: #5729. Just don't know how to replicate.

I can't share the private repo but we could set up a Google Meet and walk you through it.

bjornharvold · 2021-06-12T04:57:33Z

Following up on our investigations

Cleaning out the cache and dist folders and trying to build the entire monorepo with all 11 SPAs, the build finishes but without saying "Nx build success". It just stops. 4 apps are able to build before the process halts.
Cleaning out the cache and dist folders and trying to build a single app (+ deps) that failed when trying to build all apps, it builds to a certain point and also just stops. There is no output under dist/apps; only the libs it managed to build before halting.

The results are the same with using parallel builds or not.

bjornharvold · 2021-06-12T07:57:02Z

Following up on our investigations

We downgraded Angular from latest to 12.0.0
We downgraded nx.dev from latest to 12.3.5
We downgraded Angular CLI / builder from latest to 12.0.0
We downgraded css-loader from latest to 5.2.5
We downgraded prettier from latest to 2.3.0
We downgraded sass from latest to 1.32.13
We downgraded sass-loader from latest to 11.1.0

Same error and consistently building the same 4 SPAs. No other errors.

bjornharvold · 2021-06-13T01:50:35Z

CI build log - No errors - You just see it failing at the end because it's expecting a directory to exist under app/dist.
plan-116555777-JOB1-162.log

bjornharvold · 2021-06-15T03:26:56Z

I added --skip-nx-cache to my build command. Also did not have an effect.

I tried building a single SPA with no luck. The build needed to compile 425 modules but the build just stopped after about a 100.

nx build extranet --with-deps --skip-nx-cache

I noticed a deprecation warning half way through when I tried to build the single SPA

------------------------------------------------------------------------------
Building entry point '@iko/media/ui-card-manager'
------------------------------------------------------------------------------
Writing package metadata
DEPRECATION WARNING: Using / for division is deprecated and will be removed in Dart Sass 2.0.0.

Recommendation: math.div($spacer, 2)

More info and automated migrator: https://sass-lang.com/d/slash-div

    ╷
302 │ $headings-margin-bottom:      $spacer / 2 !default;
    │                               ^^^^^^^^^^^
    ╵
    node_modules/bootstrap/scss/_variables.scss 302:31                                               @import
    libs/media/ui-card-manager/src/lib/components/internal/image-list/image-list.component.scss 7:9  root stylesheet

DEPRECATION WARNING: Using / for division is deprecated and will be removed in Dart Sass 2.0.0.

Recommendation: math.div($input-padding-y, 2)

More info and automated migrator: https://sass-lang.com/d/slash-div

    ╷
498 │ $input-height-inner-quarter:            add($input-line-height * .25em, $input-padding-y / 2) !default;
    │                                                                         ^^^^^^^^^^^^^^^^^^^^
    ╵
    node_modules/bootstrap/scss/_variables.scss 498:73                                               @import
    libs/media/ui-card-manager/src/lib/components/internal/image-list/image-list.component.scss 7:9  root stylesheet

DEPRECATION WARNING: Using / for division is deprecated and will be removed in Dart Sass 2.0.0.

Recommendation: math.div($custom-control-indicator-size, 2)

More info and automated migrator: https://sass-lang.com/d/slash-div

    ╷
568 │ $custom-switch-indicator-border-radius:         $custom-control-indicator-size / 2 !default;
    │                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ╵
    node_modules/bootstrap/scss/_variables.scss 568:49                                               @import
    libs/media/ui-card-manager/src/lib/components/internal/image-list/image-list.component.scss 7:9  root stylesheet

DEPRECATION WARNING: Using / for division is deprecated and will be removed in Dart Sass 2.0.0.

Recommendation: math.div($spacer, 2)

More info and automated migrator: https://sass-lang.com/d/slash-div

    ╷
713 │ $nav-divider-margin-y:              $spacer / 2 !default;
    │                                     ^^^^^^^^^^^
    ╵
    node_modules/bootstrap/scss/_variables.scss 713:37                                               @import
    libs/media/ui-card-manager/src/lib/components/internal/image-list/image-list.component.scss 7:9  root stylesheet

DEPRECATION WARNING: Using / for division is deprecated and will be removed in Dart Sass 2.0.0.

Recommendation: math.div($spacer, 2)

More info and automated migrator: https://sass-lang.com/d/slash-div

    ╷
718 │ $navbar-padding-y:                  $spacer / 2 !default;
    │                                     ^^^^^^^^^^^
    ╵
    node_modules/bootstrap/scss/_variables.scss 718:37                                               @import
    libs/media/ui-card-manager/src/lib/components/internal/image-list/image-list.component.scss 7:9  root stylesheet

WARNING: 2 repetitive deprecation warnings omitted.

DEPRECATION WARNING: Using / for division is deprecated and will be removed in Dart Sass 2.0.0.

Recommendation: math.div($spacer, 2)

More info and automated migrator: https://sass-lang.com/d/slash-div

    ╷
302 │ $headings-margin-bottom:      $spacer / 2 !default;
    │                               ^^^^^^^^^^^
    ╵
    node_modules/bootstrap/scss/_variables.scss 302:31                                               @import
    libs/media/ui-card-manager/src/lib/components/internal/video-list/video-list.component.scss 7:9  root stylesheet

DEPRECATION WARNING: Using / for division is deprecated and will be removed in Dart Sass 2.0.0.

Recommendation: math.div($input-padding-y, 2)

More info and automated migrator: https://sass-lang.com/d/slash-div

    ╷
498 │ $input-height-inner-quarter:            add($input-line-height * .25em, $input-padding-y / 2) !default;
    │                                                                         ^^^^^^^^^^^^^^^^^^^^
    ╵
    node_modules/bootstrap/scss/_variables.scss 498:73                                               @import
    libs/media/ui-card-manager/src/lib/components/internal/video-list/video-list.component.scss 7:9  root stylesheet

DEPRECATION WARNING: Using / for division is deprecated and will be removed in Dart Sass 2.0.0.

Recommendation: math.div($custom-control-indicator-size, 2)

More info and automated migrator: https://sass-lang.com/d/slash-div

    ╷
568 │ $custom-switch-indicator-border-radius:         $custom-control-indicator-size / 2 !default;
    │                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ╵
    node_modules/bootstrap/scss/_variables.scss 568:49                                               @import
    libs/media/ui-card-manager/src/lib/components/internal/video-list/video-list.component.scss 7:9  root stylesheet

DEPRECATION WARNING: Using / for division is deprecated and will be removed in Dart Sass 2.0.0.

Recommendation: math.div($spacer, 2)

More info and automated migrator: https://sass-lang.com/d/slash-div

    ╷
713 │ $nav-divider-margin-y:              $spacer / 2 !default;
    │                                     ^^^^^^^^^^^
    ╵
    node_modules/bootstrap/scss/_variables.scss 713:37                                               @import
    libs/media/ui-card-manager/src/lib/components/internal/video-list/video-list.component.scss 7:9  root stylesheet

DEPRECATION WARNING: Using / for division is deprecated and will be removed in Dart Sass 2.0.0.

Recommendation: math.div($spacer, 2)

More info and automated migrator: https://sass-lang.com/d/slash-div

    ╷
718 │ $navbar-padding-y:                  $spacer / 2 !default;
    │                                     ^^^^^^^^^^^
    ╵
    node_modules/bootstrap/scss/_variables.scss 718:37                                               @import
    libs/media/ui-card-manager/src/lib/components/internal/video-list/video-list.component.scss 7:9  root stylesheet

WARNING: 2 repetitive deprecation warnings omitted.

Other than that there is no indication of failure. Build is green but no artifact was built.

bjornharvold · 2021-06-15T03:38:00Z

Notice the long list of libraries that need to get built [assuming in that order] at the beginning of the log and that it continually builds down to and including advertising-ui-banner and then skips a ton of libraries. No indication as to why.

plan-116555777-JOB1-167.log

bjornharvold · 2021-06-15T13:39:05Z

Now if I try to build the next library under advertising-ui-banner, namely: web-components-feature-hotel-card, it fails because a lot of dependencies never got built.

Some of the project web-components-feature-hotel-card's dependencies have not been built yet. Please build these libraries before:
- inventory-domain
- meeting-room-domain
- inventory-interfaces-domain
- property-domain
- channel-domain
- activity-domain
- add-on-domain
- attraction-domain
- package-domain
- place-domain
- restaurant-domain
- spa-domain
- inventory-ui-hotel-card
- inventory-ui-hotel-best-price-card
- media-ui-slideshow
- analytics-domain
- booking-domain
- property-ui-hotel-rating-carousel
- property-util-hotel-rating-pipe
- bucket-list-ui-like-button
- bucket-list-util-liked-pipe
- bucket-list-domain
- property-ui-cancellation-policy-read-only
- property-ui-unavailable-card
- property-util-category-score-pipe
- inventory-ui-guest-room-best-price-card
- web-components-domain
- inventory-ui-quick-book-confirmation-modal
- booking-feature-booking-manager-modal
- booking-ui-booking-list
- booking-ui-booking-single-view
- property-util-perk-guaranteed-pipe
- booking-ui-booking-confirm-cancellation
- bucket-list-feature-bucket-list-modal
- bucket-list-ui-bucket-list-list
- profile-feature-profile-modal
- profile-ui-hotel-preferences
- profile-ui-room-preferences
- profile-ui-personal-preferences

You can see the libraries listed in the correct order at the beginning of the log, but they are never mentioned as having been built. E.g. meeting-room-domain. These libraries would get mentioned even if they were taken from the nx cache. But instead they just disappear and are never heard of.

So I think this is no longer a cache issue but a silent error or similar that fails in a way that lets the build go on so we don't even know where to look.

Thoughts? I haven't been able to build for almost a week and colleagues and customers are waiting on my team to try and solve this. Is there a way for my team to debug this in more depth? Otherwise, we are unsure on how to proceed.

Cheers

leosvelperez · 2021-06-15T15:02:50Z

@bjornharvold sorry to hear that, but without a reproduction, it's very hard for us to troubleshoot this, especially this type of issue where is not clear what the problem could be, we'd be blind guessing things. Last time, I was able to troubleshoot the #5729 issue because a way to reproduce it was shared.

I just checked, just in case, the previous issue reproduction repos with the latest Nx and Angular packages versions and those are working correctly, so it doesn't seem to be the same issue.

You mentioned that after the fix for #5729 everything was working fine, I know you've tried several things, but just in case, I'd recommend you to try and pin the versions of the packages to the exact version the fix had and see if it works. Make sure to only update the version of the packages and not modify any other code:

If the problem persists, this would tell us the problem is related to the apps' code and not to the version of the packages (it was working before with those packages version and now it's not).
If the problem goes away, then some package version introduced the issue and you'd need to do some detective work updating them one at a time and see where the error starts to appear.

You can also try to check out the latest working commit and see if it's still working. This could also confirm is not related to a package version, but to a code change if that commit is still working correctly.

Some questions to think about:

Is this only happening on CI or does it happen locally as well? In general, try to determine if it's specific to the environment or not.
Does it always build successfully the same apps? Or is it random? Try to assert if the results are deterministic or if there is some random factor.
If you try to only build one of the apps that fail, does it works or does it still fail? Is it related to multiple projects building?

As I said, it's really hard for us to troubleshoot it without a clear reproduction and preferably a minimal one.

If you can reproduce the issue locally and isolate it to a project, try to pick the smallest project you have, remove as many deps as you can while still having the issue, this would help the debugging process.

Hopefully, some of the points above can help you troubleshoot it.

bjornharvold · 2021-06-16T05:47:14Z

Hi @leosvelperez,

Thank you for your detailed response! Very appreciated :-)

We started pulling the build apart and here is what we learned:

All our 1000+ libraries are created with the --buildable flag FYI
All libraries build correctly by themselves
All apps build correctly if we build them individually WITHOUT the --with-deps flag. E.g. nx build extranet. What is strange is that, in the past, the app wouldn't build if we didn't add --with-deps because it would be complaining abut missing dependencies and tell us to build those first.
Our simplest apps [that have very few dependencies] build with the --with-deps flag.
Our larger apps [with many monorepo dependencies] fail when we use the --with-deps flag. E.g. nx build extranet --with-deps. No errors, the build will just stop building and it will stop building consistently at the same place for each app respectively. So, different SPA will stop at their own same specific spot in the build process every time. The build will omit building certain libraries [as explained above] without notifying us and we think that stops the build when the next libraries, that are dependent on those libraries, can't find them.
The command run-many also works fine WITHOUT the --with-deps flag. E.g. nx run-many --target build --projects extranet,booking-engine --configuration production.

In a nutshell, I think --with-deps gets confused with large dependency graphs [since when I am unsure]. What's also confusing is that we don't have to add --with-deps any longer as the apps are intelligent enough to build its own dependencies. Was this always the case. I am pretty certain I remember my apps complaining about missing dependencies and to build them first when I tried to build them before.

So my build runs and all we had to do was remove the --with-deps flag. So strange.

smasala · 2021-08-09T14:16:30Z

@FrozenPandaz This is happening for us on 12.3.6 when using @nrwl/workspace:run-commands - it returns the cached output to the console (success|failure) but doesn't restore the build files (dist)

github-actions · 2023-03-23T17:05:19Z

This issue has been closed for more than 30 days. If this issue is still occuring, please open a new issue with more recent context.

bjornharvold added the type: bug label Jun 10, 2021

FrozenPandaz added blocked: more info needed scope: core core nx functionality labels Jun 10, 2021

FrozenPandaz assigned leosvelperez Jun 10, 2021

bjornharvold closed this as completed Jun 16, 2021

github-actions bot added the outdated label Mar 23, 2023

github-actions bot locked as resolved and limited conversation to collaborators Mar 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Caching errors are back a la 5729 #5962

Caching errors are back a la 5729 #5962

bjornharvold commented Jun 10, 2021 •

edited

Loading

FrozenPandaz commented Jun 10, 2021

bjornharvold commented Jun 11, 2021

bjornharvold commented Jun 11, 2021 •

edited

Loading

bjornharvold commented Jun 12, 2021 •

edited

Loading

bjornharvold commented Jun 12, 2021

bjornharvold commented Jun 13, 2021

bjornharvold commented Jun 15, 2021

bjornharvold commented Jun 15, 2021 •

edited

Loading

bjornharvold commented Jun 15, 2021

leosvelperez commented Jun 15, 2021

bjornharvold commented Jun 16, 2021

smasala commented Aug 9, 2021

github-actions bot commented Mar 23, 2023

Caching errors are back a la 5729 #5962

Caching errors are back a la 5729 #5962

Comments

bjornharvold commented Jun 10, 2021 • edited Loading

Current Behavior

Expected Behavior

Steps to Reproduce

Failure Logs

Environment

FrozenPandaz commented Jun 10, 2021

bjornharvold commented Jun 11, 2021

bjornharvold commented Jun 11, 2021 • edited Loading

bjornharvold commented Jun 12, 2021 • edited Loading

bjornharvold commented Jun 12, 2021

bjornharvold commented Jun 13, 2021

bjornharvold commented Jun 15, 2021

bjornharvold commented Jun 15, 2021 • edited Loading

bjornharvold commented Jun 15, 2021

leosvelperez commented Jun 15, 2021

bjornharvold commented Jun 16, 2021

smasala commented Aug 9, 2021

github-actions bot commented Mar 23, 2023

bjornharvold commented Jun 10, 2021 •

edited

Loading

bjornharvold commented Jun 11, 2021 •

edited

Loading

bjornharvold commented Jun 12, 2021 •

edited

Loading

bjornharvold commented Jun 15, 2021 •

edited

Loading