Reduce IO calls in extractDependencies #18266

bishabosha · 2023-07-21T15:38:27Z

fix two hotspots in extractDependencies - checking if depFile is a class, (does file IO on isDirectory), and resolving sibling class file of a TASTy file (more file IO).

To fix this, only check the file extension, and also avoid unnecessary resolving of sibling .class file (always failing) when we have a .scala file as the associated file.

Second we can cache the associated class file of a TASTy file.

before:

after:

as you can see recordClassDependency after optimisation is now completely dominated by calling to zinc

bishabosha · 2023-07-21T15:38:44Z

test performance please

dottybot · 2023-07-21T15:40:42Z

performance test scheduled: 207 job(s) in queue, 1 running.

bishabosha · 2023-07-21T15:41:45Z

compiler/src/dotty/tools/dotc/core/Contexts.scala

@@ -262,6 +262,19 @@ object Contexts {
    /** AbstractFile with given path, memoized */
    def getFile(name: String): AbstractFile = getFile(name.toTermName)

+    def getSiblingClassfile(tastyFile: AbstractFile): AbstractFile =
+      base.siblingClassfiles.getOrElseUpdate(tastyFile, {


we could possibly check before caching to see if Zinc is enabled.

Did you check how many cache hits we get?

dottybot · 2023-07-21T17:28:59Z

Performance test finished successfully:

Visit https://dotty-bench.epfl.ch/18266/ to see the changes.

Benchmarks is based on merging with main (7613234)

bishabosha · 2023-07-21T17:42:33Z

Performance test finished successfully:

Visit https://dotty-bench.epfl.ch/18266/ to see the changes.

Benchmarks is based on merging with main (7613234)

note that the sbt phases are not ran in the benchmark

compiler/src/dotty/tools/dotc/core/Contexts.scala

bishabosha · 2023-07-24T13:18:33Z

is there any objection to adding new caches to ContextBase like this?

compiler/src/dotty/tools/dotc/core/Contexts.scala

nicolasstucki · 2023-07-25T09:18:48Z

compiler/src/dotty/tools/dotc/core/Contexts.scala

@@ -262,6 +262,19 @@ object Contexts {
    /** AbstractFile with given path, memoized */
    def getFile(name: String): AbstractFile = getFile(name.toTermName)

+    def getSiblingClassfile(tastyFile: AbstractFile): AbstractFile =
+      base.siblingClassfiles.getOrElseUpdate(tastyFile, {


Did you check how many cache hits we get?

compiler/src/dotty/tools/dotc/core/SymbolLoaders.scala

compiler/src/dotty/tools/dotc/classpath/FileUtils.scala

nicolasstucki · 2023-07-25T09:29:22Z

is there any objection to adding new caches to ContextBase like this?

Not sure if that is a good idea.

@smarter WDYT?

compiler/src/dotty/tools/dotc/sbt/ExtractDependencies.scala

smarter · 2023-07-25T13:29:51Z

compiler/src/dotty/tools/dotc/sbt/ExtractDependencies.scala

+          // did not find associated class file, e.g. for a TASTy-only classpath.
+          // The file that Zinc recieves with binaryDependency is used to lookup any either any
+          // generated non-local classes or produced xsbti.API associated with the file.


Does incremental compilation works at all with a tasty-only classpath? What happens if we just always pass to sbt the name of a .class file even if it doesn't exist?

What happens if we just always pass to sbt the name of a .class file even if it doesn't exist?

I guess that would lead to overcompilation:
https://github.com/lampepfl/dotty/blob/a73316af791502b82525471816fe9153c36f845a/compiler/src/dotty/tools/dotc/core/SymDenotations.scala#L536-L540
But what happens if we pass the .tasty file instead?

We decided that whatever we record as the binaryDependency will need to be emitted eventually as a real class file with generatedNonLocalClass

bishabosha · 2023-07-31T19:10:54Z

one CI failure so far for ea75836 (scala3-staging/Compile/doc task) but I couldn't reproduce, running CI again

compiler/src/dotty/tools/dotc/sbt/ExtractDependencies.scala

bishabosha · 2023-08-02T13:22:57Z

test performance with #sbt please: ec59c31 2dfc28f

dottybot · 2023-08-02T13:24:47Z

performance test scheduled for githubcomlampepfldottypull18266commits2dfc28ff27c14e33cf110456a7ddb6b596fb4525: 92 job(s) in queue, 1 running.

dottybot · 2023-08-02T14:20:48Z

performance test failed:

Please check https://dotty-bench.epfl.ch/logs/ for more information

Kordyjan · 2023-12-01T13:16:33Z

@bishabosha Does this also need to wait for #17594 backport?

bishabosha · 2023-12-01T13:21:43Z

Does this also need to wait for #17594 backport?

that's right

bishabosha requested a review from nicolasstucki July 21, 2023 15:38

bishabosha assigned nicolasstucki Jul 21, 2023

bishabosha commented Jul 21, 2023

View reviewed changes

nicolasstucki reviewed Jul 24, 2023

View reviewed changes

compiler/src/dotty/tools/dotc/core/Contexts.scala Outdated Show resolved Hide resolved

bishabosha force-pushed the reduce-extract-deps-io branch from e4d25ef to 314e731 Compare July 24, 2023 09:16

bishabosha requested a review from nicolasstucki July 24, 2023 13:17

nicolasstucki reviewed Jul 25, 2023

View reviewed changes

smarter reviewed Jul 25, 2023

View reviewed changes

compiler/src/dotty/tools/dotc/sbt/ExtractDependencies.scala Outdated Show resolved Hide resolved

smarter reviewed Jul 25, 2023

View reviewed changes

bishabosha force-pushed the reduce-extract-deps-io branch from dd5d0df to 1e5ac78 Compare July 31, 2023 13:47

bishabosha added 4 commits July 31, 2023 15:51

reduce file-io in extractDependencies

087fdd0

cache associated classfile of tasty

6f37050

compute class name once

1bc36a6

try only plain classfile

529eaf4

bishabosha force-pushed the reduce-extract-deps-io branch from 1e5ac78 to 529eaf4 Compare July 31, 2023 13:51

revert context cache, use local cache, always give classfile

ea75836

bishabosha requested review from smarter and nicolasstucki July 31, 2023 16:31

bishabosha assigned smarter and unassigned nicolasstucki Jul 31, 2023

bishabosha force-pushed the reduce-extract-deps-io branch from 91e2b7e to daf159c Compare August 2, 2023 12:12

smarter approved these changes Aug 2, 2023

View reviewed changes

compiler/src/dotty/tools/dotc/sbt/ExtractDependencies.scala Outdated Show resolved Hide resolved

compiler/src/dotty/tools/dotc/sbt/ExtractDependencies.scala Outdated Show resolved Hide resolved

compiler/src/dotty/tools/dotc/sbt/ExtractDependencies.scala Outdated Show resolved Hide resolved

smarter assigned bishabosha Aug 2, 2023

smarter removed their assignment Aug 2, 2023

add todo, address comments

2dfc28f

bishabosha force-pushed the reduce-extract-deps-io branch from daf159c to 2dfc28f Compare August 2, 2023 12:35

bishabosha requested a review from smarter August 2, 2023 12:35

bishabosha enabled auto-merge (squash) August 2, 2023 12:41

smarter approved these changes Aug 2, 2023

View reviewed changes

bishabosha merged commit 1b79cb3 into scala:main Aug 2, 2023

bishabosha deleted the reduce-extract-deps-io branch August 2, 2023 16:14

bishabosha added the backport:nominated If we agree to backport this PR, replace this tag with "backport:accepted", otherwise delete it. label Aug 3, 2023

Kordyjan removed the backport:nominated If we agree to backport this PR, replace this tag with "backport:accepted", otherwise delete it. label Oct 10, 2023

bishabosha added the backport:nominated If we agree to backport this PR, replace this tag with "backport:accepted", otherwise delete it. label Oct 11, 2023

Kordyjan added this to the 3.4.0 milestone Dec 20, 2023

WojciechMazur removed the backport:nominated If we agree to backport this PR, replace this tag with "backport:accepted", otherwise delete it. label Nov 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reduce IO calls in extractDependencies #18266

Reduce IO calls in extractDependencies #18266

bishabosha commented Jul 21, 2023 •

edited

Loading

bishabosha commented Jul 21, 2023

dottybot commented Jul 21, 2023

bishabosha Jul 21, 2023

nicolasstucki Jul 25, 2023

dottybot commented Jul 21, 2023

bishabosha commented Jul 21, 2023

bishabosha commented Jul 24, 2023 •

edited

Loading

nicolasstucki Jul 25, 2023

nicolasstucki commented Jul 25, 2023

smarter Jul 25, 2023

smarter Jul 29, 2023

bishabosha Jul 31, 2023 •

edited

Loading

bishabosha commented Jul 31, 2023 •

edited

Loading

bishabosha commented Aug 2, 2023

dottybot commented Aug 2, 2023

dottybot commented Aug 2, 2023

Kordyjan commented Dec 1, 2023

bishabosha commented Dec 1, 2023

Reduce IO calls in extractDependencies #18266

Reduce IO calls in extractDependencies #18266

Conversation

bishabosha commented Jul 21, 2023 • edited Loading

bishabosha commented Jul 21, 2023

dottybot commented Jul 21, 2023

bishabosha Jul 21, 2023

Choose a reason for hiding this comment

nicolasstucki Jul 25, 2023

Choose a reason for hiding this comment

dottybot commented Jul 21, 2023

bishabosha commented Jul 21, 2023

bishabosha commented Jul 24, 2023 • edited Loading

nicolasstucki Jul 25, 2023

Choose a reason for hiding this comment

nicolasstucki commented Jul 25, 2023

smarter Jul 25, 2023

Choose a reason for hiding this comment

smarter Jul 29, 2023

Choose a reason for hiding this comment

bishabosha Jul 31, 2023 • edited Loading

Choose a reason for hiding this comment

bishabosha commented Jul 31, 2023 • edited Loading

bishabosha commented Aug 2, 2023

dottybot commented Aug 2, 2023

dottybot commented Aug 2, 2023

Kordyjan commented Dec 1, 2023

bishabosha commented Dec 1, 2023

bishabosha commented Jul 21, 2023 •

edited

Loading

bishabosha commented Jul 24, 2023 •

edited

Loading

bishabosha Jul 31, 2023 •

edited

Loading

bishabosha commented Jul 31, 2023 •

edited

Loading