Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove the need for intermediate .zip file in PackageSupplier #2876

Merged
merged 3 commits into from
Feb 24, 2024

Conversation

Geod24
Copy link
Member

@Geod24 Geod24 commented Feb 22, 2024

By using a different overload of `retryDownload`,
we return the data instead of writing to disk,
ensuring that we don't have a lingering file
in memory and reducing IO.

This follow the same approach for deprecations as previous `PackageSupplier` deprecations: it's an inevitable, but easily fixable, breaking change for classes that implement `PackageSupplier`, but it's a proper deprecation for client code, based on the assumption that all the classes implementing `PackageSupplier` are in Dub.

@Geod24 Geod24 requested a review from s-ludwig February 22, 2024 02:15
Copy link

github-actions bot commented Feb 22, 2024

✅ PR OK, no changes in deprecations or warnings

Total deprecations: 0

Total warnings: 0

Build statistics:

 statistics (-before, +after)
-executable size=5368480 bin/dub
-rough build time=61s
+executable size=5356064 bin/dub
+rough build time=62s
Full build output
DUB version 1.35.1, built on Jan  6 2024
LDC - the LLVM D compiler (1.36.0):
  based on DMD v2.106.1 and LLVM 17.0.6
  built with LDC - the LLVM D compiler (1.36.0)
  Default target: x86_64-unknown-linux-gnu
  Host CPU: znver3
  http://dlang.org - http://wiki.dlang.org/LDC


  Registered Targets:
    aarch64     - AArch64 (little endian)
    aarch64_32  - AArch64 (little endian ILP32)
    aarch64_be  - AArch64 (big endian)
    amdgcn      - AMD GCN GPUs
    arm         - ARM
    arm64       - ARM64 (little endian)
    arm64_32    - ARM64 (little endian ILP32)
    armeb       - ARM (big endian)
    avr         - Atmel AVR Microcontroller
    bpf         - BPF (host endian)
    bpfeb       - BPF (big endian)
    bpfel       - BPF (little endian)
    hexagon     - Hexagon
    lanai       - Lanai
    loongarch32 - 32-bit LoongArch
    loongarch64 - 64-bit LoongArch
    mips        - MIPS (32-bit big endian)
    mips64      - MIPS (64-bit big endian)
    mips64el    - MIPS (64-bit little endian)
    mipsel      - MIPS (32-bit little endian)
    msp430      - MSP430 [experimental]
    nvptx       - NVIDIA PTX 32-bit
    nvptx64     - NVIDIA PTX 64-bit
    ppc32       - PowerPC 32
    ppc32le     - PowerPC 32 LE
    ppc64       - PowerPC 64
    ppc64le     - PowerPC 64 LE
    r600        - AMD GPUs HD2XXX-HD6XXX
    riscv32     - 32-bit RISC-V
    riscv64     - 64-bit RISC-V
    sparc       - Sparc
    sparcel     - Sparc LE
    sparcv9     - Sparc V9
    spirv32     - SPIR-V 32-bit
    spirv64     - SPIR-V 64-bit
    systemz     - SystemZ
    thumb       - Thumb
    thumbeb     - Thumb (big endian)
    ve          - VE
    wasm32      - WebAssembly 32-bit
    wasm64      - WebAssembly 64-bit
    x86         - 32-bit X86: Pentium-Pro and above
    x86-64      - 64-bit X86: EM64T and AMD64
    xcore       - XCore
   Upgrading project in /home/runner/work/dub/dub/
    Starting Performing "release" build using /opt/hostedtoolcache/dc/ldc2-1.36.0/x64/ldc2-1.36.0-linux-x86_64/bin/ldc2 for x86_64.
    Building dub 1.36.0+commit.117.gca352040: building configuration [application]
     Linking dub
STAT:statistics (-before, +after)
STAT:executable size=5356064 bin/dub
STAT:rough build time=62s

@thewilsonator
Copy link
Contributor

This fails CI

@Geod24
Copy link
Member Author

Geod24 commented Feb 22, 2024

Yeah, because the registry actually issue a redirection to Github and retryDownload doesn't follow redirection. Will fix.

Actually it was just broken...

This makes the source of the issue more obvious when using
the verbose flag.
We were using the default template parameter, `char`,
and as the documentation for `get` states:
> If asking for char, content will be converted from
> the connection character set (specified in HTTP
> response headers or FTP connection properties,
> both ISO-8859-1 by default) to UTF-8.

However we are downloading a binary so converting
it to UTF-8 will fail. Especially if we intend
to cast it to `ubyte[]` afterwards.
By using a different overload of `retryDownload`,
we return the data instead of writing to disk,
ensuring that we don't have a lingering file
in memory and reducing IO.

This follow the same approach for deprecations as previous
`PackageSupplier` deprecations: it's an inevitable, but
easily fixable, breaking change for classes that implement
`PackageSupplier`, but it's a proper deprecation for client
code, based on the assumption that all the classes implementing
`PackageSupplier` are in Dub.
@Geod24 Geod24 force-pushed the mlang/FetchPackage branch from e6233b7 to c6a12b2 Compare February 22, 2024 10:56
@Geod24
Copy link
Member Author

Geod24 commented Feb 22, 2024

Yep that did it

@Geod24
Copy link
Member Author

Geod24 commented Feb 22, 2024

@s-ludwig : Your input here would be appreciated. Do you see a reason why this is the wrong direction ?

In the long run I think having the PackageSupplier do the extraction makes more sense, so we're not tied to .zip files, but I wanted to remove the intermediate file in the meantime. It seems wasteful considering the data will be loaded in memory multiple times (first via download, then via readFile).

@Geod24 Geod24 merged commit dd9e2f6 into dlang:master Feb 24, 2024
31 checks passed
@Geod24 Geod24 deleted the mlang/FetchPackage branch February 24, 2024 15:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants