Skip to content
This repository has been archived by the owner on Oct 7, 2020. It is now read-only.

Use preview of pipeline caching in azure builds #1335

Merged
merged 13 commits into from
Jul 30, 2019
Merged

Use preview of pipeline caching in azure builds #1335

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
cab65cd
Add cache for stack-root and stack-work
jneira Jul 16, 2019
78b443c
Add name as key and move STACK_ROOT to variables
jneira Jul 16, 2019
efb4838
Delete ghc at the end
jneira Jul 18, 2019
19f3b12
Delete setup-exe-cache
jneira Jul 18, 2019
7f895f3
Clean exec at the start
jneira Jul 18, 2019
65c1249
Pack/unpack cache to keep files attributes, see https://github.com/mi…
jneira Jul 19, 2019
b639b21
Add cacheHitVar to make conditions work
jneira Jul 19, 2019
3db2e7f
Change file extension to avoid docker auto extraction
jneira Jul 19, 2019
7ddebe1
Put tar files inside a dir
jneira Jul 19, 2019
8c787c6
Fix cache hit var name
jneira Jul 20, 2019
6dde4cc
Extract in the correct dir
jneira Jul 21, 2019
6a976ab
Apply changes from linux to macos
jneira Jul 21, 2019
e077fd8
Avoid unnecessary error creating cache dir
jneira Jul 22, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 23 additions & 1 deletion .azure/linux-stack.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,13 +25,30 @@ jobs:
YAML_FILE: stack-8.4.2.yaml
stack-8.2.2:
YAML_FILE: stack-8.2.2.yaml
variables:
STACK_ROOT: /home/vsts/.stack
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should be $(HOME)/.stack but it seems it cant be used in build variables: https://github.com/MicrosoftDocs/vsts-docs/issues/4337

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two possibilities:
If you can execute a command, something like stack path --stack-root should give you the root.
Otherwise, I think you can change the location of ~/.stack which may also be a solution.

Copy link
Member Author

@jneira jneira Jul 22, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

variables is defined prior any other command, so its possible values are limited to predefined variables and expressions. We would have to set in in the .bashrc to use stack path.
I used the default one cause in macos write outside the home directory throwed permission errors (if it doesnt i would use /.stack).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, just in case, values defined in variables are set in all script steps, so stack must use the definition.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest adding a line to the bashrc-files that sets the stack-root since we have other env-variables there. The path uses the same trick: The path variable could only be overwritten, not appended when using the variables section on the yaml

Copy link
Member Author

@jneira jneira Jul 23, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now we could move the variable to bashrc but if we want to use it directly in the CacheBeta task, like it is in the windows script (and hopefully like we will able to use in linux and macos) the variable only can be in the variables block, cause we can't load the bashrc file in any task but bash one, and it limits somewhat its possible use.
Otoh there is an issue to make possible use the home directory as initial var (we only need the home directory to make sure we can write in the given path for macos)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't think of the issue that the task cannot load the bashrc file. Having a variable does indeed seem to be the best solution.

steps:
- task: CacheBeta@0
inputs:
key: |
"cache"
$(Agent.OS)
$(Build.SourcesDirectory)/$(YAML_FILE)
path: .azure-cache
cacheHitVar: CACHE_RESTORED
displayName: "Download cache"
- bash: |
mkdir -p $STACK_ROOT
tar -xzf .azure-cache/stack-root.tar.gz -C /
mkdir -p .stack-work
tar -xzf .azure-cache/stack-work.tar.gz
displayName: "Unpack cache"
condition: eq(variables.CACHE_RESTORED, 'true')
- bash: |
git submodule sync
git submodule update --init
displayName: Sync submodules
- bash: |
export STACK_ROOT="$(Build.SourcesDirectory)"/.stack-root
mkdir -p ~/.local/bin
curl -L https://get.haskellstack.org/stable/linux-x86_64.tar.gz | \
tar xz --wildcards --strip-components=1 -C ~/.local/bin '*/stack'
Expand Down Expand Up @@ -66,3 +83,8 @@ jobs:
# source .azure/linux.bashrc
# stack test --stack-yaml $(YAML_FILE)
# displayName: Run Test
- bash: |
mkdir -p .azure-cache
tar -czf .azure-cache/stack-root.tar.gz $STACK_ROOT
tar -czf .azure-cache/stack-work.tar.gz .stack-work
displayName: "Pack cache"
24 changes: 23 additions & 1 deletion .azure/macos-stack.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,13 +21,30 @@ jobs:
YAML_FILE: stack-8.4.2.yaml
stack-8.2.2:
YAML_FILE: stack-8.2.2.yaml
variables:
STACK_ROOT: /Users/vsts/.stack
steps:
- task: CacheBeta@0
inputs:
key: |
"cache"
$(Agent.OS)
$(Build.SourcesDirectory)/$(YAML_FILE)
path: .azure-cache
cacheHitVar: CACHE_RESTORED
displayName: "Download cache"
- bash: |
mkdir -p $STACK_ROOT
tar -xzf .azure-cache/stack-root.tar.gz -C /
mkdir -p .stack-work
tar -xzf .azure-cache/stack-work.tar.gz
displayName: "Unpack cache"
condition: eq(variables.CACHE_RESTORED, 'true')
- bash: |
git submodule sync
git submodule update --init
displayName: Sync submodules
- bash: |
export STACK_ROOT="$(Build.SourcesDirectory)"/.stack-root
mkdir -p ~/.local/bin
curl -skL https://get.haskellstack.org/stable/osx-x86_64.tar.gz | \
tar xz --strip-components=1 --include '*/stack' -C ~/.local/bin;
Expand Down Expand Up @@ -63,3 +80,8 @@ jobs:
# source .azure/macos.bashrc
# stack test --stack-yaml $(YAML_FILE)
# displayName: Run Test
- bash: |
mkdir -p .azure-cache
tar -czf .azure-cache/stack-root.tar.gz $STACK_ROOT
tar -czf .azure-cache/stack-work.tar.gz .stack-work
displayName: "Pack cache"
19 changes: 19 additions & 0 deletions .azure/windows-stack.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,26 @@ jobs:
YAML_FILE: stack-8.4.2.yaml
stack-8.2.2:
YAML_FILE: stack-8.2.2.yaml
variables:
STACK_ROOT: "C:\\sr"

steps:
- task: CacheBeta@0
inputs:
key: |
"stack-root"
$(Agent.OS)
$(Build.SourcesDirectory)/$(YAML_FILE)
path: $(STACK_ROOT)
displayName: "Cache stack-root"
- task: CacheBeta@0
inputs:
key: |
"stack-work"
$(Agent.OS)
$(Build.SourcesDirectory)/$(YAML_FILE)
path: .stack-work
displayName: "Cache stack-work"
- bash: |
git submodule sync
git submodule update --init
Expand Down
1 change: 0 additions & 1 deletion .azure/windows.bashrc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
export STACK_ROOT="C:\\sr"
export LOCAL_BIN_PATH=$(cygpath $APPDATA\\local\\bin)
export Z3_BIN_PATH=/usr/local/z3-4.8.5-x64-win/bin
export PATH=$Z3_BIN_PATH:$LOCAL_BIN_PATH:$PATH