Skip to content

Commit

Permalink
[Test-Proxy] Windows Image, Additional Documentation, Certificate Upd…
Browse files Browse the repository at this point in the history
…ate. (#1879)

* Lots of documentation updates (around SSL and the like)
* Adding `windows` version of the container, so that our windows machines in devops can load
* Renaming from `ubuntu_testproxy` to `testproxy` because the manifest will figure out which one you need
* Updating the default `pfx` that we install on the docker images to have a password, such that you don't hit issues when you try to import it with `-p ""`. On pshell it can take that as "not passing" a required argument. Lets just avoid that.
  • Loading branch information
scbedd authored Aug 6, 2021
1 parent 09ae0a8 commit 653e51f
Show file tree
Hide file tree
Showing 15 changed files with 304 additions and 78 deletions.
1 change: 1 addition & 0 deletions .github/CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
Expand Up @@ -13,3 +13,4 @@
/tools/webhook-router/ @weshaggard @praveenkuttappan
/tools/github-issues/ @weshaggard @praveenkuttappan @AlexGhiondea
/tools/pipeline-generator/ @weshaggard @benbp
/tools/test-proxy/ @scbedd @mikeharder
85 changes: 47 additions & 38 deletions eng/containers/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,18 @@ parameters:
- name: DockerDeployments
type: object
default:
- name: test-proxy
dockerRepo: 'engsys/ubuntu_testproxy_server'
- name: test_proxy_linux
pool: 'ubuntu-20.04'
dockerRepo: 'engsys/testproxy-lin'
dockerFile: 'tools/test-proxy/docker/dockerfile'
stableTags:
- 'latest'
- name: test_proxy_windows
pool: 'windows-2019'
dockerRepo: 'engsys/testproxy-win'
dockerFile: 'tools/test-proxy/docker/dockerfile-win'
stableTags:
- 'latest'

trigger:
branches:
Expand All @@ -15,48 +22,50 @@ trigger:
paths:
include:
- eng/containers/
- tools/test-proxy/dockerfile
- tools/test-proxy/docker/

pr: none

pool:
vmImage: 'ubuntu-20.04'

variables:
- name: containerRegistry
value: 'azsdkengsys'
- name: imageTag
value: $(build.buildid)
- template: ../pipelines/templates/variables/globals.yml

steps:
- ${{ each config in parameters.DockerDeployments }}:
- task: Docker@2
displayName: Build ${{ config.name }}:$(imageTag)
inputs:
command: build
Dockerfile: ${{ config.dockerFile }}
tags: $(imageTag)
arguments: '-t $(containerRegistry).azurecr.io/${{ config.dockerRepo }}:$(imageTag)'
- task: Docker@2
displayName: Push ${{ config.name }}:$(imageTag)
inputs:
containerRegistry: $(containerRegistry)
repository: ${{ config.dockerRepo }}
command: push
tags: $(imageTag)
- ${{ each stableTag in config.stableTags }}:
- task: Docker@2
displayName: Build ${{ config.name }}:${{ stableTag }}
inputs:
command: build
Dockerfile: ${{ config.dockerFile }}
tags: ${{ stableTag }}
arguments: '-t $(containerRegistry).azurecr.io/${{ config.dockerRepo }}:${{ stableTag }}'
- task: Docker@2
displayName: Push ${{ config.name }}:${{ stableTag }}
inputs:
containerRegistry: $(containerRegistry)
repository: ${{ config.dockerRepo }}
command: push
tags: ${{ stableTag }}

jobs:
- ${{ each config in parameters.DockerDeployments }}:
- job: container_build_${{ config.name }}
displayName: Deploy ${{ config.name }} Image
pool:
vmImage: ${{ config.pool }}
steps:
- task: Docker@2
displayName: Build ${{ config.name }}:$(imageTag)
inputs:
command: build
Dockerfile: ${{ config.dockerFile }}
tags: $(imageTag)
arguments: '-t $(containerRegistry).azurecr.io/${{ config.dockerRepo }}:$(imageTag)'
- task: Docker@2
displayName: Push ${{ config.name }}:$(imageTag)
inputs:
containerRegistry: $(containerRegistry)
repository: ${{ config.dockerRepo }}
command: push
tags: $(imageTag)
- ${{ each stableTag in config.stableTags }}:
- task: Docker@2
displayName: Build ${{ config.name }}:${{ stableTag }}
inputs:
command: build
Dockerfile: ${{ config.dockerFile }}
tags: ${{ stableTag }}
arguments: '-t $(containerRegistry).azurecr.io/${{ config.dockerRepo }}:${{ stableTag }}'
- task: Docker@2
displayName: Push ${{ config.name }}:${{ stableTag }}
inputs:
containerRegistry: $(containerRegistry)
repository: ${{ config.dockerRepo }}
command: push
tags: ${{ stableTag }}
2 changes: 1 addition & 1 deletion tools/test-proxy/Azure.Sdk.Tools.TestProxy/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ Feel free to build the docker file locally within working directory `/tools/test
Or, leverage the azure sdk eng sys container registry.

```powershell
> docker run -v <your-volume-name-or-location>:/etc/testproxy -p 5001:5001 -p 5000:5000 azsdkengsys.azurecr.io/engsys/ubuntu_testproxy_server:latest
> docker run -v <your-volume-name-or-location>:/etc/testproxy -p 5001:5001 -p 5000:5000 azsdkengsys.azurecr.io/engsys/testproxy:latest
```

Note in both cases you will need to provide the port and volume mapping as arguments.
Expand Down
35 changes: 16 additions & 19 deletions tools/test-proxy/docker/README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,12 @@
# Common Docker Questions


## Local Generation of the test-proxy image

### Build and Run

First, navigate to the folder containing the test-proxy dockerfile: `tools/test-proxy/docker`.

Invoke to generate a container (with optional tag):

```docker
Expand All @@ -24,34 +27,27 @@ If you _don't_ provide a volume bound to `/etc/testproxy`, it's not actually the
docker cp <containerid>:/etc/testproxy/ <target local path local path>
```

### Certificates
### Windows Container

All necessary components for dev-certificate usage are present within the `dev_certificate` directory.
Actions images do not support swapping between `Linux` and `Windows` containers. This lack of support means that the azure-sdk engsys must also provide a test-proxy container for windows containers.

**Note that this certificate was generated with no password.**
To build _that_, use the following command.

Within are components of a **dev certificate** that has no usage outside of keeping your local usage of SSL happy. When running the container, you will need to trust this certificate if you want to connect to `https://localhost:5001` without cert validation failures. These certificates have no usage outside of your local box.

```powershell
# ensure root access
> $rootCert = $(Import-PfxCertificate -FilePath ./dev_certificate/dotnet-devcert.pfx -CertStoreLocation 'Cert:\LocalMachine\Root')
```docker
docker build . -f dockerfile-win -t test-proxy
```

or via `dotnet`

```powershell
dotnet dev-certs https --clean --import ./dotnet-devcert.pfx --password=""
```
### Certificates

Or add and trust with the toolchain of your choice.
All necessary components for dev-certificate usage are present within the `dev_certificate` directory. Reference [trusting-cert-per-language.md](../documentation/trusting-cert-per-language.md) to learn how to add and trust with the toolchain of your choice.

On a ubuntu-flavored distro of linux, feel free to re-use the import mechanism in the local file `import-dev-cert.sh`. Prior to using locally, ensure $CERT_FOLDER environment variable is set to the local directory `dev_certificate` to access necessary files!
Please note that each language + its SSL stack will provide different mechanisms for validating SSL certificates. Again, reference [trusting-cert-per-language.md](../documentation/trusting-cert-per-language.md) to understand the process beyond the most general case.

### Confirm Success

Run the container and attempt a `curl https://localhost:5001/Admin/IsAlive`.

Do you see activity in docker logs + a successfully 200 response?
Do you see activity in docker logs + a 200 response?

### Acknowledgement

Expand All @@ -64,8 +60,8 @@ Most issues we've seen are related to having a prior `az acr login` or the like.
If your error looks something like this:

```
> docker pull azsdkengsys.azurecr.io/engsys/ubuntu_testproxy_server:latest
Error response from daemon: Head https://azsdkengsys.azurecr.io/v2/engsys/ubuntu_testproxy_server/manifests/latest: unauthorized: authentication required
> docker pull azsdkengsys.azurecr.io/engsys/testproxy:latest
Error response from daemon: Head https://azsdkengsys.azurecr.io/v2/engsys/testproxy/manifests/latest: unauthorized: authentication required
```

Need to clear multiple sets of credentials.
Expand All @@ -79,7 +75,7 @@ This occurs when a user has a **prior login** to `azsdkengsys.azurecr.io`. `az a
For errors that look like:

```
> docker pull azsdkengsys.azurecr.io/engsys/ubuntu_testproxy_server:latest
> docker pull azsdkengsys.azurecr.io/engsys/testproxy:latest
Error response from daemon: Get https://azsdkengsys.azurecr.io/v2/: x509: certificate has expired or is not yet valid
```

Expand All @@ -88,3 +84,4 @@ Open up docker desktop and click the bug.
![image](https://user-images.githubusercontent.com/45376673/126579279-5048132c-39c0-4b40-a3b2-6da03553097b.png)

Then click `Restart`. Reference [this stack overflow](https://stackoverflow.com/questions/35289802/docker-pull-error-x509-certificate-has-expired-or-is-not-yet-valid) post. The docker daemon clock doesn't stay synced with windows, which causes these certificate failures.

2 changes: 1 addition & 1 deletion tools/test-proxy/docker/dev_certificate/apply-dev-cert.sh
Original file line number Diff line number Diff line change
Expand Up @@ -28,4 +28,4 @@ fi
$SUDO cp $CRTFILE "/usr/local/share/ca-certificates"
$SUDO update-ca-certificates

dotnet dev-certs https --clean --import $PFXFILE -p ""
dotnet dev-certs https --clean --import $PFXFILE -p "password"
30 changes: 15 additions & 15 deletions tools/test-proxy/docker/dev_certificate/dotnet-devcert.crt
Original file line number Diff line number Diff line change
@@ -1,20 +1,20 @@
-----BEGIN CERTIFICATE-----
MIIDSDCCAjCgAwIBAgIUQNoz8JVYNTmMQF17s1FN2QH5iD4wDQYJKoZIhvcNAQEL
BQAwFDESMBAGA1UEAwwJbG9jYWxob3N0MB4XDTIxMDUxNjIzMjgxN1oXDTIyMDUx
NjIzMjgxN1owFDESMBAGA1UEAwwJbG9jYWxob3N0MIIBIjANBgkqhkiG9w0BAQEF
AAOCAQ8AMIIBCgKCAQEAwbJFEHvlNYkOEXh6KGhHufLG2ue7BFuM2MgEUSW7cJLN
8mNiFAEikR7iPqcC9NSKoQXLq4b3WYA3+pIk4f/32U/SoxTvjktUc3UZ+Lz8Tj6c
itMacLyWlFyk9aPu26z9awQisXA+pNbLsgwkYkfyCOTCK9pK72EAop8Sw16euTun
Emdk8it7t/oSD5E3qGeHakVLy2qHpRFYR/i/0Vph8f0Prfdfvoyl/s9WlJjhC10z
DGk+pZ18Bw/CKKjJ3KYBvUEtaCrRlODInBLP784IXDN6fqyBLxMzg4nZRDEQC0jp
f0Dks473MWswZBjASoEiQygy8OXeOS6o9aXTi/+HwwIDAQABo4GRMIGOMA8GA1Ud
MIIDSDCCAjCgAwIBAgIUPMKpJ/j10eQrcQBNnkImIaOYHakwDQYJKoZIhvcNAQEL
BQAwFDESMBAGA1UEAwwJbG9jYWxob3N0MB4XDTIxMDgwNTAwMzU1NloXDTIyMDgw
NTAwMzU1NlowFDESMBAGA1UEAwwJbG9jYWxob3N0MIIBIjANBgkqhkiG9w0BAQEF
AAOCAQ8AMIIBCgKCAQEAxe/ZseXgOTVoF7uTjX5Leknk95jIoyGc+VlxA8BhzGOr
r4u6VNQZRCMq+svHY36tW4+u/xHNe2kvbwy2mnS8cFFLfst+94qBZVJDBxSGZ9I/
wekErNsjFsik4UrMvcC+ZlGPh7hb3f7tSx29tn1DIkAUXVnbZ6TT5s+mYRQpZ6fW
6kR3RNfc0A1IUM7Zs9yfNEr0O2H41P2HcLKoOPtvd7GvTQm9Ofh3srKvII+sZn/J
WH7r76oRQMX904mOMdryQwZLObsqX4dXIEbafKVSecB3PBVIhv8gVtJhcZbQP1pI
mMiWd6PHv46ZhGf7+cKnYUSa8Ia2t/wetK1wd00dFwIDAQABo4GRMIGOMA8GA1Ud
EwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgGmMBYGA1UdJQEB/wQMMAoGCCsGAQUF
BwMBMBcGA1UdEQEB/wQNMAuCCWxvY2FsaG9zdDA6BgorBgEEAYI3VAEBBCwMKkFT
UC5ORVQgQ29yZSBIVFRQUyBkZXZlbG9wbWVudCBjZXJ0aWZpY2F0ZTANBgkqhkiG
9w0BAQsFAAOCAQEAfCNALyjTjiOeCVXJ7UCozjovH5LXxiIrkQISYKI36UV7vUDr
iZHrTZvLYhXRgdGs56YOUUajWItl+6W0lIBS6yKAN3Dq6UvJJ2a6UYuaq36thQdk
zNa7HTWhEan9Zri1KCsTrhCLjDd1X6txanbwsn1GLI4F0Wt0FwZoYxVB1FmsTT4c
sqSslz4zUTISBMAzaUQbFKFPUkxieg4AtB+et5qM1ADccrm5nm5zMMWNiP1kPrw9
jg5jmTqdAcIa5w1bv1CnorAwbhghPRrPVaoSvocwVimPTWrhQHPNNyQsQyeMuQkr
OSDIoP0MMiJ92/8pvwoE373qEW9A+0ImIo2YEw==
9w0BAQsFAAOCAQEAIj2VlBVcXGSly6KCBg6lgwFi+henWfSox77iuGAaAxDjN3jd
9lZahW4MPNLHKSrPRb4YNSLZ2jh7zdcttQrqd4qH65o1q56q5JrCmli99iIzY9Y8
RdYyxK4Zzr31wjpsyFiWQfqJTuSFUUg9uDDj0negwEZLIGlt7nr12wflt2+QOJtD
byMeSZLbB5dPzn341DK0qfJEJMMgL0XsPEVZ3TQ6Alc9zq5wI608C/mXnz3xJE05
UTYD8pRJJ/DyG0empvOVE8Sg93msHPquAbgqO9aqCpykgg/a8CFvI4wRdfvGEFlv
8XJKL8Y/PFsmFeO3axq3zUYKFVdc9Un4dFIaag==
-----END CERTIFICATE-----
Binary file modified tools/test-proxy/docker/dev_certificate/dotnet-devcert.pfx
Binary file not shown.
23 changes: 23 additions & 0 deletions tools/test-proxy/docker/dev_certificate/localhost.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
[req]
prompt = no
default_bits = 2048
distinguished_name = subject
req_extensions = req_ext
x509_extensions = x509_ext

[ subject ]
commonName = localhost

[req_ext]
basicConstraints = critical, CA:true
subjectAltName = @alt_names

[x509_ext]
basicConstraints = critical, CA:true
keyUsage = critical, keyCertSign, cRLSign, digitalSignature,keyEncipherment
extendedKeyUsage = critical, serverAuth
subjectAltName = critical, @alt_names
1.3.6.1.4.1.311.84.1.1 = ASN1:UTF8String:ASP.NET Core HTTPS development certificate # Needed to get it imported by dotnet dev-certs

[alt_names]
DNS.1 = localhost
2 changes: 1 addition & 1 deletion tools/test-proxy/docker/dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ RUN chmod +x $CERT_FOLDER/$CERT_IMPORT_SH \
RUN dotnet tool install azure.sdk.tools.testproxy \
--global \
--add-source https://pkgs.dev.azure.com/azure-sdk/public/_packaging/azure-sdk/nuget/v3/index.json \
--version 1.0.0-dev.20210722.1 \
--version 1.0.0-dev.20210729.1 \
&& mkdir /etc/testproxy

EXPOSE 5001
Expand Down
93 changes: 93 additions & 0 deletions tools/test-proxy/docker/dockerfile-win
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
FROM mcr.microsoft.com/powershell:nanoserver AS installer-env

ARG PS_VERSION=7.1.0
ARG PS_PACKAGE_URL=https://github.com/PowerShell/PowerShell/releases/download/v$PS_VERSION/PowerShell-$PS_VERSION-win-x64.zip

ENV POWERSHELL_TELEMETRY_OPTOUT="1"

SHELL ["pwsh", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]

ARG PS_PACKAGE_URL_BASE64

RUN Write-host "Verifying valid Version..."; \
if (!($env:PS_VERSION -match '^\d+\.\d+\.\d+(-\w+(\.\d+)?)?$' )) { \
throw ('PS_Version ({0}) must match the regex "^\d+\.\d+\.\d+(-\w+(\.\d+)?)?$"' -f $env:PS_VERSION) \
} \
$ProgressPreference = 'SilentlyContinue'; \
Write-host "using url: $env:PS_PACKAGE_URL" ;\
$url = $env:PS_PACKAGE_URL ; \
Write-host "downloading: $url" ; \
[Net.ServicePointManager]::SecurityProtocol = [Net.ServicePointManager]::SecurityProtocol -bor [Net.SecurityProtocolType]::Tls12; \
New-Item -ItemType Directory /installer > $null ; \
Invoke-WebRequest -Uri $url -outfile /installer/powershell.zip -verbose; \
Expand-Archive /installer/powershell.zip -DestinationPath \PowerShell

FROM mcr.microsoft.com/dotnet/sdk:5.0.302-nanoserver-1809 AS build

# Copy PowerShell Core from the installer container
ENV ProgramFiles="C:\Program Files" \
# set a fixed location for the Module analysis cache
PSModuleAnalysisCachePath="C:\Users\Public\AppData\Local\Microsoft\Windows\PowerShell\docker\ModuleAnalysisCache" \
# Persist %PSCORE% ENV variable for user convenience
PSCORE="$ProgramFiles\PowerShell\pwsh.exe" \
# Set the default windows path so we can use it
WindowsPATH="C:\Windows\system32;C:\Windows" \
NO_AT_BRIDGE=1 \
ASPNETCORE_ENVIRONMENT=Development

USER ContainerAdministrator
RUN setx PATH "%PATH%;%ProgramFiles%\PowerShell;" /M
USER ContainerUser

COPY --from=installer-env ["\\PowerShell\\", "$ProgramFiles\\PowerShell"]

# intialize powershell module cache
RUN pwsh \
-NoLogo \
-NoProfile \
-Command " \
$stopTime = (get-date).AddMinutes(15); \
$ErrorActionPreference = 'Stop' ; \
$ProgressPreference = 'SilentlyContinue' ; \
while(!(Test-Path -Path $env:PSModuleAnalysisCachePath)) { \
Write-Host "'Waiting for $env:PSModuleAnalysisCachePath'" ; \
if((get-date) -gt $stopTime) { throw 'timout expired'} \
Start-Sleep -Seconds 6 ; \
}"

RUN mkdir certwork
RUN mkdir testproxy

ADD dev_certificate/dotnet-devcert.pfx certwork
ADD dev_certificate/dotnet-devcert.crt certwork

USER ContainerAdministrator
RUN dotnet dev-certs https --clean --import /certwork/dotnet-devcert.pfx -p "password"
USER ContainerUser

# this is the workaround for the inability to "trust" a certificate due to permissions issues on the windows platform
# we would obviously prefer to use...
# RUN dotnet dev-certs https --trust
# ...to trust the cert we imported above. However, due to the fact that a UI is REQUIRED to assent to this specific tool,
# it won't work on docker. ADDITIONALLY, we can't use the methodology of using powershell to insert into the LocalMachine/Root
# as even if we surround it in a USER ContainerAdministrator to elevate our access, we STILL hit weird permission denied errors
# and he ASP.NET startup still can't load it as a dev certificate.
ENV ASPNETCORE_Kestrel__Certificates__Default__Path="/certwork/dotnet-devcert.pfx" \
ASPNETCORE_Kestrel__Certificates__Default__Password="password"

SHELL ["pwsh", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]

# install the package
RUN dotnet tool install azure.sdk.tools.testproxy \
--tool-path /proxyserver \
--add-source https://pkgs.dev.azure.com/azure-sdk/public/_packaging/azure-sdk/nuget/v3/index.json \
--version 1.0.0-dev.20210729.1

EXPOSE 5001
EXPOSE 5000

# this override allows the tool server to listen to traffic over the docker bridge.
# default URL of localhost:5000 or localhost:50001 are not usable from outside the container
ENV DOTNET_URLS="http://0.0.0.0:5000;https://0.0.0.0:5001"

ENTRYPOINT ["/proxyserver/test-proxy.exe", "--storage-location", "/testproxy"]
Loading

0 comments on commit 653e51f

Please sign in to comment.