Skip to content

Commit

Permalink
Facilitate running load tests inside WebHost (#55)
Browse files Browse the repository at this point in the history
* Extract Store CompositionRoot/Builders to common project so Web and CLI can share
* Add Web App Test target to CLI
  • Loading branch information
bartelink authored Dec 6, 2018
1 parent 5c6a822 commit ae9b2ee
Show file tree
Hide file tree
Showing 24 changed files with 935 additions and 418 deletions.
9 changes: 8 additions & 1 deletion Equinox.sln
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,9 @@ Project("{6EC3EE1D-3C4E-46DD-8F32-0CC8E7565705}") = "Equinox.Cosmos", "src\Equin
EndProject
Project("{6EC3EE1D-3C4E-46DD-8F32-0CC8E7565705}") = "Equinox.Cosmos.Integration", "tests\Equinox.Cosmos.Integration\Equinox.Cosmos.Integration.fsproj", "{DE0FEBF0-72DC-4D4A-BBA7-788D875D6B4B}"
EndProject
Project("{F2A71F9B-5D33-465A-A702-920D77279786}") = "Web", "samples\Store\Web\Web.fsproj", "{1B0D4568-96FD-4083-8520-CD537C0B2FF0}"
Project("{6EC3EE1D-3C4E-46DD-8F32-0CC8E7565705}") = "Web", "samples\Store\Web\Web.fsproj", "{1B0D4568-96FD-4083-8520-CD537C0B2FF0}"
EndProject
Project("{6EC3EE1D-3C4E-46DD-8F32-0CC8E7565705}") = "Infrastructure", "samples\Store\Infrastructure\Infrastructure.fsproj", "{ACE52D04-2FE3-4FD6-A066-9C81429C3997}"
EndProject
Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution
Expand Down Expand Up @@ -110,6 +112,10 @@ Global
{1B0D4568-96FD-4083-8520-CD537C0B2FF0}.Debug|Any CPU.Build.0 = Debug|Any CPU
{1B0D4568-96FD-4083-8520-CD537C0B2FF0}.Release|Any CPU.ActiveCfg = Release|Any CPU
{1B0D4568-96FD-4083-8520-CD537C0B2FF0}.Release|Any CPU.Build.0 = Release|Any CPU
{ACE52D04-2FE3-4FD6-A066-9C81429C3997}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{ACE52D04-2FE3-4FD6-A066-9C81429C3997}.Debug|Any CPU.Build.0 = Debug|Any CPU
{ACE52D04-2FE3-4FD6-A066-9C81429C3997}.Release|Any CPU.ActiveCfg = Release|Any CPU
{ACE52D04-2FE3-4FD6-A066-9C81429C3997}.Release|Any CPU.Build.0 = Release|Any CPU
EndGlobalSection
GlobalSection(SolutionProperties) = preSolution
HideSolutionNode = FALSE
Expand All @@ -120,6 +126,7 @@ Global
{406A280E-0708-4B12-8443-8FD5660CD271} = {D67D5A5F-2E59-4514-A997-FEBDC467AAF6}
{0B2D5815-D6A5-4AAC-9B75-D57B165E2A92} = {D67D5A5F-2E59-4514-A997-FEBDC467AAF6}
{1B0D4568-96FD-4083-8520-CD537C0B2FF0} = {D67D5A5F-2E59-4514-A997-FEBDC467AAF6}
{ACE52D04-2FE3-4FD6-A066-9C81429C3997} = {D67D5A5F-2E59-4514-A997-FEBDC467AAF6}
EndGlobalSection
GlobalSection(ExtensibilityGlobals) = postSolution
SolutionGuid = {177E1E7B-E275-4FC6-AE3C-2C651ECCF71E}
Expand Down
65 changes: 44 additions & 21 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,25 +89,45 @@ Run, including running the tests that assume you've got a local EventStore and p
## build, skip EventStore tests, skip auto-provisioning + de-provisioning Cosmos

./build -se -scp

# BENCHMARKS

A key facility of this repo is beoing able to run load tests, either in process against a nominated store, or via HTTP to a nominated Web app. The following tests are implemented at present:

- `Favorite` - Simulate a very enthusiastic user that favorites things once per Second - triggering an ever-growing state which can only work efficiently if you:
- apply a snapshotting scheme (although being unbounded, it will eventually hit the store's limits - 4MB/event for EventStore, 3MB/document for CosmosDb)
- apply caching on CosmosDb (so re-reading and transporting the snapshots is eliminated from the RU/bandwidth/latency costs)
- `SaveForLater` - Simulate a happy shopper that saves 3 items per second, and empties the Save For Later list whenever it is full (when it hits 50 items)
- Snapshotting helps a lot
- Caching is not as essential as it is for the `Favorite` test

## Run EventStore benchmark (when provisioned)

& .\cli\Equinox.Cli\bin\Release\net461\Equinox.Cli.exe es run
& dotnet run -f netcoreapp2.1 -p cli/equinox.cli -- es run
& .\cli\Equinox.Cli\bin\Release\net461\Equinox.Cli.exe run es
& dotnet run -f netcoreapp2.1 -p cli/equinox.cli -- run es

## run CosmosDb benchmark (when provisioned)

```
$env:EQUINOX_COSMOS_CONNECTION="AccountEndpoint=https://....;AccountKey=....=;"
$env:EQUINOX_COSMOS_DATABASE="equinox-test"
$env:EQUINOX_COSMOS_COLLECTION="equinox-test"
cli/Equinox.cli/bin/Release/net461/Equinox.Cli `
cosmos -s $env:EQUINOX_COSMOS_CONNECTION -d $env:EQUINOX_COSMOS_DATABASE -c $env:EQUINOX_COSMOS_COLLECTION `
run
dotnet run -f netcoreapp2.1 -p cli/equinox.cli -- `
cosmos -s $env:EQUINOX_COSMOS_CONNECTION -d $env:EQUINOX_COSMOS_DATABASE -c $env:EQUINOX_COSMOS_COLLECTION `
run
```
$env:EQUINOX_COSMOS_CONNECTION="AccountEndpoint=https://....;AccountKey=....=;"
$env:EQUINOX_COSMOS_DATABASE="equinox-test"
$env:EQUINOX_COSMOS_COLLECTION="equinox-test"

cli/Equinox.cli/bin/Release/net461/Equinox.Cli run `
cosmos -s $env:EQUINOX_COSMOS_CONNECTION -d $env:EQUINOX_COSMOS_DATABASE -c $env:EQUINOX_COSMOS_COLLECTION
dotnet run -f netcoreapp2.1 -p cli/equinox.cli -- run `
cosmos -s $env:EQUINOX_COSMOS_CONNECTION -d $env:EQUINOX_COSMOS_DATABASE -c $env:EQUINOX_COSMOS_COLLECTION

## run Web benchmark

The CLI can drive the Store/Web ASP.NET Core app. Doing so requires starting a web process with an appropriate store (Cosmos in this example, but can be `memory`/omitted, `es` etc as in the other examples)

### in Window 1

& dotnet run -c Release -f netcoreapp2.1 -p samples/Store/Web -- -C -U cosmos

### in Window 2

& dotnet run -c Release -f netcoreapp2.1 -p cli/Equinox.Cli -- run -t saveforlater -f 200 web

# PROVISIONING

Expand All @@ -120,20 +140,23 @@ For EventStore, the tests assume a running local instance configured as follows
# run as a single-node cluster to allow connection logic to use cluster mode as for a commercial cluster
& $env:ProgramData\chocolatey\bin\EventStore.ClusterNode.exe --gossip-on-single-node --discover-via-dns 0 --ext-http-port=30778

## CosmosDb (when not using -sc)

```
dotnet run -f netcoreapp2.1 -p cli/equinox.cli -- init -ru 10000 `
cosmos -s $env:EQUINOX_COSMOS_CONNECTION -d $env:EQUINOX_COSMOS_DATABASE -c $env:EQUINOX_COSMOS_COLLECTION
```

# DEPROVISIONING

## Deprovisioning (aka nuking) EventStore data resulting from tests to reset baseline

While EventStore rarely shows any negative effects from repeated load test runs, it can be useful for various reasons to drop all the data generated by the load tests by casting it to the winds:-

# requires admin privilege
rm $env:ProgramData\chocolatey\lib\eventstore-oss\tools\data

## COSMOSDB (when not using -sc)

```
dotnet run -f netcoreapp2.1 -p cli/equinox.cli -- cosmos -s $env:EQUINOX_COSMOS_CONNECTION -d $env:EQUINOX_COSMOS_DATABASE -c $env:EQUINOX_COSMOS_COLLECTION provision -ru 10000
```

## DEPROVISIONING COSMOSDB
## Deprovisioning CosmosDb

The above provisioning step provisions RUs in DocDB for the collection, which add up quickly. *When finished running any test, it's critical to drop the RU allocations back down again via some mechanism*.

Expand Down
2 changes: 1 addition & 1 deletion build.proj
Original file line number Diff line number Diff line change
Expand Up @@ -25,4 +25,4 @@

<Target Name="Build" DependsOnTargets="VSTest;Pack" />

</Project>
</Project>
6 changes: 3 additions & 3 deletions build.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,8 @@ $env:EQUINOX_INTEGRATION_SKIP_EVENTSTORE=[string]$skipEs
if ($skipEs) { warn "Skipping EventStore tests" }

function cliCosmos($arghs) {
Write-Host "dotnet run cli/Equinox.Cli cosmos -s <REDACTED> -d $cosmosDatabase -c $cosmosCollection $arghs"
dotnet run -p cli/Equinox.Cli -f netcoreapp2.1 cosmos -s $cosmosServer -d $cosmosDatabase -c $cosmosCollection @arghs
Write-Host "dotnet run cli/Equinox.Cli -- $arghs cosmos -s <REDACTED> -d $cosmosDatabase -c $cosmosCollection"
dotnet run -p cli/Equinox.Cli -f netcoreapp2.1 -- @arghs cosmos -s $cosmosServer -d $cosmosDatabase -c $cosmosCollection
}

if ($skipCosmos) {
Expand All @@ -30,7 +30,7 @@ if ($skipCosmos) {
warn "Skipping Provisioning Cosmos"
} else {
warn "Provisioning cosmos..."
cliCosmos @("provision", "-ru", "1000")
cliCosmos @("init", "-ru", "1000")
$deprovisionCosmos=$true
}
$env:EQUINOX_INTEGRATION_SKIP_COSMOS=[string]$skipCosmos
Expand Down
62 changes: 62 additions & 0 deletions cli/Equinox.Cli/Clients.fs
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
module Equinox.Cli.Clients

open Domain
open Equinox.Cli.Infrastructure
open System
open System.Net
open System.Net.Http

type Session(client: HttpClient, clientId: ClientId) =

member __.Send(req : HttpRequestMessage) : Async<HttpResponseMessage> =
let req = req |> HttpReq.withHeader "COMPLETELY_INSECURE_CLIENT_ID" clientId.Value
client.Send(req)

type Favorited = { date: System.DateTimeOffset; skuId: SkuId }

type FavoritesClient(session: Session) =

member __.Favorite(skus: SkuId[]) = async {
let request = HttpReq.post () |> HttpReq.withPath "api/favorites" |> HttpReq.withJsonNet skus
let! response = session.Send request
do! response.EnsureStatusCode(HttpStatusCode.NoContent)
}

member __.List = async {
let request = HttpReq.get () |> HttpReq.withPath "api/favorites"
let! response = session.Send request
return! response |> HttpRes.deserializeOkJsonNet<Favorited[]>
}

type Saved = { skuId : SkuId; dateSaved : DateTimeOffset }

type SavesClient(session: Session) =

// this (returning a bool indicating whether it got saved) is fine for now
// IRL we don't want to be leaning on the fact we get a 400 when we exceed the max imems limit as a core API design element
member __.Save(skus: SkuId[]) : Async<bool> = async {
let request = HttpReq.post () |> HttpReq.withPath "api/saves" |> HttpReq.withJsonNet skus
let! response = session.Send request
if response.StatusCode = HttpStatusCode.BadRequest then
return false
else
do! response.EnsureStatusCode(HttpStatusCode.NoContent)
return true
}

member __.Remove(skus: SkuId[]) : Async<unit> = async {
let request = HttpReq.delete () |> HttpReq.withPath "api/saves" |> HttpReq.withJsonNet skus
let! response = session.Send request
return! response.EnsureStatusCode(HttpStatusCode.NoContent)
}

member __.List = async {
let request = HttpReq.get () |> HttpReq.withPath "api/saves"
let! response = session.Send request
return! response |> HttpRes.deserializeOkJsonNet<Saved[]>
}

type Session with

member session.Favorites = FavoritesClient session
member session.Saves = SavesClient session
12 changes: 5 additions & 7 deletions cli/Equinox.Cli/Equinox.Cli.fsproj
Original file line number Diff line number Diff line change
Expand Up @@ -20,37 +20,35 @@
<Compile Include="Infrastructure\Aggregate.fs" />
<Compile Include="Infrastructure\LoadTestRunner.fs" />
<Compile Include="Infrastructure\LocalLoadTestRunner.fs" />
<Compile Include="Clients.fs" />
<Compile Include="Tests.fs" />
<Compile Include="Program.fs" />
</ItemGroup>

<ItemGroup>
<!-- workaround for not being able to make Backend and Domain as inlined in a complete way https://github.com/nuget/home/issues/3891#issuecomment-377319939 -->
<ProjectReference Include="..\..\src\Equinox.Cosmos\Equinox.Cosmos.fsproj" />
<ProjectReference Include="..\..\src\Equinox.MemoryStore\Equinox.MemoryStore.fsproj" PrivateAssets="all" />
<ProjectReference Include="..\..\src\Equinox.MemoryStore\Equinox.MemoryStore.fsproj" />
<ProjectReference Include="..\..\samples\Store\Backend\Backend.fsproj" PrivateAssets="all" />
<ProjectReference Include="..\..\samples\Store\Domain\Domain.fsproj" PrivateAssets="all" />
<ProjectReference Include="..\..\src\Equinox.EventStore\Equinox.EventStore.fsproj" />
<ProjectReference Include="..\..\samples\Store\Infrastructure\Infrastructure.fsproj" PrivateAssets="all" />
</ItemGroup>

<ItemGroup>
<PackageReference Include="Argu" Version="5.1.0" />
<PackageReference Include="Destructurama.FSharp" Version="1.0.14" Condition=" '$(TargetFramework)' == 'net461' " />
<PackageReference Include="Destructurama.FSharp.NetCore" Version="1.0.14" Condition=" '$(TargetFramework)' == 'netcoreapp2.1' " />
<!--Handle TypeShape-restriction; would otherwise use 3.1.2.5-->
<PackageReference Include="FSharp.Core" Version="4.0.0.1" Condition=" '$(TargetFramework)' == 'net461' " />
<PackageReference Include="FSharp.Core" Version="4.3.4" Condition=" '$(TargetFramework)' == 'netcoreapp2.1' " />
<PackageReference Include="MathNet.Numerics" Version="4.6.0" />
<PackageReference Include="Newtonsoft.Json" Version="11.0.2" />
<PackageReference Include="Serilog.Sinks.Console" Version="3.1.1" />
<PackageReference Include="Serilog.Sinks.Seq" Version="4.0.0" />
<PackageReference Include="System.Reactive" Version="4.0.0" />
<Reference Include="System.Runtime.Caching" Condition=" '$(TargetFramework)' != 'netstandard2.0' " />
</ItemGroup>

<!-- workaround for not being able to make Backend and Domain as inlined in a complete way https://github.com/nuget/home/issues/3891#issuecomment-377319939 -->
<Target Name="CopyProjectReferencesToPackage" DependsOnTargets="ResolveReferences">
<ItemGroup>
<BuildOutputInPackage Include="@(ReferenceCopyLocalPaths->WithMetadataValue('ReferenceSourceTarget', 'ProjectReference'))" />
<BuildOutputInPackage Include="@(ReferenceCopyLocalPaths-&gt;WithMetadataValue('ReferenceSourceTarget', 'ProjectReference'))" />
</ItemGroup>
</Target>

Expand Down
Loading

0 comments on commit ae9b2ee

Please sign in to comment.