Feature/merge configuration files (#316)

* #296 renamed configuration.json to ocelot.json in preparation

* removed things we dont need for tests

* another file we dont need

* removed some async we dont need

* refactoring to consolidate configuration code

* removed another pointless abstraction

* #296 started writing merge code

* #296 coming up with ideas for this config merging

* #296 still hacking this idea around

* #296 will now do a crappy merge on the configuration

* #296 change so tests pass on windows

.gitignore

@@ -243,12 +243,12 @@ tools/
 .DS_Store
 
 # Ocelot acceptance test config
-test/Ocelot.AcceptanceTests/configuration.json
+test/Ocelot.AcceptanceTests/ocelot.json
 
 # Read the docstates
 _build/
 _static/
 _templates/
 
 # JetBrains Rider
-.idea/
+.idea/

docs/features/configuration.rst

@@ -1,7 +1,7 @@
 Configuration
 ============
 
-An example configuration can be found `here <https://github.com/TomPallister/Ocelot/blob/develop/test/Ocelot.ManualTest/configuration.json>`_.
+An example configuration can be found `here <https://github.com/TomPallister/Ocelot/blob/develop/test/Ocelot.ManualTest/ocelot.json>`_.
 There are two sections to the configuration. An array of ReRoutes and a GlobalConfiguration. 
 The ReRoutes are the objects that tell Ocelot how to treat an upstream request. The Global 
 configuration is a bit hacky and allows overrides of ReRoute specific settings. It's useful
@@ -69,22 +69,6 @@ Here is an example ReRoute configuration, You don't need to set all of these thi
 
 More information on how to use these options is below..
 
-Follow Redirects / Use CookieContainer 
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Use HttpHandlerOptions in ReRoute configuration to set up HttpHandler behavior:
-
-1. AllowAutoRedirect is a value that indicates whether the request should follow redirection responses. Set it true if the request should automatically 
-follow redirection responses from the Downstream resource; otherwise false. The default value is false.
-2. UseCookieContainer is a value that indicates whether the handler uses the CookieContainer 
-property to store server cookies and uses these cookies when sending requests. The default value is false. Please note
-that if you are using the CookieContainer Ocelot caches the HttpClient for each downstream service. This means that all requests
-to that DownstreamService will share the same cookies. `Issue 274 <https://github.com/ThreeMammals/Ocelot/issues/274>`_ was created because a user
-noticed that the cookies were being shared. I tried to think of a nice way to handle this but I think it is impossible. If you don't cache the clients
-that means each request gets a new client and therefore a new cookie container. If you clear the cookies from the cached client container you get race conditions due to inflight
-requests. This would also mean that subsequent requests dont use the cookies from the previous response! All in all not a great situation. I would avoid setting 
-UseCookieContainer to true unless you have a really really good reason. Just look at your response headers and forward the cookies back with your next request! 
-
 Multiple environments
 ^^^^^^^^^^^^^^^^^^^^^
 
@@ -99,15 +83,40 @@ to you
                         .SetBasePath(hostingContext.HostingEnvironment.ContentRootPath)
                         .AddJsonFile("appsettings.json", true, true)
                         .AddJsonFile($"appsettings.{hostingContext.HostingEnvironment.EnvironmentName}.json", true, true)
-                        .AddJsonFile("configuration.json")
+                        .AddJsonFile("ocelot.json")
                         .AddJsonFile($"configuration.{hostingContext.HostingEnvironment.EnvironmentName}.json")
                         .AddEnvironmentVariables();
                 })
 
-Ocelot should now use the environment specific configuration and fall back to configuration.json if there isnt one.
+Ocelot will now use the environment specific configuration and fall back to ocelot.json if there isnt one.
 
 You also need to set the corresponding environment variable which is ASPNETCORE_ENVIRONMENT. More info on this can be found in the `asp.net core docs <https://docs.microsoft.com/en-us/aspnet/core/fundamentals/environments>`_.
 
+Merging configuration files
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+This feature was requested in `Issue 296 <https://github.com/ThreeMammals/Ocelot/issues/296>`_ and allows users to have multiple configuration files to make managing large configurations easier.
+
+Instead of adding the configuration directly e.g. AddJsonFile("ocelot.json") you can call AddOcelot() like below. 
+
+.. code-block:: csharp
+
+    .ConfigureAppConfiguration((hostingContext, config) =>
+        {
+            config
+                .SetBasePath(hostingContext.HostingEnvironment.ContentRootPath)
+                .AddJsonFile("appsettings.json", true, true)
+                .AddJsonFile($"appsettings.{hostingContext.HostingEnvironment.EnvironmentName}.json", true, true)
+                .AddOcelot()
+                .AddEnvironmentVariables();
+        })
+
+In this scenario Ocelot will look for any files that match the pattern ocleot.*.json and then merge these together. If you want to set the GlobalConfiguration property you must have a file called ocelot.global.json. 
+
+The way Ocelot merges the files is basically load them, loop over them, add any ReRoutes, add any AggregateReRoutes and if the file is called ocelot.global.json add the GlobalConfiguration aswell as any ReRoutes or AggregateReRoutes. Ocelot will then save the merged configuration to a file called ocelot.json and this will be used as the source of truth while ocelot is running.
+
+At the moment there is no validation at this stage it only happens when Ocelot validates the final merged configuration. This is something to be aware of when you are investigating problems. I would advise always checking what is in ocelot.json if you have any problems.
+
 Store configuration in consul
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -119,7 +128,7 @@ If you add the following when you register your services Ocelot will attempt to
     .AddOcelot()
     .AddStoreOcelotConfigurationInConsul();
 
-You also need to add the following to your configuration.json. This is how Ocelot
+You also need to add the following to your ocelot.json. This is how Ocelot
 finds your Consul agent and interacts to load and store the configuration from Consul.
 
 .. code-block:: json
@@ -135,3 +144,19 @@ I decided to create this feature after working on the raft consensus algorithm a
 I guess it means if you want to use Ocelot to its fullest you take on Consul as a dependency for now.
 
 This feature has a 3 second ttl cache before making a new request to your local consul agent.
+
+Follow Redirects / Use CookieContainer 
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Use HttpHandlerOptions in ReRoute configuration to set up HttpHandler behavior:
+
+1. AllowAutoRedirect is a value that indicates whether the request should follow redirection responses. Set it true if the request should automatically 
+follow redirection responses from the Downstream resource; otherwise false. The default value is false.
+2. UseCookieContainer is a value that indicates whether the handler uses the CookieContainer 
+property to store server cookies and uses these cookies when sending requests. The default value is false. Please note
+that if you are using the CookieContainer Ocelot caches the HttpClient for each downstream service. This means that all requests
+to that DownstreamService will share the same cookies. `Issue 274 <https://github.com/ThreeMammals/Ocelot/issues/274>`_ was created because a user
+noticed that the cookies were being shared. I tried to think of a nice way to handle this but I think it is impossible. If you don't cache the clients
+that means each request gets a new client and therefore a new cookie container. If you clear the cookies from the cached client container you get race conditions due to inflight
+requests. This would also mean that subsequent requests dont use the cookies from the previous response! All in all not a great situation. I would avoid setting 
+UseCookieContainer to true unless you have a really really good reason. Just look at your response headers and forward the cookies back with your next request! 

docs/features/delegatinghandlers.rst

@@ -40,7 +40,7 @@ Or transient as below...
             .AddTransientDelegatingHandler<FakeHandlerTwo>()
 
 Both of these Add methods have a default parameter called global which is set to false. If it is false then the intent of 
-the DelegatingHandler is to be applied to specific ReRoutes via configuration.json (more on that later). If it is set to true
+the DelegatingHandler is to be applied to specific ReRoutes via ocelot.json (more on that later). If it is set to true
 then it becomes a global handler and will be applied to all ReRoutes.
 
 e.g.
@@ -58,7 +58,7 @@ Or transient as below...
             .AddTransientDelegatingHandler<FakeHandler>(true)
 
 Finally if you want ReRoute specific DelegatingHandlers or to order your specific and / or global (more on this later) DelegatingHandlers
-then you must add the following json to the specific ReRoute in configuration.json. The names in the array must match the class names of your
+then you must add the following json to the specific ReRoute in ocelot.json. The names in the array must match the class names of your
 DelegatingHandlers for Ocelot to match them together.
 
 .. code-block:: json
@@ -70,8 +70,8 @@ DelegatingHandlers for Ocelot to match them together.
 
 You can have as many DelegatingHandlers as you want and they are run in the following order:
 
-1. Any globals that are left in the order they were added to services and are not in the DelegatingHandlers array from configuration.json.
-2. Any non global DelegatingHandlers plus any globals that were in the DelegatingHandlers array from configuration.json ordered as they are in the DelegatingHandlers array.
+1. Any globals that are left in the order they were added to services and are not in the DelegatingHandlers array from ocelot.json.
+2. Any non global DelegatingHandlers plus any globals that were in the DelegatingHandlers array from ocelot.json ordered as they are in the DelegatingHandlers array.
 3. Tracing DelegatingHandler if enabled (see tracing docs).
 4. QoS DelegatingHandler if enabled (see QoS docs).
 5. The HttpClient sends the HttpRequestMessage.

docs/features/headerstransformation.rst

@@ -8,7 +8,7 @@ Add to Request
 
 This feature was requestes in `GitHub #313 <https://github.com/ThreeMammals/Ocelot/issues/313>`_.
 
-If you want to add a header to your upstream request please add the following to a ReRoute in your configuration.json:
+If you want to add a header to your upstream request please add the following to a ReRoute in your ocelot.json:
 
 .. code-block:: json
 
@@ -25,7 +25,7 @@ Add to Response
 
 This feature was requested in `GitHub #280 <https://github.com/TomPallister/Ocelot/issues/280>`_.
 
-If you want to add a header to your downstream response please add the following to a ReRoute in configuration.json.
+If you want to add a header to your downstream response please add the following to a ReRoute in ocelot.json..
 
 .. code-block:: json
 
@@ -57,7 +57,7 @@ The key is "Test" and the value is "http://www.bbc.co.uk/, http://ocelot.com/".
 Pre Downstream Request
 ^^^^^^^^^^^^^^^^^^^^^^
 
-Add the following to a ReRoute in configuration.json in order to replace http://www.bbc.co.uk/ with http://ocelot.com/. This header will be changed before the request downstream and will be sent to the downstream server.
+Add the following to a ReRoute in ocelot.json in order to replace http://www.bbc.co.uk/ with http://ocelot.com/. This header will be changed before the request downstream and will be sent to the downstream server.
 
 .. code-block:: json
 
@@ -68,7 +68,7 @@ Add the following to a ReRoute in configuration.json in order to replace http://
 Post Downstream Request
 ^^^^^^^^^^^^^^^^^^^^^^^
 
-Add the following to a ReRoute in configuration.json in order to replace http://www.bbc.co.uk/ with http://ocelot.com/. This transformation will take place after Ocelot has received the response from the downstream service.
+Add the following to a ReRoute in ocelot.json in order to replace http://www.bbc.co.uk/ with http://ocelot.com/. This transformation will take place after Ocelot has received the response from the downstream service.
 
 .. code-block:: json
 

docs/features/loadbalancer.rst

@@ -16,7 +16,7 @@ You must choose in your configuration which load balancer to use.
 Configuration
 ^^^^^^^^^^^^^
 
-The following shows how to set up multiple downstream services for a ReRoute using configuration.json and then select the LeadConnection load balancer. This is the simplest way to get load balancing set up.
+The following shows how to set up multiple downstream services for a ReRoute using ocelot.json and then select the LeadConnection load balancer. This is the simplest way to get load balancing set up.
 
 .. code-block:: json
 

docs/features/ratelimiting.rst

@@ -23,7 +23,7 @@ Period - This value specifies the period, such as 1s, 5m, 1h,1d and so on.
 PeriodTimespan - This value specifies that we can retry after a certain number of seconds.
 Limit - This value specifies the maximum number of requests that a client can make in a defined period.
 
-You can also set the following in the GlobalConfiguration part of configuration.json
+You can also set the following in the GlobalConfiguration part of ocelot.json
 
 .. code-block:: json
 

docs/features/requestaggregation.rst

@@ -7,7 +7,7 @@ architecture with Ocelot.
 
 This feature was requested as part of `Issue 79 <https://github.com/TomPallister/Ocelot/pull/79>`_ and further improvements were made as part of `Issue 298 <https://github.com/TomPallister/Ocelot/issue/298>`_.
 
-In order to set this up you must do something like the following in your configuration.json. Here we have specified two normal ReRoutes and each one has a Key property. 
+In order to set this up you must do something like the following in your ocelot.json. Here we have specified two normal ReRoutes and each one has a Key property. 
 We then specify an Aggregate that composes the two ReRoutes using their keys in the ReRouteKeys list and says then we have the UpstreamPathTemplate which works like a normal ReRoute.
 Obviously you cannot have duplicate UpstreamPathTemplates between ReRoutes and Aggregates. You can use all of Ocelot's normal ReRoute options apart from RequestIdKey (explained in gotchas below).
 
@@ -17,7 +17,7 @@ Advanced register your own Aggregators
 Ocelot started with just the basic request aggregation and since then we have added a more advanced method that let's the user take in the responses from the 
 downstream services and then aggregate them into a response object.
 
-The configuration.json setup is pretty much the same as the basic aggregation approach apart from you need to add an Aggregator property like below.
+The ocelot.json setup is pretty much the same as the basic aggregation approach apart from you need to add an Aggregator property like below.
 
 .. code-block:: json
 

docs/features/requestid.rst

@@ -12,7 +12,7 @@ In order to use the reques tid feature you have two options.
 
 *Global*
 
-In your configuration.json set the following in the GlobalConfiguration section. This will be used for all requests into Ocelot.
+In your ocelot.json set the following in the GlobalConfiguration section. This will be used for all requests into Ocelot.
 
 .. code-block:: json
 
@@ -24,7 +24,7 @@ I reccomend using the GlobalConfiguration unless you really need it to be ReRout
 
 *ReRoute*
 
-If you want to override this for a specific ReRoute add the following to configuration.json for the specific ReRoute.
+If you want to override this for a specific ReRoute add the following to ocelot.json for the specific ReRoute.
 
 .. code-block:: json
 

docs/features/routing.rst

@@ -145,9 +145,9 @@ Priority
 ^^^^^^^^
 
 In `Issue 270 <https://github.com/TomPallister/Ocelot/pull/270>`_ I finally decided to expose the ReRoute priority in 
-configuration.json. This means you can decide in what order you want your ReRoutes to match the Upstream HttpRequest.
+ocelot.json. This means you can decide in what order you want your ReRoutes to match the Upstream HttpRequest.
 
-In order to get this working add the following to a ReRoute in configuration.json, 0 is just an example value here but will explain below.
+In order to get this working add the following to a ReRoute in ocelot.json, 0 is just an example value here but will explain below.
 
 .. code-block:: json
 

docs/features/tracing.rst

@@ -20,7 +20,7 @@ In your ConfigureServices method
             option.Service = "Ocelot";
         });
 
-Then in your configuration.json add the following to the ReRoute you want to trace..
+Then in your ocelot.json add the following to the ReRoute you want to trace..
 
 .. code-block:: json
 

docs/features/websockets.rst

@@ -15,7 +15,7 @@ In your Configure method you need to tell your application to use WebSockets.
         app.UseOcelot().Wait();
     })
 
-Then in your configuration.json add the following to proxy a ReRoute using websockets.
+Then in your ocelot.json add the following to proxy a ReRoute using websockets.
 
 .. code-block:: json
 

docs/introduction/gettingstarted.rst

@@ -18,7 +18,7 @@ All versions can be found `here <https://www.nuget.org/packages/Ocelot/>`_.
 
 **Configuration**
 
-The following is a very basic configuration.json. It won't do anything but should get Ocelot starting.
+The following is a very basic ocelot.json. It won't do anything but should get Ocelot starting.
 
 .. code-block:: json
 
@@ -55,7 +55,7 @@ AddOcelot() (adds ocelot services), UseOcelot().Wait() (sets up all the Ocelot m
                         .SetBasePath(hostingContext.HostingEnvironment.ContentRootPath)
                         .AddJsonFile("appsettings.json", true, true)
                         .AddJsonFile($"appsettings.{hostingContext.HostingEnvironment.EnvironmentName}.json", true, true)
-                        .AddJsonFile("configuration.json")
+                        .AddJsonFile("ocelot.json")
                         .AddEnvironmentVariables();
                 })
                 .ConfigureServices(s => {
@@ -87,7 +87,7 @@ All versions can be found `here <https://www.nuget.org/packages/Ocelot/>`_.
 
 **Configuration**
 
-The following is a very basic configuration.json. It won't do anything but should get Ocelot starting.
+The following is a very basic ocelot.json. It won't do anything but should get Ocelot starting.
 
 .. code-block:: json
 
@@ -135,7 +135,7 @@ An example startup using a json file for configuration can be seen below.
                 .SetBasePath(env.ContentRootPath)
                 .AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
                 .AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true)
-                .AddJsonFile("configuration.json")
+                .AddJsonFile("ocelot.json")
                 .AddEnvironmentVariables();
 
             Configuration = builder.Build();