Dashboard 1.0.2 version does not work with SBA 2.1.1

[B0bSapp]

Hello @MartinDevillers , I faced following problem:
SBA server version 2.1.1, spring-boot-admin-hystrix-dashboard version 1.0.2.
Hystrix stream on client side exists and is working fine - it is exposed via explicit servlet configuration:

  public ServletRegistrationBean statisticsStream() {
    return new ServletRegistrationBean(
        new HystrixMetricsStreamServlet(), "/actuator/hystrix.stream");
  }

Calling https://client-url/actuator/hystrix.stream returns valid data:

ping: 

data: {"type":"HystrixCommand","name":"ValidationCommandKey","group":"ReceiptValidationHystrixCommand","currentTime":1579713161842,"isCircuitBreakerOpen":false,"errorPercentage":0,"errorCount":0,"requestCount":0,"rollingCountBadRequests":0,"rollingCountCollapsedRequests":0,"rollingCountEmit":0,"rollingCountExceptionsThrown":0,"rollingCountFailure":0,"rollingCountFallbackEmit":0,"rollingCountFallbackFailure":0,"rollingCountFallbackMissing":0,"rollingCountFallbackRejection":0,"rollingCountFallbackSuccess":0,"rollingCountResponsesFromCache":0,"rollingCountSemaphoreRejected":0,"rollingCountShortCircuited":0,"rollingCountSuccess":0,"rollingCountThreadPoolRejected":0,"rollingCountTimeout":0,"currentConcurrentExecutionCount":0,"rollingMaxConcurrentExecutionCount":0,"latencyExecute_mean":0,"latencyExecute":{"0":0,"25":0,"50":0,"75":0,"90":0,"95":0,"99":0,"99.5":0,"100":0},"latencyTotal_mean":0,"latencyTotal":{"0":0,"25":0,"50":0,"75":0,"90":0,"95":0,"99":0,"99.5":0,"100":0},"propertyValue_circuitBreakerRequestVolumeThreshold":10,"propertyValue_circuitBreakerSleepWindowInMilliseconds":5000,"propertyValue_circuitBreakerErrorThresholdPercentage":50,"propertyValue_circuitBreakerForceOpen":false,"propertyValue_circuitBreakerForceClosed":false,"propertyValue_circuitBreakerEnabled":true,"propertyValue_executionIsolationStrategy":"THREAD","propertyValue_executionIsolationThreadTimeoutInMilliseconds":1000,"propertyValue_executionTimeoutInMilliseconds":1000,"propertyValue_executionIsolationThreadInterruptOnTimeout":true,"propertyValue_executionIsolationThreadPoolKeyOverride":null,"propertyValue_executionIsolationSemaphoreMaxConcurrentRequests":10,"propertyValue_fallbackIsolationSemaphoreMaxConcurrentRequests":10,"propertyValue_metricsRollingStatisticalWindowInMilliseconds":10000,"propertyValue_requestCacheEnabled":true,"propertyValue_requestLogEnabled":true,"reportingHosts":1,"threadPool":"ReceiptValidationHystrixCommand"}

data: {"type":"HystrixCommand","name":"DataCommitCommandKey","group":"DataCommitHystrixCommand","currentTime":1579713161842,"isCircuitBreakerOpen":false,"errorPercentage":0,"errorCount":0,"requestCount":0,"rollingCountBadRequests":0,"rollingCountCollapsedRequests":0,"rollingCountEmit":0,"rollingCountExceptionsThrown":0,"rollingCountFailure":0,"rollingCountFallbackEmit":0,"rollingCountFallbackFailure":0,"rollingCountFallbackMissing":0,"rollingCountFallbackRejection":0,"rollingCountFallbackSuccess":0,"rollingCountResponsesFromCache":0,"rollingCountSemaphoreRejected":0,"rollingCountShortCircuited":0,"rollingCountSuccess":0,"rollingCountThreadPoolRejected":0,"rollingCountTimeout":0,"currentConcurrentExecutionCount":0,"rollingMaxConcurrentExecutionCount":0,"latencyExecute_mean":0,"latencyExecute":{"0":0,"25":0,"50":0,"75":0,"90":0,"95":0,"99":0,"99.5":0,"100":0},"latencyTotal_mean":0,"latencyTotal":{"0":0,"25":0,"50":0,"75":0,"90":0,"95":0,"99":0,"99.5":0,"100":0},"propertyValue_circuitBreakerRequestVolumeThreshold":10,"propertyValue_circuitBreakerSleepWindowInMilliseconds":5000,"propertyValue_circuitBreakerErrorThresholdPercentage":50,"propertyValue_circuitBreakerForceOpen":false,"propertyValue_circuitBreakerForceClosed":false,"propertyValue_circuitBreakerEnabled":true,"propertyValue_executionIsolationStrategy":"THREAD","propertyValue_executionIsolationThreadTimeoutInMilliseconds":1000,"propertyValue_executionTimeoutInMilliseconds":1000,"propertyValue_executionIsolationThreadInterruptOnTimeout":true,"propertyValue_executionIsolationThreadPoolKeyOverride":null,"propertyValue_executionIsolationSemaphoreMaxConcurrentRequests":10,"propertyValue_fallbackIsolationSemaphoreMaxConcurrentRequests":10,"propertyValue_metricsRollingStatisticalWindowInMilliseconds":10000,"propertyValue_requestCacheEnabled":true,"propertyValue_requestLogEnabled":true,"reportingHosts":1,"threadPool":"DataCommitHystrixCommand"}

But when I open application is SBA I can't see hystrix dashboard

Can you help to resolve this issue? Thanks in advance.

[MartinDevillers]

Hi @B0bSapp thank you for posting this issue :-)
If you configure the Hystrix Stream endpoint in your client application manually, then you will also have to manually register it with Spring Boot Actuator to enable discovery. I would recommend you to use spring-cloud-starter-netflix-hystrix (see https://mvnrepository.com/artifact/org.springframework.cloud/spring-cloud-starter-netflix-hystrix )

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-netflix-hystrix</artifactId>
</dependency>

This will automatically expose the Hystrix Stream endpoint and register it with Actuator for auto discovery, so you don't have to do any of the config yourself.

[MartinDevillers]

Also, if you do want to do the registration yourself. I suggest you look at the source code of the Spring Cloud Netflix Hystrix project for inspiration, specifically the following HystrixStreamEndpoint class:
https://github.com/spring-cloud/spring-cloud-netflix/blob/2.2.x/spring-cloud-netflix-hystrix/src/main/java/org/springframework/cloud/netflix/hystrix/HystrixStreamEndpoint.java

@ServletEndpoint(id = "hystrix.stream")
public class HystrixStreamEndpoint implements Supplier<EndpointServlet> {

	private final Map<String, String> initParameters;

	public HystrixStreamEndpoint(Map<String, String> initParameters) {
		this.initParameters = initParameters;
	}

	@Override
	public EndpointServlet get() {
		return new EndpointServlet(HystrixMetricsStreamServlet.class)
				.withInitParameters(this.initParameters);
	}
}

I hope this information gets you further. If you have any questions, just post a new comment here and I’ll reply ASAP 🙂

[B0bSapp]

@MartinDevillers thank you for quick reply, but it does not seem to help
Here's my client configuration:

@Bean
  public ServletRegistrationBean statisticsStream() {
    return new ServletRegistrationBean(
        new HystrixMetricsStreamServlet(), "/actuator/hystrix.stream");
  }

  @ServletEndpoint(id = "/actuator/hystrix.stream")
  public class HystrixStreamEndpoint implements Supplier<EndpointServlet> {

    private final Map<String, String> initParameters;

    public HystrixStreamEndpoint(Map<String, String> initParameters) {
      this.initParameters = initParameters;
    }

    @Override
    public EndpointServlet get() {
      return new EndpointServlet(HystrixMetricsStreamServlet.class)
          .withInitParameters(this.initParameters);
    }
  }

I would like to mention, that I have some kind of specific base-path (application.yml snippet):

management:
  endpoints:
    web:
      base-path: /
      exposure:
        include: health, info

That's why I have to make explicit servlet configuration. From SBAS startup logs I see, that module ws loaded successfully

2020-01-22 13:22:03.482 INFO 1 --- [ main] b.a.s.u.c.AdminServerUiAutoConfiguration : Loaded Spring Boot Admin UI Extension: UiExtension(resourcePath=hystrix-dashboard/css/custom.a0de57be.css, resourceLocation=classpath:/META-INF/spring-boot-admin-server-ui/extensions/hystrix-dashboard/css/custom.a0de57be.css) 2020-01-22 13:22:03.482 INFO 1 --- [ main] b.a.s.u.c.AdminServerUiAutoConfiguration : Loaded Spring Boot Admin UI Extension: UiExtension(resourcePath=hystrix-dashboard/js/custom.ad7f373b.js, resourceLocation=classpath:/META-INF/spring-boot-admin-server-ui/extensions/hystrix-dashboard/js/custom.ad7f373b.js)
And from devtools I see that dashboard is loading actually

Calling health endpoint from SBA work fine:
https://sba/instances//actuator/health

{"status":"UP"}

But calling https://sba/instances//actuator/hystrix.stream return 404. So looks like I need expose this endpoint somewhere else.

[MartinDevillers]

management.endpoints.web.base-path=/ Aha, you're running actuator on the root path of your application. As per the Spring Boot Actuator documentation, this disables the discovery page.

2.5. Hypermedia for Actuator Web Endpoints

When a custom management context path is configured, the “discovery page” automatically moves from /actuator to the root of the management context. For example, if the management context path is /management, then the discovery page is available from /management. When the management context path is set to /, the discovery page is disabled to prevent the possibility of a clash with other mappings.

So basically, what happens now is that Spring Boot Admin no longer can use the "discovery page" to discover an instance's endpoints. Instead, it now uses a different strategy called "endpoint probing" to see which endpoints are available. This strategy simply tries various well-known endpoints for HTTP 200 responses. Since hystrix.stream is not well-known, it will never try and discover this endpoint. However, you can explicitly add hystrix.stream to the list of probed endpoints on the Spring Boot Admin Server side by configuring the following property.

spring.boot.admin.probed-endpoints=info,health,hystrix.stream

Click here for the list of default values of this property

However, I would personally recommend against this approach. Endpoint probing is a brittle strategy and was only implemented into SBA to remain compatible with Spring Boot 1.x applications who don't support the "discovery page". Instead, I would set the base-path of the actuator endpoints to something else besides the root path. Besides running into compatibility issues like this one, you also run the risk of having clashes with other endpoints. Also, from a security perspective I would prefer to keep all the actuator endpoints under some common path so you can more easily secure the whole set of actuator endpoints. (Right @joshiste ?)

[B0bSapp]

Thank you so much for your time and answers! After adding probing in SBA, dashboard appeared, but all the rest disappeared :), which is expected, I suppose (I added probing for info, health and hystrix). But dashboard didn't show anything

From dev tools I get url, from which hystrix.stream is expected, I guess https://sba/hystrix-static/monitor/monitor.html?stream=../../instances/instance/actuator/hystrix.stream&title=instance, but nothing is printed

While stream itself is working fine:

data: {"type":"HystrixCommand","name":"ValidationCommandKey","group":"ReceiptValidationHystrixCommand","currentTime":1579794226872,"isCircuitBreakerOpen":false,"errorPercentage":0,"errorCount":0,"requestCount":0,"rollingCountBadRequests":0,"rollingCountCollapsedRequests":0,"rollingCountEmit":0,"rollingCountExceptionsThrown":0,"rollingCountFailure":0,"rollingCountFallbackEmit":0,"rollingCountFallbackFailure":0,"rollingCountFallbackMissing":0,"rollingCountFallbackRejection":0,"rollingCountFallbackSuccess":0,"rollingCountResponsesFromCache":0,"rollingCountSemaphoreRejected":0,"rollingCountShortCircuited":0,"rollingCountSuccess":0,"rollingCountThreadPoolRejected":0,"rollingCountTimeout":0,"currentConcurrentExecutionCount":0,"rollingMaxConcurrentExecutionCount":0,"latencyExecute_mean":0,"latencyExecute":{"0":0,"25":0,"50":0,"75":0,"90":0,"95":0,"99":0,"99.5":0,"100":0},"latencyTotal_mean":0,"latencyTotal":{"0":0,"25":0,"50":0,"75":0,"90":0,"95":0,"99":0,"99.5":0,"100":0},"propertyValue_circuitBreakerRequestVolumeThreshold":10,"propertyValue_circuitBreakerSleepWindowInMilliseconds":5000,"propertyValue_circuitBreakerErrorThresholdPercentage":50,"propertyValue_circuitBreakerForceOpen":false,"propertyValue_circuitBreakerForceClosed":false,"propertyValue_circuitBreakerEnabled":true,"propertyValue_executionIsolationStrategy":"THREAD","propertyValue_executionIsolationThreadTimeoutInMilliseconds":1000,"propertyValue_executionTimeoutInMilliseconds":1000,"propertyValue_executionIsolationThreadInterruptOnTimeout":true,"propertyValue_executionIsolationThreadPoolKeyOverride":null,"propertyValue_executionIsolationSemaphoreMaxConcurrentRequests":10,"propertyValue_fallbackIsolationSemaphoreMaxConcurrentRequests":10,"propertyValue_metricsRollingStatisticalWindowInMilliseconds":10000,"propertyValue_requestCacheEnabled":true,"propertyValue_requestLogEnabled":true,"reportingHosts":1,"threadPool":"ReceiptValidationHystrixCommand"}

Well, I guess, I spent a lot of time trying to get it working, so I'm going to follow your recommendations - avoid setting base path to root. Will get back once make all according changes. Thanks a lot for your cooperation one more time!

[MartinDevillers]

Oh wow it looks like the IFRAME containing the Hystrix Dashboard crashed in your screenshot. I haven't seen this behavior before. Have you tried accessing the hystrix.stream that is proxied through Spring Boot Admin? The Hystrix Dashboard doesn't directly connect with the client application, but instead connects with a proxy provided by Spring Boot Admin. You should see this url pass by in your Chrome Developer Tools - Network Tab. Here's an example of such an url from my own local demo environment:

http://localhost:8080/instances/a542226c30d8/actuator/hystrix.stream

I notice in your url the name of the instance is called instance whereas in my environment it's some random generated identifier a542226c30d8

[B0bSapp]

@MartinDevillers I replaced this with 'instance' for security reasons :) Trying to access via this url
https://sba.url/#/instances/5ee5a781280d/hystrix for the first time did not return anything for the first time, but later I received this error response:

There was an unexpected error (type=Service Unavailable, status=503).
MaxConcurrentConnections reached: 5

So, looks like stream is accessible, but response is short cut somewhere inside kubernetes

[MartinDevillers]

Ah, very good for omitting the security sensitive bits 🙂 The url you posted is to access the Hystrix Dashboard page inside Spring Boot Admin. The url I posted is to access the event stream API of Spring Boot Admin that exposes the data for the dashboard. Does the event stream work?

[B0bSapp]

Yes, event stream, works fine:
https://client.url/actuator/hystrix.stream returns valid data.

[MartinDevillers]

But that's the hystrix.stream of the client, not the hystrix.stream that is proxied through Spring Boot Admin. There are two different streams:

1. The original hystrix.stream provided by your Spring Boot client applications. You've already tested this and it works.
2. The proxied hystrix.stream provided by your Spring Boot Admin Server application. This is the one used by Hystrix Dashboard to generate the GUI. You can access this stream like this:

https://{your-spring-boot-admin-server-hostname}/instances/{instance-id-here}/actuator/hystrix.stream

[B0bSapp]

Sorry for messing things up. This endpoint returns following error:

There was an unexpected error (type=Service Unavailable, status=503).
MaxConcurrentConnections reached: 5

So looks like stream is accessible from sba, but response is short cut inside kubernetes

[MartinDevillers]

Interesting, can you confirm that other endpoints do return data like:

https://{your-spring-boot-admin-server-hostname}/instances/{instance-id-here}/actuator/info
https://{your-spring-boot-admin-server-hostname}/instances/{instance-id-here}/actuator/health

[MartinDevillers]

Also I wonder if this is related to codecentric/spring-boot-admin#661 or Netflix/Hystrix#1644

[B0bSapp]

Yeah, that was the first thing, that I checked. It returns

{}

and

{"status":"UP"}

respectively

[B0bSapp]

Most probably, it is, but before I started getting this error, trying to call sba proxy endpoint did not return any data, page just hung and any 'ping:' messages appeared.

[MartinDevillers]

Interesting, I wonder what is the difference between your stack and the ones I've used that causes this behavior. At my previous client I was running all Spring Boot applications inside a Dockerized environment running on AWS Fargate, which in turn was running behind an AWS ALB. I never encountered any 503's (although I did have other problems like the stream stopped responding sometimes after a while).

[B0bSapp]

Managed to get it working finally. Problem was in kubernetes security, I added '/hystrix.stream' to unsecured endpoints list and get following results:
https://sba/instances/ab222159bf8e/actuator/hystrix.stream

data: {"type":"HystrixCommand","name":"ValidationCommandKey","group":"ReceiptValidationHystrixCommand","currentTime":1579876277469,"isCircuitBreakerOpen":false,"errorPercentage":0,"errorCount":0,"requestCount":0,"rollingCountBadRequests":0,"rollingCountCollapsedRequests":0,"rollingCountEmit":0,"rollingCountExceptionsThrown":0,"rollingCountFailure":0,"rollingCountFallbackEmit":0,"rollingCountFallbackFailure":0,"rollingCountFallbackMissing":0,"rollingCountFallbackRejection":0,"rollingCountFallbackSuccess":0,"rollingCountResponsesFromCache":0,"rollingCountSemaphoreRejected":0,"rollingCountShortCircuited":0,"rollingCountSuccess":0,"rollingCountThreadPoolRejected":0,"rollingCountTimeout":0,"currentConcurrentExecutionCount":4,"rollingMaxConcurrentExecutionCount":0,"latencyExecute_mean":0,"latencyExecute":{"0":0,"25":0,"50":0,"75":0,"90":0,"95":0,"99":0,"99.5":0,"100":0},"latencyTotal_mean":0,"latencyTotal":{"0":0,"25":0,"50":0,"75":0,"90":0,"95":0,"99":0,"99.5":0,"100":0},"propertyValue_circuitBreakerRequestVolumeThreshold":10,"propertyValue_circuitBreakerSleepWindowInMilliseconds":5000,"propertyValue_circuitBreakerErrorThresholdPercentage":50,"propertyValue_circuitBreakerForceOpen":false,"propertyValue_circuitBreakerForceClosed":false,"propertyValue_circuitBreakerEnabled":true,"propertyValue_executionIsolationStrategy":"THREAD","propertyValue_executionIsolationThreadTimeoutInMilliseconds":1000,"propertyValue_executionTimeoutInMilliseconds":1000,"propertyValue_executionIsolationThreadInterruptOnTimeout":true,"propertyValue_executionIsolationThreadPoolKeyOverride":null,"propertyValue_executionIsolationSemaphoreMaxConcurrentRequests":10,"propertyValue_fallbackIsolationSemaphoreMaxConcurrentRequests":10,"propertyValue_metricsRollingStatisticalWindowInMilliseconds":10000,"propertyValue_requestCacheEnabled":true,"propertyValue_requestLogEnabled":true,"reportingHosts":1,"threadPool":"ReceiptValidationHystrixCommand"}

https://sba/hystrix-static/monitor/monitor.html?stream=../../instances/ab222159bf8e/actuator/hystrix.stream&title=ab222159bf8e

But main SBA dashboard still shows nothing )

[MartinDevillers]

Cool good job! I can help you to diagnose your final problem with the IFRAME. It is currently error'ing out with a "xxx refused to connect" error. This is Chrome blocking the IFRAME from loading because a cross-origin policy is telling the browser to refuse this content. If you open the Chrome Developer Console I think it also prints an error in red stating that it blocked the IFRAME from loading. You can debug this using the Chrome Developer Tools on the Network tab. Look up the following two Doc requests:

1. The one for Spring Boot Admin itself https://sba/
2. The one used by the iframe https://sba/hystrix-static/monitor/monitor.html?....

For both requests, inspect the response headers sent by the server. Look for headers like:

• Content-Security-Policy: frame-ancestors 'self' https://awesome-domain.com
• X-Frame-Options: ALLOW-FROM https://awesome-domain.com

More info:

• https://stackoverflow.com/questions/31944552/iframe-refuses-to-display
• https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Frame-Options
• https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy

[B0bSapp]

Sorry for annoying, but looks like I have problem with headers as well (
Here's snipped from my SBA security config

    http.headers()
        .frameOptions()
        .disable()
        .and()
        .authorizeRequests()

So I disable sending X-Frame-options header. And according to devtool logs it seems to be working for /health endpoint :

But for /hystrix-static/monitor/monitor.html situation didn't change:

Does plugin hardcode this header? Because the same situation was when I set frameOption to SAME_ORIGIN

[MartinDevillers]

Hi @B0bSapp , don't worry you're not annoying at all. I want your Hystrix Dashboard to work as well :-)

As far as I am aware, neither the plugin nor Spring Boot Admin adds the x-frame-options header. So this header is either coming from a custom security configuration inside your application or somewhere else (maybe a load-balancer, API-gateway or firewall?).

To verify this, I've tested SBA locally (version 2.1.5) with the plugin enabled. The response headers on the static monitor page look like this:

Accept-Ranges: bytes
Cache-Control: max-age=3600
Content-Length: 7371
Content-Type: text/html
Last-Modified: Thu, 23 Jan 2020 16:12:47 GMT

I would suggest disabling all custom security policies and then enabling them again 1-by-1 to see which one triggers the x-frame-options.

Looking around the security config of SBA didn't yield anything related to the x-frame-options headers:

https://github.com/codecentric/spring-boot-admin/blob/1680b57e036917f9d888d27b95efbd5f7c3a1b93/spring-boot-admin-samples/spring-boot-admin-sample-servlet/src/main/java/de/codecentric/boot/admin/SecuritySecureConfig.java

[B0bSapp]

Yeah, you were right, as always) Problem was in security policy, configured on SBA side. Once corrected everything works as a charm!

So, I'm closing this issue, thanks a lot for help!