The log-manager cannot work well with Nginx reverse proxy.

[siaimes]

Organization Name:

Short summary about the issue/question:
The marketplace cannot work well with Nginx reverse proxy.

Brief what process you are following:
My master node does not have an Internet IP, and I am configuring it to be accessible by clients on the Internet. So I configured nginx on a server located in a demilitarized zone.

My nginx configuration file for openpai is as follows:

server {

	# SSL configuration
	listen 443 ssl;
        
	server_name openpai.domain.com;

        ssl_certificate "/path/to/certificate.crt";
	ssl_certificate_key "/path/to/private.key";

	location / {
            proxy_pass https://172.17.1.100;
	}

}

Everything is behaving normally, except for the marketplace.

The market cannot be opened on the Internet, and if I am on the LAN, I can open the market, but the lock in front of the URL of the Chrome browser turns red (it was black before accessing the market).

I know this is because I wrote the IP instead of the domain name for the marketplace part in the services-configuration.yaml configuration file like is.

webportal:
  server-port: 9286
  plugins:
  - title: Marketplace
    uri: https://172.17.1.100/marketplace/plugin.js
    config:
      repo: Microsoft/pai

marketplace-webportal:
  marketplace_api_uri: https://172.17.1.100/marketplace/api

Because when using Nginx as a reverse proxy, Nginx will rewrite the host filed in the web page to the current host. However, most of the host filed of openpai can be modified except for the marketplace. Therefore, I think that the source code of the marketplace may violate some regulations.

How to reproduce it:

OpenPAI Environment:

• OpenPAI version: v1.4.1
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release): Ubuntu 16.04.7 LTS
• Kernel (e.g. uname -a): Linux devbox 4.15.0-123-generic Removing private SSH key #126~16.04.1-Ubuntu SMP Wed Oct 21 13:48:05 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
• Hardware (e.g. core number, memory size, storage size, GPU type etc.):
• Others:

Anything else we need to know:

[siaimes]

Forgive my carelessness. Just after I submitted this issue, I saw that the key in the configuration file was uri instead of url, so the problem was solved. The new configuration file is as follows.

webportal:
  server-port: 9286
  plugins:
  - title: Marketplace
    uri: /marketplace/plugin.js
    config:
      repo: Microsoft/pai

marketplace-webportal:
  marketplace_api_uri: /marketplace/api

[siaimes]

Well, the log manager also has this problem. But I didn't find a place to optimize it. Can anyone help me solve this problem?

[Binyang2014]

@siaimes Can you follow this? #5252 (comment)

[siaimes]

@Binyang2014 Thank you for your reply.

I can't fully understand the solution you gave me in this issue.

I tried to add uri-https to the pylon section, but it doesn’t seem to work for me.

Another situation is that because I want to access the k8s dashboard, I have configured a self-signed certificate for openpai on the LAN.

[siaimes]

Another situation is that because I want to access the k8s dashboard, I have configured a self-signed certificate for openpai on the LAN.

This means that I use a self-signed certificate to access openpai on the LAN and a CA certificate to access openpai on the Internet.

[siaimes]

I can't fully understand the solution you gave me in this issue.

Okay, this seems to require me to modify the source code and hard-write the domain name into the source code.

If I understand correctly, it will put a particularly heavy service pressure on my DMZ server, because all log requests in the LAN will be proxied by the DMZ server instead of being sent directly to the master node.

[siaimes]

I found another '404 not found' error.

[Binyang2014]

I found another '404 not found' error.

You can safely ignore this error for ssh API. This API already deprecated. We will remote this request from frontend.

[Binyang2014]

I can't fully understand the solution you gave me in this issue.

Okay, this seems to require me to modify the source code and hard-write the domain name into the source code.

If I understand correctly, it will put a particularly heavy service pressure on my DMZ server, because all log requests in the LAN will be proxied by the DMZ server instead of being sent directly to the master node.

You are correct. But since log stored inside the cluster, and for your environment, client can not connect PAI cluster directly. All requests must pass through the gateway. From our practice, the traffic is not too heavy and we limited the bandwidth usage for each log request.
If you find performance issue, you can use a powerful machine to deploy gateway, or use multi gateway to load balance the traffic

[siaimes]

I can't fully understand the solution you gave me in this issue.

Okay, this seems to require me to modify the source code and hard-write the domain name into the source code.
If I understand correctly, it will put a particularly heavy service pressure on my DMZ server, because all log requests in the LAN will be proxied by the DMZ server instead of being sent directly to the master node.

You are correct. But since log stored inside the cluster, and for your environment, client can not connect PAI cluster directly. All requests must pass through the gateway. From our practice, the traffic is not too heavy and we limited the bandwidth usage for each log request.
If you find performance issue, you can use a powerful machine to deploy gateway, or use multi gateway to load balance the traffic

What I mean by this is that most of my client requests come from the LAN. These requests do not actually need to be sent to the DMZ server but can be sent directly to the master node.

You know, many services will be deployed on the DMZ server, not just openpai, so I hope to save bandwidth as much as possible.

[Binyang2014]

I hope I understand your question correctly. Most of your client access the openPAI form LAN. The client in LAN works correctly and can see the logs.
But some of the clients access openPAI form internet. The requests need to go through the DMZ first.
And you want only the request from internet to go through the DMZ.

If so, you should add rules in your reverse proxy.
rewrite the response with https://172.17.1.100/log-manager/.... to https://dmz-server/log-manager

You can refer nginx subfilter: http://nginx.org/en/docs/http/ngx_http_sub_module.html#sub_filter

This a workaround we will solve this in future release

[siaimes]

I hope I understand your question correctly. Most of your client access the openPAI form LAN. The client in LAN works correctly and can see the logs.
But some of the clients access openPAI form internet. The requests need to go through the DMZ first.
And you want only the request from internet to go through the DMZ.

If so, you should add rules in your reverse proxy.
rewrite the response with https://172.17.1.100/log-manager/.... to https://dmz-server/log-manager

You can refer nginx subfilter: http://nginx.org/en/docs/http/ngx_http_sub_module.html#sub_filter

This a workaround we will solve this in future release

Yes, this is exactly what I need.

I just modified the configuration of Nginx, the new configuration file is as below.

        location / {
            proxy_pass https://172.17.1.100;
            sub_filter 'https://172.17.1.100/log-manager/' 'https://$host/log-manager/';
        }

However, this does not seem to solve the problem. I looked at the console and found that its link was not replaced correctly.

[siaimes]

I hope I understand your question correctly. Most of your client access the openPAI form LAN. The client in LAN works correctly and can see the logs.
But some of the clients access openPAI form internet. The requests need to go through the DMZ first.
And you want only the request from internet to go through the DMZ.
If so, you should add rules in your reverse proxy.
rewrite the response with https://172.17.1.100/log-manager/.... to https://dmz-server/log-manager
You can refer nginx subfilter: http://nginx.org/en/docs/http/ngx_http_sub_module.html#sub_filter
This a workaround we will solve this in future release

Yes, this is exactly what I need.

I just modified the configuration of Nginx, the new configuration file is as below.

        location / {
            proxy_pass https://172.17.1.100;
            sub_filter 'https://172.17.1.100/log-manager/' 'https://$host/log-manager/';
        }

However, this does not seem to solve the problem. I looked at the console and found that its link was not replaced correctly.

        location / {
            proxy_pass https://172.17.1.100;
            sub_filter 'https://172.17.1.100/log-manager/' 'https://$host/log-manager/';
            proxy_set_header Accept-Encoding "";
            sub_filter_types *;
            sub_filter_once off;
        }

http-sub-module-sub-filter-of-nginx-and-reverse-proxy-not-working

@Binyang2014 The problem has been solved, thank you very much for your help. If there are related optimizations in subsequent versions, I hope you will remind me in this issue so that I can delete related useless nginx configurations.