Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ISSUE#3315]Nacos client support https #3654

Merged
merged 5 commits into from
Aug 24, 2020

Conversation

wangweizZZ
Copy link
Collaborator

@wangweizZZ wangweizZZ commented Aug 20, 2020

Please do not create a Pull Request without creating an issue first.

What is the purpose of the change

#3315

client-side support https

Brief changelog

  1. common module add tls part
  2. JdkHttpClientRequest support https

The http client now uses a unified AbstractNacosRestTemplate, while NacosRestTemplate uses JdkHttpClientRequest as the internal client

TLS classes

SSLContext

  • TlsHelper
    Utils for build SSLContext

  • SelfKeyManager
    Utils for build customized KeyManager

  • SelfTrustManager
    Utils for build customized TrustManager

HostnameVerifier

  • SelfHostnameVerifier
    A HostnameVerifier verify ipv4 and localhost.

AbstractHttpClientFactory

Developers need to modify the createXXXXRestTemplate method to create an HTTP client supporting https by calling the initTls method

TLS mode

  1. Not support TLS (default)

  2. Making your client support TLS without authentication

 System.setProperty(TlsSystemConfig.TLS_ENABLE, "true");
  1. Making your client support TLS one-way authentication
 System.setProperty(TlsSystemConfig.TLS_ENABLE, "true");
 System.setProperty(TlsSystemConfig.CLIENT_AUTH, "true");
 System.setProperty(TlsSystemConfig.CLIENT_TRUST_CERT, "trustCert");

TLS file watch
default check interval is 10 min.
customize with the following parameters

System.setProperty(TlsSystemConfig.CHECK_INTERVAL,"10")

Verifying this change

XXXX

Follow this checklist to help us incorporate your contribution quickly and easily:

  • Make sure there is a Github issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
  • Format the pull request title like [ISSUE #123] Fix UnknownException when host config not exist. Each commit in the pull request should have a meaningful subject line and body.
  • Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
  • Write necessary unit-test to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module.
  • Run mvn -B clean package apache-rat:check findbugs:findbugs -Dmaven.test.skip=true to make sure basic checks pass. Run mvn clean install -DskipITs to make sure unit-test pass. Run mvn clean test-compile failsafe:integration-test to make sure integration-test pass.

* common module add tls related classes
* JdkHttpClientRequest support https
* unified IpUtils
* common module add tls related classes
* JdkHttpClientRequest support https
* unified IpUtils
@wangweizZZ wangweizZZ changed the title Feature https [# 3315]Nacos client support https Aug 20, 2020
@wangweizZZ wangweizZZ changed the title [# 3315]Nacos client support https [ISSUE#3315]Nacos client support https Aug 20, 2020

private static final Logger LOGGER = LoggerFactory.getLogger(SelfTrustManager.class);

@SuppressWarnings("checkstyle:WhitespaceAround")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why SuppressWarnings checkstyle:WhitespaceAround?

Copy link
Collaborator Author

@wangweizZZ wangweizZZ Aug 20, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

第60行那两个}}报的这个checkstyle错误,但是 格式化工具就自动格式化成这样

Copy link
Collaborator

@KomachiSion KomachiSion Aug 21, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

感觉可能是checkstyle的bug 官方文档说不会把双括弧认为是非法的。
https://checkstyle.sourceforge.io/config_whitespace.html#WhitespaceAround

或者是我们这种写法不属于双括弧初始化器。

@Maijh97
Copy link
Collaborator

Maijh97 commented Aug 20, 2020

我提个建议: 感觉不能只是在JdkHttpClientRequest里来支持 HTTPS,需要考虑下把支持HTTPS的实现放到更上一层,例如放到NacosResttemplate里,或者是AbstractNacosRestTemplate,因为后续可能会把依赖的HTTP client组件实现替换为其它的实现(比如apache http)。

@wangweizZZ
Copy link
Collaborator Author

我提个建议: 感觉不能只是在JdkHttpClientRequest里来支持 HTTPS,需要考虑下把支持HTTPS的实现放到更上一层,例如放到NacosResttemplate里,或者是AbstractNacosRestTemplate,因为后续可能会把依赖的HTTP client组件实现替换为其它的实现(比如apache http)。

因为这个需要不同的HTTPClient 自己去使用对应的SSLContext方可。这里我的理解是更应该说是一个https 的插入点,因为这些比如apache http 本来也就支持https 只不过需要使用一些自定义构造。比如 在 AbstractHttpClientFactory 里进行判断 是否开启https认证 从而构建对应的 HttpAsyncClients.custom().setSSlxx(SSLContext) 实例这样。

我觉得现在是因为这个 JdkHttpClientRequest 不太支持 https。使得一些SSLContext的生成逻辑判断夹杂其中。 是否可以在JdkHttpClientRequest里暴露一个方法出来接收这个 SSLContext 。
即在AbstractHttpClientFactory 里加一个buildSslContext方法,而在 createNacosRestTemplate方法中 new JdkHttpClientRequest 后setSslXXX 这样?

@Maijh97
Copy link
Collaborator

Maijh97 commented Aug 20, 2020 via email

assignLogger().error("add tls file listener fail", e);
}
}

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里的逻辑是不是可以独立为一个函数,看起来相对简洁点。

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

是指tlsFileWatcher这一段吗?addSSLContextChangeListener(FileChangeListener Listener)?
独立成一个函数,这样至少对使用AbstractHttpClientFactory.java 类的开发者来说屏蔽一些tlsClientTrustCertPath这样的细节。是好点,我改下。

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

从51行开始到68行,感觉这里可以独立为一个函数。

@yanlinly yanlinly self-requested a review August 24, 2020 01:59
yanlinly
yanlinly previously approved these changes Aug 24, 2020
Copy link
Collaborator

@yanlinly yanlinly left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

Copy link
Collaborator

@yanlinly yanlinly left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

@yanlinly yanlinly merged commit da8fa57 into alibaba:develop Aug 24, 2020
chuntaojun added a commit that referenced this pull request Sep 14, 2020
* fix-#3595, delete the unnecessary code (#3596)

* [ISSUE #3566] move the permission code of nacos-core module to nacos-auth module (#3593)

* move the permission code of nacos-core module to nacos-auth module.

* Fix some code style issues

* address server module auth package name change.

* test change

* Incorrect package name correction

* [ISSUE #3592] Fix incorrect prompt when accessing the restricted namespace (#3603)

* Fix incorrect prompt when accessing the restricted namespace

* Modify variable name

* [ISSUE #3600] Replace the deprecated api of jwt (#3616)

* replace the deprecated api of jwt

* transfer secretKey to byte array just using String encode with utf-8

* [ISSUE #3613]  Fix `unit test method not be static` & update publish config listener  in `ConfigTest.java` (#3614)

* fix `unit test method not be static` & update publish config listener in `ConfigTest.java`

* fix `unit test method not be static` & update publish config listener in `ConfigTest.java`

* move jwt dependency from console,core to auth. (#3624)

* refactor: unified implementation of http client api adjustment. (#3639)

* [ISSUE #3628] set naming client updateTask interval more flexible (#3637)

* 1.use server cacheMillis event service deleted
2.naming client UpdateTask's interval will inc by failCount that connect with server

* 1.move failCount to updateTask
2.redefine the updateService method name. updateServiceNow -> updateService, wrap updateService in updateServiceNow when first getServiceInfo

* 1.create push client even service is not exist
2.serviceInfo's hosts is empty or can't connect to server both add the updateTsk interval

* format the indent

* fix: create kvstorage

* refactor: create kv storage]

* refactor: refactor rocksdb storage code

* Fix issue 3661 (#3662)

* improvement: merge upstream/develop

* feat: merge upstream develop

* fix: fix issue #3661

* [ISSUE#3658] refactor TaskManager and move to nacos-common module (#3663)

* Add TaskManagerTest

* Move Abstract Task and Task processor to nacos-common

* Add Nacos execute engine interface and nacos task interface

* Refactor Task Manager to NacosDelayTaskExecuteEngine

* for code style

* [ISSUE #3671] move some tools class into common package (#3672)

* for #3621 (#3668)

* [ISSUE#3315]Nacos client support https (#3654)

* [ISSUE #3315] nacos client support https
* common module add tls related classes
* JdkHttpClientRequest support https
* unified IpUtils

* [ISSUE #3315] nacos client support https
* common module add tls related classes
* JdkHttpClientRequest support https
* unified IpUtils

* [ISSUE #3315] nacos client support https

* format code

* fix typo and doc format of README file (#3688)

1. It’s a little weird to use Chinese comma in English
2. Fix typo:   change 'reposity' to  'repository', 'instance' to 'instances'
3. Format other details of the doc

* Try to fix nacos server CLOSE_WAIT (#3703)

* Fix revert chunk isn't work in Content Comparison page (#3686)

* fix: fixed cluster node version issue

* Fix Logging in with the wrong username or password will cause jackson serialization results to fail (#3695) (#3721)

* [ISSUE#3712] add apache http client factory (#3716)

* refactor: Add apache http client Factory.

* refactor: Add apache http client Factory.

* add license

* refactor: class name change

* [Issue#3692] Use new distro task engine replace old task dispatcher. (#3715)

* Add ServiceManagerTest

* Add DistroConsistencyServiceImplTest

* Add new distro entities

* Add new distro sync data change

* add new VerifyTask to do checksum.

* Replace old sync task and checksum task

* Add retry sync change task.

* Fix high cpu load

* For checkstyle

* Combine naming sync task to reduce http cost

* Move some classes

* Refactor keys combined logic

* enhance package construct

* Fix unit test

* For pmd

* [Issue #3692] Use new distro implmentation to handle sync datum and checksum request (#3734)

* Add receive data and verify data distro protocol interface

* Use new distro implmentation to handle sync datum and checksum request

* Use new distro implmentation to handle get datum request.

* refactor: adjust the data loading logic when a new protocol is started

* [ISSUE#3692] Use new distro implmentation to handle init all datum request. (#3744)

* Add load data task in DistroProtocol

* Use new distro implmentation to handle init all datum request.

* Fix unit test

* fix: #3617 (#3678)

* fix #3617

* 调整代码格式

* 修改抛出的Exception类型

* 调整逻辑

* 移除没用到的方法

* 修改注释

* [ISSUE#3658] Move distro sync code to nacos-core module (#3750)

* Move distro sync code to nacos-core module

* Update unit test

* feat: none

* fix: fixing serialization problems

* fix: fixed some logic errors

Co-authored-by: 赵延 <[email protected]>
Co-authored-by: mai.jh <[email protected]>
Co-authored-by: ljhrot <[email protected]>
Co-authored-by: Xarrow <[email protected]>
Co-authored-by: 杨翊 SionYang <[email protected]>
Co-authored-by: Gagharv <[email protected]>
Co-authored-by: syapollo <[email protected]>
Co-authored-by: S2W <[email protected]>
Co-authored-by: Tboy <[email protected]>
Co-authored-by: 邪影oO <[email protected]>
chuntaojun added a commit that referenced this pull request Oct 14, 2020
* fix-#3595, delete the unnecessary code (#3596)

* [ISSUE #3566] move the permission code of nacos-core module to nacos-auth module (#3593)

* move the permission code of nacos-core module to nacos-auth module.

* Fix some code style issues

* address server module auth package name change.

* test change

* Incorrect package name correction

* [ISSUE #3592] Fix incorrect prompt when accessing the restricted namespace (#3603)

* Fix incorrect prompt when accessing the restricted namespace

* Modify variable name

* [ISSUE #3600] Replace the deprecated api of jwt (#3616)

* replace the deprecated api of jwt

* transfer secretKey to byte array just using String encode with utf-8

* [ISSUE #3613]  Fix `unit test method not be static` & update publish config listener  in `ConfigTest.java` (#3614)

* fix `unit test method not be static` & update publish config listener in `ConfigTest.java`

* fix `unit test method not be static` & update publish config listener in `ConfigTest.java`

* move jwt dependency from console,core to auth. (#3624)

* refactor: unified implementation of http client api adjustment. (#3639)

* [ISSUE #3628] set naming client updateTask interval more flexible (#3637)

* 1.use server cacheMillis event service deleted
2.naming client UpdateTask's interval will inc by failCount that connect with server

* 1.move failCount to updateTask
2.redefine the updateService method name. updateServiceNow -> updateService, wrap updateService in updateServiceNow when first getServiceInfo

* 1.create push client even service is not exist
2.serviceInfo's hosts is empty or can't connect to server both add the updateTsk interval

* format the indent

* fix: create kvstorage

* refactor: create kv storage]

* refactor: refactor rocksdb storage code

* Fix issue 3661 (#3662)

* improvement: merge upstream/develop

* feat: merge upstream develop

* fix: fix issue #3661

* [ISSUE#3658] refactor TaskManager and move to nacos-common module (#3663)

* Add TaskManagerTest

* Move Abstract Task and Task processor to nacos-common

* Add Nacos execute engine interface and nacos task interface

* Refactor Task Manager to NacosDelayTaskExecuteEngine

* for code style

* [ISSUE #3671] move some tools class into common package (#3672)

* for #3621 (#3668)

* [ISSUE#3315]Nacos client support https (#3654)

* [ISSUE #3315] nacos client support https
* common module add tls related classes
* JdkHttpClientRequest support https
* unified IpUtils

* [ISSUE #3315] nacos client support https
* common module add tls related classes
* JdkHttpClientRequest support https
* unified IpUtils

* [ISSUE #3315] nacos client support https

* format code

* fix typo and doc format of README file (#3688)

1. It’s a little weird to use Chinese comma in English
2. Fix typo:   change 'reposity' to  'repository', 'instance' to 'instances'
3. Format other details of the doc

* Try to fix nacos server CLOSE_WAIT (#3703)

* Fix revert chunk isn't work in Content Comparison page (#3686)

* fix: fixed cluster node version issue

* Fix Logging in with the wrong username or password will cause jackson serialization results to fail (#3695) (#3721)

* [ISSUE#3712] add apache http client factory (#3716)

* refactor: Add apache http client Factory.

* refactor: Add apache http client Factory.

* add license

* refactor: class name change

* [Issue#3692] Use new distro task engine replace old task dispatcher. (#3715)

* Add ServiceManagerTest

* Add DistroConsistencyServiceImplTest

* Add new distro entities

* Add new distro sync data change

* add new VerifyTask to do checksum.

* Replace old sync task and checksum task

* Add retry sync change task.

* Fix high cpu load

* For checkstyle

* Combine naming sync task to reduce http cost

* Move some classes

* Refactor keys combined logic

* enhance package construct

* Fix unit test

* For pmd

* [Issue #3692] Use new distro implmentation to handle sync datum and checksum request (#3734)

* Add receive data and verify data distro protocol interface

* Use new distro implmentation to handle sync datum and checksum request

* Use new distro implmentation to handle get datum request.

* refactor: adjust the data loading logic when a new protocol is started

* [ISSUE#3692] Use new distro implmentation to handle init all datum request. (#3744)

* Add load data task in DistroProtocol

* Use new distro implmentation to handle init all datum request.

* Fix unit test

* fix: #3617 (#3678)

* fix #3617

* 调整代码格式

* 修改抛出的Exception类型

* 调整逻辑

* 移除没用到的方法

* 修改注释

* [ISSUE#3658] Move distro sync code to nacos-core module (#3750)

* Move distro sync code to nacos-core module

* Update unit test

* feat: none

* fix: fixing serialization problems

* replace the deprecated api com.fasterxml.jackson.databind.node.ObjectNode#put(java.lang.String, com.fasterxml.jackson.databind.JsonNode)

* Revert "[#3368]Cancel empty Long polling thread to improve performance. (#ISSUE3432)" (#3778)

This reverts commit 95c8bf2.

* [ISSUE #3658] Some enhance refactor for naming distro (#3765)

* Some enhance refactor for naming distro

* Remove null code

* [ISSUE #3687] check serviceName's format(groupName@@serviceName) in server and client (#3767)

* 1.in server, check serviceName's format 'groupName@@serviceName', groupName and serviceName can't be blank
2.in client, check 'groupName@@serviceName', groupName and serviceName can't be blank

* ignore the check to groupName

* check split's length instead of exception to check argument

* 1.add some notes
2.remove unnecessary code

* modify the notes

* [ISSUE#3790] Supplement http response Content-Encoding processing. (#3791)

* bug: fix issue #3790, Supplement http response Content-Encoding processing

* bug: fix issue #3790, Supplement http response Content-Encoding processing.

* bug: fix issue #3790, Supplement http response Content-Encoding processing.

* Update ConvertUtils.java (#3789)

* fix ConvertUtils can not handle FormatException.

* fix: fixed some logic errors

* [ISSUE#3192] naming module replace http client (#3763)

* naming module replace http client

* refactor: naming module replace http client.

* refactor: naming module replace http client.

* refactor: Add apache http client Factory.

* refactor: naming module replace http client.

* fix code style

* refactor: Add http client config

* refactor: naming module HttpClientManager change

* refactor: naming module HttpClientManager change

* refactor: naming module replace http client.

* fix code style

* refactor: fix JDK http client Use error problem.

* refactor: Query And Header entity init Add non-empty judgment

* Enhance the asynchronous http delete request method to support body passing parameters.

* refactor: apache http client set MaxConnTotal and maxConnPerRoute.

* Fix NullPointerException when no subscriber for slow event (#3835)

* Set mediaType charset as utf8 (#3837)

* Fix code style problem in DiskUtils (#3842)

* Fix http client contentType charset problem (#3848)

* feature issue #3804 (#3805)

* improvement: merge upstream/develop

* feat: merge upstream develop

* feat: manage the loading of configuration files uniformly

* fix: fix copyright style

* style: fix code style

* fix: fix code style

* bug: fix the problem of incorrect judgment of http response code in SubscribeManager#getSubscribers() method (#3879)

* [ISSUE #3867] replace the way which get version (#3872)

* replace the way which get version

* remove version sign application.properties in nacos-api

* [ISSUE #3871] fix description don't match the error (#3886)

* Fix jraft problem

* Fix chinese string are truncated in ConcunrrentDiskUtil (#3883)

* Temp fix raft server can't refresh raft configuration problem

* update spring boot dependencies version (#3900)

Co-authored-by: yanlinly <[email protected]>
Co-authored-by: 杨翊 SionYang <[email protected]>

* [ISSUE #3781]Fix service list intermittently lost service (#3891)

* update service init

* commit futureMap.remove()

* update serviceManager

Co-authored-by: yanlinly <[email protected]>
Co-authored-by: 杨翊 SionYang <[email protected]>

* Fix Listener do not listen new consistency problem

* Fix Performance logger thread call old raft error

* Use datum in new raft processor to compatible old data

* Use multiple kv storage in new raft processor to compatible old data

* Revert PR#2849.

* Move datum key check to KeyBuilder

* Fix 1.3.2 upgrade 1.4.0 can't notify service change problem

* [ISSUE #3850] ignore socket exception when client destroy already (#3906)

* if client destroy already, ignore socket exception.

* remove the exception's judgement

* remove unuseful import

* fix: fix merge conflict

* isAddressServerHealth set as true when request success (#3952)

* [ISSUE#3533] change cache dir with namespace -- part 1: unify the cache dir (#3859)

* [ISSUE#3533] change cache dir with namespace -- part 2: unify the log dir (#3882)

* fix typo error (#3954)

* [ISSUE #3909] add domain's judgement (#3913)

* add domain's judgement

* modify domain's judgement, can resolve = true

* remove judgement in 'if' code block

* replace Ip to IP in InetUtils

* log warn info when domain can not be resolved

* fix vaiable name

* 1.fix unit test can't pass (#3956)

2.update the unit test

* Move remove listener logic to ServiceManager

* Move remove listener logic to ServiceManager

* Fix#3973 (#3974)

* fix #3973

* 重复代码抽取到一个方法

* 删除私有方法的注释

* 处理namespace参数的方法提出到一个工具类中

* 修改注释

* 添加licences

* 增加 TenantUtil 的测试

* TenantUtil 改名为 NamespaceUtil

* For #3384, Fix member extend info do not update error. (#3982)

* refactor: code refactor

Co-authored-by: 赵延 <[email protected]>
Co-authored-by: mai.jh <[email protected]>
Co-authored-by: ljhrot <[email protected]>
Co-authored-by: Xarrow <[email protected]>
Co-authored-by: 杨翊 SionYang <[email protected]>
Co-authored-by: Gagharv <[email protected]>
Co-authored-by: syapollo <[email protected]>
Co-authored-by: S2W <[email protected]>
Co-authored-by: Tboy <[email protected]>
Co-authored-by: 邪影oO <[email protected]>
Co-authored-by: yanlinly <[email protected]>
Co-authored-by: Mark4z <[email protected]>
Co-authored-by: Marcus <[email protected]>
Co-authored-by: shizhengxing <[email protected]>
Co-authored-by: ljhrot <[email protected]>
Co-authored-by: sanxun0325 <[email protected]>
Co-authored-by: JackSun-Developer <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants