Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support proxy user with kerberos #1027

Merged
merged 1 commit into from
Oct 13, 2021
Merged

Support proxy user with kerberos #1027

merged 1 commit into from
Oct 13, 2021

Conversation

saLeox
Copy link
Contributor

@saLeox saLeox commented Oct 6, 2021

What is the purpose of the change

Enable proxy user mechanism with kerberos, when access hdfs or submit spark / hive job.
Related issues: #479, #465

Brief change log

  • Add two variables KEYTAB_PROXYUSER_ENABLED and KEYTAB_PROXYUSER_SUPERUSER, that can be read from properties file in each engine folder;
  • When KEYTAB_PROXYUSER_ENABLED is true, will load keytab file of specific super user in HDFSUtils;
  • When KEYTAB_PROXYUSER_ENABLED is true, will add the "--proxy-user" option when submit spark job (proxy user will be the one submit the task in this context);
  • Add Property("javax.security.auth.useSubjectCredsOnly", "false") in HiveEngineConnExecutor.scala, to support Kerberos authentication and impersonation in Hive.

Verifying this change

This change added tests and can be verified as follows:

  • Make sure your cluster has integrated with kerberos;
  • Remove the remark of kerberos setting, switch the wds.linkis.keytab.host.enabled and wds.linkis.keytab.proxyuser.enable onto true;
  • Declare the value of wds.linkis.keytab.proxyuser.superuser;
  • Prepare the keytab of superuser and make it available to the other users (chmod 110 or 111);
  • Try to run the linkis-cli (-engineType spark) or linkis-cli-hive

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (no)
  • Anything that affects deployment: (no)
  • The MGS(Microservice Governance Services), i.e., Spring Cloud Gateway, OpenFeign, Eureka.: (no)

Documentation

  • Does this pull request introduce a new feature? (no)

wushengyeyouya
wushengyeyouya reviewed Oct 11, 2021
View reviewed changes
.gitignore Outdated
@@ -1,7 +1,7 @@
*.iml
.idea
.DS_Store

.*
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line must be deleted.

...-engineconn-plugins/engineconn-plugins/flink/src/main/resources/linkis-engineconn.properties Outdated
Comment on lines 31 to 35

#wds.linkis.keytab.file=/opt/keytab/
#wds.linkis.keytab.host.enabled=true
#wds.linkis.keytab.host=hadoop
#wds.linkis.keytab.proxyuser.enable=true
#wds.linkis.keytab.proxyuser.superuser=hadoop
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These lines must be deleted.

...s-engineconn-plugins/engineconn-plugins/hive/src/main/resources/linkis-engineconn.properties Outdated
Comment on lines 30 to 35
#wds.linkis.keytab.file=/opt/keytab/
#wds.linkis.keytab.host.enabled=true
#wds.linkis.keytab.host=hadoop
#wds.linkis.keytab.proxyuser.enable=true
#wds.linkis.keytab.proxyuser.superuser=hadoop
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These lines must be deleted.

...ngineconn-plugins/engineconn-plugins/io_file/src/main/resources/linkis-engineconn.properties Outdated
Comment on lines 35 to 39
#wds.linkis.keytab.host.enabled=true
#wds.linkis.keytab.host=hadoop
#wds.linkis.keytab.proxyuser.enable=true
#wds.linkis.keytab.proxyuser.superuser=hadoop
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These lines must be deleted.

...s-engineconn-plugins/engineconn-plugins/jdbc/src/main/resources/linkis-engineconn.properties Outdated
Comment on lines 30 to 34
#wds.linkis.keytab.file=/opt/keytab/
#wds.linkis.keytab.host.enabled=true
#wds.linkis.keytab.host=hadoop
#wds.linkis.keytab.proxyuser.enable=true
#wds.linkis.keytab.proxyuser.superuser=hadoop
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These lines must be deleted.

...gineconn-plugins/engineconn-plugins/pipeline/src/main/resources/linkis-engineconn.properties Outdated
Comment on lines 30 to 34
#wds.linkis.keytab.file=/opt/keytab/
#wds.linkis.keytab.host.enabled=true
#wds.linkis.keytab.host=hadoop
#wds.linkis.keytab.proxyuser.enable=true
#wds.linkis.keytab.proxyuser.superuser=hadoop
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These lines must be deleted.

...engineconn-plugins/engineconn-plugins/python/src/main/resources/linkis-engineconn.properties Outdated
Comment on lines 30 to 34
#wds.linkis.keytab.host.enabled=true
#wds.linkis.keytab.host=hadoop
#wds.linkis.keytab.proxyuser.enable=true
#wds.linkis.keytab.proxyuser.superuser=hadoop
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These lines must be deleted.

...-engineconn-plugins/engineconn-plugins/spark/src/main/resources/linkis-engineconn.properties Outdated
Comment on lines 31 to 35
#wds.linkis.keytab.file=/opt/keytab/
#wds.linkis.keytab.host.enabled=true
#wds.linkis.keytab.host=hadoop
#wds.linkis.keytab.proxyuser.enable=true
#wds.linkis.keytab.proxyuser.superuser=hadoop
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These lines must be deleted.

...scala/com/webank/wedatasphere/linkis/engineplugin/hive/executor/HiveEngineConnExecutor.scala Outdated
@@ -90,6 +90,7 @@ class HiveEngineConnExecutor(id: Int,

override def init(): Unit = {
LOG.info(s"Ready to change engine state!")
System.setProperty("javax.security.auth.useSubjectCredsOnly", "false");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please set it when the keytab.proxyuser is enabled.

peacewong
peacewong reviewed Oct 12, 2021
View reviewed changes
...edatasphere/linkis/engineplugin/spark/launch/SparkSubmitProcessEngineConnLaunchBuilder.scala Outdated
@@ -387,6 +389,13 @@ class SparkSubmitProcessEngineConnLaunchBuilder private extends JavaProcessEngin
}
}
}

if (!SparkConfiguration.KEYTAB_PROXYUSER_ENABLED.getValue) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should use the same configuration as HadoopConf

@saLeox saLeox force-pushed the dev-1.0.3 branch 2 times, most recently from 6dec492 to b94758a Compare October 13, 2021 03:58
peacewong
peacewong approved these changes Oct 13, 2021
View reviewed changes
Copy link
Contributor

@peacewong peacewong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@peacewong peacewong merged commit b8e2017 into apache:dev-1.0.3 Oct 13, 2021
@saLeox saLeox mentioned this pull request Nov 10, 2021
Closed
@casionone casionone mentioned this pull request Nov 24, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wushengyeyouya wushengyeyouya wushengyeyouya left review comments

@peacewong peacewong peacewong approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@saLeox @peacewong @wushengyeyouya