Support HDFS Artifacts

[dtaniwaki]

Is this a BUG REPORT or FEATURE REQUEST?:

FEATURE REQUEST

What happened:

My company uses HDFS w/ Kerberos auth as a storage for artifacts and want to use it easily in argo.

Anything else we need to know?:

I'm implementing it, so I will send a PR.

I'm also adding the HDFS support in argo-events.
argoproj/argo-events#142

[dtaniwaki]

I have implemented the HDFS support in https://github.com/dtaniwaki/argo/tree/support-hdfs-artifacts but the HDFS go client error with kubernetes-HDFS, which I use for my local development, prevent me from testing Kerberos authentication. I'll wait or work with the repo owner of the client.

[dtaniwaki]

I succeeded to use this HDFS artifact with the following patch for kubernetes-HDFS.

diff --git a/charts/hdfs-config-k8s/templates/configmap.yaml b/charts/hdfs-config-k8s/templates/configmap.yaml
index 379dab8..f02b692 100644
--- a/charts/hdfs-config-k8s/templates/configmap.yaml
+++ b/charts/hdfs-config-k8s/templates/configmap.yaml
@@ -34,7 +34,7 @@ data:
       </property>
       <property>
         <name>hadoop.rpc.protection</name>
-        <value>privacy</value>
+        <value>authentication</value>
       </property>
       <property>
         <name>hadoop.user.group.static.mapping.overrides</name>
@@ -74,7 +74,7 @@ data:
       </property>
       <property>
         <name>dfs.encrypt.data.transfer</name>
-        <value>true</value>
+        <value>false</value>
       </property>
       <property>
         <name>dfs.namenode.kerberos.principal</name>

[alexmt]

This is done. Thanks @dtaniwaki !

[BobLiu20]

@dtaniwaki Hi guys. Thanks for your great work. Could you share me how to config the ConfigMap to use HDFS? I can't find any doc for this case.

[dtaniwaki]

@BobLiu20
The commented-out lines in https://github.com/dtaniwaki/argo/blob/34ed28dda5a693d6d37a3c1d544d28a432a19f21/examples/hdfs-artifact.yaml#L41-L52 may help.