Add image for hive3 and hadoop3 #13

shubhamtagra · 2019-06-26T11:09:49Z

Images based on hdp3.

I have use image created via this change for Hive ACID tests in trinodb/trino#1034

findepi

My understanding is that most of the files were created as a copy from hdp2.6 and then applying certain changes. Now, these changes are not visible.

Could you please make 2 separate commits? One that is plain copy and second which changes the file contents as necessary?

findepi · 2019-06-26T13:37:43Z

prestodev/centos7-oj8/README.md

+
+## Oracle license
+
+By using this image, you accept the Oracle Binary Code License Agreement for Java SE available here:


Doesn't apply, the image uses openjdk

findepi · 2019-06-26T13:38:24Z

prestodev/centos7-oj8/README.md

@@ -0,0 +1,13 @@
+# centos7-oj8 [![][layers-badge]][layers-link] [![][version-badge]][dockerhub-link]
+
+[layers-badge]: https://images.microbadger.com/badges/image/prestodev/centos7-oj8.svg


@kokosing i'd like to abandon this READMEs. I don't find them useful.
WDYT?

I noticed the link are dead as of now but wasnt sure if there are plan to get them working so extended the readmes from existing ones.

@stagraqubole i understand. You did the reasonable thing. Let's wait for @kokosing 's opinion before dropping this.

I agree. Especially, if they have outdated information then they do more harm than good.

I will remove these then.

prestodev/centos7-oj8/Dockerfile

findepi · 2019-06-26T13:43:23Z

prestodev/centos7-oj8/Dockerfile

+    \
+    # install supervisor
+    yum --enablerepo=extras install -y setuptools epel-release && \
+    sed -i 's/https/http/g' /etc/yum.repos.d/epel.repo && \


Should not be needed. Why so?

will double check.

Thanks @stagraqubole for working on this!

By any chance, did you also prepare kerberized images?
They would be required when we want to run all the tests on hdp3.

I didnt prepare kerberized image, my requirement was limited to running tests using hive3.

findepi · 2019-06-26T13:45:02Z

Thanks @stagraqubole for working on this!

By any chance, did you also prepare kerberized images?
They would be required when we want to run all the tests on hdp3.

shubhamtagra · 2019-06-26T14:01:51Z

My understanding is that most of the files were created as a copy from hdp2.6 and then applying certain changes. Now, these changes are not visible.

Could you please make 2 separate commits? One that is plain copy and second which changes the file contents as necessary?

Makes sense, I will separate it out.

shubhamtagra · 2019-06-27T05:15:15Z

Updates as per the comments @findepi

kokosing

Nice piece of work!

kokosing · 2019-06-27T08:05:34Z

prestodev/centos7-oj8/README.md

@@ -0,0 +1,13 @@
+# centos7-oj8 [![][layers-badge]][layers-link] [![][version-badge]][dockerhub-link]
+
+[layers-badge]: https://images.microbadger.com/badges/image/prestodev/centos7-oj8.svg


I agree. Especially, if they have outdated information then they do more harm than good.

prestodev/hdp3-hive3/Dockerfile

kokosing · 2019-06-27T08:06:53Z

prestodev/hdp3-hive3/Dockerfile

@@ -59,7 +66,7 @@ RUN chmod 755 /root && chmod 700 /root/.ssh
 RUN passwd --unlock root

 # HDFS ports
-EXPOSE 1004 1006 8020 50010 50020 50070 50075 50470
+EXPOSE 1004 1006 8020 9866 9867 9870 9864 50470


What happened to these ports? Is 50070 no longer exposed? Was it migrated somehow? I used to use it in other versions.

Port have changed. I used https://issues.apache.org/jira/browse/HDFS-9427 and updated the ports that were in use.

prestodev/hdp3-hive3/files/etc/hadoop/conf/yarn-site.xml

prestodev/hdp3-hive3/files/etc/tez/conf/tez-site.xml

shubhamtagra · 2019-06-27T09:27:43Z

Handled your comments @kokosing

kokosing · 2019-06-27T11:57:59Z

Ah... I forgot about tests. Please see https://github.com/prestosql/docker-images/blob/master/bin/test.sh. It would be great if you could add a test for this image. Where you could do some simple queries against hive (CREATE TABLE, INSERT, SELECT, TRUNCATE, DROP).

shubhamtagra · 2019-06-27T16:58:52Z

Ah... I forgot about tests. Please see https://github.com/prestosql/docker-images/blob/master/bin/test.sh. It would be great if you could add a test for this image. Where you could do some simple queries against hive (CREATE TABLE, INSERT, SELECT, TRUNCATE, DROP).

@kokosing I see that the test.sh file already has the test queries and can run for any profile which has compose file defined. Do you want me to make changes to ensure that it runs on hdp3-hive3 (like having compose file for hdp3-hive3 etc) or something more?

shubhamtagra · 2019-06-28T07:13:20Z

Added docker-compose.yml for hdp3-hive3 for tests
Added beeline-site.xml to let hive -e work without specifying connection url

Ran bin/test.sh hdp3-hive3 successfully. Let me know if something more is needed @kokosing

shubhamtagra · 2019-06-28T07:14:57Z

I also noticed that in test.sh, set +e is set around the function that runs the hive queries. Wouldnt we want the test run to fail if hive queries fail for some reason?

shubhamtagra · 2019-06-28T07:31:27Z

I also noticed that in test.sh, set +e is set around the function that runs the hive queries. Wouldnt we want the test run to fail if hive queries fail for some reason?

The expectation is to return the error code and do cleanup, hence the set +e. But we are currently returning last hive query's error code which is not correct. I have added a commit to fix that as well

findepi

Partial review

(until prestodev/hdp3-hive3/files/etc/supervisord.d/mysql-metastore.conf)

prestodev/hdp3-hive3/Dockerfile

prestodev/centos7-oj8/Dockerfile

prestodev/hdp3-hive3/Dockerfile

prestodev/hdp3-hive3/files/etc/hadoop/conf/yarn-site.xml

prestodev/hdp3-hive3/files/etc/hive/conf/hive-site.xml

findepi · 2019-07-01T10:36:02Z

prestodev/hdp3-hive3/files/etc/hive/conf/hive-site.xml

+
+    <property>
+      <name>hive.support.concurrency</name>
+      <value>true</value>


What is this for?
(I think we'll never run any concurrent workload in hive directly.)

This is a requirement for using DbTxnManager and DbTxnManager is a requirement for ACID.

findepi · 2019-07-01T10:39:19Z

prestodev/hdp3-hive3/files/etc/supervisord.d/mysql-metastore.conf

@@ -1,5 +1,5 @@
 [program:mysql-metastore]
-command=/usr/bin/pidproxy /var/run/mysqld/mysqld.pid /usr/bin/mysqld_safe
+command=/bin/bash -c "(chown -R mysql:mysql /var/lib/mysql) && (/usr/bin/pidproxy /var/run/mysqld/mysqld.pid /usr/bin/mysqld_safe)"


I guess this is not sufficient to chown once in Dockerfile. Did you try? i wouldn't be surprised -- i think i encountered such a problem recently somewhere else.

no need for ( ... ), please remove them

I wasnt able to get it running with chown in Dockerfile. Tried again by placing another instance of it after full setup but still there are failures when container starts.

shubhamtagra · 2019-07-02T05:39:37Z

@findepi handled your latest comments. Only the part about mysql vs mariadb is left.

shubhamtagra · 2019-07-08T06:43:20Z

@findepi @kokosing ping!! I have a PR in presto blocked on this, can we move forward with this please.

kokosing · 2019-07-08T07:12:42Z

@findepi @kokosing ping!! I have a PR in presto blocked on this, can we move forward with this please.

Can you please post a docker image under different repository name to docker hub for now? I would like to merge this PR and the one in Presto together. Notice that merging this PR is not enough, we still need to do a release.

shubhamtagra · 2019-07-08T07:53:50Z

@findepi @kokosing ping!! I have a PR in presto blocked on this, can we move forward with this please.

Can you please post a docker image under different repository name to docker hub for now? I would like to merge this PR and the one in Presto together. Notice that merging this PR is not enough, we still need to do a release.

I have done that, PR is open in presto (with a bit outdated docker image): trinodb/trino#1034

I will update it with the latest image.

kokosing · 2019-07-08T12:41:12Z

I will update it with the latest image.

Super. Thanks!

shubhamtagra · 2019-07-09T06:39:58Z

I will update it with the latest image.

Super. Thanks!

This is done. Image is at shubhamtagra/hdp3.1-hive:1

findepi

Looks good overall, a bunch of comments.
I'm testing this locally now.

etc/compose/hdp3.1-hive/docker-compose.yml

etc/compose/hdp3-hive3/docker-compose.yml

prestodev/hdp3.1-hive/files/etc/supervisord.d/mysql-metastore.conf

prestodev/hdp3.1-hive/Dockerfile

prestodev/hdp3.1-hive/files/root/setup.sh

findepi · 2019-07-24T21:57:53Z

I manually tested locally the metastore part of the image

docker run --rm -it --name hadoop-master --hostname hadoop-master -p 9083:9083 -p 1180:1180 prestodev/hdp3.1-hive:latest

echo "127.0.0.1 hadoop-master" > /etc/hosts

connector.name=hive-hadoop2
hive.metastore.uri=thrift://hadoop-master:9083
hive.metastore.thrift.client.socks-proxy=hadoop-master:1180

It worked like a charm.

Good job!

findepi · 2019-07-25T09:08:19Z

i will take over this PR & merge it, if you don't mind.

findepi · 2019-07-25T12:54:40Z

Thanks @stagraqubole, this is merged as 51c48e9

cla-bot bot added the cla-signed label Jun 26, 2019

shubhamtagra mentioned this pull request Jun 26, 2019

Read support for Hive's INSERT ONLY ACID tables trinodb/trino#1034

Closed

shubhamtagra force-pushed the hive3 branch from ec1cee4 to 18cc364 Compare June 26, 2019 11:12

findepi reviewed Jun 26, 2019

View reviewed changes

shubhamtagra force-pushed the hive3 branch 2 times, most recently from 5c90577 to 2a46baa Compare June 27, 2019 05:02

kokosing reviewed Jun 27, 2019

View reviewed changes

shubhamtagra force-pushed the hive3 branch 2 times, most recently from 0dec365 to 8584ce4 Compare June 27, 2019 09:25

shubhamtagra force-pushed the hive3 branch from 8584ce4 to d3c4c0e Compare June 28, 2019 07:10

findepi reviewed Jul 1, 2019

View reviewed changes

Replicate hdp2.6-hive as the starting point for hdp3.1-hive

b5ee758

shubhamtagra force-pushed the hive3 branch from 1a3e1a7 to 2b36730 Compare July 2, 2019 05:35

Shubham Tagra added 2 commits July 3, 2019 17:44

Use hdp3 on centos7 in hdp3.1-hive

9db924f

Fix test.sh to return correct error code for hive query failures

af55708

shubhamtagra force-pushed the hive3 branch from 2b36730 to af55708 Compare July 3, 2019 12:15

findepi reviewed Jul 24, 2019

View reviewed changes

findepi mentioned this pull request Jul 25, 2019

Add hdp3.1 image (v2) #14

Merged

findepi closed this Jul 25, 2019


		## Oracle license

		By using this image, you accept the Oracle Binary Code License Agreement for Java SE available here:

		@@ -0,0 +1,13 @@
		# centos7-oj8 [![][layers-badge]][layers-link] [![][version-badge]][dockerhub-link]

		[layers-badge]: https://images.microbadger.com/badges/image/prestodev/centos7-oj8.svg

Add image for hive3 and hadoop3 #13

Add image for hive3 and hadoop3 #13

Conversation

shubhamtagra commented Jun 26, 2019

findepi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

findepi commented Jun 26, 2019

shubhamtagra commented Jun 26, 2019

shubhamtagra commented Jun 27, 2019

kokosing left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shubhamtagra commented Jun 27, 2019

kokosing commented Jun 27, 2019

shubhamtagra commented Jun 27, 2019 • edited Loading

shubhamtagra commented Jun 28, 2019

shubhamtagra commented Jun 28, 2019

shubhamtagra commented Jun 28, 2019

findepi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shubhamtagra commented Jul 2, 2019

shubhamtagra commented Jul 8, 2019

kokosing commented Jul 8, 2019

shubhamtagra commented Jul 8, 2019

kokosing commented Jul 8, 2019

shubhamtagra commented Jul 9, 2019 • edited Loading

findepi left a comment

Choose a reason for hiding this comment

findepi commented Jul 24, 2019

findepi commented Jul 25, 2019

findepi commented Jul 25, 2019

shubhamtagra commented Jun 27, 2019 •

edited

Loading

shubhamtagra commented Jul 9, 2019 •

edited

Loading