[AIRFLOW-4296][AIP-3-subtask-18] Remove py2 in ci process #5090

zhongjiajie · 2019-04-12T14:11:02Z

Make sure you have checked all steps below.

Jira

My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-4296

Description

My PR #5020 CI failed, and the sample log https://api.travis-ci.org/v3/job/519213992/log.txt

Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 21, in <module>
    from airflow import configuration
  File "/usr/local/lib/python2.7/dist-packages/airflow/__init__.py", line 36, in <module>
    from airflow import settings, configuration as conf
  File "/usr/local/lib/python2.7/dist-packages/airflow/settings.py", line 32, in <module>
    from airflow.utils.sqlalchemy import setup_event_handlers
  File "/usr/local/lib/python2.7/dist-packages/airflow/utils/sqlalchemy.py", line 22, in <module>
    import json
  File "/usr/local/lib/python2.7/dist-packages/airflow/utils/json.py", line 27, in <module>
    class AirflowJsonEncoder(json.JSONEncoder):
AttributeError: 'module' object has no attribute 'JSONEncoder'

And I found out we ci process still have py2, So remove from __future__ import absolute_import will cause error. This PR are try to fix this.

Code Quality

Passes flake8

zhongjiajie · 2019-04-12T14:12:34Z

@Fokko @ashb @XD-DENG PTAL.

kaxil · 2019-04-12T16:45:54Z

This is already done and merged in master:

3020d97

zhongjiajie · 2019-04-12T16:51:42Z

@kaxil I rebase on master but PR-5019 and PR-5020 still failed.

Fokko · 2019-04-14T18:44:37Z

Restarted the CI

zhongjiajie · 2019-04-15T01:49:23Z

Still failed, I try to fix it.

jmcarp · 2019-04-15T04:51:54Z

scripts/ci/kubernetes/docker/airflow-test-env-init.sh

@@ -19,7 +19,7 @@

 set -x

-cd /usr/local/lib/python2.7/dist-packages/airflow && \
+cd /usr/local/lib/python3.6/dist-packages/airflow && \


It looks like this path should use python3.5 instead of python3.6. Xenial LTS installs python3.5 unless we add a new PPA.

Thanks for catching. But I change base docker images to python:3.6-slim because we have a jira ticket in https://issues.apache.org/jira/browse/AIRFLOW-4210. And I think we should change our ci k8s as well.

zhongjiajie · 2019-04-15T06:28:47Z

Add gcc because of failed in https://travis-ci.org/apache/airflow/jobs/520123725

codecov-io · 2019-04-15T08:57:22Z

Codecov Report

Merging #5090 into master will decrease coverage by 0.47%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #5090      +/-   ##
==========================================
- Coverage   77.23%   76.76%   -0.48%     
==========================================
  Files         465      465              
  Lines       29837    29837              
==========================================
- Hits        23046    22905     -141     
- Misses       6791     6932     +141

Impacted Files	Coverage Δ
airflow/contrib/kubernetes/volume_mount.py	`33.33% <0%> (-66.67%)`	⬇️
airflow/contrib/kubernetes/pod_launcher.py	`38.21% <0%> (-52.04%)`	⬇️
airflow/contrib/kubernetes/volume.py	`50% <0%> (-50%)`	⬇️
airflow/contrib/kubernetes/pod_generator.py	`43.24% <0%> (-43.25%)`	⬇️
...rflow/contrib/operators/kubernetes_pod_operator.py	`56.16% <0%> (-42.47%)`	⬇️
airflow/contrib/kubernetes/kube_client.py	`27.58% <0%> (-37.94%)`	⬇️
airflow/configuration.py	`85.95% <0%> (-4.8%)`	⬇️
airflow/models/taskinstance.py	`92.59% <0%> (+0.17%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1c6aa5e...9ccf73f. Read the comment docs.

codecov-io · 2019-04-15T08:57:23Z

Codecov Report

Merging #5090 into master will decrease coverage by 0.47%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #5090      +/-   ##
==========================================
- Coverage   77.23%   76.76%   -0.48%     
==========================================
  Files         465      465              
  Lines       29837    29837              
==========================================
- Hits        23046    22905     -141     
- Misses       6791     6932     +141

Impacted Files	Coverage Δ
airflow/contrib/kubernetes/volume_mount.py	`33.33% <0%> (-66.67%)`	⬇️
airflow/contrib/kubernetes/pod_launcher.py	`38.21% <0%> (-52.04%)`	⬇️
airflow/contrib/kubernetes/volume.py	`50% <0%> (-50%)`	⬇️
airflow/contrib/kubernetes/pod_generator.py	`43.24% <0%> (-43.25%)`	⬇️
...rflow/contrib/operators/kubernetes_pod_operator.py	`56.16% <0%> (-42.47%)`	⬇️
airflow/contrib/kubernetes/kube_client.py	`27.58% <0%> (-37.94%)`	⬇️
airflow/configuration.py	`85.95% <0%> (-4.8%)`	⬇️
airflow/models/taskinstance.py	`92.59% <0%> (+0.17%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1c6aa5e...9ccf73f. Read the comment docs.

zhongjiajie · 2019-04-15T09:08:53Z

CI green. But before merge I'd like to test if this change work in my personal repo https://github.com/zhongjiajie/airflow/tree/test_if_ci_work

Travis CI log is here https://travis-ci.com/zhongjiajie/airflow/builds/108322089

I use this PR and cherry-pick #5020. submit https://github.com/zhongjiajie/airflow/tree/test_if_ci_work

Fokko · 2019-04-15T10:13:36Z

@zhongjiajie Still a sad CI :'( Can you rerun the job?

zhongjiajie · 2019-04-15T10:28:45Z

Seem failed with un-related test https://travis-ci.com/zhongjiajie/airflow/jobs/192913602
But I will try again to ensure.

zhongjiajie · 2019-04-15T11:41:43Z

https://travis-ci.com/zhongjiajie/airflow/builds/108333920

K8S ci pass, but postgre failed and I think the failed is un-related

======================================================================
46) ERROR: test_execution_limited_parallelism (tests.executors.test_local_executor.LocalExecutorTest)
----------------------------------------------------------------------
   Traceback (most recent call last):
    tests/executors/test_local_executor.py line 75 in test_execution_limited_parallelism
      self.execution_parallelism(parallelism=test_parallelism)
    tests/executors/test_local_executor.py line 61 in execution_parallelism
      self.assertTrue(executor.event_buffer['fail'], State.FAILED)
   KeyError: 'fail'

And this postgre job sucess in https://travis-ci.com/zhongjiajie/airflow/builds/108322089

Should I restart the CI in my repo or just accept/review this PR
WDYT @Fokko

milton0825 · 2019-04-16T00:13:33Z

@zhongjiajie can you restart the tests in your CI? Probably just a flaky test.

zhongjiajie · 2019-04-16T01:13:18Z

@milton0825 I think is flaky test, but in job https://travis-ci.com/zhongjiajie/airflow/builds/108333920 we could found out K8S test job pass(we just want to prove K8S test pass). So I wonder if we could accept this PR

zhongjiajie · 2019-04-16T02:33:33Z

https://travis-ci.org/zhongjiajie/airflow/builds/520548126

FYI @Fokko @milton0825 , CI green in my own repo https://github.com/zhongjiajie/airflow/tree/test_if_ci_work

milton0825 · 2019-04-16T04:14:30Z

lgtm

jmcarp · 2019-04-19T18:53:20Z

Anything left to do on this? It looks good to me.

mik-laj · 2019-04-19T18:57:18Z

PTAL @potiuk

milton0825 · 2019-04-19T19:11:50Z

Yeah let's ship this as it blocks the PY2 deprecation.

potiuk

Looks good. Hopefully I got to fix the last issue with multi-staging docker soon and we get rid of it altogether ;)

zhongjiajie · 2019-04-20T14:15:16Z

Ping @Fokko @ashb @kaxil @XD-DENG PTAL.

zhongjiajie · 2019-04-20T16:05:56Z

@potiuk Do you have the rights to merge PR now. I check my dev-email and notice you become committer. BTW Congratulations 👍

zhongjiajie · 2019-04-21T03:57:15Z

@potiuk Thanks for merging.

zhongjiajie · 2019-04-21T04:18:45Z

@potiuk Notice that you don't change jira ticket status as RESOLVED, maybe you should take a look as #4966 (comment). FYI

potiuk · 2019-04-21T06:35:30Z

Yeah..I realised I have no necessary rights in JIRA yet. I am waiting for infra team to grant me the right. But I keep the list of tickets I merged so no worries !

Happy Easter :)

Fokko · 2019-04-21T10:18:19Z

Thanks @potiuk for picking this up. Happy Easter as well! 👍

This was referenced Apr 12, 2019

[AIRFLOW-4200][AIP-3-subtask-4] Remove all __future__ imports #5020

Merged

[AIRFLOW-4199][AIP-3-step3] Remove all sys version info #5019

Merged

jmcarp mentioned this pull request Apr 12, 2019

[AIRFLOW-4227] Use python3-style type annotations. #5030

Closed

6 tasks

This was referenced Apr 14, 2019

[AIRFLOW-4198] Remove all try/import compatibility imports #5091

Merged

[AIRFLOW-4208] Replace @abstractproperty by @abstractmethod #5041

Merged

jmcarp reviewed Apr 15, 2019

View reviewed changes

zhongjiajie force-pushed the ci_remove_py2_content branch 2 times, most recently from 59aa927 to a4097bb Compare April 15, 2019 06:28

[AIRFLOW-4296] Remove py2 in ci process

9ccf73f

zhongjiajie force-pushed the ci_remove_py2_content branch from a4097bb to 9ccf73f Compare April 15, 2019 08:16

potiuk approved these changes Apr 20, 2019

View reviewed changes

potiuk merged commit 0ac501f into apache:master Apr 20, 2019

zhongjiajie deleted the ci_remove_py2_content branch April 21, 2019 03:56

andriisoldatenko pushed a commit to andriisoldatenko/airflow that referenced this pull request Jul 26, 2019

[AIRFLOW-4296] Remove py2 in ci process (apache#5090)

33cf0b8

wmorris75 pushed a commit to modmed/incubator-airflow that referenced this pull request Jul 29, 2019

[AIRFLOW-4296] Remove py2 in ci process (apache#5090)

c57b694

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AIRFLOW-4296][AIP-3-subtask-18] Remove py2 in ci process #5090

[AIRFLOW-4296][AIP-3-subtask-18] Remove py2 in ci process #5090

zhongjiajie commented Apr 12, 2019

zhongjiajie commented Apr 12, 2019

kaxil commented Apr 12, 2019

zhongjiajie commented Apr 12, 2019

Fokko commented Apr 14, 2019

zhongjiajie commented Apr 15, 2019

jmcarp Apr 15, 2019

zhongjiajie Apr 15, 2019

zhongjiajie commented Apr 15, 2019

codecov-io commented Apr 15, 2019

codecov-io commented Apr 15, 2019 •

edited

Loading

zhongjiajie commented Apr 15, 2019 •

edited

Loading

Fokko commented Apr 15, 2019

zhongjiajie commented Apr 15, 2019

zhongjiajie commented Apr 15, 2019 •

edited

Loading

milton0825 commented Apr 16, 2019

zhongjiajie commented Apr 16, 2019

zhongjiajie commented Apr 16, 2019

milton0825 commented Apr 16, 2019

jmcarp commented Apr 19, 2019

mik-laj commented Apr 19, 2019

milton0825 commented Apr 19, 2019

potiuk left a comment

zhongjiajie commented Apr 20, 2019 •

edited

Loading

zhongjiajie commented Apr 20, 2019

zhongjiajie commented Apr 21, 2019

zhongjiajie commented Apr 21, 2019 •

edited

Loading

potiuk commented Apr 21, 2019

Fokko commented Apr 21, 2019

[AIRFLOW-4296][AIP-3-subtask-18] Remove py2 in ci process #5090

[AIRFLOW-4296][AIP-3-subtask-18] Remove py2 in ci process #5090

Conversation

zhongjiajie commented Apr 12, 2019

Jira

Description

Code Quality

zhongjiajie commented Apr 12, 2019

kaxil commented Apr 12, 2019

zhongjiajie commented Apr 12, 2019

Fokko commented Apr 14, 2019

zhongjiajie commented Apr 15, 2019

jmcarp Apr 15, 2019

Choose a reason for hiding this comment

zhongjiajie Apr 15, 2019

Choose a reason for hiding this comment

zhongjiajie commented Apr 15, 2019

codecov-io commented Apr 15, 2019

Codecov Report

codecov-io commented Apr 15, 2019 • edited Loading

Codecov Report

zhongjiajie commented Apr 15, 2019 • edited Loading

Fokko commented Apr 15, 2019

zhongjiajie commented Apr 15, 2019

zhongjiajie commented Apr 15, 2019 • edited Loading

milton0825 commented Apr 16, 2019

zhongjiajie commented Apr 16, 2019

zhongjiajie commented Apr 16, 2019

milton0825 commented Apr 16, 2019

jmcarp commented Apr 19, 2019

mik-laj commented Apr 19, 2019

milton0825 commented Apr 19, 2019

potiuk left a comment

Choose a reason for hiding this comment

zhongjiajie commented Apr 20, 2019 • edited Loading

zhongjiajie commented Apr 20, 2019

zhongjiajie commented Apr 21, 2019

zhongjiajie commented Apr 21, 2019 • edited Loading

potiuk commented Apr 21, 2019

Fokko commented Apr 21, 2019

codecov-io commented Apr 15, 2019 •

edited

Loading

zhongjiajie commented Apr 15, 2019 •

edited

Loading

zhongjiajie commented Apr 15, 2019 •

edited

Loading

zhongjiajie commented Apr 20, 2019 •

edited

Loading

zhongjiajie commented Apr 21, 2019 •

edited

Loading