Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cleanup ftp publisher code #64

Merged
merged 1 commit into from
Jan 21, 2022
Merged

cleanup ftp publisher code #64

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 11 additions & 17 deletions src/publishers/publishers/ftp_publisher.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,12 +20,6 @@ class FTPPublisher(BasePublisher):

def publish(self, publisher_input):

def get_prefix_and_ftp(ftp_server_url):
url = ftp_server_url.split(' ')
prefix_ftp = url[0].strip()
url_ftp = url[1].lstrip()
return prefix_ftp, url_ftp

try:
ftp_url = publisher_input.parameter_values_map['FTP_URL']

Expand All @@ -49,32 +43,32 @@ def get_prefix_and_ftp(ftp_server_url):
f.write(bytes_data)
f.close()

ftp_prefix = get_prefix_and_ftp(ftp_url)[0]
ftp_url = get_prefix_and_ftp(ftp_url)[1]

ftp_data = urlsplit(ftp_url)

ftp_hostname = ftp_data.scheme + '.' + ftp_data.hostname
ftp_hostname = ftp_data.hostname
ftp_username = ftp_data.username
ftp_password = ftp_data.password
ftp_port = ftp_data.port

remote_path = ftp_data.path + filename

if ftp_prefix == 'sftp:':
if ftp_data.scheme == 'sftp':
ssh_port = ftp_data.port if ftp_data.port else 22
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(ftp_hostname, ftp_port, ftp_username, ftp_password)
ssh.connect(hostname=ftp_hostname, port=ssh_port, username=ftp_username, password=ftp_password)
sftp = ssh.open_sftp()
sftp.put(filename, remote_path)
sftp.close()
os.remove(filename)
else:
elif ftp_data.scheme == 'ftp':
ftp_port = ftp_data.port if ftp_data.port else 21
ftp = ftplib.FTP()
ftp.connect(ftp_hostname, ftp_port)
ftp.connect(host=ftp_hostname, port=ftp_port)
ftp.login(ftp_username, ftp_password)
ftp.storbinary('STOR ' + remote_path, open(filename, 'rb'))
ftp.quit()
os.remove(filename)
else:
raise Exception("Schema '{}' not supported, choose 'ftp' or 'sftp'".format(ftp_data.scheme))
except Exception as error:
BasePublisher.print_exception(self, error)
finally:
os.remove(filename)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initially I thought finally is not a good place for os.remove(), because it wipes the file even if the upload was not successful. That's why in the original version it was inline, and also the reason why someone thought temporary local file names should be predictable (to allow troubleshooting).

However on the other read of our code, we should probably handle this better upstream, and also prevent a race condition with naming at the same time. I'll open an issue for that, and accept this code so the failed uploads don't clutter the container forever.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or use else: in the try-except clause

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we could have just left it as it was, but I like this approach, followed by more cleanup which I propose in #66