Add check for PEM exists for Hive and Livy DAG.

[rajaths010494]

While running hive or livy dag where we use ssh to login into the machine pem file was missing in /tmp folder.
Now PEM is downloaded into /tmp in SSH task so PEM is available at the same worker node.
This PR checks whether the file exists or not while copying the pem into /tmp and downloading before SSH into the machine.
If it doesn't exist then it will raise an exception.

closes: #620

[codecov]

Codecov Report

Base: 98.30% // Head: 98.30% // No change to project coverage 👍

Coverage data is based on head (513c50d) compared to base (8e11ca6).
Patch has no changes to coverable lines.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #621   +/-   ##
=======================================
  Coverage   98.30%   98.30%           
=======================================
  Files          79       79           
  Lines        4131     4131           
=======================================
  Hits         4061     4061           
  Misses         70       70           

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[pankajkoti]

I am thinking that the tasks in the DAG could be running on different workers and hence there could be failure as the file system would not be common between them. Can we think if this could be the reason and if yes, how we can resolve this?

[rajaths010494]

I am thinking that the tasks in the DAG could be running on different workers and hence there could be failure as the file system would not be common between them. Can we think if this could be the reason and if yes, how we can resolve this?

when I checked in the worker node the pem file wasn't present. So added the check if the file exists or not. as you mentioned if a different worker for tasks but I see only one worker node.

if that's the case multiple workers are present then files created in one worker should also be made available in the workers as well.

[pankajkoti]

I am thinking that the tasks in the DAG could be running on different workers and hence there could be failure as the file system would not be common between them. Can we think if this could be the reason and if yes, how we can resolve this?

when I checked in the worker node the pem file wasn't present. So added the check if the file exists or not. as you mentioned if a different worker for tasks but I see only one worker node.

if that's the case multiple workers are present then files created in one worker should also be made available in the workers as well.

yes, currently we have one worker so that may not be the problem. However, I believe there is no sync mechanism available which makes files created on one worker node available on other worker nodes. Even if such a sync existed, there would be delays in sync and we should not rely on a such sync with lags.

Although this change is needed and helps us in raising errors earlier, it still won't solve the problem that the DAG is failing when the copy fails. I believe we need something more to solve this problem of intermittent failures.

[rajaths010494]

I am thinking that the tasks in the DAG could be running on different workers and hence there could be failure as the file system would not be common between them. Can we think if this could be the reason and if yes, how we can resolve this?

when I checked in the worker node the pem file wasn't present. So added the check if the file exists or not. as you mentioned if a different worker for tasks but I see only one worker node. if that's the case multiple workers are present then files created in one worker should also be made available in the workers as well.

yes, currently we have one worker so that may not be the problem. However, I believe there is no sync mechanism available which makes files created on one worker node available on other worker nodes. Even if such a sync existed, there would be delays in sync and we should not rely on a such sync with lags.

Although this change is needed and helps us in raising errors earlier, it still won't solve the problem that the DAG is failing when the copy fails. I believe we need something more to solve this problem of intermittent failures.

yes, we can merge the copy task into the task which uses PEM and make it one so whenever task on any worker it copies the PEM at that time and then ssh into the machine.

[rajaths010494]

I am thinking that the tasks in the DAG could be running on different workers and hence there could be failure as the file system would not be common between them. Can we think if this could be the reason and if yes, how we can resolve this?

when I checked in the worker node the pem file wasn't present. So added the check if the file exists or not. as you mentioned if a different worker for tasks but I see only one worker node. if that's the case multiple workers are present then files created in one worker should also be made available in the workers as well.

yes, currently we have one worker so that may not be the problem. However, I believe there is no sync mechanism available which makes files created on one worker node available on other worker nodes. Even if such a sync existed, there would be delays in sync and we should not rely on a such sync with lags.
Although this change is needed and helps us in raising errors earlier, it still won't solve the problem that the DAG is failing when the copy fails. I believe we need something more to solve this problem of intermittent failures.

yes, we can merge the copy task into the task which uses PEM and make it one so whenever task on any worker it copies the PEM at that time and then ssh into the machine.

@pankajkoti I have modified the code to download the PEM as part of the SSH task, so now PEM gets downloaded into the same worker where SSH tasks run.