[JENKINS-67963] Add option to save bandwidth and resources, which are wasted unnecessarily

[parvit]

[fixes #65]

It introduces the requested feature by which it is now possible to activate a configuration item under the SSH Host page (disabled by default) that will track the selected files for upload and check if it is really necessary to upload it again.

The tracking is done in a json file in job-specific folder and two values are tracked for every file:

• Last modified timestamp
• MD5 hash

The files are uploaded only if either:

• The file was not previously uploaded
• The new file timestamp is newer AND the MD5 hash has changed

As an optimization if the timestamp is not newer than the MD5 hash is not recomputed (which could be slow for big files).

• Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
• Ensure that the pull request title represents the desired changelog entry
• Please describe what you did
• Link to relevant issues in GitHub or Jira
• Link to relevant pull requests, esp. upstream and downstream changes
• Ensure you have provided tests - that demonstrates feature works or fixes the issue

[parvit]

For reference the job configuration i have used is attached also:
remote-deploy-test.xml.txt

• 1-check-uploads-no-file.log
  Initial scenario, the file has not been uploaded previously, is uploaded correctly.

• 2-check-uploads-same-file.log
  Same file is generated again, is skipped correctly.

• 3-check-uploads-only-touch.log
  Same file is selected again but has newer modification date, is skipped correctly.

• 4-check-uploads-new-file.log
  A file with new content is selected, is uploaded correctly.

Following are the same tests but with feature disabled, in all cases the file is uploaded correctly.

• 1-no-check-uploads-no-file.log
• 2-no-check-uploads-same-file.log
• 3-no-check-uploads-only-touch.log
• 4-no-check-uploads-new-file.log

[gmcdonald]

Thanks, tests look ok, except the output of each log shows 'SSH:Transferred 1 file(s)' - even when the file was skipped - a bug ?

[parvit]

@gmcdonald Hi, no that seems to be a behavior in BPTransfer class of the parent plugin publish-over. To me there does not seem to a way to influence that count from the ssh plugin.
To change that would have been a bit out of scope for this change so i've opted to add the warning.

[ttodua]

@parvit in the top post of this PR, include such phrase in the bottom (or anywhere):

[fixes #65]

and it will link this PR to that issue's closure.

[parvit]

Changed, I thought however that i did already link it with my first message.

[gmcdonald]

Merging this, though to note here as in the Issue that it would be much more useful if this was configurable at the job level rather than one big switch on/off for every job that only an Admin can configure - a possible future feature request,

[ttodua]

@gmcdonald can you tell, when the updated version (which would have this merged PR) will be released? the official page https://plugins.jenkins.io/publish-over-ssh/#documentation still shown 1.24 (which was "9 months ago")

[gmcdonald]

Hi @ttodua yes I am planning on doing a new release soon, just have a few bits to tidy up.

[ttodua]

@gmcdonald thanks for quick reply, how long it would take? passed several months and I have been hoping to get that released. any approximates?
thank you !

[ttodua]

@parvit would you be able to make a new PR (I would make a small bounty for that too), so the plugin can have option to check/grab only those files which are presented as "changed/added" from github push action? i.e. from payload it's visible which files are changed on the last commit https://pastebin.com/raw/u8u0B4PB so plugin could download only affected files. what do you think (also @gmcdonald), maybe it will speed up the process, or not that much?

[parvit]

Hi, thanks for reaching out to me. I think it would make sense if you have that info so you can skip checking the md5. But would you ignore the other files as not changed or do we check them the same? What i mean is: would this new info be just a hint or the actual list to upload? Il Mar 15 Nov 2022, 07:55 T. Todua ***@***.***> ha scritto:

…

@parvit <https://github.com/parvit> would you be able to make a new PR (I would make a small bounty for that too), so the plugin can have option to check/grab only those files which are presented as "changed/added" from github push action? i.e. from payload it's visible which files are changed on the last commit https://pastebin.com/raw/u8u0B4PB so plugin could download only affected files. idk, maybe it will speed up the process, or not at all? — Reply to this email directly, view it on GitHub <#245 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWGJD4HYDUM7MFTBGOQSWADWIMXUVANCNFSM5WWQOMAA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[ttodua]

@gmcdonald sorry for bothering, may I ask if there is any way we could help you to push forward the release?

[ttodua]

@gmcdonald sorry for bothering, may I ask if there is any way we could help you to push forward the release?

[ttodua]

@gmcdonald
can you please respond, say at least two words? thanks.

[gmcdonald]

Hey @ttodua , and others, apologies, I have fallen behind on releases. Currently there is a bit of a lift to get back upto date on being compatible with current Jenkins versions and release practices. I will get back on this and let you know a timeline asap

[ttodua]

Hey @ttodua , and others, apologies, I have fallen behind on releases. Currently there is a bit of a lift to get back upto date on being compatible with current Jenkins versions and release practices. I will get back on this and let you know a timeline asap

thanks for the heads up !

[ttodua]

@gmcdonald if you have some thoughts on this.

[ttodua]

Hey @parvit,
is it possible that you published this extension to Jenkins repo?

[parvit]

Hey @parvit, is it possible that you published this extension to Jenkins repo?

No i haven't, i was still waiting for your response to the comment of Nov 15 of last year.

[NicolasSanchez80]

Hi @parvit , can you test it in pipeline? In my case, it's doesn't works. No md5 reference inside logs. Thanks,

[ttodua]

@parvit please drop me your mail (or contact me at toduatt@ gmail). I want you to open "issues" functionality on your fork: https://github.com/parvit/publish-over-ssh-plugin/
and then there I would submit a new issue with bounty.
so, development continued on your fork and possibly you published the plugin on jenkins repo, as current plugin development seems abandoned by author.

[parvit]

Hi, i'm open to this but please know that the bountysource platform is not usable anymore. If you're open to discussing directly with me the bounty then please confirm it and i'll send you my email. Regards, Il Dom 22 Ott 2023, 13:23 T. Todua ***@***.***> ha scritto:

…

@parvit <https://github.com/parvit> please drop me your mail (or contact me at toduatt@ gmail). I want you to open "issues" functionality on your fork: https://github.com/parvit/publish-over-ssh-plugin/ and then there I would submit a new issue with bounty. — Reply to this email directly, view it on GitHub <#245 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWGJD4GDRTL55HHTAXC7ZQDYAT6ZXAVCNFSM5WWQOMAKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCNZXGQYDMNZRGMYA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[ttodua]

@parvit

ok, i've opened here: #327