Remote Archive

[scimerman]

This is first PR of Remote Archive solution

It implements

• a standardized and secure ssh way access to the surf archive
• accessible as any other storage that we have currently in use
• connections are established on-demand when accessed, and unmounted after an idle period
• all mount connections are multiplexed (aggregated) into single one with longer timeout period

Performance

• the connection is established by the one (elevated) user
• depends on speed of network and storage (local disk and disks on Surf side)
• tested on Hyperchicken and Talos (where local disk speed was main difference)

Security

• users are not able to control their access credentials
• users are not able to access data of another groups

Data retrieval

• users who can browse the archive files, can also submit a command to
  • list files state on the archive (is on disks, on tape etc)
  • move the data between disk and tape
• works without a direct access of the user: works in a queuing system, which checks for the permissions

TODO

• move to production account from the archive test account
• write user documentation
• find test users to use test archive account, teach them how to use archiving solution - collect feedback
• test performance on one of the production machines