Skip to content

Assignment 3 Proposal

Jump to bottom
Aniruddha Patil edited this page Apr 23, 2020 · 10 revisions

Automating file transfers for Airavata's Managed File Transfer

Members

  • Nikita Bafna
  • Shivali Jejurkar
  • Aniruddha Patil

Problem Statement

  • Currently, the file transfers through Airavata's Managed File Transfer (MFT) are requested manually.
  • In order for a user to automate backups/file transfers, the user would have to write their own scripts.
  • This would result in the automation of file transfers becoming a grueling task for users, along with the use of unnecessary resources for potentially transferring unnecessary files.
  • We intend to automate these transfers so that this grueling task can be circumvented and unnecessary resource usage can be avoided.

Benefits to Community

Any user or a set of users using Airavata requiring a regular backup/transfer of files would be benefited. This eliminates the need for the users to write their own scripts to schedule file transfers. If the functionality of scheduled file transfers is provided by Airavata, the overload in terms of redundant file transfers can be minimized. This will put the users to ease, conserve compute resources and save storage.

Use Case

  • A user specifies a list of files/folders to watch and a period of time for syncing them.
  • File Autotransfer is responsible for making these transfers and notifying the users about its status over mail.
  • An entity qualifies for auto-transfer if it has been newly created or modified (based on md5 checksum).

Deliverables

  • A well-defined API that enables users to schedule file transfers.
  • Email notification of file transfer status.

Implementation Plan

Phase 1

  • Scaffolding the existing implementation of Airavata's MFT for development purposes.
  • Designing an architecture for the scheduling module.
  • Defining APIs that would be exposed to the users, so that it can be integrated into the dashboard in the future.

Phase 2

  • Scheduling a single file transfer from a source to a destination.
  • Extending the single file transfer to a list of files and folders.
  • Email notification on completion or failure of file transfer.

Phase 3

  • Extending the task of scheduling transfers to syncing entities.
  • Writing clear and concise documentation.
  • Extensive testing to ensure reliability.
  • Creating a pull request to apache/airavata-mft

External communication links

Clone this wiki locally