Add sftp connector

[fvankrieken]

The problems

• We need to distribute PLUTO major releases to an SFTP server
• We need to ingest source data from SFTP servers
• We want to showcase the potential for various destination types with dcpy.lifecycle/metadata

Acceptance Criteria

• the SFTP connector can push and pull files form SFTP servers
• handle the variety of credentials for the servers we need to push/pull from
• easy to run only unit tests or only integration tests

Not In Scope

• Deciding exactly what "integration" means. New tests here will require dependencies to exist a server) unlike unit tests.
• Refactoring existing tests that require external dependencies

Solution

...

---

There are a few needed use cases for this.

1. distributing pluto to ginger server - see FTP (Ginger) destination for PLUTO #1226
2. ingesting pluto inputs from same server - see Ingest migration/Ingest v2 - migrate "Pluto Input" datasets #1261

We'll also need to add ginger credentials to onepass if they're not there already, and configure a known host for ginger as well

There's some code in a few private repos that might be useful to use...

Tagging @alexrichey for visibility - didn't see an issue yet and wanted this tracked as a blocker for #1261

[damonmcc]

specification (expected "files changed")

• add functionality to connects.ftp.FTPConnector to push and pull from an SFTP server
• add tests to test/conenctors/test_sftp.py
  • create an SFTP server for test to interact with via docker compose during test_helper.yml

notes

• vocab: prefer sftp over ftp, so will rename things to be consistent
• types of tests
  • we've talked about unit vs integration vs e2e but we haven't decided what each of those should mean/be in our repo
  • these SFTP tests are related to this discussion and will certainly inform it
  • but deciding what our test types are, how they work, and where they live is out of scope

[alexrichey]

@damonmcc This is a good start. I think it'd be good to get a little more details about your intentions for a few different things:

1. What (if any) library are you thinking of using for sftp?
2. How are we going to accommodate different profiles/credentials for different SFTP servers? Related: how are we going to dispatch differently based on the destination in the metadata?
3. Related... when would we want to pull credentials for, say, Ginger from 1pw? It would have to be somewhat dynamic, unless we want to have very specific Ginger distribution jobs.
4. We'd be adding a fairly new type of test (🎉!) but how are we going to separate it from the other tests? (To your point, figuring everything out for testing vocab is out of scope, but this does force us to make at least one choice here)
5. What needs to happen to integrate this connector into Ingest and Distribution?

[alexrichey]

Just a potentially relevant note for when you're solutioning out how to invoke subsets of tests: for those that shouldn't make http callouts, this plugin is very slick, and might be a nice safeguard for true unit tests.