Creating the EC2Architect

[JackUrb]

Overview

Introduces our new available backend for deploying Mephisto tasks. This version uses AWS EC2 instances, along with a host of other services, to set up an automated structure relying on just one SSL certificate. This allows multiple users to host tasks, and includes a fallback server that workers can see if one of the task servers is down (plus the server logs these bad requests, so we can trace down broken tasks!).

Implementation

The primary implementation is split into the pre-setup, which is handled by prepare_ec2_servers.py, and the deploy of task-related instances, which is handled by the EC2Architect. Both of these rely heavily on helpers written in ec2_helpers.py.

The core of the pre-work is to set up a VPC to host our servers, a load balancer to point to the fallback by default, and various auxilliary components to make this happen (like requesting and registering the SSL certificate). The fallback takes anything routed to *.<registered_domain.com>.

When an instance is being launched for a specific task, we register it to <task_name>.<registered_domain.com>, and route traffic from the load balancer based on that subdomain to the task server.

When a server is actually deployed, we set it up to run with systemctl remotely (which is what the .service files are for), and this methodology is used for both the fallback server and for task servers.

Files are stored in the servers directory containing information about launched servers, such that they can be referenced in the future and cleaned up when their usage is done (even if there's an issue with the python process).

For the future

An in-depth document on the architecture of what services are set up to run this architect, how the pre-setup work (involving the VPC and load balancer) differs from what is done to launch instances later, and more.

Testing

Tested by launching the parlai_chat with an ec2 architect after having launched the pre-setup with one of our internal roles. Servers successfully launch, deploy, then cleanup after the task is done. After executing everything, the only remaining resources are those for the fallback server.

[codecov-commenter]

Codecov Report

Merging #603 (fd5f629) into main (4937db1) will decrease coverage by 2.46%.
The diff coverage is 22.69%.

@@            Coverage Diff             @@
##             main     #603      +/-   ##
==========================================
- Coverage   66.36%   63.90%   -2.47%     
==========================================
  Files          83       85       +2     
  Lines        7605     8050     +445     
==========================================
+ Hits         5047     5144      +97     
- Misses       2558     2906     +348     

Impacted Files	Coverage Δ
...ueprints/static_html_task/static_html_blueprint.py	41.79% <ø> (ø)
...ephisto/abstractions/architects/ec2/ec2_helpers.py	16.72% <16.72%> (ø)
...histo/abstractions/architects/ec2/ec2_architect.py	35.25% <35.25%> (ø)
mephisto/operations/registry.py	87.17% <100.00%> (+0.16%)	⬆️
...tractions/architects/channels/websocket_channel.py	79.06% <0.00%> (-1.17%)	⬇️
mephisto/operations/supervisor.py	80.26% <0.00%> (-0.67%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4937db1...fd5f629. Read the comment docs.

[kushalchawla]

Dear @JackUrb, thank you so much for this PR. I have been facing some pushback from my IT team to use Heroku. Using EC2Architect would mean an all-AWS + AMT solution, which is perfect for me.

Requests:

1. Please let us know when EC2Architect is stable enough for us to explore and use for our studies.
2. When do you expect 1) to happen? What is the timeline in your mind?
3. The documentation that you plan for the future would be super helpful. But apart from that, let us know what additional steps are required at our end such as setting up a domain name, etc.
4. I am willing to provide feedback on this by trying it out for my use case. Let me know if you need me there. Otherwise, I will just wait for a while for this PR to mature.

Thanks!

[JackUrb]

Hi @kushalchawla, I'm hoping to have it committed Monday. The only thing you need is to register a domain name from somewhere, either with a standard registrar or somewhere that offers free domains. After that, running the setup script will tell you where to delegate your domain to, and then it will set up the rest (including registering certificates).

The one downside of the setup is it requires around 2 hours for the initial setup to propagate, after which you do incur costs while that is running (on the order of a dollar or so a day, given my current testing).

[JackUrb]

Merging without documentation for now after extended testing, will make official usage documentation next week.