Cloud Based API Instance for Developers

[palisadoes]

We need to have a cloud based API instance for developers to test against. It must:

1. Be integrated into our CI/CD pipeline so that the app is rebuilt with every newly merged PR.
2. Initialize the database to a known working state with appropriate services will be restarted when this happens: (there are existing API calls for this):
   1. with each CI/CD update
   2. every 24 hours
3. Use a talawa.io sub-domain as the endpoint
4. The API instance will have a rate limiting mechanism to reduce the risk of abuse
5. Preferably use a free service
6. Use a palisadoes role based account for management
7. The demo cloud API must be easily switched between branches as the code source. We will be migrating to master as the default branch soon.

Please coordinate these efforts with @noman2002 and @kb-0311. Ask to be assigned this task by them.

[kirtanchandak]

@noman2002 @kb-0311 I can give this issue a try

[kirtanchandak]

there is file named push-documentation.yml in github/workflows folder. Do we need to make changes here or create a new file?

[palisadoes]

1. We used to have this working with Heroku using this file. https://github.com/PalisadoesFoundation/talawa-api/blob/develop/.github/workflows/ci.yml.archive
2. There was some setup that was required with the Heroku cloud to make it work correctly whenever we did a merge. We decided to stop using Heroku due to the cost of having the instance running. We would like to use a fremium service.

[noman2002]

@kirtanchandak Please proceed with this. You can use that file, you just need to do some minor changes. But I would suggest please check other solutions like render.com instead of Heroku.

[vasujain275]

Hi @noman2002 @kirtanchandak, I have been working on this same issue containerization talawa-api using docker here - #1418 that was merged yesterday and was working on developing a ci/cd pipeline for this exact use case, @kirtanchandak can I work on this issue if you haven't made any significant progress yet?

[palisadoes]

One issue per person

[vasujain275]

One issue per person

Okay

[github-actions]

This issue did not get any activity in the past 10 days and will be closed in 180 days if no update occurs. Please check if the develop branch has fixed it and report again or close the issue.

[palisadoes]

@kirtanchandak Are you still working on this?

[vasujain275]

@palisadoes I can work on this if @kirtanchandak is not working on this

[palisadoes]

Assigning to@vasujain275. Please work with @noman2002 and @kb-0311 for comments, advice and reviewing the PR

[vasujain275]

Hi @noman2002 and @kb-0311, I did some research and found that the MongoDB Atlas free tier is good enough for our use case. I looked into redis free tier at render.com, it has 25 MB of memory with a 50 connection limit and no persistence storage. We can also host our API on render.com but it will be shut down after an inactivity of 5 minutes and will be restarted when called again. Will that render.com's inactivity delay or redis free tier will be a problem?

[kb-0311]

@vasujain275

1. Instead of render are there other free tier cloud instances that we can use to deploy our application?
2. Well in production the Redis server was to be used as an in-memory cache, not a global cache so we also need to have the option to access the file system/ services of the server where the API is deployed.
3. Render and stuff is fine but if possible can you also explore free credit use of GCP and/or Azure? Do let me know if those are feasible.

[vasujain275]

@kb-0311 We can use the AWS free tier, which will have a t2.micro ec2 instance free for 12 months per account ( we will need a credit card to avail this 12-month free tier ). We can use the docker file and docker-compose file I created in a recent issue, we can set a ci/cd pipeline to push our image to the docker hub on each push and then pull to e2c using GitHub actions. We can then run docker-compose on e2c also using GitHub ci/cd that can run containers of both redis and MongoDB, although I think it will be best to run MongoDB on Atlas as it has a good free tier and will reduce the load on our t2.micro ec2 instance if we run it on atlas. We can also use azure/gcp, they also have this exact free tier of 12 months that requires a credit card on signup.

[palisadoes]

@vasujain275 Ideally we also want:

1. A single account to manage all the required resources.
2. The ability to have trusted contributors update the cloud service parameters to make sure the CI/CD functions
3. Ensure that only one person has the right to purchase new services or upgrade any existing ones. This will help reduce long term costs and potential abuse.

Other:

1. @kb-0311 @noman2002 Do you have anything else to add?
2. Is the most recent comment from @vasujain275 in keeping with the desired outcome?

[vasujain275]

@palisadoes

1. Only a single aws account is required to manage this cloud instance.
2. We only need to add the cloud service parameters as github secrets of this talawa-api repo. Then the secrets can be safely and easily used in our github actions using environment variables.
3. Addition of github secrets can be done by any trusted contributors.
4. Only the person who will sign up for aws instance will have the right to purchase new services or upgrade any existing ones.
5. No one will need to directly ssh into the instance to make changes, as they will be done via ci/cd pipeline.
6. As for MongoDB Atlas, we will only need to add the MongoDB Atlas Cluster url as github secret of this repo.

Q. How can github actions do all the changes in our aws instance?
Ans. Docker Containers, we will never install the talwa api directly to our aws instance instead we will use docker container of our api that can be rebuilt and restarted every time we push changes to our repo.

My proposed solution -

1. Firstly, I will create all github actions files and docker compose files required to completely set up talawa-api instance on aws via ci/cd.
2. I will then test all the github actions and compose files on my personal aws ec2 instance to make sure all the things are working as intended. This way I will not be required to have direct access to talawa-api actual production ec2 instance.
3. When all the github actions are completed and tested by me then one of the trusted contributors can signup for aws instance and MongoDB Atlas. Then he will set up the required env variables as github secrets in this repo.

@noman2002 @kb-0311 , appreciate your feedback on possible improvements in this approach

[palisadoes]

1. Based on your answer we will need to manage two cloud accounts. Can this be done using only one account?
2. We need to make sure that more than one person can access the AWS dashboard for purposes of troubleshooting. Can this be done without them having access to the credit card information to add new services?
3. If we use a MongoDB Atlas account, how can we guarantee the same AWS access requirement above?

[kb-0311]

@vasujain275

1. Yes your solution and flow make sense to me.
2. At this stage it would be easier to showcase the CI/CD pipeline using your forked talawa-api instance as the base for now and your free tier AWS ec2 instance as the deployment server.
3. Create a new branch and configure the docker-compose file as necessary, since you are the owner of the forked repo you can also add the GitHub secrets for Altas and aws.
4. Then make demo PR merges to that branch and demonstrate how to CI/CD pipeline will work using a video.
5. This will help us understand potential problems that could occur at a given point more precisely.

@palisadoes

1. Can you elaborate on the two accounts? Are you referring to one account for aws and another for MongoDB Atlas or something else?
2. In aws and MongoDB I think there are certain ABAC and RBAC features for this purpose. Maybe we can take a look at that? @vasujain275

[vasujain275]

@palisadoes

1. By two accounts, If you referring to one account for aws and another for MongoDB Atlas then in my opinion it's better to have them separated as it will decrease the load on our ec2 machine.
2. As @kb-0311 suggests there is role-based access in both aws and MongoDB Atlas that we can use to give limited access to another person for troubleshooting purposes.

@kb-0311
I will start working on this by creating a new branch "deployment" in my forked repo and will follow the process you mentioned in the previous comment.

[palisadoes]

1. Yes one in AWS and the other with Mongo
2. Remember this will be solely for demo purposes not a heavily used system with only a few transactions per hour for developers to test against. Keep this in mind with your final recommendation.

[vasujain275]

@palisadoes Okay I will keep that in mind

[palisadoes]

@vasujain275 @SiddheshKukade The server has been upgraded to this:

Please coordinate on getting this to work. Remember we want this to be completely automated:

1. Every 24 hours:
   1. The DB is reset
   2. The local repos are reset to the latest develop
2. The API and Admin instances must reload with every new PR merge into the develop branch

I'm hoping that this will be sufficient as an initial security measure.

Additionally:

1. I have created a talawa-admin user for Admin to use when running
2. I have created a talawa-API user for API to use when running
3. Use Apache as a reverse proxy for both apps

When it is all working we'll need the posix shell for the talawa-* users to be /usr/sbin/nologin for added security. Please investigate how to get docker running as a user running with this shell. It's very important.

[palisadoes]

The DNS entries to use are:

1. demo.talawa.io (Admin)
2. api-demo.talawa.io (API)

They are the same server, but the separate DNS entries should help us in future if we want to host them on separate machines.

[palisadoes]

@SiddheshKukade The API didn't survive the machine reboot when it was upgraded.

[palisadoes]

Team,

You can use these files for guidance in setting up Admin and the API as linux daemons. They are from our switchmap repo and work correctly. This will allow you to control the apps using the systemctl command

• https://github.com/PalisadoesFoundation/switchmap-ng/tree/main/examples/linux/systemd

You can use this file as an configuration example for setting up an Apache reverse proxy.

• https://github.com/PalisadoesFoundation/switchmap-ng/tree/main/examples/linux/apache

You can use this file as an configuration example for setting up an Nginx reverse proxy.

• https://github.com/PalisadoesFoundation/switchmap-ng/tree/main/examples/linux/nginx

You'll have to adjust them accordingly.

Unless you can find a way to reduce the Docker resource utilization we'll need to run the apps natively.

[vasujain275]

@palisadoes Let me and @SiddheshKukade discuss the potential solution on meet and get back to you.

@SiddheshKukade Can you contact me on slack/mail to setup a meet on a suitable timing this weekend.

[palisadoes]

I've created two issues that hopefully will help with the long term solution

System Daemon

• API: Create systemd file so the app can run as a Linux system daemon. #2737
• Admin: Create systemd file so the app can run as a Linux system daemon. talawa-admin#2595

Docker Performance

• Admin: Extremely Poor Docker Resource Utilization Efficiency talawa-admin#2596
• API: Extremely Poor Docker Resource Utilization Efficiency #2730

[vasujain275]

@palisadoes

The Docker performance is extremely poor due to issues with our build process. This was the root cause earlier when we encountered a block, and it still is. We need to prioritize fixing the build process first.

To address this, we require someone with good GraphQL experience to resolve the build process issues. Once that is done, we can easily deploy the API admin on the same VPS using Docker alone. This approach will be highly efficient.

With Docker and Caddy (as the reverse proxy), there will be no need to create a systemd files too.

[palisadoes]

@vasujain275 Please create an issue detailing what is required. Hopefully we can find someone who will be willing to dedicate the time. In the issue, please state that we need an experienced person with a long and active GitHub history.

[palisadoes]

@SiddheshKukade We still need a running API instance. Have you managed to do more testing of your approach? I'd like the community to start using it before the end of the year

[vasujain275]

@palisadoes Okay let me create an issue with error logs and requirements

[github-actions]

This issue did not get any activity in the past 10 days and will be closed in 180 days if no update occurs. Please check if the develop branch has fixed it and report again or close the issue.

[vasujain275]

@palisadoes

1. I have verifed that recent the recently merged pr by @adithyanotfound fixed the build issue. Now the npm run prod is working as aspected.
2. Now we can run the talawa-api and talawa-admin both on a single VPS using docker with the performance issues.
3. I am currently working on the modified docker files and will submit a pr with those and new scripts.
4. Here is the worflow I came with, listing it here to avoid confusion -

• Both talawa-api and talawa-admin will run on talawa user with caddy serving as reverse proxy in both of there docker image.
• By using Caddy instead of nginx as a revese proxy we elemimate the need for certbot and its realted scripts.
• Every 24hr a cron job will run that will pull the latest commits and rebuild the docker containers in the VPS
• Every 24hr another cron job will also run to reset the mongodb database.

Do you suggest any changes or should I proceed with this?

[palisadoes]

@vasujain275

Based on this, do we need to differentiate between prod and dev environments? The poor performance of the dev environment for the API is a real cause of hardship. I'd prefer a great user experience for both end users and developers alike.

Maintaining two environments as an open source project with limited resources doesn't seem feasible to me. Is this a practical solution? Do you see any potential downsides to this approach, and if so do they outweigh the benefits in your opinion?

I welcome comments from others

[vasujain275]

@palisadoes

We don't need two different environments for talawa api, the production one will be sufficient to handle both requests from talawa admin/app and developers.

Do we also need Apollo Graphql Playground Hosted on the VPS? Because it will be resource intensive as it requires self hosting a standalone apollo server. Currently the prod build disables the Apollo playground i.e. /graphql endpoint.

[adithyanotfound]

@palisadoes

The production environment should be enough. Although having a development instance would be nice since we can then implement CD pipelines which would make deployment more easier. But due to limited resources, the best course of action would be to just have the prod instance. The current PR workflow should be sufficient to handle most cases. Occasionally, any errors can be addressed when the main branch is merged with the develop branch, if necessary.

[palisadoes]

1. Lets get rid of the development environment all together. I'll create an issue for it.
2. The resource intensity is too great in dev.

[vasujain275]

@palisadoes There is no need to remove dev docker files, developers can use that on their local machines for local development. We will use production docker files in cloud instance anyway.

[palisadoes]

Thanks

[palisadoes]

Here is the issue

• Remove the Development environment from the code base #2800

[vasujain275]

@palisadoes

I will in the meantime update the cloud instance scripts, production docker files and test them on vasu user on the vps.

[palisadoes]

This will be an interesting issue. Get ready for the migration!

• MinIO Async Deletion Service - develop-postgres branch #2808

[vasujain275]

@palisadoes

Should I wait for this postgres migration to complete first? Or are we planning to go with mongodb for now?

[palisadoes]

1. We'll run everything on the server using develop.
2. Just before the migration starts I'll merge develop-postgres into develop

[vasujain275]

@palisadoes

Okay, I will wait for the merge then modify the docker files and scripts for postgres only.

[palisadoes]

@vasujain275

1. We need this functioning. Only on develop.
2. I've just merged the latest Admin develop-postgres into develop
   1. Merge the latest develop-postgres to develop for Cloud instance talawa-admin#3183
3. Please try to get this running by the weekend.