-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #11 from Rafsan7238/large-file-fission-ingestion
Large file fission ingestion
- Loading branch information
Showing
72 changed files
with
1,527 additions
and
1,731 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,36 +1,71 @@ | ||
| FOR COMP90024 ASSIGNMENT 2 | ||
| BY TEAM 45: | ||
| William Chen 1400081 | ||
| Petr Andreev 1375858 | ||
| Rafsan Al Mamun 1407776 | ||
| Ojaswi Dheer 1447227 | ||
|
|
||
| # COMP90024_Assignment-2 | ||
| This repo contains the assignment 2 files for COMP90024: Cluster and Cloud Programming at the University of Melbourne. | ||
|
|
||
| ## Environment | ||
| Install stuff, set up clusters, install software. | ||
|
|
||
| 1. Connect to unimelb VPN. | ||
| 2. Start a docker runtime on your machine. | ||
| 3. Run Docker Compose in root folder. | ||
| - `docker-compose up` | ||
| - This will compose a local docker container that install all dev dependencies. | ||
| 4. Place a private ssh key as generated in MRC into a folder called `.ssh` in the root project. | ||
| - Note that the .ssh folder is gitignore'd and this should always be the case forever and always. | ||
| - Remember to restrict file access to this ssh key by running `chmod 600 ~/.ssh/{key name}.pem`. | ||
| - `chmod 600` prevents read and write access from any other users on your machine. | ||
| 5. Using a shell attached to this docker container, connect to the bastion node. | ||
| - `ssh -i ~/.ssh/{key name}.pem -L 6443:192.168.10.153:6443 [email protected]` | ||
| - Remember that after any ssh connection or port forward request, you should keep that terminal open and open a new one for further interaction with the cloud system. | ||
| 6. Obtain the Kube config from an Admin and add it to a folder called `.kube` in the root project. | ||
| ## Admins | ||
| William Chen [email protected] | ||
| Petr Andreev [email protected] | ||
| Rafsan Al Mamun [email protected] | ||
| Ojaswi Dheer [email protected] | ||
|
|
||
| ## Setup | ||
| ### Installation | ||
| - As with the assignment class repository, you should install the below: | ||
| - OpenStack clients 6.3.x ([Installation instructions](https://docs.openstack.org/newton/user-guide/common/cli-install-openstack-command-line-clients.html)). | ||
| > Note: Please ensure the following Openstack clients are installed: `python-cinderclient`, `python-keystoneclient`, `python-magnumclient`, `python-neutronclient`, `python-novaclient`, `python-octaviaclient`. See: [Install the OpenStack client](https://docs.openstack.org/newton/user-guide/common/cli-install-openstack-command-line-clients.html). | ||
| - JQ 1.6.x ([Installation instructions](https://jqlang.github.io/jq/download/)). | ||
| - Kubectl 1.26.8 ([Installation instructions](https://kubernetes.io/docs/tasks/tools/)). | ||
| - Helm 3.6.3 ([Installation instructions](https://helm.sh/docs/intro/install/)). | ||
| - Connect to [Campus network](https://studentit.unimelb.edu.au/wifi-vpn#uniwireless) if on-campus or [UniMelb Student VPN](https://studentit.unimelb.edu.au/wifi-vpn#vpn) if off-campus | ||
| - Get access to this project and create a key pair in MRC. Send the **public** key to an admin. Place the private key in you machines `~/.ssh` directory. | ||
| - Tighten access to this key `chmod 600 ~/.ssh/{key name}.pem`. | ||
| - Once your access key has been added to the project, you will be given a Kube config file. Add it to a folder called `.kube` in this project. | ||
| - As with the ssh key, run `chmod 600 ~/.kube/config`. | ||
| ??? I couldn't actually get the docker to work, I'm just using wsl right now. | ||
|
|
||
| ## Setting up | ||
| 1. Set up fission by running fission_startup.sh | ||
| - This will generate the packages used by fission | ||
| - These packages will be used to create fission functions | ||
| - These functions will get exposed in the endpoints described in the readme in `backend` | ||
| ### Connecting to the cluster | ||
| - Ensure you are on the Unimelb campus network, either using the campus wifi or through the [student VPN](https://studentit.unimelb.edu.au/wifi-vpn#vpn)! | ||
| - ssh into the cluster bastion node | ||
| `ssh -i ~/.ssh/{key name as made in installation}.pem -L 6443:192.168.10.153:6443 [email protected]` | ||
|
|
||
| - Port-forward the cluster's Fission router to your localhost. Run the below in a separate terminal. | ||
| `kubectl port-forward service/router -n fission 9090:80` | ||
|
|
||
| - Optionally, forward the cluster's kibana port to your localhost. Run the below in yet another separate terminal. This will give you local access to the kibana dashboard. | ||
| `kubectl port-forward service/kibana-kibana -n elastic 5601:5601` | ||
|
|
||
| ### Seeing function logs | ||
| - You can see the running logs of the repeating harvesters by using the lines | ||
|
|
||
| `fission function log -f --name addobservations` | ||
| `fission function log -f --name mharvester` | ||
|
|
||
| ## Usage | ||
| ### Static data manipulation | ||
| The notebook for manipulating elastic search indexes and documents is `data/elastic_client.ipynb`. This notebook contains functioning requests that can be used to create all indexes (existing indexes will be ignored) and/or upload data to these indexes. | ||
|
|
||
| Note that this notebook can only upload data to the static file indexes. Therefore it has intentional limitations: | ||
| - It can only insert data if the index is empty. | ||
| - The exception is the very large hourly air quality file which requires uploads in batches. | ||
| - The only means to delete an index is to delete it through Kibana, this makes it more difficult to accidentally delete any indexes. | ||
| - It can create the indexes for the two data sources with regular harvesting (see below), but it cannot add any data to these indexes. | ||
|
|
||
| ### Automatic data manipulation | ||
| There are two regularly updating data sources. Mastodon, and BOM (Bureau of Meteorology). These data sources are regularly added to via fission functions triggered by timers. The Mastodon set is updated every 5 minutes, the BOM set is updated every 15 minutes. | ||
|
|
||
| todo: | ||
| - add content description to this readme. | ||
| - add installation instructions??? | ||
| - add instructions on how to use the client. | ||
| ### Analytics | ||
| There are two notebooks for requesting data from elastic search and displaying analytics. (Rafsan and Ojawsi to write this section.) | ||
|
|
||
| ## Development | ||
| To upload any backend changes, run `sh publish_backend.sh` in the root directory of this project. This will: | ||
| - Package everything in `backend/` into a zip file. | ||
| - Wipe all fission packages, functions, and triggers. | ||
| - Recreate all fission packages, functions, and triggers using the zipped archive. | ||
|
|
||
| ## Tests | ||
| To run tests execute `pytest tests/fission_tests.py` from project directory | ||
| ## Tests | ||
| To run tests execute `pytest tests/fission_tests.py` from project directory |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.