moved all tests from src/it to src/test/it #10
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
|
|
||
| image: | ||
| file: .gitpod.Dockerfile | ||
|
|
||
| tasks: | ||
| - init: | | ||
| sbt clean compile | ||
| sbt test | ||
| sbt package |
|
There was a problem hiding this comment. Why are we still installing pre-requisites for batect? There was a problem hiding this comment. Lets also remove this from README local Additionally, it would be great to include somewhere with a big heading on how to run Test/IntegrationTest How can we trigger a single test? There was a problem hiding this comment. Removed pre-requisites for batect and local readme. I have added the command to run a testClass, not able to find one to run single test though. Also since we move all the tests under IT folder, we can run IntegrationTest via sbt test command only |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,87 +1,80 @@ | ||
| # Data transformations with Scala | ||
|
|
||
| This is a collection of jobs that are supposed to transform data. | ||
| These jobs are using _Spark_ to process larger volumes of data and are supposed to run on a _Spark_ cluster ( | ||
| via `spark-submit`). | ||
|
|
||
| ## Gearing Up for the Pairing Session | ||
|
|
||
| **✅ Goals** | ||
|
|
||
| 1. **Get a working environment** | ||
| Either local ([local](#local-setup), or using [gitpod](#gitpod-setup)) | ||
| 2. **Get a high-level understanding of the code and test dataset structure** | ||
| 3. Have your preferred text editor or IDE setup and ready to go. | ||
|
|
||
| **❌ Non-Goals** | ||
|
|
||
| - solving the exercises / writing code | ||
| > ⚠️ The exercises will be given at the time of interview, and solved by pairing with the interviewer. | ||
|
|
||
| ## Pre-requisites | ||
|
|
||
| Please make sure you have the following installed | ||
|
|
||
| * Java 11 | ||
| * Scala 2.12.16 | ||
| * Sbt 1.7.x | ||
| * Apache Spark 3.3 with ability to run spark-submit | ||
|
|
||
| ## Local Setup Process | ||
|
|
||
| * Clone the repo | ||
| * Package the project with `sbt package` | ||
| * Ensure that you're able to run the tests with `sbt test` (some are ignored) | ||
| * Sample data is available in the `src/test/resource/data` directory | ||
|
|
||
| > 💡 If you don't manage to run the local setup or you have restrictions to install software in your laptop, use | ||
| > the [gitpod](#gitpod-setup) one | ||
|
|
||
| ### Gitpod setup | ||
|
|
||
| Alternatively, you can setup the environment using | ||
|
|
||
| [](https://gitpod.io/#https://github.com/techops-recsys-lateral-hiring/dataengineer-transformations-scala) | ||
|
|
||
| There's an initialize script setup that takes around 3 minutes to complete. Once you use paste this repository link in | ||
| new Workspace, please wait until the packages are installed. | ||
| After everything is setup, select Poetry's environment by clicking on thumbs up icon and navigate to Testing tab and hit | ||
| refresh icon to discover tests. | ||
|
|
||
| Note that you can use gitpod's web interface or | ||
| setup [ssh to Gitpod](https://www.gitpod.io/docs/references/ides-and-editors/vscode#connecting-to-vs-code-desktop) so | ||
| that you can use VS Code from local to remote to Gitpod | ||
|
|
||
| Remember to stop the vm and restart it just before the interview. | ||
|
|
||
| ### Verify setup | ||
|
|
||
| > All of the following commands should be running successfully | ||
|
|
||
| #### Run all tests | ||
|
|
||
| ```bash | ||
| sbt test | ||
| ``` | ||
|
|
||
| #### Run specific tests class | ||
|
|
||
| ```bash | ||
| sbt "test:testOnly *MySuite" | ||
| ``` | ||
|
|
||
| #### Run style checks | ||
|
|
||
| ```bash | ||
| sbt scalastyle | ||
| ``` | ||
|
|
||
| --- | ||
| # STOP HERE: Do not code before the interview begins. | ||
|
|
@@ -117,23 +110,13 @@ A single `*.csv` file containing data similar to: | |
| ... | ||
| ``` | ||
|
|
||
| #### Run the job | ||
|
|
||
| ```bash | ||
| spark-submit --master local --class thoughtworks.wordcount.WordCount \ | ||
| target/scala-2.12/tw-pipeline_2.12-0.1.0-SNAPSHOT.jar \ | ||
| "./src/main/resources/data/words.txt" \ | ||
| ./output | ||
| ``` | ||
|
|
||
| ### Citibike | ||
|
|
@@ -170,23 +153,13 @@ Historical bike ride `*.csv` file: | |
| ... | ||
| ``` | ||
|
|
||
| ##### Run the job | ||
|
|
||
| ```bash | ||
| spark-submit --master local --class thoughtworks.ingest.DailyDriver \ | ||
| target/scala-2.12/tw-pipeline_2.12-0.1.0-SNAPSHOT.jar \ | ||
| "./src/main/resources/data/citibike.csv" \ | ||
| "./output_int" | ||
| ``` | ||
|
|
||
| #### Distance calculation | ||
|
|
@@ -221,26 +194,9 @@ Historical bike ride `*.parquet` files | |
|
|
||
| ##### Run the job | ||
|
|
||
| ```bash | ||
| spark-submit --master local --class thoughtworks.citibike.CitibikeTransformer \ | ||
| target/scala-2.12/tw-pipeline_2.12-0.1.0-SNAPSHOT.jar \ | ||
| "./output_int" \ | ||
| ./output | ||
| ``` | ||
|
There was a problem hiding this comment. Please add a repeat here for |
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it would be better to start from the gitpod base image so we guarantee that all the vscode / intellij things work.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
make sense