moved all tests from src/it to src/test/it #10
Conversation
… AnyFunSuite test
This reverts commit d1d8098.
There was a problem hiding this comment.
Why are we still installing pre-requisites for batect?
There was a problem hiding this comment.
Lets also remove this from README local
Additionally, it would be great to include somewhere with a big heading on how to run Test/IntegrationTest
How can we trigger a single test?
There was a problem hiding this comment.
Removed pre-requisites for batect and local readme. I have added the command to run a testClass, not able to find one to run single test though. Also since we move all the tests under IT folder, we can run IntegrationTest via sbt test command only
| @@ -35,7 +35,7 @@ object CitibikeTransformer { | |||
| spark.stop() | |||
| } | |||
|
|
|||
| private def getInputAndOutputPaths(args: Array[String]) = { | |||
| private def getInputAndOutputPaths(args: Array[String]): (String, String) = { | |||
There was a problem hiding this comment.
Lets simplify this with a Path case class?
There was a problem hiding this comment.
This is still pending, I think you want me to take it out as a class right rather than a function?
There was a problem hiding this comment.
I have reviewed the non-scala code changes
please wait for atif's review for the scala/test part of the PR.
Overall looks good! Thanks a lot for it: the main differences will be the graphs & the code walkthrough, but I would be happy to add these in another PR :)
| FROM sbtscala/scala-sbt:eclipse-temurin-11.0.15_1.7.1_2.12.16 AS build | ||
|
|
||
| #FROM gitpod/workspace-full |
There was a problem hiding this comment.
I think it would be better to start from the gitpod base image so we guarantee that all the vscode / intellij things work.
| @@ -10,7 +10,10 @@ RUN wget https://archive.apache.org/dist/spark/spark-3.3.0/spark-3.3.0-bin-hadoo | |||
| RUN tar xvf spark-3.3.0-bin-hadoop3.tgz | |||
| ENV PATH="/opt/spark-3.3.0-bin-hadoop3/bin:$PATH" | |||
|
|
|||
| #RUN brew install sbt | |||
There was a problem hiding this comment.
| #RUN brew install sbt |
| @@ -35,7 +35,7 @@ object DailyDriver { | |||
| spark.stop() | |||
There was a problem hiding this comment.
Could we split the scala code changes (and testing way from the README.md change? right now this is a bit hard to review what is a code improvement and what is the update of the readme and docker removal)
There was a problem hiding this comment.
TBH I tried, but couldn't do it. All my commits were coming in 1 PR since nothing were accepted.
README.md
Outdated
| target/scala-2.12/tw-pipeline_2.12-0.1.0-SNAPSHOT.jar \ | ||
| "./output_int" \ | ||
| ./output | ||
| ``` |
There was a problem hiding this comment.
Please add a repeat here for
> ⚠️ The exercises will be given at the time of interview, and solved by pairing with the interviewer.
As part of this issue,