Experiment scripts #447
Experiment scripts #447
Conversation
| @@ -0,0 +1,163 @@ | |||
| from PIL import Image | |||
There was a problem hiding this comment.
Do not copy this file, just reuse the file under demo
There was a problem hiding this comment.
Very much agree that it's good to avoid duplicate code. However the demo/train.py will perhaps change for different reason and at different time comparing to mnist/train.py, maybe they are not truly the duplicate code? (even they currently have exact same code, but in the future they will diverge)
For example, mnist/train.py probably will never change after the experiment, but the demo/train.py maybe will keep evolving (and breaking the experiment code if we only have one train.py).
I think we need to avoid only the truly duplicate code, otherwise if two irrelevant components share the same code, it adds the unnecessary coupling of the two components.
Just my thoughts, open for discussion.
There was a problem hiding this comment.
Mabye we can change the folder name for a general name, such as demo/job/..., or someone will feel uncertain about the different this experiemnt/mnist and demo/mnist.
| @@ -0,0 +1,169 @@ | |||
| from PIL import Image | |||
There was a problem hiding this comment.
Do not copy this file, just reuse the file under demo.
|
|
||
|
|
||
| # NOTE: must change this to your own username on paddlecloud. | ||
| USERNAME = "[email protected]" |
There was a problem hiding this comment.
Can we put mnist dataset to the public folder so we don't need this hardcoded username anymore?
| # NOTE: must change this to your own username on paddlecloud. | ||
| USERNAME = "[email protected]" | ||
| DC = os.getenv("PADDLE_CLOUD_CURRENT_DATACENTER") | ||
| #common.DATA_HOME = "/pfs/%s/home/%s" % (DC, USERNAME) |
There was a problem hiding this comment.
Is this unintentionally commented out?
There was a problem hiding this comment.
Done, delete the unnecessary commented code.
| #!/bin/bash | ||
| DEFAULT_JOBNAME_PREFIX="mnist" | ||
|
|
||
| function submit_general_job() { |
There was a problem hiding this comment.
We can keep submit_general_job, but do we need to use it in the experiment?
There was a problem hiding this comment.
Following the TestCase1 of the design, we will compare the gernatl jobs and fault-tolerant jobs, so maybe we need this function?
There was a problem hiding this comment.
I see, I meant currently we are using tolerant job to indicate scaling the job or not. We can also do the experiment by just not scaling the fault tolerant job (but always start a tolerant job), so we do not need the general job. But that's fine. This is just a minor difference.
|
|
||
|
|
||
| DC = os.getenv("PADDLE_CLOUD_CURRENT_DATACENTER") | ||
| common.DATA_HOME = "/pfs/%s/public/idl/users/dl/paddlecloud/public/dataset" % DC |
There was a problem hiding this comment.
The especial path is only used for the internal CPU cluster, I will changed this one to a general path before merged.
Fixed #446