torchx - profile scheduler validate call (pytorch#972)

Summary:

track torchx scheduler validate() call. Will capture elapse time and whether validation failed. There is no behavior impact to torchx users as this change only add logging for monitoring purpose

verified scuba log has the validate record:
https://fburl.com/scuba/pytorch_elastic_tsm_log/jza7f33n

Differential Revision: D66166216

torchx/runner/api.py

@@ -404,10 +404,11 @@ def dryrun(
                     role.env[tracker_config_env_var_name(name)] = config
 
         cfg = cfg or dict()
+        runcfg = json.dumps(cfg) if cfg else None
         with log_event(
             "dryrun",
             scheduler,
-            runcfg=json.dumps(cfg) if cfg else None,
+            runcfg=runcfg,
             workspace=workspace,
         ):
             sched = self._scheduler(scheduler)
@@ -433,7 +434,13 @@ def dryrun(
                         " Either a patch was built or no changes to workspace was detected."
                     )
 
-            sched._validate(app, scheduler)
+            with log_event(
+                "validate",
+                scheduler,
+                runcfg=runcfg,
+                workspace=workspace,
+            ):
+                sched._validate(app, scheduler)
             dryrun_info = sched.submit_dryrun(app, resolved_cfg)
             dryrun_info._scheduler = scheduler
             return dryrun_info