mgr/cephadm: retry after JSONDecodeError in wait_for_mgr_restart()

'ceph mgr dump' does not always return valid JSON so cephadm will throw an exception sometimes when applying a spec as per the issue this PR closes. Add a try/except to catch a possible JSONDecodeError and retry after sleeping. Fixes: https://tracker.ceph.com/issues/49870 Signed-off-by: John Fulton <[email protected]>
ceph · Mar 17, 2021 · 076157e · 076157e
1 parent 1af1a7c
commit 076157e
Showing 1 changed file with 16 additions and 7 deletions.
diff --git a/src/cephadm/cephadm b/src/cephadm/cephadm
@@ -3926,16 +3926,25 @@ def command_bootstrap(ctx):
         # first get latest mgrmap epoch from the mon.  try newer 'mgr
         # stat' command first, then fall back to 'mgr dump' if
         # necessary
-        try:
-            out = cli(['mgr', 'stat'])
-        except Exception:
-            out = cli(['mgr', 'dump'])
-        j = json.loads(out)
-        epoch = j['epoch']
-
         # wait for mgr to have it
         logger.info('Waiting for the mgr to restart...')
 
+        for sleep_secs in [1, 4, 25]:
+            try:
+                try:
+                    cmd = ['mgr', 'stat']
+                    out = cli(cmd)
+                except Exception:
+                    cmd = ['mgr', 'dump']
+                    out = cli(cmd)
+                j = json.loads(out)
+                break
+            except json.decoder.JSONDecodeError:
+                cmd_str = 'ceph ' + ' '.join(cmd)
+                logger.info('`%s` failed transiently. Retrying. waiting %s seconds...' \
+                            % (cmd_str, sleep_secs))
+                time.sleep(sleep_secs)
+        epoch = j['epoch']
         def mgr_has_latest_epoch():
             # type: () -> bool
             try: