Remove close flush for mosaicml logger

[mvpatel2000]

What does this PR do?

Remove close flush for mosaicml logger. All data is force flushed on fit/predict/eval_end. Attempting to log on close may result in an error as it tries to schedule a future while the python interpreter is shutting down. Instead, we should only wait on existing futures already scheduled.

While we attempt to not schedule futures and do a blocking op currently, under the hood mcli still uses asyncio which uses futures.

[XiaohanZhangCMU]

Can you add an example run name to the PR description which failed because the original flush?

[mvpatel2000]

Can you add an example run name to the PR description which failed because the original flush?

I only was able to reliably reproduce on interactive runs :(. I have an example trace:

Traceback (most recent call last):
  File "/mnt/workdisk/mihirp/composer/composer/core/engine.py", line 553, in _close
    callback.close(state, logger)
  File "/mnt/workdisk/mihirp/composer/composer/loggers/mosaicml_logger.py", line 149, in close
    self._flush_metadata(force_flush=True, future=False)
  File "/mnt/workdisk/mihirp/composer/composer/loggers/mosaicml_logger.py", line 173, in _flush_metadata
    mcli.update_run_metadata(self.run_name, self.buffered_metadata, future=False, protect=True)
  File "/usr/lib/python3/dist-packages/mcli/api/runs/api_update_run_metadata.py", line 139, in update_run_metadata
    response = run_singular_mapi_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/mcli/api/engine/engine.py", line 688, in run_singular_mapi_request
    future = connection.pool.submit(
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 167, in submit
    raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown