FileNotFoundError on Skill removal

[krisgesling]

Describe the bug
I hit this error when doing an uninstall using PR #2803

2021-02-04 11:35:41.063 | ERROR    | 20080 | mycroft.skills.skill_manager:run:260 | Something really unexpected has occured and the skill manager loop safety harness was hit.
Traceback (most recent call last):
  File "/home/gez/mycroft-core/mycroft/skills/skill_manager.py", line 248, in run
    self._load_new_skills()
  File "/home/gez/mycroft-core/mycroft/skills/skill_manager.py", line 296, in _load_new_skills
    for skill_dir in self._get_skill_directories():
  File "/home/gez/mycroft-core/mycroft/skills/skill_manager.py", line 322, in _get_skill_directories
    if SKILL_MAIN_MODULE in os.listdir(skill_dir):
FileNotFoundError: [Errno 2] No such file or directory: '/home/gez/.local/share/mycroft/skills/youtube-music-skill.forslund/'
2021-02-04 11:36:11.184 | INFO     | 20080 | mycroft.skills.skill_manager:_unload_removed_skills:338 | removing youtube-music-skill.forslund

The directory did exist in the new location, and was removed correctly, so it's possible that this is unrelated to that PR and the hot reloader tried to load the Skill as it was being removed which could have happened regardless of where it was located.

Currently when we run the SkillManager the order of operations is:

                self._reload_modified_skills()
                self._load_new_skills()
                self._unload_removed_skills()
                self._update_skills()

Wondering if we just need to reorder to make sure that we first unload removed skills.

To Reproduce
Quite hard to reproduce reliably...

Expected behavior
No exception should be raised when removing a Skill.