xonsh shell

[ahundt]

I just wanted to mention the xonsh shell (tutorial) to you guys because it seems like it might be the perfect way to use and integrate sacred. This is because they have similar principles where they have a very rich history to improve reproducibility. Xonsh is a full shell you can use much like bash, but it is implemented in python and you can call python code from the command line much like ipython.

This is just an FYI I'm writing because a light bulb just went off where I thought these two packages might compliment each other beautifully. :-)

No action needs to be taken so this can be closed unless there was specific interest in discussing things.

[Qwlouse]

xonsh looks indeed pretty cool, thanks for the pointer. Unfortunately, I currently don't have the time to look into opportunities for integrating it into sacred. I'll leave the issue open as a feature request, but I won't work on it anytime soon.

[ahundt]

After remarking on #193, I started looking into how xonsh integration could work. There are a few "ingredients":

1. extension
2. events
3. history backend, plus an understanding of xonsh history (such as replay, lazyjson might be useful too)
4. import local python modules sys.path.insert(0, '')

Todo:

• create initial xontrib https://github.com/ahundt/sacredx
• create a xonsh history backend that feeds the relevant xonsh shell history to sacred's observers
• create sacred events that trigger xonsh events, and vice versa
• ensure interactive is enabled and experiment.run() does not block xonsh, consider xonsh.proc
• let config variables be changed during run
• check out /incorporate assistant
• run experiments in parallel
  • possibly using the queue
  • distributed library
  • xonsh distributed xontrib
• sanctuary based functionality in command line
  • incorporate database status & etc as xonsh command line, see sanctuary
  • model loading & mangodb interface commands
  • keras integration see sanctuary

Here is a first conceptual session with some of the above:

$ xonsh
$ cd /path/to/my/experiments
$ ls
my_experiment.py my_dataset.py
$ import my_dataset
$ ds = my_
$ import my_experiment
$ ex = my_experiment.MyExperiment('experiment', interactive=True)
$ config = {'a': 3, 'b': 4}
$ sacredx(ex, config_updates=config)

[ahundt]

@Qwlouse It seems an observer isn't what I want to log the command line history, because observers cannot trigger events. Is that correct?

Can sacred be changed a bit so observers are able to trigger events?

Essentially, at the time run is called I'd like to save the shell history json file, perhaps instead I'll needs something like what's done for the tfrecordwriter, but I looked in stflow and that works with @ decorators, which won't really be sensible on the command line.

Suggestions?

[ahundt]

I've started writing things, see https://github.com/ahundt/sacredx. Would you be interested in incorporating it directly if it reaches a good point?

Also, which entry would be ideal for the shell history json file, a run.info xonsh_history key to the json value, an artifact, a resource, something else?

[Qwlouse]

Hey, I'm not sure I understand your questions. And I'm afraid I would need to dig into xonsh to really give you helpful feedback.

Observers do not trigger events, that is right. Observers listen to events. So when the experiment is started each observers started_event is called. And I would probably go for meta-info. A commandline option might be a good fit for what you want to do.

class StoreHistoryOption(CommandLineOption):
    """Save the shell history as meta information."""
    short_flag = 's'

    @classmethod
    def apply(cls, args, run):
        """Set this run to not warn about suspicous changes."""
        run.meta_info['history'] = xonsh.somehow_retrieve_history()

You could then add options={'--store_history': True} to the ex.run() call and have this triggered right before the run is started.
If you expect the history to become very large, then adding it as a file is probably a better idea. In that case you'd better use a pre_run_hook to add a resource.

[ahundt]

Hmm, the difference is xonsh is actually a shell implemented in python, so you can run executables like you would in bash or call python functions like you would in ipython. In other words, it is like an ipython prompt fused with a bash prompt so you might as well use the python session you're currently in. For that reason, using the existing command line API may not completely make sense because you can just make the python calls yourself.

On the other hand, maybe it still does make sense to provide a direct command line, and I'd just have to add a tiny bit of extra glue to reorganize and pass things along correctly....

Regarding my other question: In essence, if the separate repository gets to a good point where it is more useful than just some minor sandbox code, would you be interested in merging it directly into sacred or should I keep it a separate repository permanently?

[Qwlouse]

Well, some of these options make sense even if called from the python interface. So I guess calling them Commandline options is a bit of a misnomer here.

Regarding my other question: In essence, if the separate repository gets to a good point where it is more useful than just some minor sandbox code, would you be interested in merging it directly into sacred or should I keep it a separate repository permanently?

At this point, I don't feel comfortable making promises. Using xonsh seems like a less common usecase. So depending on the effort of integrating it and the added convenience, I'll consider it. But currently, I know too little about xonsh, to make a good predicition.