Refactor executor main

[andygrove]

Which issue does this PR close?

Part of #613

Rationale for this change

Make it easier to build custom versions of Ballista with extensions and to wrap the executor in Python.

What changes are included in this PR?

Refactor existing code:

• Add new ExecutorProcessConfig struct
• Move logic from main.rs to executor_process.rs
• Move main.rs out of lib and into src/bin/main.rs

Are there any user-facing changes?

No

[andygrove]

@avantgardnerio You have experience in building custom executors that extend Ballista. Is this approach reasonable as a starting point to simplify doing this?

@jdye64 Would be good to get your thoughts too

[jdye64]

I like this. Moving the start_executor_process(...) to executor_process.rs makes it more clean to delegate the starting of the executor from a custom driver program.

[avantgardnerio]

Is this approach reasonable as a starting point to simplify doing this?

Yes, I definitely think we should make this change.

That being said, it can be painful extending the dependency graph to this many layers deep:

                         +-> delta-rs -+
                         |             |
                         |             |
arrow-rs ----> datafusion -------------+--> ballista --> custom-app

So it's my hope at S&T that we can get as much functionality merged upstream as possible to minimize TTL on arrow-rs changes.