Initial code of runtimetest command

[zenlint]

I have add the initial cod of runtimetest command, to replace runtime_test.sh.
Reference issuses:
Support test cases #7
Convert test_runtime.sh into a go program #8

Now, the result shows as below:

# ./ocitools runtimetest -r runc -o all
INFO[0001] successful Details:                          
INFO[0001] =============================================================================================== 
INFO[0001] 
BundleName:
  process1
BundleDir:
  bundles/process1
CaseArgs:
  --args=./runtimetest --rootfs=rootfs --read-only=true
TestResult:
  SUCCESS

INFO[0001] 
BundleName:
  hostname0
BundleDir:
  bundles/hostname0
CaseArgs:
  --args=./runtimetest --rootfs=rootfs --hostname=zenlin
TestResult:
  SUCCESS

INFO[0001] 
BundleName:
  process0
BundleDir:
  bundles/process0
CaseArgs:
  --args=./runtimetest --rootfs=rootfs --read-only=false
TestResult:
  SUCCESS

INFO[0001] failure Details:                             
INFO[0001] =============================================================================================== 
INFO[0001] =============================================================================================== 
INFO[0001] statistics:  3 bundles success, 0 bundles failed

My plan as below,

1. Reach cases in cases.conf
2. Support different runtime
   what is your suggestion about this? @mrunalp

[mrunalp]

We can add more test cases and support other runtimes if they have wrappers to CLI similar to runc.

[zenlint]

@mrunalp
yes, quite agree with u, I am just adding more testcases with my local test, once the PR Add initial version of runtimetest command: #9 is merged, I will add them to cases.conf.

To "if they have wrappers to CLI similar to runc", do u mean by a lib like factory created in ocitools repo or a 3rd tools? Any reference disscution about this?

[wking]

On Thu, Jan 28, 2016 at 11:20:07PM -0800, LinZhinan(Zen Lin) wrote:

And what do u mean by "if they have wrappers to CLI similar to
runc"? Any reference disscution about this?

My preference here would be to use 1 to define the command line API
that the test suite uses to drive runtimes. In places where that
differs from runC, and runC wants to stick with their difference, we
can either PR the command-line API or write a shell wrapper around
runC to give it the API-defined interface.

[zenlint]

@wking I am going to pay close attention on [1]: https://github.com/wking/oci-command-line-api.

[zenlint]

@mrunalp
To support different runtimes, there are two ways to reach,

1. Best final way,
   Support different runtimes after there is a official command-line API for runtimes or a official conversion tool(official means accepted by opencontainers here).
2. Temporary way,
   Just support different runtimes with abstract factrory like https://github.com/huawei-openlab/oct/tree/master/factory and conversion tools for temporary, and finally change to [1].

What do u think about these two ways?

[liangchenye]

@zenlinTechnofreak close since we have another PR to work on this.