Complete doc on running functional tests

[rhoboat]

Addresses #11797

[ycombinator]

I'm interpreting these instructions as having to run 3 commands, npm run test:ui, then npm run test:ui:server, and then finally node scripts/functional_test_runner. Is this interpretation correct?

[spalger]

@ycombinator I think that how you interpreted the instructions is logical, but that is not correct.

I think this section should be given three sub-sections:

• There are three ways to run the tests depending on your goals:

  1. Easiest option, much slower when running the tests multiple times, runs elasticsearch/kibana locally:

     • npm run test:ui
       • does everything in a single command
       • tears down everything after the tests run
       • exit code reports success/failure of the tests

  2. Optimized for development, tests can be re-run much faster, runs es/kibana locally:

     • two commands, run in separate terminals, separates the components that are long-running and slow from those that are ephemeral and fast.
       • npm run test:ui:server
         • starts elasticsearch and the kibana server
         • slow to start
         • can be reused for multiple executions of the tests
         • automatically restarts the kibana server when relevant changes are detected
       • node scripts/functional_test_runner
         • runs the tests against kibana/elasticsearch that were started npm run test:ui:server
         • exit code reports success/failure of the tests

  3. Runs test tests against instances of Elasticsearch/Kibana started some other way (like elastic cloud, or an instance you are managing some other way)

     • just executes the functional tests
     • url, credentials, etc. for Elasticsearch and Kibana are specified via environment variables

     export TEST_KIBANA_PROTOCOL=https
     export TEST_KIBANA_HOSTNAME=my-kibana-instance.internal.net
     export TEST_KIBANA_PORT=443
     export TEST_KIBANA_USER=kibana
     export TEST_KIBANA_PASS=password
     
     export TEST_ES_PROTOCOL=http
     export TEST_ES_HOSTNAME=my-es-cluster.internal.net
     export TEST_ES_PORT=9200
     export TEST_ES_USER=elastic
     export TEST_ES_PASS=password
     node scripts/functional_test_runner

[ycombinator]

Perfect, I suspected something wasn't quite right. Thanks for clarifying and the suggestions @spalger!

[rhoboat]

Cool. So @spalger I will update the docs to reflect your comment. Essentially, your comment will become the docs.

[rhoboat]

Take another look, please? @spalger @ycombinator

[rhoboat]

{blob}test/functional/config.js[test/functional/config.js] isn't rendering as I'd like. Maybe because github doesn't know how to properly render asciidoc. I'd like to publish and see how it renders there, tweaking if needed at that point.

[ycombinator]

Love the descriptions of the options! LGTM.

[epixa]

@archanid Github can't parse asciidoc. For doc changes, you should always compile the docs locally to make sure they do what you want and can actually compile, since a broken docs page in Kibana will prevent docs from publishing across all projects.

https://github.com/elastic/docs is the docs repo, and its readme shows instructions for getting it running.

Note that when building docs for Kibana, you'll want to use the --chunk=1 option as well, which isn't documented.

[spalger]

LGTM, though please make sure things look right by building locally before merging

[rhoboat]

@spalger or @epixa is there documentation for how to compile the docs locally?
Edit: nevermind, I think this is it https://github.com/elastic/docs#for-a-local-repo

[epixa]

@archanid Those are the correct docs. I have a PR up to improve the doc building experience for Kibana, and I expect that to get merged in the next couple of days: #13164

[tylersmalley]

I am getting a warning when building

asciidoc: WARNING: development-functional-tests.asciidoc: line 53: section title out of sequence: expected level 3, got level 4

[rhoboat]

@tylersmalley try again?

[tylersmalley]

LGTM