Emit periodic keepalive events from Worker

[nitzmahone]

fixes #1187

This approach is less "fancy" than some others, but also doesn't require yet another interposing output handle.

Open issues:

• short-circuit handling: Ideally there'd be a short-circuit in the Process layer for this, or any other "malformed"/bogus event, but as-is, Process will bomb out with a KeyError on any dictionary missing uuid or counter. Ideally, these keepalives could be as simple as a newline, dot, or {}, but unless/until we can be sure that an older Process isn't going to barf on them, we have to have a larger payload to keep it happy.

• other consumers: In cases where the event actually gets persisted by Process, will anything else be upset by the empty 0-0.json event it writes as? (EDIT: consensus was to go for a "fail-fast" approach instead)

• unit tests: there are a number of potential output corruption races this attempts to prevent, but simulating them with unit tests to ensure they stay "prevented" would probably be a Good Thing. (EDIT: added crazy way-subsecond keepalive tests to try and flush out problems and races here)

• tests: Don't want to gold-plate this until we're sure we like this approach. Also, the transmit/worker/process tests need some refactoring to be able to introduce delays so the keepalives have a chance to fire, and also to ensure that they don't fire when they're not supposed to.

This specifically is not trying to solve the generic "I want to do periodic random stuff in runner" issue- it seems like that would be better handled in a future asyncio layer where such things are trivial to do without extra threading overhead.

[AlanCoding]

I put up nitzmahone#1 to suggest a test. You can take it or leave it regarding the patch, but this definitely needs an integration test like that.

[shanemcd]

I didn't try running it - but the code makes sense. LGTM pending approval from someone that actually tested it.

[hswong3i]

I can install https://github.com/ansible/ansible-runner/releases/tag/2.3.2 with Ubuntu 18.04, which is Python 3.6, but with following runtime error message:

$ sudo podman run -ti --rm ubuntu:18.04

root@0772e205b3d1:/# apt update && apt -y install python3-pip

root@0772e205b3d1:/# pip3 install ansible-runner
Collecting ansible-runner
  Downloading https://files.pythonhosted.org/packages/8a/ce/fd842b0d464eafad9c88fce0573e638c806211d579a7cdd5d65b417939b4/ansible_runner-2.3.2-py3-none-any.whl (80kB)
    100% |################################| 81kB 10.6MB/s 
Collecting packaging (from ansible-runner)
  Downloading https://files.pythonhosted.org/packages/05/8e/8de486cbd03baba4deef4142bd643a3e7bbe954a784dc1bb17142572d127/packaging-21.3-py3-none-any.whl (40kB)
    100% |################################| 40kB 17.3MB/s 
Collecting pexpect>=4.5 (from ansible-runner)
  Downloading https://files.pythonhosted.org/packages/39/7b/88dbb785881c28a102619d46423cb853b46dbccc70d3ac362d99773a78ce/pexpect-4.8.0-py2.py3-none-any.whl (59kB)
    100% |################################| 61kB 17.7MB/s 
Collecting python-daemon (from ansible-runner)
  Downloading https://files.pythonhosted.org/packages/89/38/c223036ee8104ae95118d4481b5cacccf4547d7e5b8d1ee1c63d2cd57e5d/python_daemon-2.3.2-py3-none-any.whl
Requirement already satisfied: six in /usr/lib/python3/dist-packages (from ansible-runner)
Collecting pyyaml (from ansible-runner)
  Downloading https://files.pythonhosted.org/packages/36/2b/61d51a2c4f25ef062ae3f74576b01638bebad5e045f747ff12643df63844/PyYAML-6.0.tar.gz (124kB)
    100% |################################| 133kB 17.7MB/s 
Collecting pyparsing!=3.0.5,>=2.0.2 (from packaging->ansible-runner)
  Downloading https://files.pythonhosted.org/packages/6c/10/a7d0fa5baea8fe7b50f448ab742f26f52b80bfca85ac2be9d35cdd9a3246/pyparsing-3.0.9-py3-none-any.whl (98kB)
    100% |################################| 102kB 18.4MB/s 
Collecting ptyprocess>=0.5 (from pexpect>=4.5->ansible-runner)
  Downloading https://files.pythonhosted.org/packages/22/a6/858897256d0deac81a172289110f31629fc4cee19b6f01283303e18c8db3/ptyprocess-0.7.0-py2.py3-none-any.whl
Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (from python-daemon->ansible-runner)
Collecting lockfile>=0.10 (from python-daemon->ansible-runner)
  Downloading https://files.pythonhosted.org/packages/c8/22/9460e311f340cb62d26a38c419b1381b8593b0bb6b5d1f056938b086d362/lockfile-0.12.2-py2.py3-none-any.whl
Collecting docutils (from python-daemon->ansible-runner)
  Downloading https://files.pythonhosted.org/packages/8d/14/69b4bad34e3f250afe29a854da03acb6747711f3df06c359fa053fae4e76/docutils-0.18.1-py2.py3-none-any.whl (570kB)
    100% |################################| 573kB 3.9MB/s 
Building wheels for collected packages: pyyaml
  Running setup.py bdist_wheel for pyyaml ... done
  Stored in directory: /root/.cache/pip/wheels/95/84/67/ebeac632c63797cfbeb90128ca41073117721540dad526d213
Successfully built pyyaml
Installing collected packages: pyparsing, packaging, ptyprocess, pexpect, lockfile, docutils, python-daemon, pyyaml, ansible-runner
Successfully installed ansible-runner-2.3.2 docutils-0.18.1 lockfile-0.12.2 packaging-21.3 pexpect-4.8.0 ptyprocess-0.7.0 pyparsing-3.0.9 python-daemon-2.3.2 pyyaml-6.0

root@0772e205b3d1:/# ansible-runner --version
Traceback (most recent call last):
  File "/usr/local/bin/ansible-runner", line 7, in <module>
    from ansible_runner.__main__ import main
  File "/usr/local/lib/python3.6/dist-packages/ansible_runner/__init__.py", line 3, in <module>
    from .interface import run, run_async, \
  File "/usr/local/lib/python3.6/dist-packages/ansible_runner/interface.py", line 33, in <module>
    from ansible_runner.streaming import Transmitter, Worker, Processor
  File "/usr/local/lib/python3.6/dist-packages/ansible_runner/streaming.py", line 1
    from __future__ import annotations  # allow newer type syntax until 3.10 is our minimum
    ^
SyntaxError: future feature annotations is not defined

Seems like changes from https://github.com/ansible/ansible-runner/pull/1191/files#diff-81c3df3f520280c8548b6499f287446e289a81ecad30288e0c561d3aa95456c5R1 require Python >= 3.7, am I correct?