Better assertion with less wait time

[tompng]

Need to merge ruby/irb#1000 and ruby/reline#746 first

There are good E2E tests and bad E2E tests.

require 'capybara'

# Good E2E test
click_link('bar')
expect(page).to have_content('baz') # This will automatically retries

# Bad and slow E2E test
click_link('bar')
sleep 1 # Need to sleep enough time because there is no retry in the assertion below
expect(page.content).to include('baz')

Yamatanooroti should provide a way to write good E2E test for fast and stable testing.

• write should not sleep 0.1sec(before write) + 0.1sec(after write). It should be faster.
• assert_screen should retry with a very short wait time.
• assert_screen should accept regexp to avoid sleep t; assert_match(result.join, /pattern/)
• Should be able to call assert_screen and result several times in a single test case.

Example using the new feature

start_terminal(rows, cols, command, startup_message: /irb\(main\):001>/)
write "sleep 1\n"
# Wait until irb is ready
assert_screen(/irb\(main\):002>/)
write "debug\n"
# Wait until rdbg is ready
assert_screen(/irb:rdbg/)
write "foobar\n"
# Assert result
assert_screen(expected_result)
close

IRB and Reline test

IRB and Reline's test implicitly depend on wait=0.1, so test will fail. We need to properly set startup_message and/or add an assert_screen before some write operations.

Expected performance improvements are:
Reline test_yamatanooroti: 3:30 -> 0:50
IRB test_yamatanooroti: 0:38 -> 0:12

Branches:
https://github.com/tompng/reline/tree/new_yamatanooroti
https://github.com/tompng/irb/tree/new_yamatanooroti

[YO4]

It seems to require 'io/wait' required for RUBY_VERSION <3.2.0

[tompng]

Thank you 👍 added

[ima1zumi]

📝 added by io/wait
https://docs.ruby-lang.org/ja/latest/method/IO/i/wait.html
https://docs.ruby-lang.org/en/3.3/IO.html#method-i-wait

[ima1zumi]

Is exception handling no longer necessary?

[tompng]

@vterm.write(chunk)

It always succeeds because it is not an IO, it won't be closed.

Errno::EAGAIN, Errno::EWOULDBLOCK, IO::EAGAINWaitReadable

Not needed because there is always wait_readable before read_nonblock

Errno::EIO

def write(str)

Failed write (always called from test) should always raise an error and make the test fail.

def test_foo
  write("exit!\n") # Process terminates. pty_input will be closed.
  write("1+2\n") # This should raise error
end

@pty_input.write(@vterm.read)

There is a chance that error might raise. I added rescue Errno::EIO.
(It rarely happens. I can't reproduce raising Errno::EIO without inserting sleep 0.0004 before calling vterm_write)

[ima1zumi]

What does a mean?

[tompng]

It is actual_lines or convert_proc.call(actual_lines). I changed it to simply actual

[ima1zumi]

It feels like using assert gives the impression of a public method for testing, so maybe a different name would be better. (Sorry, I don’t have any ideas right now.)

[tompng]

Changed to retryable_screen_assertion_with_proc

[ima1zumi]

Excellent work! Thank you!