Add timeout for starting tarantool server #302

VitaliyaIoffe · 2021-05-17T11:00:15Z

This patch includes 4 commits:

Use gevent sleep for switching context

time.sleep was changed to gevent.sleep to allow current greenlet to
sleep and others to run. When time.sleep is used, greenlet's context
is not changed from the main process to test's greenlet.

As a result of this, there is no data received by the main process while
hanging the tarantool server process and the test is fallen down by the
common timeout (NO_OUTPUT_TIMEOUT). Even worse, the tarantool server
process is not killed by test-run.

Using gevent.sleep makes the test fail by the test timeout and kill
the farantool server process.

Part of #276

Add timeout for starting tarantool server

When a tarantool server starts, it waits for a special pattern in the
log file to proceed. If there is no pattern present, the server hangs.
After the test timeout (TEST_TIMEOUT) runs out, the test fails.

This patch adds the --server-start-timeout option to test-run (by
default it equals to 90 seconds). Now when the server hangs and the
time (SERVER_START_TIMEOUT) runs out, a comprehensible exception is
raised with the message that the server failed to start within the
timeout.

Fixes: #276

Kill processes of non-started tarantool servers

It was found that processes of non-started tarantool servers are not
killed by test-run and leave to hang. This situation can be reproduced
by creating the main server, then creating a replica server, but the
replica server is unable to join the master, for example, due to lack
of user permissions. In this case, the test fails by the server start
timeout and test-run kills the main server process only.

Fixes #256
Follows up #276

Add unit test checking no hanging procs present

This patch adds a simple unit test checking that if a tarantool server
failed to start within a certain amount of seconds, test-tun raises a
comprehensible exception and kills the server process.

Follows up #256
Follows up #276

lib/tarantool_server.py

lib/preprocessor.py

lib/tarantool_server.py

lib/preprocessor.py

ligurio · 2021-05-27T11:48:16Z

@VitaliyaIoffe could you rebase to master? Latest changes includes code coverage measurement and it would be interesting how code coverage numbers are changed with your tests.

test/negative_tests/test_reconcile_logs.py

lib/preprocessor.py

When a tarantool server starts, it waits for a special pattern in the log file to proceed. If there is no pattern present, the server hangs. After the test timeout (TEST_TIMEOUT) runs out, the test fails. This patch adds the `--server-start-timeout` option to test-run (by default it equals to 90 seconds). Now when the server hangs and the time (SERVER_START_TIMEOUT) runs out, a comprehensible exception is raised with the message that the server failed to start within the timeout. Fixes: #276

It was found that processes of non-started tarantool servers are not killed by test-run and leave to hang. This situation can be reproduced by creating the main server, then creating a replica server, but the replica server is unable to join the master, for example, due to lack of user permissions. In this case, the test fails by the server start timeout and test-run kills the main server process only. This patch fixes the issue. Fixes #256 Follows up #276

ylobankov · 2022-03-10T20:13:02Z

Thanks for fixes!

See my comments inline and one more note: please put commit "Add unit test checking no hanging procs present" right after commit "Add timeout for starting tarantool server" with fix or squash them to a single commit.

It is not possible. The test checks that a process of a non-started tarantool server will not hang and will be killed after server start timeout runs out. The problem with hanging processes of non-started tarantool servers was resolved in the Kill processes of non-started tarantool servers commit and the test can be added only after this commit.

lib/options.py

lib/tarantool_server.py

test-run.py

This patch adds a simple unit test checking that if a tarantool server failed to start within a certain amount of seconds, test-tun raises a comprehensible exception and kills the server process. Follows up #256 Follows up #276

Totktonada

LGTM.

ligurio

LGTM

Bump test-run to new version with the following improvements: - Add timeout for starting tarantool server (tarantool/test-run#302) - Kill hanging processes of not started servers (tarantool/test-run#332) - Rerun all failed tests, not only marked as fragile (tarantool/test-run#329) NO_DOC=testing NO_TEST=testing NO_CHANGELOG=testing

Bump test-run to new version with the following improvements: - Add timeout for starting tarantool server (tarantool/test-run#302) - Kill hanging processes of not started servers (tarantool/test-run#332) - Rerun all failed tests, not only marked as fragile (tarantool/test-run#329) NO_DOC=testing NO_TEST=testing NO_CHANGELOG=testing (cherry picked from commit 0b46154)

kyukhin requested a review from Totktonada May 17, 2021 11:07

VitaliyaIoffe force-pushed the VitaliyaIoffe/gh-276-hangs-timeout-doesnt-work branch 2 times, most recently from 5864c82 to 7135e26 Compare May 17, 2021 15:05