In order to solve issue 702 we have to check whether a work-horse
terminated unexpectedly (by inspecting the exit code of the work-horse
process). If it exited unexpectedly we check if the job has either been
marked as finished, failed or other valid states. If it's not in any
valid state we mark it as failed and move it to the failed queue.
Since the process was terminated unexpectedly (think OOM) we do not
have any exception context and we can't run any custom exception handlers.
There is still a chance that the job will finish successfully but the
work-horse process will be killed before the job is marked as finished
and we will erroneously mark it as failed. The users should take care
to write idempotent jobs.
Test that demonstrates that if a work-horse process is terminated unexpectedly
the job being processed could be stuck at the "Started" state
(https://github.com/nvie/rq/issues/702)
Otherwise, there's no linear history between this old tag and any new,
future, tags. This is likely been a manual mistake when I released
0.6.0, so this is corrected now.
- run with an empty queue
- schedule one job (which uses get_current_connection and get_current_job) and
run `rqworker`
- schedule a job that itself schedules `access_self` and run `rqworker`
- Make sure the job didn't fail by assuring the failed queue is still empty
afterwards.
- Install this package locally when running in travis.
This actually unifies the behaviour of tox and travis as tox also builds the
package and then installs it into each test environment.
- fix flake8 (as run by tox)
This allows jobs to use get_current_connection() with a resolvable connection.
And then these jobs can schedule new jobs for example (my use-case). Or attach
information to their job-object.
Also pop the pushed connection after running the jobs. This is needed for some
tests that check the _connection_stack afterwards;-) And also for use-cases
where the workers are used multiple times.
fixesnvie/rq#479