* tests: updated github worklow for tests to use requirements.txt and dev-requirements.txt
* build: updated dev-requirements.txt
Co-authored-by: Ruslan Mullakhmetov <ruslan@twentythree.net>
* feat: added job heartbeat to track whether job is actually executing
heartbeat might be needed in cases when worker was hardkilled or the whole VM/docker was forcibly rebooted.
* fixed tests
* fixed test coverage issue
* chore: renamed job.heartbeat stuff according to review feedback
* chore: pipelined worker heartbeat and job heartbeat
* docs: documented job.heartbeat property
* fixes after review
* docs: updated last_heartbeat description
* chore: review
Co-authored-by: Ruslan Mullakhmetov <ruslan@twentythree.net>
* tests: added ability to run tests in Docker
useful to run full test suit on Mac
* tests: minor improvement in dockerfile for tests
* tests: typo in Dockerfile
* tests: updated dev requirements
Co-authored-by: Ruslan Mullakhmetov <ruslan@twentythree.net>
* scheduler: now operates with chunks of jobs
* scheduler: set default chunk_size for ScheduledJobRegistry.get_jobs_to_schedule
* scheduler: fixed missing indent
* scheduler: added test for get_jobs_to_schedule() with chunk_size parameter
* scheduler: fixed test for passing python 3.5 (no f-strings)
* scheduler: fixed chunk_size in test make it lighter to run
* feat: avoided "zombie" processes after killing work horse by setting work horse process group and killing this group
* fixed tests
* tests: added test to check that all workhorse subprocesses are killed
* tests: updated guthub run tests dependencies since they are not using (dev-)requirements.txt
Co-authored-by: Ruslan Mullakhmetov <ruslan@twentythree.net>
* handled unhandled exceptions in horse to prevent a job from being silently dropped without going into FailedRegistry
* changes after review
* made sure that work_horse always terminates in a proper way with tests
* minor refactoring
* fix for failing test
* fixes for the other tests
- removed exception handling (done in monitor_work_horse)
- adjusted some tests for the checks that are not relevant anymore
* review suggested changes
* cleanup
Co-authored-by: Ruslan Mullakhmetov <ruslan@twentythree.net>
* Initial implementation of Retry class
* Fixes job.refresh() under Python 3.5
* Remove the use of text_type in job.py
* Retry can be scheduled
* monitor_work_horse() should call handle_job_failure() with queue argument.
* Flake8 fixes
* Added docs for job retries
In our systems, this bug seemed the be the cause of the disappearing workers: worker keys would get a very small TTL in Redis and would eventually expire, thus mysteriously "disappearing" from dashboards.
The variable contains the server version and allows to determine
available features. This is relevant for API changes like HSET mappings
in version 4.0.0 or LPOS in version 6.0.6.
To keep the number of connection.info() calls low, the information is
*cached* once determined, as a server version unlikely changes while
keeping the connection up.
Signed-off-by: Paul Spooren <mail@aparcar.org>
* CI: replace travis with GitHub action matrix
The GitHub action runs the same tests as travis however using a more
complex matrix:
Python versions: 3.5, 3.6, 3.7, 3.8
Python 3.4 is dropped by redis-py, therefore not required in the tests.
Redis versions: 3, 4, 5, 6
The different Redis versions offer different features and this allows
checks for compatibility.
redis-py versions: 3.5.0, 3.5.3
3.5.0 is the oldest supported version, 3.5.3 is the latest upstream
verison
Signed-off-by: Paul Spooren <mail@aparcar.org>
* CI: Add flake8 lint action
This actions runs `flake8` and shows style problems of the code. It uses
th GitHub default options which handle most problems as warnings. These
rules could become slowly more strict.
Signed-off-by: Paul Spooren <mail@aparcar.org>
As `redis-py` in version 3.5.0 is now required, whoch only support
Python 3.5+, we should remove Python 3.4 from the supported Python
versions as well.
Signed-off-by: Paul Spooren <mail@aparcar.org>
* setup: read requirements.txt for dependencies
This makes it easier to keep required packages in sync.
Signed-off-by: Paul Spooren <mail@aparcar.org>
* requirements: Update to click 5.0 and redis 3.5.0
Click 5.0 was already required by the `setup.py` and is not brought in
sync. Redis Python library 3.5.0 introduces the `HSET` command with
mapping support which replaces the previous `hmset`. By lifting the
minimal required version to 3.5.0 we can remove the combat function if
Redis server 4.0 is guaranteed.
Signed-off-by: Paul Spooren <mail@aparcar.org>
* ci: remove Python3.4 testing
`redis-py` 3.5.0 does no longer support Python 3.4, so drop it in CI.
Signed-off-by: Paul Spooren <mail@aparcar.org>
Selwin Ong
Chris de Graaf
Ruslan Mullakhmetov
Nikita Romaniuk
Orkhan
Qiming Sun
Vladimir Ulupov
Evgeni Golov
Ahmed Elmaghallawy
maces
Sergei Zherevchuk
nerok
Aaron Tainter
Zhicheng Cai
David Murray
wevsty
Paul Spooren
mbarkhau
Konstantin Schubert