* handled unhandled exceptions in horse to prevent a job from being silently dropped without going into FailedRegistry
* changes after review
* made sure that work_horse always terminates in a proper way with tests
* minor refactoring
* fix for failing test
* fixes for the other tests
- removed exception handling (done in monitor_work_horse)
- adjusted some tests for the checks that are not relevant anymore
* review suggested changes
* cleanup
Co-authored-by: Ruslan Mullakhmetov <ruslan@twentythree.net>
* Initial implementation of Retry class
* Fixes job.refresh() under Python 3.5
* Remove the use of text_type in job.py
* Retry can be scheduled
* monitor_work_horse() should call handle_job_failure() with queue argument.
* Flake8 fixes
* Added docs for job retries
In our systems, this bug seemed the be the cause of the disappearing workers: worker keys would get a very small TTL in Redis and would eventually expire, thus mysteriously "disappearing" from dashboards.
The variable contains the server version and allows to determine
available features. This is relevant for API changes like HSET mappings
in version 4.0.0 or LPOS in version 6.0.6.
To keep the number of connection.info() calls low, the information is
*cached* once determined, as a server version unlikely changes while
keeping the connection up.
Signed-off-by: Paul Spooren <mail@aparcar.org>
* CI: replace travis with GitHub action matrix
The GitHub action runs the same tests as travis however using a more
complex matrix:
Python versions: 3.5, 3.6, 3.7, 3.8
Python 3.4 is dropped by redis-py, therefore not required in the tests.
Redis versions: 3, 4, 5, 6
The different Redis versions offer different features and this allows
checks for compatibility.
redis-py versions: 3.5.0, 3.5.3
3.5.0 is the oldest supported version, 3.5.3 is the latest upstream
verison
Signed-off-by: Paul Spooren <mail@aparcar.org>
* CI: Add flake8 lint action
This actions runs `flake8` and shows style problems of the code. It uses
th GitHub default options which handle most problems as warnings. These
rules could become slowly more strict.
Signed-off-by: Paul Spooren <mail@aparcar.org>
As `redis-py` in version 3.5.0 is now required, whoch only support
Python 3.5+, we should remove Python 3.4 from the supported Python
versions as well.
Signed-off-by: Paul Spooren <mail@aparcar.org>
* setup: read requirements.txt for dependencies
This makes it easier to keep required packages in sync.
Signed-off-by: Paul Spooren <mail@aparcar.org>
* requirements: Update to click 5.0 and redis 3.5.0
Click 5.0 was already required by the `setup.py` and is not brought in
sync. Redis Python library 3.5.0 introduces the `HSET` command with
mapping support which replaces the previous `hmset`. By lifting the
minimal required version to 3.5.0 we can remove the combat function if
Redis server 4.0 is guaranteed.
Signed-off-by: Paul Spooren <mail@aparcar.org>
* ci: remove Python3.4 testing
`redis-py` 3.5.0 does no longer support Python 3.4, so drop it in CI.
Signed-off-by: Paul Spooren <mail@aparcar.org>
* Took into account DST when computing localtime zones. This wasn't accounted for when using non-UTC datetimes with queue.enqueue_at()
* Updates tests with mocked timezones to test both scenarios
The exception handling block was raising the caught exception in-place, which caused the original traceback info to be lost. Rather than replace `raise e` with `raise`, I simply removed the whole try / except, since no action was being taken in the except block.
* Rename `dependent_jobs` to `jobs_to_enqueue` in queue.py
* Rename `dependencies_job_ids` to `dependency_ids`.
* Remove `as_text` (no more python2 support). Use `bytes.decode`