* Add started job registry cleanup job failure callback call
* WIP - need to fix test
* fix test
* rename, tests and docs
* better log message
* use class name
* Update registry.py
* Enhanced Redis Connection Reliability
The Redis connection may fail for several reasons. As the connection can be
(1) explicitly passed to the worker or (2) implicity set, this will improve the
Connection configuration by setting a timeout to the socket, and adding
an ExponentialBackoff Retry logic.
* Simpler Connection logic
* Add simple retry logic to Redis Connection Error
* Make retry exponential, add keepalive & socket_connect_timeout
* Handles configuration on Redis' connection pool
* Simplifies timeout exception logic
* Fix burst bug, add test
* Add docs related to `socket_timeout`, improve compatibility with older RedisPy versions
* Fixes
* New timeout private method
* Fix timeout
* WIP job results
* Result can now be saved
* Successfully saved and restored result
* result.save() should accept pipeline
* Successful results are saved
* Failures are now saved properly too.
* Added test for Result.get_latest()
* Checkpoint
* Got Result.all() to work
* Added Result.count(), Result.delete()
* Backward compatibility for job.result and job.exc_info
* Added some typing
* More typing stuff
* Fixed typing in job.py
* More typing updates
* Only keep the last 10 results
* Documented job.results()
* Got results test to pass
* Don't run test_results.py on Redis server < 5.0
* Fixed mock import on some Python versions
* Remove Redis 3 from test matrix
* Jobs should never use the new Result implementation if server is < 5.0
* Results should only be created is Redis stream is supported.
* Added back Redis 3 to test matrix
* Fixed job.supports_redis_streams
* Fixed worker test
* Updated docs.
* Add feature to enqueue dependents at the front of queues
* Add documentation for the Dependency(enqueue_at_front=...) parameter
* docs: Add `enqueue_at_front` to list of parameters for Dependency
* test: Update dependency test to not rely on Redis ordering
* refactor: Save enqueue_at_front boolean in job.meta instead of separate instance attr
* fix: Made enqueue_at_front an instance attribute instead of putting it inside meta
* doc: default queues order is a priority order
In introduction, we say that queue order matters, but we are not very specific.
Added a paragraph to explicit that.
* doc: fix typo
* added Dependency class with allow_failures
* Requested changes
* Check type before setting `job.dependency_allow_fail` within `Job.create`
* Set `job.dependency_allow_fail` within `Job.create`
* Added test to ensure persistence of `dependency_allow_fail`
* Removed typing and allow mixed list of ints and Job objects
* Convert dependency_allow_fail boolean to integer during serialization to avoid redis DataError
* Updated `test_multiple_dependencies_are_accepted_and_persisted` test to include `Dependency` cases
* Adding placeholder test to test actual behavior of new `Dependency` usage in `depends_on`
* Updated `test_job_dependency` to include cases using `Dependency`
* Added dependency_allow_fail logic to `Job.restore`
* Renamed `dependency_allow_fail` to a simpler `allow_failure`
* Update docs to add section about the new `Dependency` class and use-case
* Updated `Job.dependencies_are_met` logic to take `FAILED` and `STOPPED` jobs into account when `allow_failure=True`
* Updated `test_job_dependency` test. Still failing with `Dependency` case.
* Fix `allow_failure` type coercion in `Job.restore`
* Re-arrange tests, so default `Dependency.allow_failure` is before explicit `allow_failure=True`
* Fixed Dependency, so it works correctly when allow_failure=True
* Attempt to execute pipeline prior to queueing a failed job's dependents. test_create_and_cancel_job_enqueue_dependents_in_registry test now passes.
* Added `Depedency` test utilizing multiple dependencies
* Removed irrelevant on_success and on_failure keyword arguments in example
* Replaced use of long_running_job
* Add test to verify `Dependency.jobs` contraints
* Suppress connection error in handle_job_failure
* test_dependencies have passed
* All tests pass if enqueue_dependents called without pipeline.watch()
* All tests now pass
* Removed print statements
* Cleanup Dependency implementation
* Renamed job.allow_failure to job.allow_dependency_failures
Co-authored-by: mattchan <mattchan@tencent.com>
Co-authored-by: Mike Hill <mhilluniversal@gmail.com>
There are small typos in:
- docs/docs/exceptions.md
- docs/docs/jobs.md
- rq/queue.py
- tests/fixtures.py
- tests/test_job.py
Fixes:
- Should read `slightly` rather than `slighty`.
- Should read `requeuing` rather than `requeueing`.
- Should read `implementers` rather than `implementors`.
- Should read `definition` rather than `defition`.
- Should read `canceled` rather than `canceld`.
Signed-off-by: Tim Gates <tim.gates@iress.com>
* Added CrossPlatformDeathPenalty that doesn't rely on signals
* Updated `SimpleWorker`'s `death_penalty_class` to utilize `CrossPlatformDeathPenalty` to allow use on Windows machines
* Changed `CrossPlatformDeathPenalty` to `TimerDeathPenalty`
* Removed overridden `death_penalty_class` in `SimpleWorker` until feature matures
* Added section in testing.md explaining how to utilize `SimpleWorker` on Windows OS
* Replaced usage of chatting with .format for python 3.5 compatibility
* Add tests for new timeout feature
* Explicitly set defaults.CALLBACK_TIMEOUT
* Implemented cross-thread method of raising errors by using ctypes
* Finished writing tests for new TimerDeathPenalty
* job: add get_meta() function
The newly introduced function returns meta data stored for the job. This
is required since job.meta stays an empty dict until the job is
finished or failed.
With the new function it's possible to store arbiatraty states/stages of
the job and allow the user to track progress. A long running job may
return custom stages like `downloading_data`, `unpacking_data`,
`processing_data`, etc.
This may allow better interfaces since users can track progress.
Signed-off-by: Paul Spooren <mail@aparcar.org>
* docs: add missing `refresh` arg to get_status()
This was previously missing.
Signed-off-by: Paul Spooren <mail@aparcar.org>
* Support enqueueing with on_success_callback
* Got success callback to execute properly
* Added on_failure callback
* Document success and failure callbacks
* More Flake8 fixes
* Mention that options can also be passed in via environment variables
* Increase coverage on test_callbacks.py
* Make `enqueue_*` and `Queue.parse_args` accept `pipeline` arg
* undo bad docs
* Fix lints in new code
* Implement enqueue_many, refactor dependency setup out to method
* Make name consistant
* Refactor enqueue_many, update tests, add docs
* Refactor out enqueueing from dependency setup
* Move new docs to queue page
* Fix section header
* Fix section header again
* Update rq version to 1.9.0
* Newer pip install command from git
Updated cutting edge pip install command to use HTTPS rather than insecure git+git protocol, as recommended by pip documentation. This allows the command to work with pip >= 21.0.1
* Updated pip command in README
* implemented round-robin and random access to queues
* added tests for RoundRobinQueue
* reverted change in gitignore
* removed linebreak
* added tests for random queues
* added documentation for round robin and random queues
* moved round robin strategy to worker
* reverted changes to queue.py
* reverted changes to workers.md
* reverted changes to test_queue
* added tests for RoundRobinWorker and RandomWorker
* added doc for round robin and random workers
* removed f-strings for backward compatibility
* corrected a mistake
* minor changes (code style)
* now using _ordered_queues instead of queues for reordering queues
* Added send_stop_job_command().
* send_stop_job_command now accepts just connection and job_id
* Document send_job_job_command
* Updated test coverage
* feat: added job heartbeat to track whether job is actually executing
heartbeat might be needed in cases when worker was hardkilled or the whole VM/docker was forcibly rebooted.
* fixed tests
* fixed test coverage issue
* chore: renamed job.heartbeat stuff according to review feedback
* chore: pipelined worker heartbeat and job heartbeat
* docs: documented job.heartbeat property
* fixes after review
* docs: updated last_heartbeat description
* chore: review
Co-authored-by: Ruslan Mullakhmetov <ruslan@twentythree.net>
Tianci Hu Marrero
lowercase00
Rony Lutsky
Krzysztof Jeziorny
Selwin Ong
Jahn Thomas Fidje
Alex Garel
Tim Gates
Pavel Levchuk
urmikab
Rajesh Khadka
Mateusz Soszyński
Michael Hill
mgcdanny
luto
Josh Cohen
Paul Spooren
rpkak
Joseph
Antonino Sabetta
James Cobb
Biel Cardona
Daniel Alley
f0cker
C. Mangla
Kyle Verhoog
Arturo Bracero
Ruslan Mullakhmetov