This bug has opened a lot of possible race-conditions, since the
watch-logic from redis did not fail anymore, if dependencies have been
changed in parallel.
* Fix `job.cancel` to remove job from registiries if not in queue
* Remove old queue remove call
* Block the ability to cancel job that are already canceled
* Fix py35 compat
* Rename helper method
* job: add get_meta() function
The newly introduced function returns meta data stored for the job. This
is required since job.meta stays an empty dict until the job is
finished or failed.
With the new function it's possible to store arbiatraty states/stages of
the job and allow the user to track progress. A long running job may
return custom stages like `downloading_data`, `unpacking_data`,
`processing_data`, etc.
This may allow better interfaces since users can track progress.
Signed-off-by: Paul Spooren <mail@aparcar.org>
* docs: add missing `refresh` arg to get_status()
This was previously missing.
Signed-off-by: Paul Spooren <mail@aparcar.org>
* Support enqueueing with on_success_callback
* Got success callback to execute properly
* Added on_failure callback
* Document success and failure callbacks
* More Flake8 fixes
* Mention that options can also be passed in via environment variables
* Increase coverage on test_callbacks.py
* adds unit test for a deserialization error
This tests that deserialization exceptions are properly logged, and fails in
the manner described in #1422 .
* Catch deserializing errors in Worker.handle_exception()
This fixes#1422 , and makes
tests/test_worker.py::TestWorker::test_deserializing_failure_is_handled
pass.
* made unit test less specific
This is required to get the test to pass under other serializers / other
python versions.
* Added generic DeserializationError
* switched ValueError to DeserializationError in a test
The changed test is creating an invalid job, which now raises
DeserializationError when data is accessed, as opposed to ValueError.
* cleanup jobs that are not really running due to zombie workers
* remove registry entries for zombie jobs
* return only the job ids on cleanup
* test zombie job cleanup
* format code
* rename variable to explain that second element in tuple is expiry, not score
* remove worker_key
* detect zombie jobs using old heartbeats
* reuse get_expired_job_ids
* set score using current_timestamp
* test idle jobs using stale heartbeats
* extract timeout into variable
* move heartbeats into StartedJobRegistry
* use registry.heartbeat in tests
* remove heartbeats when job removed from StartedJobRegistry
* remove idle and expired jobs from both wip and heartbeats set
* send heartbeat_ttl to registry.add
* typo
* revert everything 😶
* only keep job heartbeats as score (and get rid of job timeouts as scores
* calculate heartbeat_ttl in an overrideable function + override it in SimpleWorker + move storing StartedJobRegistry scores to job.heartbeat()
* set heartbeat to monitoring interval for infinite timeouts
* track elapsed_execution_time as part of worker
* reset current job working time when work on a job is done
* persisting the job working time as part of monitoring
* Also accept lists and tuples as value of `depends_on`.
* The elements of the lists/tuples may be either Jobs or Job IDs.
* A single Job / Job ID is still accepted as well.
* Represent _all_ job dependencies in `Job.to_dict()`.
We now represent the entire list, instead of just the first element.
* Fix some doctext regarding plurality of dependencies.
* Add unit tests for job dependencies.
* One unit test establishes a pattern for checking execution order as affected by dependencies.
* Another unit test applies this pattern to the new capability to name multiple dependencies.
* Add unit test for new `depends_on` input formats.
Also test that these are properly persisted.
* Repair `Job.restore()`.
Need to convert bytes back to strings when reloading `dependency_ids`.
* Maintain backwards compat. in `Job.to_dict()`.
Keep the old `dependency_id` (singular) key.
* Provide coverage for new test fixture.
* Simplify some code.
Cut some superfluous `as_text()` calls left over from an earlier commit.
* Check for `dependency_id` in `Job.restore()` for backwd. compat.
Also eliminate use of `as_text()` here, in favor of `.decode()`.
* Switch to snake case instead of camel case.
* Eliminate some usages of `as_text()`.
Also cut some `print` statements.
* Cleanup.
* Accept arbitrary iterables for `Job`'s `depends_on` kwarg.
Instead of requiring a list or tuple, we now make use of `ensure_list()`.
* Add test fixtures.
* Provide a system to get two workers working simultaneously, using `multiprocessing`.
* Define a simple job that just says whether its dependencies are met.
* In `rpush`, make an option to record the name of the worker.
* Improve unit tests on execution order with dependencies.
These now actually have two workers going, which makes a more thorough test.
* Add unit test examining `Job.dependencies_are_met()` at execution time.
* Redesign dependency execution order unit tests.
* Simplify assertions.
* Improve doctext and formatting.
* Move fixture tests to new, dedicated module `test_fixtures.py`.
* Use `enqueue` instead of `enqueue_call` in new unit tests.
* feat: added job heartbeat to track whether job is actually executing
heartbeat might be needed in cases when worker was hardkilled or the whole VM/docker was forcibly rebooted.
* fixed tests
* fixed test coverage issue
* chore: renamed job.heartbeat stuff according to review feedback
* chore: pipelined worker heartbeat and job heartbeat
* docs: documented job.heartbeat property
* fixes after review
* docs: updated last_heartbeat description
* chore: review
Co-authored-by: Ruslan Mullakhmetov <ruslan@twentythree.net>
* Initial implementation of Retry class
* Fixes job.refresh() under Python 3.5
* Remove the use of text_type in job.py
* Retry can be scheduled
* monitor_work_horse() should call handle_job_failure() with queue argument.
* Flake8 fixes
* Added docs for job retries
1) Check if `created_at` when checking if dependencies are met.
If `created_at` is `None` then the job has been deleted. This is sort of hack - we just need one of the fields on the job's hash that is ALWAYS populated. You can persist a job to redis without setting status...
2) Job#fetch_dependencies no longer raises NoSuchJob.
If one of a job's dependencies has been deleted from Redis, it is not returned from `fetch_dependencies` and no exception is raised.
* Add job status setting in enqueue_at (and in enqueue_in) methods
Update tests for this change
Closes: #1179
* Add status param to create_job func, rework enqueue_at status setting
* Multi Dependency Support - Registration & Enqueue Call
Internal API changes to support multiple dependencies.
* Store all of a job's _dependencies_ in a redis set. Delete that set when a job is deleted.
* Add Job#fetch_dependencies method - which return all jobs a job is dependent upon and optionally _WATCHES_ all dependency ids.
* Use Job#fetch_dependencies in Queue#call_enqueue. `fetch_dependencies` now sets WATCH and raises InvalidJobDependency, rather than call_enqueue.
`Queue` and `Job` public APIs still expect single ids of jobs for `depends_on` but internally register them in a way that could support multiple jobs being passed as dependencies.
Next up: need to update Queue#enqueue_dependents
* Use existing fetch_many method to get dependencies.
Modify fetch_dependencies to use fetch_many.
* Remove default value for fetch_many's connection parameter
* PR review housekeeping
* Remove a duplicate test
* Oneline something
* Fix missing colon in dependencies key
* Delete job key, dependents and dependencies at once
* More Fixes From Code Review
Updates to Job, Queue and associated tests.
* When Checking dependencies Avoid, trip to Redis
* When checking the status of a job, we have a 'clean' status of all dependencies(returned from Job#fetch_dependencies) and the job keys are WATCHed, so there's no reason to go back to Redis to get the status _again_.
* Looks as though, the `_status` set in `Job#restore` was bytes while it was converted to text(`as_text`) in `Job#get_status` - for consistency(and tests) converting to text in `restore` as well.
* In `Queue#enqueue_call`, moved WATCH of dependencies_key to before fetching dependencies. This doesn't really matter but seems more _correct_ - one can imagine some rogue API adding a dependency after they've been fetched but before they've been WATCHEed.
* Update Job#get_status to get _local_ status
* If refresh=False is passed, don't get status from Redis; return the value of _status. This is to avoid a trip to Redis if the caller can guarantee that the value of `_status` is _clean_.
* More Fixups
* Expire dependency keys in Job#cleanup
* Consistency in Job#fetch_dependencies
* Convert `_dependency_id` to `_dependency_ids`
Change `Job`s tracking from a single id of it's dependencies from a single _id_ to a list of _id_s. This change should be private to `Job` - especially leaving `Job#to_dict` and `Job#restore`s treatment of a single 'dependency_id' intact.
This change modifies existing tests.
* Remove reliance upon dependency property in tests
... use dependency.id not `_dependency_id`
* Re-add assertions for Falsey Values
* Add _dependency_id property
For backwards compatibility with other libs such as django-rq and rq-scheduler
* Added FailedJobRegistry.
* Added job.failure_ttl.
* queue.enqueue() now supports failure_ttl
* Added registry.get_queue().
* FailedJobRegistry.add() now assigns DEFAULT_FAILURE_TTL.
* StartedJobRegistry.cleanup() now moves expired jobs to FailedJobRegistry.
* Failed jobs are now added to FailedJobRegistry.
* Added FailedJobRegistry.requeue()
* Document the new `FailedJobRegistry` and changes in custom exception handler behavior.
* Added worker.disable_default_exception_handler.
* Document --disable-default-exception-handler option.
* Deleted worker.failed_queue.
* Deleted "move_to_failed_queue" exception handler.
* StartedJobRegistry should no longer move jobs to FailedQueue.
* Deleted requeue_job
* Fixed test error.
* Make requeue cli command work with FailedJobRegistry
* Added .pytest_cache to gitignore.
* Custom exception handlers are no longer run in reverse
* Restored requeue_job function
* Removed get_failed_queue
* Deleted FailedQueue
* Updated changelog.
* Document `failure_ttl`
* Updated docs.
* Remove job.status
* Fixed typo in test_registry.py
* Replaced _pipeline() with pipeline()
* FailedJobRegistry no longer fails on redis-py>=3
* Fixes test_clean_registries
* Worker names are now randomized
* Added a note about random worker names in CHANGES.md
* Worker will now stop working when encountering an unhandled exception.
* Worker should reraise SystemExit on cold shutdowns
* Added anchor.js to docs
* Support for Sentry-SDK (#1045)
* Updated RQ to support sentry-sdk
* Document Sentry integration
* Install sentry-sdk before running tests
* Improved rq info CLI command to be more efficient when displaying lar… (#1046)
* Improved rq info CLI command to be more efficient when displaying large number of workers
* Fixed an rq info --by-queue bug
* Fixed worker.total_working_time bug (#1047)
* queue.enqueue() no longer accepts `timeout` argument (#1055)
* Clean worker registry (#1056)
* queue.enqueue() no longer accepts `timeout` argument
* Added clean_worker_registry()
* Show worker hostname and PID on cli (#1058)
* Show worker hostname and PID on cli
* Improve test coverage
* Remove Redis version check when SSL is used
* Bump version to 1.0
* Removed pytest_cache/README.md
* Changed worker logging to use exc_info=True
* Removed unused queue.dequeue()
* Fixed typo in CHANGES.md
* setup_loghandlers() should always call logger.setLevel() if specified
* Replaced async keyword with is_async in the Queue class to fix reserved keyword syntax errors in Python 3.7
* Updated tests to use is_async keyword when instantiating Queue objects
* Updated docs to reference is_async keyword for Queue objects
* Updated tox.ini, setup.py and .travis.yml with references to Python 3.7
* Define redis key prefix as class variable
Some prefixes were hardcoded in several places. This made it hard to
use custom prefixes via subclasses.
Resolves#920
* fixup! Define redis key prefix as class variable
* Initial take on delete_dependents
* Add tests including corner cases
* No need to canel dependents since they are not in a queue yet anyway
* The dependents keys can be deleted in all cases
* Update tests to included saved jobs in the deletion tests
* Correctly use pipeline in cancel method
* Unused connection
* Include dependents into dict format of job
* Add TODO
* Address comments from selwin
* Delete dependents key in redis if delete_dependents is called on its own
* Address recent comments from selwin
* Small change to trigger travis
* Remove TODO referring to canceled job state
* Remove dependent_ids from to_dict
* Address recent comments from selwin
Currently, the job is being performed inside `enqueue_call()`, which
means that `async=False` has no effect if `enqueue_job()` is called
directly. This commit fixes that.
This includes:
- a partial refactor of the CLI to organize the shared options
- extends the tests in areas where passing custom backend classes makes sense
- allow setting the core CLI options as env vars
- minor cosmetic changes here and there
- run with an empty queue
- schedule one job (which uses get_current_connection and get_current_job) and
run `rqworker`
- schedule a job that itself schedules `access_self` and run `rqworker`
- Make sure the job didn't fail by assuring the failed queue is still empty
afterwards.
- Install this package locally when running in travis.
This actually unifies the behaviour of tox and travis as tox also builds the
package and then installs it into each test environment.
- fix flake8 (as run by tox)
th3hamm0r
Josh Cohen
Selwin Ong
Paul Spooren
Cyrille Lavigne
pwws
Omer Lachish
MyrikLD
skieffer
Ruslan Mullakhmetov
thomas
Thomas Matecki
Babatunde Olusola
Ivan Kiryanov
chevell
Theofanis Despoudis
stj
Christophe Olinger
Samuel Colvin
Theo
Alexey Katichev
Felipe Lacerda
Jannis Leidel
Arnold Krille
Javier Lopez
Eduard Carreras