You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Thomas Matecki 80c82f731f Multi Dependency Support - Registration & Enqueue Call (#1155)
* Multi Dependency Support - Registration & Enqueue Call

Internal API changes to support multiple dependencies.
* Store all of a job's _dependencies_ in a redis set. Delete that set when a job is deleted.
* Add Job#fetch_dependencies method - which return all jobs a job is dependent upon and optionally _WATCHES_ all dependency ids.
* Use Job#fetch_dependencies in Queue#call_enqueue. `fetch_dependencies` now sets WATCH and raises InvalidJobDependency, rather than call_enqueue.

`Queue` and `Job` public APIs still expect single ids of jobs for `depends_on` but internally register them in a way that could support multiple jobs being passed as dependencies.

Next up: need to update Queue#enqueue_dependents

* Use existing fetch_many method to get dependencies.

Modify fetch_dependencies to use fetch_many.

* Remove default value for fetch_many's connection parameter

* PR review housekeeping

* Remove a duplicate test
* Oneline something
* Fix missing colon in dependencies key
* Delete job key, dependents and dependencies at once

* More Fixes From Code Review

Updates to Job, Queue and associated tests.

* When Checking dependencies Avoid, trip to Redis

* When checking the status of a job, we have a 'clean' status of all dependencies(returned from Job#fetch_dependencies) and the job keys are WATCHed, so there's no reason to go back to Redis to get the status _again_.
* Looks as though, the `_status` set in `Job#restore` was bytes while it was converted to text(`as_text`) in `Job#get_status` - for consistency(and tests) converting to text in `restore` as well.
* In `Queue#enqueue_call`, moved WATCH of dependencies_key to before fetching dependencies. This doesn't really matter but seems more _correct_ - one can imagine some rogue API adding a dependency after they've been fetched but before they've been WATCHEed.

* Update Job#get_status to get _local_ status

* If refresh=False is passed, don't get status from Redis; return the value of _status. This is to avoid a trip to Redis if the caller can guarantee that the value of `_status` is _clean_.

* More Fixups

* Expire dependency keys in Job#cleanup
* Consistency in Job#fetch_dependencies
5 years ago
..
config_files RQ v1.0! (#1059) 6 years ago
__init__.py refactor: use try ImportError instead of py-version check (#1034) 6 years ago
fixtures.py RQ v1.0! (#1059) 6 years ago
test_cli.py Fix rq info not found workers information error, Fixes #1139 (#1149) 5 years ago
test_connection.py Fixed #870 Improved test coverage for connections.py and utils.py 7 years ago
test_decorator.py Multi Dependency Support [Internal API Changes] (#1147) 5 years ago
test_helpers.py Restored the ability to specify connection params in config 10 years ago
test_job.py Multi Dependency Support - Registration & Enqueue Call (#1155) 5 years ago
test_queue.py Multi Dependency Support - Registration & Enqueue Call (#1155) 5 years ago
test_registry.py Added `delete_job` argument to registry.remove()` (#1161) 5 years ago
test_sentry.py RQ v1.0! (#1059) 6 years ago
test_utils.py add milliseconds into timestamps, fix #721 7 years ago
test_worker.py Store worker's RQ and Python versions (#1125) 5 years ago
test_worker_registration.py RQ v1.0! (#1059) 6 years ago