* Add job status setting in enqueue_at (and in enqueue_in) methods
Update tests for this change
Closes: #1179
* Add status param to create_job func, rework enqueue_at status setting
* First RQScheduler prototype
* WIP job scheduling
* Fixed Python 2.7 tests
* Added ScheduledJobRegistry.get_scheduled_time(job)
* WIP on scheduler's threading mechanism
* Fixed test errors
* Changed scheduler.acquire_locks() to instance method
* Added scheduler.prepare_registries()
* Somewhat working implementation of RQ scheduler
* Only call stop_scheduler if there's a scheduler present
* Use OSError rather than ProcessLookupError for PyPy compatibility
* Added `auto_start` argument to scheduler.acquire_locks()
* Make RQScheduler play better with timezone
* Fixed test error
* Added --with-scheduler flag to rq worker CLI
* Fix tests on Python 2.x
* More Python 2 fixes
* Only call `scheduler.start` if worker is run in non burst mode
* Fixed an issue where running worker with scheduler would fail sometimes
* Make `worker.stop_scheduler()` more resilient to errors
* worker.dequeue_job_and_maintain_ttl() should also periodically run maintenance tasks
* Scheduler can now work with worker in both burst and non burst mode
* Fixed scheduler logging message
* Always log scheduler errors when running
* Improve scheduler error logging message
* Removed testing code
* Scheduler should periodically try to acquire locks for other queues it doesn't have
* Added tests for scheduler.should_reacquire_locks
* Added queue.enqueue_in()
* Fixes queue.enqueue_in() in Python 2.7
* First stab at documenting job scheduling
* Remove unused methods
* Remove Python 2.6 logging compatibility code
* Remove more unused imports
* Added convenience methods to access job registries from queue
* Added test for worker.run_maintenance_tasks()
* Simplify worker.queue_names() and worker.queue_keys()
* Updated changelog to mention RQ's new job scheduling mechanism.
* Multi Dependency Support - Registration & Enqueue Call
Internal API changes to support multiple dependencies.
* Store all of a job's _dependencies_ in a redis set. Delete that set when a job is deleted.
* Add Job#fetch_dependencies method - which return all jobs a job is dependent upon and optionally _WATCHES_ all dependency ids.
* Use Job#fetch_dependencies in Queue#call_enqueue. `fetch_dependencies` now sets WATCH and raises InvalidJobDependency, rather than call_enqueue.
`Queue` and `Job` public APIs still expect single ids of jobs for `depends_on` but internally register them in a way that could support multiple jobs being passed as dependencies.
Next up: need to update Queue#enqueue_dependents
* Use existing fetch_many method to get dependencies.
Modify fetch_dependencies to use fetch_many.
* Remove default value for fetch_many's connection parameter
* PR review housekeeping
* Remove a duplicate test
* Oneline something
* Fix missing colon in dependencies key
* Delete job key, dependents and dependencies at once
* More Fixes From Code Review
Updates to Job, Queue and associated tests.
* When Checking dependencies Avoid, trip to Redis
* When checking the status of a job, we have a 'clean' status of all dependencies(returned from Job#fetch_dependencies) and the job keys are WATCHed, so there's no reason to go back to Redis to get the status _again_.
* Looks as though, the `_status` set in `Job#restore` was bytes while it was converted to text(`as_text`) in `Job#get_status` - for consistency(and tests) converting to text in `restore` as well.
* In `Queue#enqueue_call`, moved WATCH of dependencies_key to before fetching dependencies. This doesn't really matter but seems more _correct_ - one can imagine some rogue API adding a dependency after they've been fetched but before they've been WATCHEed.
* Update Job#get_status to get _local_ status
* If refresh=False is passed, don't get status from Redis; return the value of _status. This is to avoid a trip to Redis if the caller can guarantee that the value of `_status` is _clean_.
* More Fixups
* Expire dependency keys in Job#cleanup
* Consistency in Job#fetch_dependencies
* Added FailedJobRegistry.
* Added job.failure_ttl.
* queue.enqueue() now supports failure_ttl
* Added registry.get_queue().
* FailedJobRegistry.add() now assigns DEFAULT_FAILURE_TTL.
* StartedJobRegistry.cleanup() now moves expired jobs to FailedJobRegistry.
* Failed jobs are now added to FailedJobRegistry.
* Added FailedJobRegistry.requeue()
* Document the new `FailedJobRegistry` and changes in custom exception handler behavior.
* Added worker.disable_default_exception_handler.
* Document --disable-default-exception-handler option.
* Deleted worker.failed_queue.
* Deleted "move_to_failed_queue" exception handler.
* StartedJobRegistry should no longer move jobs to FailedQueue.
* Deleted requeue_job
* Fixed test error.
* Make requeue cli command work with FailedJobRegistry
* Added .pytest_cache to gitignore.
* Custom exception handlers are no longer run in reverse
* Restored requeue_job function
* Removed get_failed_queue
* Deleted FailedQueue
* Updated changelog.
* Document `failure_ttl`
* Updated docs.
* Remove job.status
* Fixed typo in test_registry.py
* Replaced _pipeline() with pipeline()
* FailedJobRegistry no longer fails on redis-py>=3
* Fixes test_clean_registries
* Worker names are now randomized
* Added a note about random worker names in CHANGES.md
* Worker will now stop working when encountering an unhandled exception.
* Worker should reraise SystemExit on cold shutdowns
* Added anchor.js to docs
* Support for Sentry-SDK (#1045)
* Updated RQ to support sentry-sdk
* Document Sentry integration
* Install sentry-sdk before running tests
* Improved rq info CLI command to be more efficient when displaying lar… (#1046)
* Improved rq info CLI command to be more efficient when displaying large number of workers
* Fixed an rq info --by-queue bug
* Fixed worker.total_working_time bug (#1047)
* queue.enqueue() no longer accepts `timeout` argument (#1055)
* Clean worker registry (#1056)
* queue.enqueue() no longer accepts `timeout` argument
* Added clean_worker_registry()
* Show worker hostname and PID on cli (#1058)
* Show worker hostname and PID on cli
* Improve test coverage
* Remove Redis version check when SSL is used
* Bump version to 1.0
* Removed pytest_cache/README.md
* Changed worker logging to use exc_info=True
* Removed unused queue.dequeue()
* Fixed typo in CHANGES.md
* setup_loghandlers() should always call logger.setLevel() if specified
* modify zadd calls for redis-py 3.0
redis-py 3.0 changes the zadd interface that accepts a single
mapping argument that is expected to be a dict.
https://github.com/andymccurdy/redis-py#mset-msetnx-and-zadd
* change FailedQueue.push_job_id to always push a str
redis-py 3.0 does not attempt to cast values to str and is left
to the user.
* remove Redis connection patching
Since in redis-py 3.0, Redis == StrictRedis class, we no longer
need to patch _zadd and other methods.
Ref: https://github.com/rq/rq/pull/1016#issuecomment-441010847
* Replaced async keyword with is_async in the Queue class to fix reserved keyword syntax errors in Python 3.7
* Updated tests to use is_async keyword when instantiating Queue objects
* Updated docs to reference is_async keyword for Queue objects
* Updated tox.ini, setup.py and .travis.yml with references to Python 3.7
* Define redis key prefix as class variable
Some prefixes were hardcoded in several places. This made it hard to
use custom prefixes via subclasses.
Resolves#920
* fixup! Define redis key prefix as class variable
* First stab at implementing worker statistics.
* Moved worker data restoration logic to worker.refresh().
* Failed and successfull job counts are now properly incremented.
* Worker now keeps track of total_working_time
* Ensure job.ended_at is set in the case of unhandled job failure.
* handle_job_failure shouldn't crash if job.started_at is not present.
* Added a custom exception for timeout transfer
* Added a util to transfer timeout to a united format
* Transfer timeout format when creating a queue or enqueue jobs
* Fixed typos
* Fixed bug in transfer_timeout function
* Added test for function transfer_timeout
* Updated transfer_timeout to allow uppercase unit
* Renamed function in utils
Currently, the job is being performed inside `enqueue_call()`, which
means that `async=False` has no effect if `enqueue_job()` is called
directly. This commit fixes that.
This includes:
- a partial refactor of the CLI to organize the shared options
- extends the tests in areas where passing custom backend classes makes sense
- allow setting the core CLI options as env vars
- minor cosmetic changes here and there
Current implementation is using the first job queue
to insert the dependent jobs. This makes RQ to ignore
the original queue where the jobs were enqueued.
With this change, we will use job origin attribute to
fetch the original queue, then insert the dependent
job to the fetched queue.
fixes issue #512
In the case of many sequential jobs having been deleted, a recursive
implementation of `Queue#dequeue` is prone to blowing the stack in the
absence of tail-recursion support. Change the implementation from
recursive to iterative to work around this issue in CPython.
Other parts of the code (i.e.: the `.create()` method) allow the `depends_on` kwarg to be a `Job` object *or* a job id. This is an attempt to allow that same idea within the `.enqueue_call()` method for a queue. Since this part of the code requires actually knowing the precise redis key for the job that's depended on, my intuition is that a `.fetch()` is the only way to solve this.
Thomas Matecki
Babatunde Olusola
Selwin Ong
Ivan Kiryanov
Thomas Matecki
Ivan Piskunov
Wolfgang Langner
Darshan Rai
Paul Robertson
chevell
stj
Theo
Amjith Ramanujam
Yaniv Greenberg
Alexey Katichev
luojiebin
Felipe Lacerda
Marko Mrdjenovic
Benjamin Root
Jannis Leidel
Michael DeWulf
Julien Surloppe
Stefan Hammer
Samuel Colvin
Stefan Hammer
Eduard Carreras
Tornstrom
Javier Lopez
Jean-Christophe SAAD-DUPUY
Dustin Falgout
Vincent Driessen
RyanMTB
Tamir Duberstein
Vlad Pronsky
glaslos
Taras Semenenko
Vincent Driessen
Jonathan Tushman
Travis Johnson
foxx