* Add support for dependent jobs in enqueue_many
* Add module to register dependencies for multiple jobs
The get_ready_jobs function will process dependencies for an array of
jobs passed in. If any jobs' dependencies are already met, those jobs
are returned so they can be enqueued.
* Add check for jobs without dependencies
* Remove extra colon in dependencies key
This seems like a bug, but if I'm mistaken please let me know.
* Add bulk deferred jobs to Redis
Need to call queue.enqueue_job to create the job hash in redis. Since all of
these jobs are deferred, they won't be added to the queue and processed
by a worker.
* Revert "Remove extra colon in dependencies key"
This reverts commit 5ebf7a35009fcca410c43b9327203915ddfd0628.
* Enqueue jobs without dependencies separately
Any jobs without dependencies will be enqueued before handling
* Fix enqueue_many return value
* Rename ready_jobs function
* Fix enqueue_many return value
* Instantiate job category arrays before if statement
* Execute pipe to enqueue jobs with met dependencies
* Add tests for enqueue_many with dependencies
* Change dependency sorting function name
* Use common kwargs dict to create jobs
* Remove redundant tests for dependent jobs
* Alphebetize imports
* Test job with met dependencies using enqueue_many
* Fix typo
* Format with black
* Sort imports
* Use lmove() when working on a single queue
* Skip reliable queue tests if Redis server doesn't support LMOVE
* Better test coverage
* job.origin should be string
* Added test for job that gets orphaned if worker.execute_job() fails
* Fix job tests
* worker.run_maintenance_tasks() now cleans intermediate queues
* Fixed import ordering
* No need to run slow tests and flake8 on SSL tests
* Minor typing fixes
* Fixed linting
* Update cli.py to support custom loggers
Allows a config.py file (via rq worker --config) to support things like this for logfmt logging:
DICT_CONFIG = {
"version": 1,
"formatters": {"logfmt": {
"()": "logfmter.Logfmter",
"keys": ["level","when","pid"],
"mapping": {"level":"levelname","when":"asctime","pid":"process"},
"datefmt": "%Y-%m-%dT%H:%M:%S%z"
}},
"handlers": {"console": {"class": "logging.StreamHandler","formatter": "logfmt" }},
"loggers": {
"root": {"handlers":["console"], "level": "INFO"},
"rq": {"handlers":["console"], "level": "INFO", "propagate": False},
}
}
* added simple test and documentation for DICT_CONFIG
* further attempt to get testing right for dictConfig
* move import to correct location
* fix
* remove meaningless options.get() usage
* linting checks and added missing test config file
* Scheduler should release and heartbeat only acquired locks.
* Added tests for heartbeat and release only acquired locks.
* Changed test description to correct one.
* Update linting configuration
This removes flake8 in favor of ruff, which also provides isort support, and
updates all files to be black, isort, and ruff compliant. This also adds black
and ruff checks to the tox and Github linting workflow.
* Tweak the code coverage config and calls
* Don't fail CI if codecov fails
* Remove codecov from dev-requirements.txt
* Pin packaging dependency to 21.3
* Pin coverage to 6.5.0
* Pin coverage to 6.2
* Downgrade packaging
* Drop Python 3.6 from test matrix
* Reenable Python 3.6 in tests
* Downgrade coverage req
* Make a different dev-requirements file for Python 3.6
* Moved dev-requirements-36 to root dir
* First stab at implementating worker pool
* Use process.is_alive() to check whether a process is still live
* Handle shutdown signal
* Check worker loop done
* First working version of `WorkerPool`.
* Added test for check_workers()
* Added test for pool.start()
* Better shutdown process
* Comment out test_start() to see if it fixes CI
* Make tests pass
* Make CI pass
* Comment out some tests
* Comment out more tests
* Re-enable a test
* Re-enable another test
* Uncomment check_workers test
* Added run_worker test
* Minor modification to dead worker detection
* More test cases
* Better process name for workers
* Added back pool.stop_workers() when signal is received
* Cleaned up cli.py
* WIP on worker-pool command
* Fix test
* Test that worker pool ignores consecutive shutdown signals
* Added test for worker-pool CLI command.
* Added timeout to CI jobs
* Fix worker pool test
* Comment out test_scheduler.py
* Fixed worker-pool in burst mode
* Increase test coverage
* Exclude tests directory from coverage.py
* Improve test coverage
* Renamed `Pool(num_workers=2) to `Pool(size=2)`
* Revert "Renamed `Pool(num_workers=2) to `Pool(size=2)`"
This reverts commit a1306f89ad0d8686c6bde447bff75e2f71f0733b.
* Renamed Pool to WorkerPool
* Added a new TestCase that doesn't use LocalStack
* Added job_class, worker_class and serializer arguments to WorkerPool
* Use parse_connection() in WorkerPool.__init__
* Added CLI arguments for worker-pool
* Minor WorkerPool and test fixes
* Fixed failing CLI test
* Document WorkerPool
* Added BaseWorker class
* Moved logging related functions to logutils
* Remove uneeded colorize function
* Updated worker.get_current_job_id() to not fail when job ID is None
* job.restore() shouldn't crash if is not present
* Fix various as_text() related crashes
* Added parse_connection function
* feat: allow custom connection pool class (#1885)
* Added test for SSL
---------
Co-authored-by: Cyril Chapellier <tchapi@users.noreply.github.com>
* improve the docs on in-flight jobs of killed/lost workers
* improve the docs on in-flight jobs of killed/lost workers
* document abandoned job behavior before 0.14
* consolidate job failure callback execution
* fix
* success callback as well
* merge fix
* create Callback class and change how callbacks serde
* deprecation
* add tests
* string format
* pr fix
* revert serialization changes
* fix timeout typing
* add documentation
* add test
* fix a bug
* fix tests
* move job heartbeat call to worker and make sure there is always a callback timeout
* fix test
* fix test
* test: check dependencies when enqueue via Queue.enqueue_job()
Signed-off-by: Simó Albert i Beltran <sim6@bona.gent>
* fix: check dependencies when enqueue via Queue.enqueue_job()
Signed-off-by: Simó Albert i Beltran <sim6@bona.gent>
Co-authored-by: Selwin Ong <selwin.ong@gmail.com>
---------
Signed-off-by: Simó Albert i Beltran <sim6@bona.gent>
Co-authored-by: Selwin Ong <selwin.ong@gmail.com>
* Updated arguments passed to the Sentinel Object when created from the settings
- added `USERNAME` key
- added `CONNECTION_KWARGS` key to allow passing additionals arguments to the Redis connections
- updated the documentation
* added missing comma
* tests(helpers): Added tests for Sentinel
---------
Co-authored-by: Joachim Burket <joachim.burket@hopitalvs.ch>
* Add started job registry cleanup job failure callback call
* WIP - need to fix test
* fix test
* rename, tests and docs
* better log message
* use class name
* Update registry.py
* Fix TimerDeathPenalty not properly handling negative/infinite timeout
* revert back to using exc_info
---------
Co-authored-by: Marcus <marcus@us2.ai>
* test: Dependency list in depends_on
Signed-off-by: Simó Albert i Beltran <sim6@bona.gent>
* fix: Dependency list in depends_on
Signed-off-by: Simó Albert i Beltran <sim6@bona.gent>
---------
Signed-off-by: Simó Albert i Beltran <sim6@bona.gent>
Ethan Wolinsky
Selwin Ong
juur
xzander
Rob Hudson
dependabot[bot]
Tom Hamilton Stubber
Cyril Chapellier
gabriels1234
nhairs
luto
Tianci Hu Marrero
Joris
Rony Lutsky
Jochen Kupperschmidt
Simó Albert i Beltran
Jesse P. Johnson
lowercase00
Joachim Burket
Marcus Ong
amks1