5b95725dc4
* added Dependency class with allow_failures * Requested changes * Check type before setting `job.dependency_allow_fail` within `Job.create` * Set `job.dependency_allow_fail` within `Job.create` * Added test to ensure persistence of `dependency_allow_fail` * Removed typing and allow mixed list of ints and Job objects * Convert dependency_allow_fail boolean to integer during serialization to avoid redis DataError * Updated `test_multiple_dependencies_are_accepted_and_persisted` test to include `Dependency` cases * Adding placeholder test to test actual behavior of new `Dependency` usage in `depends_on` * Updated `test_job_dependency` to include cases using `Dependency` * Added dependency_allow_fail logic to `Job.restore` * Renamed `dependency_allow_fail` to a simpler `allow_failure` * Update docs to add section about the new `Dependency` class and use-case * Updated `Job.dependencies_are_met` logic to take `FAILED` and `STOPPED` jobs into account when `allow_failure=True` * Updated `test_job_dependency` test. Still failing with `Dependency` case. * Fix `allow_failure` type coercion in `Job.restore` * Re-arrange tests, so default `Dependency.allow_failure` is before explicit `allow_failure=True` * Fixed Dependency, so it works correctly when allow_failure=True * Attempt to execute pipeline prior to queueing a failed job's dependents. test_create_and_cancel_job_enqueue_dependents_in_registry test now passes. * Added `Depedency` test utilizing multiple dependencies * Removed irrelevant on_success and on_failure keyword arguments in example * Replaced use of long_running_job * Add test to verify `Dependency.jobs` contraints * Suppress connection error in handle_job_failure * test_dependencies have passed * All tests pass if enqueue_dependents called without pipeline.watch() * All tests now pass * Removed print statements * Cleanup Dependency implementation * Renamed job.allow_failure to job.allow_dependency_failures Co-authored-by: mattchan <mattchan@tencent.com> Co-authored-by: Mike Hill <mhilluniversal@gmail.com> |
2 years ago | |
---|---|---|
.github | 2 years ago | |
docs | 2 years ago | |
examples | 2 years ago | |
rq | 2 years ago | |
tests | 2 years ago | |
.coveragerc | 11 years ago | |
.deepsource.toml | 5 years ago | |
.gitignore | 6 years ago | |
.mailmap | 9 years ago | |
CHANGES.md | 3 years ago | |
Dockerfile | 4 years ago | |
LICENSE | 13 years ago | |
MANIFEST.in | 4 years ago | |
Makefile | 10 years ago | |
README.md | 3 years ago | |
codecov.yml | 3 years ago | |
dev-requirements.txt | 2 years ago | |
requirements.txt | 5 years ago | |
run_tests_in_docker.sh | 4 years ago | |
setup.cfg | 6 years ago | |
setup.py | 4 years ago | |
tox.ini | 2 years ago |
README.md
RQ (Redis Queue) is a simple Python library for queueing jobs and processing them in the background with workers. It is backed by Redis and it is designed to have a low barrier to entry. It should be integrated in your web stack easily.
RQ requires Redis >= 3.0.0.
Full documentation can be found here.
Support RQ
If you find RQ useful, please consider supporting this project via Tidelift.
Getting started
First, run a Redis server, of course:
$ redis-server
To put jobs on queues, you don't have to do anything special, just define your typically lengthy or blocking function:
import requests
def count_words_at_url(url):
"""Just an example function that's called async."""
resp = requests.get(url)
return len(resp.text.split())
You do use the excellent requests package, don't you?
Then, create an RQ queue:
from redis import Redis
from rq import Queue
queue = Queue(connection=Redis())
And enqueue the function call:
from my_module import count_words_at_url
job = queue.enqueue(count_words_at_url, 'http://nvie.com')
Scheduling jobs are also similarly easy:
# Schedule job to run at 9:15, October 10th
job = queue.enqueue_at(datetime(2019, 10, 10, 9, 15), say_hello)
# Schedule job to run in 10 seconds
job = queue.enqueue_in(timedelta(seconds=10), say_hello)
Retrying failed jobs is also supported:
from rq import Retry
# Retry up to 3 times, failed job will be requeued immediately
queue.enqueue(say_hello, retry=Retry(max=3))
# Retry up to 3 times, with configurable intervals between retries
queue.enqueue(say_hello, retry=Retry(max=3, interval=[10, 30, 60]))
For a more complete example, refer to the docs. But this is the essence.
The worker
To start executing enqueued function calls in the background, start a worker from your project's directory:
$ rq worker --with-scheduler
*** Listening for work on default
Got count_words_at_url('http://nvie.com') from default
Job result = 818
*** Listening for work on default
That's about it.
Installation
Simply use the following command to install the latest released version:
pip install rq
If you want the cutting edge version (that may well be broken), use this:
pip install git+https://github.com/rq/rq.git@master#egg=rq
Related Projects
Check out these below repos which might be useful in your rq based project.
Project history
This project has been inspired by the good parts of Celery, Resque and this snippet, and has been created as a lightweight alternative to the heaviness of Celery or other AMQP-based queueing implementations.