Changed docs to use Github compatible code block markup

main
Selwin Ong 6 years ago
parent fc8dd95377
commit 4b8e615644

@ -23,20 +23,20 @@ pass in Redis connection references to queues directly.
In development mode, to connect to a default, local Redis server:
{% highlight python %}
```python
from rq import use_connection
use_connection()
{% endhighlight %}
```
In production, to connect to a specific Redis server:
{% highlight python %}
```python
from redis import Redis
from rq import use_connection
redis = Redis('my.host.org', 6789, password='secret')
use_connection(redis)
{% endhighlight %}
```
Be aware of the fact that `use_connection` pollutes the global namespace. It
also implies that you can only ever use a single connection.
@ -59,7 +59,7 @@ Each RQ object instance (queues, workers, jobs) has a `connection` keyword
argument that can be passed to the constructor. Using this, you don't need to
use `use_connection()`. Instead, you can create your queues like this:
{% highlight python %}
```python
from rq import Queue
from redis import Redis
@ -68,7 +68,7 @@ conn2 = Redis('remote.host.org', 9836)
q1 = Queue('foo', connection=conn1)
q2 = Queue('bar', connection=conn2)
{% endhighlight %}
```
Every job that is enqueued on a queue will know what connection it belongs to.
The same goes for the workers.
@ -85,7 +85,7 @@ default connection to be used.
An example will help to understand it:
{% highlight python %}
```python
from rq import Queue, Connection
from redis import Redis
@ -98,7 +98,7 @@ with Connection(Redis('localhost', 6379)):
assert q1.connection != q2.connection
assert q2.connection != q3.connection
assert q1.connection == q3.connection
{% endhighlight %}
```
You can think of this as if, within the `Connection` context, every newly
created RQ object instance will have the `connection` argument set implicitly.
@ -112,7 +112,7 @@ If your code does not allow you to use a `with` statement, for example, if you
want to use this to set up a unit test, you can use the `push_connection()` and
`pop_connection()` methods instead of using the context manager.
{% highlight python %}
```python
import unittest
from rq import Queue
from rq import push_connection, pop_connection
@ -127,8 +127,7 @@ class MyTest(unittest.TestCase):
def test_foo(self):
"""Any queues created here use local Redis."""
q = Queue()
...
{% endhighlight %}
```
### Sentinel support
@ -136,10 +135,10 @@ To use redis sentinel, you must specify a dictionary in the configuration file.
Using this setting in conjunction with the systemd or docker containers with the
automatic restart option allows workers and RQ to have a fault-tolerant connection to the redis.
{% highlight python %}
```python
SENTINEL: {'INSTANCES':[('remote.host1.org', 26379), ('remote.host2.org', 26379), ('remote.host3.org', 26379)],
'SOCKET_TIMEOUT': None,
'PASSWORD': 'secret',
'DB': 2,
'MASTER_NAME': 'master'}
{% endhighlight %}
```

@ -23,29 +23,27 @@ exception occurs.
This is how you register custom exception handler(s) to an RQ worker:
{% highlight python %}
```python
from rq.handlers import move_to_failed_queue # RQ's default exception handler
w = Worker([q], exception_handlers=[my_handler, move_to_failed_queue])
...
{% endhighlight %}
```
The handler itself is a function that takes the following parameters: `job`,
`exc_type`, `exc_value` and `traceback`:
{% highlight python %}
```python
def my_handler(job, exc_type, exc_value, traceback):
# do custom things here
# for example, write the exception info to a DB
...
{% endhighlight %}
```
You might also see the three exception arguments encoded as:
```python
def my_handler(job, *exc_info):
# do custom things here
...
```
## Chaining exception handlers
@ -63,7 +61,7 @@ as `True` (i.e. continue with the next handler).
To replace the default behaviour (i.e. moving the job to the `failed` queue),
use a custom exception handler that doesn't fall through, for example:
{% highlight python %}
```python
def black_hole(job, *exc_info):
return False
{% endhighlight %}
```

@ -13,13 +13,13 @@ arguments onto a queue. This is called _enqueueing_.
To put jobs on queues, first declare a function:
{% highlight python %}
```python
import requests
def count_words_at_url(url):
resp = requests.get(url)
return len(resp.text.split())
{% endhighlight %}
```
Noticed anything? There's nothing special about this function! Any Python
function call can be put on an RQ queue.
@ -27,7 +27,7 @@ function call can be put on an RQ queue.
To put this potentially expensive word count for a given URL in the background,
simply do this:
{% highlight python %}
```python
from rq import Queue
from redis import Redis
from somewhere import count_words_at_url
@ -43,14 +43,14 @@ print(job.result) # => None
# Now, wait a while, until the worker is finished
time.sleep(2)
print(job.result) # => 889
{% endhighlight %}
```
If you want to put the work on a specific queue, simply specify its name:
{% highlight python %}
```python
q = Queue('low', connection=redis_conn)
q.enqueue(count_words_at_url, 'http://nvie.com')
{% endhighlight %}
```
Notice the `Queue('low')` in the example above? You can use any queue name, so
you can quite flexibly distribute work to your own desire. A common naming
@ -78,28 +78,28 @@ job function.
In the last case, it may be advantageous to instead use the explicit version of
`.enqueue()`, `.enqueue_call()`:
{% highlight python %}
```python
q = Queue('low', connection=redis_conn)
q.enqueue_call(func=count_words_at_url,
args=('http://nvie.com',),
timeout=30)
{% endhighlight %}
```
For cases where the web process doesn't have access to the source code running
in the worker (i.e. code base X invokes a delayed function from code base Y),
you can pass the function as a string reference, too.
{% highlight python %}
```python
q = Queue('low', connection=redis_conn)
q.enqueue('my_package.my_module.my_func', 3, 4)
{% endhighlight %}
```
## Working with Queues
Besides enqueuing jobs, Queues have a few useful methods:
{% highlight python %}
```python
from rq import Queue
from redis import Redis
@ -117,7 +117,7 @@ job = q.fetch_job('my_id') # Returns job having ID "my_id"
# Deleting the queue
q.delete(delete_jobs=True) # Passing in `True` will remove all jobs in the queue
# queue is unusable now unless re-instantiated
{% endhighlight %}
```
### On the Design
@ -151,7 +151,7 @@ of course).
If you're familiar with Celery, you might be used to its `@task` decorator.
Starting from RQ >= 0.3, there exists a similar decorator:
{% highlight python %}
```python
from rq.decorators import job
@job('low', connection=my_redis_conn, timeout=5)
@ -161,7 +161,7 @@ def add(x, y):
job = add.delay(3, 4)
time.sleep(1)
print(job.result)
{% endhighlight %}
```
## Bypassing workers
@ -170,12 +170,12 @@ For testing purposes, you can enqueue jobs without delegating the actual
execution to a worker (available since version 0.3.1). To do this, pass the
`is_async=False` argument into the Queue constructor:
{% highlight pycon %}
```python
>>> q = Queue('low', is_async=False, connection=my_redis_conn)
>>> job = q.enqueue(fib, 8)
>>> job.result
21
{% endhighlight %}
```
The above code runs without an active worker and executes `fib(8)`
synchronously within the same process. You may know this behaviour from Celery
@ -188,11 +188,11 @@ a redis instance for storing states related to job execution and completion.
New in RQ 0.4.0 is the ability to chain the execution of multiple jobs.
To execute a job that depends on another job, use the `depends_on` argument:
{% highlight python %}
```python
q = Queue('low', connection=my_redis_conn)
report_job = q.enqueue(generate_report)
q.enqueue(send_report, depends_on=report_job)
{% endhighlight %}
```
The ability to handle job dependencies allows you to split a big job into
several smaller ones. A job that is dependent on another is enqueued only when

@ -13,14 +13,14 @@ jobs.
All job information is stored in Redis. You can inspect a job and its attributes
by using `Job.fetch()`.
{% highlight python %}
```python
from redis import Redis
from rq.job import Job
connection = Redis()
job = Job.fetch('my_job_id', connection=redis)
print('Status: %s' $ job.get_status())
{% endhighlight %}
```
Some interesting job attributes include:
* `job.status`
@ -38,14 +38,14 @@ Some interesting job attributes include:
Since job functions are regular Python functions, you have to ask RQ for the
current job ID, if any. To do this, you can use:
{% highlight python %}
```python
from rq import get_current_job
def add(x, y):
job = get_current_job()
print('Current job: %s' % (job.id,))
return x + y
{% endhighlight %}
```
## Storing arbitrary data on jobs
@ -56,7 +56,7 @@ To add/update custom status information on this job, you have access to the
`meta` property, which allows you to store arbitrary pickleable data on the job
itself:
{% highlight python %}
```python
import socket
def add(x, y):
@ -67,7 +67,7 @@ def add(x, y):
# do more work
time.sleep(1)
return x + y
{% endhighlight %}
```
## Time to live for job in queue
@ -77,13 +77,13 @@ _New in version 0.4.7._
A job has two TTLs, one for the job result and one for the job itself. This means that if you have
job that shouldn't be executed after a certain amount of time, you can define a TTL as such:
{% highlight python %}
```python
# When creating the job:
job = Job.create(func=say_hello, ttl=43)
# or when queueing a new job:
job = q.enqueue(count_words_at_url, 'http://nvie.com', ttl=43)
{% endhighlight %}
```
## Failed Jobs
@ -92,7 +92,7 @@ If a job fails and raises an exception, the worker will put the job in a failed
On the Job instance, the `is_failed` property will be true. To fetch all failed jobs, scan
through the `get_failed_queue()` queue.
{% highlight python %}
```python
from redis import StrictRedis
from rq import push_connection, get_failed_queue, Queue
from rq.job import Job
@ -115,4 +115,4 @@ fq.requeue(job.id)
assert fq.count == 0
assert Queue('fake').count == 1
{% endhighlight %}
```

@ -13,10 +13,10 @@ which looks like this:
To install, just do:
{% highlight console %}
```console
$ pip install rq-dashboard
$ rq-dashboard
{% endhighlight %}
```
It can also be integrated easily in your Flask app.
@ -25,7 +25,7 @@ It can also be integrated easily in your Flask app.
To see what queues exist and what workers are active, just type `rq info`:
{% highlight console %}
```console
$ rq info
high |██████████████████████████ 20
low |██████████████ 12
@ -36,14 +36,14 @@ Bricktop.19233 idle: low
Bricktop.19232 idle: high, default, low
Bricktop.18349 idle: default
3 workers, 3 queues
{% endhighlight %}
```
## Querying by queue names
You can also query for a subset of queues, if you're looking for specific ones:
{% highlight console %}
```console
$ rq info high default
high |██████████████████████████ 20
default |█████████ 8
@ -52,7 +52,7 @@ default |█████████ 8
Bricktop.19232 idle: high, default
Bricktop.18349 idle: default
2 workers, 2 queues
{% endhighlight %}
```
## Organising workers by queue
@ -60,7 +60,7 @@ Bricktop.18349 idle: default
By default, `rq info` prints the workers that are currently active, and the
queues that they are listening on, like this:
{% highlight console %}
```console
$ rq info
...
@ -68,12 +68,12 @@ Mickey.26421 idle: high, default
Bricktop.25458 busy: high, default, low
Turkish.25812 busy: high, default
3 workers, 3 queues
{% endhighlight %}
```
To see the same data, but organised by queue, use the `-R` (or `--by-queue`)
flag:
{% highlight console %}
```console
$ rq info -R
...
@ -82,7 +82,7 @@ low: Bricktop.25458 (busy)
default: Bricktop.25458 (busy), Mickey.26421 (idle), Turkish.25812 (busy)
failed:
3 workers, 4 queues
{% endhighlight %}
```
## Interval polling
@ -90,16 +90,16 @@ failed:
By default, `rq info` will print stats and exit.
You can specify a poll interval, by using the `--interval` flag.
{% highlight console %}
```console
$ rq info --interval 1
{% endhighlight %}
```
`rq info` will now update the screen every second. You may specify a float
value to indicate fractions of seconds. Be aware that low interval values will
increase the load on Redis, of course.
{% highlight console %}
```console
$ rq info --interval 0.5
{% endhighlight %}
```
[dashboard]: https://github.com/nvie/rq-dashboard

@ -93,15 +93,15 @@ If a job requires more (or less) time to complete, the default timeout period
can be loosened (or tightened), by specifying it as a keyword argument to the
`enqueue()` call, like so:
{% highlight python %}
```python
q = Queue()
q.enqueue(mytask, args=(foo,), kwargs={'bar': qux}, timeout=600) # 10 mins
{% endhighlight %}
```
You can also change the default timeout for jobs that are enqueued via specific
queue instances at once, which can be useful for patterns like this:
{% highlight python %}
```python
# High prio jobs should end in 8 secs, while low prio
# work may take up to 10 mins
high = Queue('high', default_timeout=8) # 8 secs
@ -109,7 +109,7 @@ low = Queue('low', default_timeout=600) # 10 mins
# Individual jobs can still override these defaults
low.enqueue(really_really_slow, timeout=3600) # 1 hr
{% endhighlight %}
```
Individual jobs can still specify an alternative timeout, as workers will
respect these.

@ -9,7 +9,7 @@ You may wish to include your RQ tasks inside unit tests. However many frameworks
Therefore, you must use the SimpleWorker class to avoid fork();
{% highlight python %}
```python
from redis import Redis
from rq import SimpleWorker, Queue
@ -18,7 +18,7 @@ queue.enqueue(my_long_running_job)
worker = SimpleWorker([queue], connection=queue.connection)
worker.work(burst=True) # Runs enqueued job
# Check for result...
{% endhighlight %}
```
## Running Jobs in unit tests
@ -31,11 +31,11 @@ Additionally, we can use fakeredis to mock a redis instance, so we don't have to
run a redis server separately. The instance of the fake redis server can
be directly passed as the connection argument to the queue:
{% highlight python %}
```python
from fakeredis import FakeStrictRedis
from rq import Queue
queue = Queue(is_async=False, connection=FakeStrictRedis())
job = queue.enqueue(my_long_running_job)
assert job.is_finished
{% endhighlight %}
```

@ -13,14 +13,14 @@ to perform inside web processes.
To start crunching work, simply start a worker from the root of your project
directory:
{% highlight console %}
```console
$ rq worker high normal low
*** Listening for work on high, normal, low
Got send_newsletter('me@nvie.com') from default
Job ended normally without result
*** Listening for work on high, normal, low
...
{% endhighlight %}
```
Workers will read jobs from the given queues (the order is important) in an
endless loop, waiting for new work to arrive when all jobs are done.
@ -37,14 +37,14 @@ new work when they run out of work. Workers can also be started in _burst
mode_ to finish all currently available work and quit as soon as all given
queues are emptied.
{% highlight console %}
```console
$ rq worker --burst high normal low
*** Listening for work on high, normal, low
Got send_newsletter('me@nvie.com') from default
Job ended normally without result
No more work, burst finished.
Registering death.
{% endhighlight %}
```
This can be useful for batch work that needs to be processed periodically, or
just to scale up your workers temporarily during peak periods.
@ -106,7 +106,7 @@ yourself before starting the work loop.
To do this, provide your own worker script (instead of using `rq worker`).
A simple implementation example:
{% highlight python %}
```python
#!/usr/bin/env python
import sys
from rq import Connection, Worker
@ -121,7 +121,7 @@ with Connection():
w = Worker(qs)
w.work()
{% endhighlight %}
```
### Worker names
@ -139,7 +139,7 @@ starting the worker, using the `--name` option.
`Worker` instances store their runtime information in Redis. Here's how to
retrieve them:
{% highlight python %}
```python
from redis import Redis
from rq import Queue, Worker
@ -150,14 +150,14 @@ workers = Worker.all(connection=redis)
# Returns all workers in this queue (new in version 0.10.0)
queue = Queue('queue_name')
workers = Worker.all(queue=queue)
{% endhighlight %}
```
_New in version 0.10.0._
If you only want to know the number of workers for monitoring purposes, using
`Worker.count()` is much more performant.
{% highlight python %}
```python
from redis import Redis
from rq import Worker
@ -169,8 +169,7 @@ workers = Worker.count(connection=redis)
# Count the number of workers for a specific queue
queue = Queue('queue_name', connection=redis)
workers = Worker.all(queue=queue)
{% endhighlight %}
```
### Worker statistics
@ -180,14 +179,14 @@ _New in version 0.9.0._
If you want to check the utilization of your queues, `Worker` instances
store a few useful information:
{% highlight python %}
```python
from rq.worker import Worker
worker = Worker.find_by_key('rq:worker:name')
worker.successful_job_count # Number of jobs finished successfully
worker.failed_job_count. # Number of failed jobs processed by this worker
worker.total_working_time # Number of time spent executing jobs
{% endhighlight %}
```
## Taking down workers
@ -209,7 +208,7 @@ If you'd like to configure `rq worker` via a configuration file instead of
through command line arguments, you can do this by creating a Python file like
`settings.py`:
{% highlight python %}
```python
REDIS_URL = 'redis://localhost:6379/1'
# You can also specify the Redis DB to use
@ -228,7 +227,7 @@ SENTRY_DSN = 'sync+http://public:secret@example.com/1'
# If you want custom worker name
# NAME = 'worker-1024'
{% endhighlight %}
```
The example above shows all the options that are currently supported.
@ -236,9 +235,9 @@ _Note: The_ `QUEUES` _and_ `REDIS_PASSWORD` _settings are new since 0.3.3._
To specify which module to read settings from, use the `-c` option:
{% highlight console %}
```console
$ rq worker -c settings
{% endhighlight %}
```
## Custom worker classes
@ -255,9 +254,9 @@ more common requests so far are:
You can use the `-w` option to specify a different worker class to use:
{% highlight console %}
```console
$ rq worker -w 'path.to.GeventWorker'
{% endhighlight %}
```
## Custom Job and Queue classes
@ -267,15 +266,15 @@ _Will be available in next release._
You can tell the worker to use a custom class for jobs and queues using
`--job-class` and/or `--queue-class`.
{% highlight console %}
```console
$ rq worker --job-class 'custom.JobClass' --queue-class 'custom.QueueClass'
{% endhighlight %}
```
Don't forget to use those same classes when enqueueing the jobs.
For example:
{% highlight python %}
```python
from rq import Queue
from rq.job import Job
@ -287,7 +286,7 @@ class CustomQueue(Queue):
queue = CustomQueue('default', connection=redis_conn)
queue.enqueue(some_func)
{% endhighlight %}
```
## Custom DeathPenalty classes
@ -307,9 +306,9 @@ _New in version 0.5.5._
If you need to handle errors differently for different types of jobs, or simply want to customize
RQ's default error handling behavior, run `rq worker` using the `--exception-handler` option:
{% highlight console %}
```console
$ rq worker --exception-handler 'path.to.my.ErrorHandler'
# Multiple exception handlers is also supported
$ rq worker --exception-handler 'path.to.my.ErrorHandler' --exception-handler 'another.ErrorHandler'
{% endhighlight %}
```

Loading…
Cancel
Save