You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
 
Vincent Driessen 90b15fd0b6 Don't silently fail when unpickling.
When a pickled job string can't be unpickled because some required
module isn't loadable, this leads to an `UnpickleError` in the worker
(not in the horse).

Currently we just assume "garbage" in the job's data field, and silently
ignore it.

This is bad.

Really bad.

Because it avoids the normal exception handling mechanism that RQ has.

Historically, this "feature" was introduced to ignore any invalid pickle
data ("bad strings") on queues, and go on. However, we must assume data
inside `job.data` to be valid pickle data.

While an invalid _format_ of pickle data (e.g. the string "blablah"
isn't valid) leads to unpickle errors, unpickling errors will also occur
when the job can't be validly constructed in memory for other reasons,
like being unable to load a specific class.

Django is a good example of this: try submitting jobs that use
`django.conf.settings` while the `DJANGO_SETTINGS_MODULE` env var isn't
set. Currently, RQ workers will drop these jobs and dismiss them like
any non-valid pickle data. You won't be notified.

This patch changes RQ's behaviour to never ignore invalid string data on
any queue and _always_ handle these errors explicitly (but without
bringing the main loop down, of course).
12 years ago
examples Rewrite of the connection setup. 13 years ago
rq Don't silently fail when unpickling. 12 years ago
tests Always call job.save even on synchronous queues so get_current_job doesn't fail 12 years ago
.env.fish Add .env.fish, for fish lovers. 12 years ago
.gitignore Ignore autoenv file. 13 years ago
.travis.yml Add way of running tests unfiltered. 13 years ago
CHANGES.md Merge branch 'job-access-within-job' 12 years ago
LICENSE Fix year. 13 years ago
README.md Add comment to the README. 13 years ago
requirements.txt Revert "Remove logbook and replace it with logging". 12 years ago
run_tests Add way of running tests unfiltered. 13 years ago
setup.cfg Remove procname dependency. 12 years ago
setup.py Use `redis.from_url()` for greater compatibility. 12 years ago

README.md

Build status

RQ (Redis Queue) is a simple Python library for queueing jobs and processing them in the background with workers. It is backed by Redis and it is designed to have a low barrier to entry. It should be integrated in your web stack easily.

Getting started

First, run a Redis server, of course:

$ redis-server

To put jobs on queues, you don't have to do anything special, just define your typically lengthy or blocking function:

import requests

def count_words_at_url(url):
    """Just an example function that's called async."""
    resp = requests.get(url)
    return len(resp.text.split())

You do use the excellent requests package, don't you?

Then, create a RQ queue:

from rq import Queue, use_connection
use_connection()
q = Queue()

And enqueue the function call:

from my_module import count_words_at_url
result = q.enqueue(count_words_at_url, 'http://nvie.com')

For a more complete example, refer to the docs. But this is the essence.

The worker

To start executing enqueued function calls in the background, start a worker from your project's directory:

$ rqworker
*** Listening for work on default
Got count_words_at_url('http://nvie.com') from default
Job result = 818
*** Listening for work on default

That's about it.

Installation

Simply use the following command to install the latest released version:

pip install rq

If you want the cutting edge version (that may well be broken), use this:

pip install -e git+git@github.com:nvie/rq.git@master#egg=rq

Project history

This project has been inspired by the good parts of Celery, Resque and this snippet, and has been created as a lightweight alternative to the heaviness of Celery or other AMQP-based queueing implementations.