When a pickled job string can't be unpickled because some required
module isn't loadable, this leads to an `UnpickleError` in the worker
(not in the horse).
Currently we just assume "garbage" in the job's data field, and silently
ignore it.
This is bad.
Really bad.
Because it avoids the normal exception handling mechanism that RQ has.
Historically, this "feature" was introduced to ignore any invalid pickle
data ("bad strings") on queues, and go on. However, we must assume data
inside `job.data` to be valid pickle data.
While an invalid _format_ of pickle data (e.g. the string "blablah"
isn't valid) leads to unpickle errors, unpickling errors will also occur
when the job can't be validly constructed in memory for other reasons,
like being unable to load a specific class.
Django is a good example of this: try submitting jobs that use
`django.conf.settings` while the `DJANGO_SETTINGS_MODULE` env var isn't
set. Currently, RQ workers will drop these jobs and dismiss them like
any non-valid pickle data. You won't be notified.
This patch changes RQ's behaviour to never ignore invalid string data on
any queue and _always_ handle these errors explicitly (but without
bringing the main loop down, of course).
redis-py now supports URL-based connection configuration. When --url
is specified, we use it to construct the Redis object. Otherwise, we
use the existing argument-based construction method.
`Redis.from_url()` is new in redis-py 2.6.2, so that prerequisite has
been adjusted accordingly.
This is to support users that are relying on the current custom property
implementation. A warning will be displayed on the console, stating
that this support will be removed from RQ version 0.4.
rqinfo was using None for the redis host and port, which was causing
exceptions when trying to connect. It was possible to supply these on
the commandline, but port was just being passed straight through as a
string, which also caused exceptions.
This commit moves some of the argument & configuration file handling out
of rqworker and into the scripts/__init__.py file, and then calls it
from rqinfo.