Vincent Driessen
4d9881eef2
Print version number when running the server.
13 years ago
Vincent Driessen
080431702d
Bump the version.
13 years ago
Vincent Driessen
1bc0c3d223
Fix bug where pickling the return value caused an uncaught exception.
13 years ago
Selwin Ong
a4f1de358f
Raise a StopException when Control+C is pressed.
13 years ago
Vincent Driessen
4057ae768b
Merge branch 'selwin-decorator'
13 years ago
Vincent Driessen
2cb058e91a
PEP8ify.
13 years ago
Vincent Driessen
d697ddb93a
Resolve connections early.
...
Fixes #101 .
13 years ago
Vincent Driessen
dbf101bc06
Update comment.
13 years ago
Vincent Driessen
a1d08a64fc
Add @wraps() call.
13 years ago
Vincent Driessen
6b79e51033
PEP8ify.
13 years ago
Vincent Driessen
ea19bdc910
Merge branch 'new-enqueue' into selwin-decorator
13 years ago
Vincent Driessen
abac4a5f41
Since we only have the timeout option, don't be too generic.
13 years ago
Selwin Ong
8c3292d35b
Make "queue" argument in job decorator required.
...
job decorator now uses Queue's "enqueue_call" method.
13 years ago
Selwin Ong
ae97f862dc
Merge branch 'master' into decorator
13 years ago
Vincent Driessen
f6374f2dfa
Add new way of invoking .enqueue(), either implicitly or explicitly.
13 years ago
Vincent Driessen
e6bb7de8c0
Get rid of the ambiguity when passing the timeout argument to .enqueue() calls.
13 years ago
Vincent Driessen
f6e67431d7
Refactor the .enqueue() API to not gobble the timeout kwargs.
...
This fixes #98 .
13 years ago
Vincent Driessen
8436d9b2c8
PEP8ify.
13 years ago
Selwin Ong
bdc1af28d1
Added a job decorator.
13 years ago
Vincent Driessen
a2b9c6ce86
Bump the version.
13 years ago
Vincent Driessen
1a178d673b
Fix bug where results wasn't correctly unpickled when refreshing from Redis.
13 years ago
Vincent Driessen
987e8fe77f
Bump version.
13 years ago
Vincent Driessen
0b66bbaab1
Merge branch 'sleekslush-master'
13 years ago
Vincent Driessen
a032896453
Add means of specifying the job function using strings.
13 years ago
Vincent Driessen
84988bdf94
Fix typo.
...
This fixes #85 .
13 years ago
Vincent Driessen
074af72536
Minor fix for empty dict check.
13 years ago
Selwin Ong
809874ee83
Don't store Job's "_instance" attribute in Redis.
13 years ago
Vincent Driessen
604fce99a1
Merge with master.
13 years ago
Selwin Ong
1fb0a1bfac
Modified ``Job`` ``save`` and ``refresh`` methods to support persisting
...
additional ``Job`` attributes into Redis.
13 years ago
Vincent Driessen
cfbbc3d0ea
Merge remote-tracking branch 'omarkhan/instance_methods'
13 years ago
Omar Khan
03bd49511d
Allow instance methods to be enqueued
...
Only works for picklable instances
13 years ago
Goran Peretin
34d161eb11
requeueing preserves job timeout
13 years ago
Goran Peretin
317a58a3b5
quarantine preserves job timeout
13 years ago
Jonas
c99cab71cd
Use a non-zero exitcode on errors in scripts
...
When a worker can't connect to Redis, return
a non-zero exit code so supervisor processes
can act accordingly.
13 years ago
Jonas
415a159ac3
Pass connection explicitly to Job.fetch
13 years ago
Vincent Driessen
074d42fe54
Remove strict procname dependency.
...
This fixes #80 .
13 years ago
Craig Slusher
2729b5435f
Uses push_connection() and pop_connection()
13 years ago
Vincent Driessen
697a4a89f8
Actually use any given default_timeout for queues.
13 years ago
Vincent Driessen
a5d9ed0a0d
Add change notes.
13 years ago
Selwin Ong
50ba003cee
Worker's "all" and "find_by_key" methods should accept "connection" as argument.
13 years ago
Vincent Driessen
c423cab770
Bump version number.
13 years ago
Vincent Driessen
7c69ad9182
Bump the version number.
13 years ago
Vincent Driessen
11f91508c6
Merge pull request #63 from selwin/master
...
Queue.lpop now takes connection as an argument. Thanks, @selwin!
13 years ago
Selwin Ong
5989228330
Queue.lpop should take connection as argument.
13 years ago
Wichert Akkerman
ec0e4776b4
Move common code out.
13 years ago
Wichert Akkerman
9f7d608da8
Renaming typo..
13 years ago
Wichert Akkerman
2d4a58631d
PyFlakes
13 years ago
Wichert Akkerman
97d2e4d784
Use console_scripts instead of scripts.
...
Unfortunately zc.buildout does not support distribute/setuptools script
option (see https://bugs.launchpad.net/zc.buildout/+bug/422724 ), but it
does support console_scripts. This also makes the scripts importable,
allowing for unittest and code reuse.
13 years ago
mattdennewitz
9f2f9e367c
Class methods now use given "cls"
13 years ago
Vincent Driessen
b886ba76de
Fix AttributeError.
13 years ago
Vincent Driessen
c337afde3a
Make the connection stack thread safe.
...
For this, I've included Werkzeug's context locals, which offers both
thread and greenlet safe local variables.
This fixes #47 .
13 years ago
Vincent Driessen
25d3f94c9f
Use the true Python 2.7 stdlib implementation instead.
13 years ago
Vincent Driessen
4885458dde
Move this hack to a separate file.
13 years ago
aiko1895
d26db69cbd
added backport of total_ordering to work with 2.6
13 years ago
Vincent Driessen
100d23b72e
Tag the first public release.
13 years ago
Vincent Driessen
a7c229ddb1
Reflect new API in the RQ scripts.
13 years ago
Vincent Driessen
2982486448
New connection management.
...
Connections can now be set explicitly on Queues, Workers, and Jobs.
Jobs that are implicitly created by Queue or Worker API calls now
inherit the connection of their creator's.
For all RQ object instances that are created now holds that the
"current" connection is used if none is passed in explicitly. The
"current" connection is thus hold on to at creation time and won't be
changed for the lifetime of the object.
Effectively, this means that, given a default Redis connection, say you
create a queue Q1, then push another Redis connection onto the
connection stack, then create Q2. In that case, Q1 means a queue on the
first connection and Q2 on the second connection.
This is way more clear than it used to be.
Also, I've removed the `use_redis()` call, which was named ugly.
Instead, some new alternatives for connection management now exist.
You can push/pop connections now:
>>> my_conn = Redis()
>>> push_connection(my_conn)
>>> q = Queue()
>>> q.connection == my_conn
True
>>> pop_connection() == my_conn
Also, you can stack them syntactically:
>>> conn1 = Redis()
>>> conn2 = Redis('example.org', 1234)
>>> with Connection(conn1):
... q = Queue()
... with Connection(conn2):
... q2 = Queue()
... q3 = Queue()
>>> q.connection == conn1
True
>>> q2.connection == conn2
True
>>> q3.connection == conn1
True
Or, if you only require a single connection to Redis (for most uses):
>>> use_connection(Redis())
13 years ago
Vincent Driessen
6bc173f122
Bugfix.
13 years ago
Vincent Driessen
28216cf63a
Implement the requeue_job() top-level API function.
...
This completely fixes #37 .
13 years ago
Vincent Driessen
c684949045
Don't expose the FailedQueue class at the top-level.
...
This fixes #36 .
13 years ago
Vincent Driessen
98ea29b15a
Don't expose the Job class at the top-level.
...
This partially fixes #37 .
13 years ago
Vincent Driessen
0b837b4b12
Don't warn. In fact, we rely on this the more.
13 years ago
Vincent Driessen
15342f14d3
Store pickled function calls as strings.
...
This aids unpacking in the case of a function that isn't importable from
the worker's runtime. The unpickling will now (almost) always succeed,
and throw an ImportError later on, when the function is actually
accessed (thus imported implicitly).
The end result is a job on the failed queue, with exc_info describing
the import error, which is tremendously useful.
13 years ago
Vincent Driessen
14ecb8e956
Add specifics on the "ImportError" to the message in case of an attribute error.
13 years ago
Vincent Driessen
844c5ed8c7
Add @slow wrapper to avoid running slow tests.
...
Use ./run_tests -f to only run the fast tests.
13 years ago
Vincent Driessen
000849c430
Initialize jobs with timeouts.
13 years ago
Vincent Driessen
b8305a818f
Safer, and shorter, version of the death penalty.
...
This case protects against JobTimeoutExceptions being raised immediately
after the job body has been (successfully) executed. Still,
JobTimeoutExceptions pass through naturally, like any other exception,
to be handled by the default exception handler that writes failed jobs
to the failed queue.
Timeouts therefore are reported like any other exception.
13 years ago
Vincent Driessen
8a856e79ea
Initial attempt at job timeouts.
13 years ago
Vincent Driessen
fb587297f6
Requeue should not error when called on a deleted job ID.
13 years ago
Vincent Driessen
7ef3b5ade8
Cleanup job hashes for jobs without result, too.
13 years ago
Vincent Driessen
5717a0ba15
Rename Job.for_call() -> Job.create().
...
This fixes #34 .
13 years ago
Vincent Driessen
7e0b843d06
Implement requeue() method on FailedQueue.
13 years ago
Vincent Driessen
8e85c7eee3
Put Job directly in the top-level `rq` module.
13 years ago
Vincent Driessen
240d2d941d
Extracted method.
...
This makes the act of moving failed jobs to the failed queue
responsibility of the FailedQueue itself, not of the Worker.
This fixes #32 .
13 years ago
Vincent Driessen
d64ad225eb
Make FailedQueue a full subclass of Queue.
...
We will add special methods on it in the future.
This fixes #33 .
13 years ago
Vincent Driessen
6f05e03293
Clean up some of the dummy jobs used for testing.
...
Also, add a random_failure test.
13 years ago
Vincent Driessen
06ce9622ea
Add compact() method on Queues, to remove dead messages.
13 years ago
Vincent Driessen
70cbf00890
Refactor out an exists() and a for_key() method.
13 years ago
Vincent Driessen
c49e564d3c
Don't break the API when jobs get deleted.
13 years ago
Vincent Driessen
bd08f24f15
Cosmetic changes to the command line output.
13 years ago
Vincent Driessen
e4055ca42f
Allow queues to be emptied.
13 years ago
Vincent Driessen
9e33e1a08f
Implement a cancel() method on jobs.
...
This fixes #29 .
13 years ago
Vincent Driessen
0a0d9d1ceb
Flake8 style fixes.
13 years ago
Vincent Driessen
1a8b80604d
Minor refactoring to make the to-failed queue code a bit more readable.
13 years ago
Vincent Driessen
11c7dbb376
Consistently renamed "failure" -> "failed" queue.
...
Fixes #28 .
13 years ago
Vincent Driessen
9f5b1545b6
Fix: store the job result in the correct key.
...
And expire job hash in Redis after 500 seconds (by default).
Fixes #27 .
13 years ago
Vincent Driessen
7bda1ca969
Encapsulate the result property (it should not directly be set).
13 years ago
Vincent Driessen
6266f68310
Style fixes (flake8).
13 years ago
Vincent Driessen
8da204f74a
Always use cPickle, never 'regular' pickle.
...
This fixes #18 .
13 years ago
Vincent Driessen
9318825429
Abstract away from the concrete pickle implementation.
...
Choose cPickle, if available, for best performance.
13 years ago
Vincent Driessen
90a458ca8e
Add more colorful terminal output.
...
For better visual parsability.
13 years ago
Vincent Driessen
63ef198fd6
Improve work generator.
13 years ago
Vincent Driessen
791f8169f5
Remove dead code.
13 years ago
Vincent Driessen
e05acfedce
Fix putting jobs on the failure queue when they fail.
13 years ago
Vincent Driessen
7c903e45ef
Simplify the persistence of jobs.
...
Fixes #23 and #24 .
13 years ago
Vincent Driessen
bffe6cbbde
Encapsulate internal function call representation.
...
This means it's not allowed anymore to directly set func, args, and
kwargs. Instead, use the for_call() constructor.
13 years ago
Vincent Driessen
370399f8f7
CHECKPOINT: dequeue_any now returns the queue that was popped from.
13 years ago
Vincent Driessen
f516f8df2e
CHECKPOINT: Handle failing and unreadable jobs.
...
Failing (or unreadable) jobs are correctly put on the failure queue by
the worker now.
13 years ago
Vincent Driessen
b1650cb9b9
CHECKPOINT: Second part of the big refactoring.
...
Jobs are now stored in separate keys, and only job IDs are put on Redis
queues. Much of the code has been hit by this change, but it is for the
good.
No really.
13 years ago
Vincent Driessen
65105b44c3
CHECKPOINT: Initial part of the big refactor.
13 years ago
Vincent Driessen
fcca48a9d7
Rename empty property -> is_empty() method.
13 years ago
Vincent Driessen
4d2f64d4b6
Mental note to self.
13 years ago
Vincent Driessen
20e908039f
Initially set the exc_info property, so it does not raise an
...
AttributeException when accessed.
13 years ago
Vincent Driessen
fdce187c27
Putting failed jobs on the failure queue.
13 years ago
Vincent Driessen
7eb8d92605
Put unreadable tasks on the failure queue.
13 years ago
Vincent Driessen
0be1cb6ac0
Change the way jobs are pickled.
...
There is no job tuple anymore, but instead Jobs are picklable by
themselves natively. Furthermore, I've added a way to annotate Jobs
with created_at and enqueued_at timestamps, to drive any future Job
performance stats. (And to enable requeueing, while keeping hold of the
queue that the Job originated from.)
This fixes #17 .
13 years ago
Vincent Driessen
0503eb2829
Clarified responsibility of the Job class.
...
The Job itself has nothing to do with queueing and dequeueing, so the
DequeueError wasn't appropriate here, either.
13 years ago
Vincent Driessen
db5753b0d6
Put Job in its own file.
13 years ago
Vincent Driessen
210477c2ab
Throw DequeueError when reading unprocessable data from queue.
13 years ago
Vincent Driessen
62ae299114
Fix mathematical meaning of the < operator.
...
This is used for alphabetical queue ordering.
13 years ago
Vincent Driessen
5c6f002878
Silently pass when trying to kill child that is already dead.
...
This fixes #16 .
13 years ago
Vincent Driessen
039a132374
Add ellipsis, to indicate we're waiting here.
13 years ago
Vincent Driessen
66775b7012
Put the version number somewhere central.
13 years ago
Vincent Driessen
aecb0a1bf0
Simplify calling .work() or .work(burst=True).
13 years ago
Vincent Driessen
636b6690d6
Add the signal name to the debug message.
13 years ago
Vincent Driessen
a154ef0bd9
Remove comment.
...
This ain't the right way to terminate when blocking by pop.
13 years ago
Vincent Driessen
62949c9adb
Extra debug output.
13 years ago
Vincent Driessen
dde3ea8ef7
Take down horse process when the worker is terminated.
13 years ago
Vincent Driessen
4ac243b3e8
Print what signal was received in a debug statement.
13 years ago
Vincent Driessen
7769d9875f
Perform a warm shutdown on SIGTERM, too.
...
Just like with Ctrl+C (SIGINT), shutdown warmly at first when kill'ed
(SIGTERM).
13 years ago
Vincent Driessen
88cbaa1df9
Slight code reshuffle + added some comments on the construction.
13 years ago
Vincent Driessen
7cba8449d9
Add comments.
13 years ago
Vincent Driessen
1cbf92c166
Workaround for os.waitpid() throwing an OSError on SIGINT.
...
When SIGINT (``Ctrl+C``) is received when inside a blocking
os.waitpid(), OSError is thrown, effectively cancelling the wait.
However, to facilitate a "warm shutdown", as we intend, Ctrl+C is
perfectly allowed and we want to keep waiting for the child. Therefore,
we perform a trick here, catching OSError, checking whether its cause
was SIGINT (errno == EINTR), and only in that case, loop to os.waitpid()
again.
13 years ago
Vincent Driessen
e278bd2967
Exit gracefully when user hits Ctrl+C in a worker.
...
The currently running task will be waited for, so it can gracefully
be finished. Further execution will be stopped.
If, during this waiting phase, Ctrl+C is hit again, the worker and the
horse will be terminated forcefully (this means work could be lost or
partially finished).
13 years ago
Vincent Driessen
ba965a1dd9
Minor text change
13 years ago
Vincent Driessen
2d2b351f7c
Change logging format.
13 years ago
Vincent Driessen
8678f26df0
Factor out call string.
13 years ago
Vincent Driessen
2ec12f1775
Fix bugje.
13 years ago
Vincent Driessen
1358246238
Better logging output.
13 years ago
Vincent Driessen
1dd73cb28d
Fix comment.
13 years ago
Vincent Driessen
cc8f05b3c6
Add __str__ to Job.
13 years ago
Vincent Driessen
55fd393626
Worker.find_by_key method now returns None for nonexisting workers.
13 years ago
Vincent Driessen
507558f6bc
Avoid forked work horse to register death of its parent worker.
13 years ago
Vincent Driessen
bd1778c610
Fix comment typo's.
13 years ago
Vincent Driessen
4ad6281641
Rewrite the ANSI color formatter a bit.
13 years ago
Vincent Driessen
edcc012b2d
Document the Queue class.
13 years ago
Vincent Driessen
45700fc788
Delete unused file.
13 years ago
Vincent Driessen
c9ba66bd59
Register workers in a central set ("rq:workers").
13 years ago
Vincent Driessen
1f12678468
Get Worker fetch methods.
13 years ago
Vincent Driessen
9e8a4d15be
Document methods.
13 years ago
Vincent Driessen
6013227f4c
Remove unused property.
13 years ago
Vincent Driessen
aebfe74630
Add polling/scaling graphs.
13 years ago
Vincent Driessen
a905961f94
Add some sample scripts.
13 years ago
Vincent Driessen
a029e5437b
Add beginnings of a rqworker script.
13 years ago
Vincent Driessen
d780c929c0
Change semantics of work(). Add work_burst().
...
work() will now start the worker run loop, and work_burst() now leads to
the burst-then-quit behaviour.
13 years ago
Vincent Driessen
a491ff248d
Add simple helper function, to avoid users to learn about the `conn`
...
stack.
This makes it easier to start using the product.
13 years ago
Vincent Driessen
0ff2984adf
Use __slots__, for minimal memory overhead.
13 years ago
Vincent Driessen
56c4445bb2
Shut up pyflakes.
13 years ago
Vincent Driessen
b5ea5f32fe
Variable renamed.
13 years ago
Vincent Driessen
8852859fc4
Improved the __repr__ and __str__ of Queues.
13 years ago
Vincent Driessen
b4c1c85276
Add equality and comparison methods.
13 years ago
Vincent Driessen
a5ea45af57
Make the dequeue methods return values consistent.
...
I merely refactored the internal calls. No external API changes have been made in this commit. In order to make the dequeueing methods consistent, each dequeue method now returns a Job instance, which is just a nice lightweight wrapper around the job tuple.
The Job class makes it easier to pass the method call info around, along with some possible meta information, like the queue the job originated from.
This fixes #7 .
13 years ago
Vincent Driessen
f492a5ae2b
Restructure some code.
...
No functional change, but leave the BLPOP'ing to the Queue, as the
queues know how to pop themselves.
13 years ago
Vincent Driessen
1a893e60cf
Have work() return whether work has been done, or not.
...
And promote Worker to the rq namespace, so you can
from rq import Worker
13 years ago
Vincent Driessen
1c9fa66bc1
Greatly simplify the setup.
...
Jobs don't even need to be tagged. Any function can be put on queues.
13 years ago
Vincent Driessen
62b6b180f3
Also allow args and kwargs to enqueue() calls on Queue.
13 years ago
Vincent Driessen
196a9815d2
Add dequeue method to Queue.
13 years ago
Vincent Driessen
b5e0377eb0
Allow jobs to be declared without a default queue.
...
This makes it impossible to use delay() on them. Instead, you should
use enqueue().
13 years ago
Vincent Driessen
407f3e8b38
Add ability to put work on alternate queues.
13 years ago
Vincent Driessen
1b8da4a861
Add test for putting work on queues.
13 years ago
Vincent Driessen
8dfdd452ef
Bugfix.
...
Yeah, it's getting late.
It's my own fault.
I know.
13 years ago
Vincent Driessen
04c88577ed
Bugfix: LPOP does not support multiple queue arguments.
...
Redis' BLPOP command takes multiple queue arguments, but LPOP can only
take a single queue. Therefore, we need to loop over all queues
manually, in order, and raise an exception is no more work is available.
13 years ago
Vincent Driessen
a77c3d9104
Support quitting when all work is done (i.e. queue is empty).
13 years ago
Vincent Driessen
8dc2b15a6e
Remove empty file.
...
It is obsoleted by rq/proxy.py
13 years ago
Vincent Driessen
98ffcd8e05
Create soft dependency on logbook.
13 years ago
Vincent Driessen
227e107a82
Oops, fix some old references to current_connection.
13 years ago
Vincent Driessen
d721f0708b
Refactor the whole Redis connection stuff to be just as easy as in RDB.
13 years ago
Vincent Driessen
518db8c24b
Add better connection management.
...
To start using RQ, push a Redis connection up its stack, like so:
from rq import push_connection
push_connection(Redis())
13 years ago
Vincent Driessen
d8d388c841
Log the results of jobs.
13 years ago
Vincent Driessen
f21b2af2b6
Make it an actual PyPI-managable Python package.
13 years ago