util
/
rq
mirror of https://github.com/peter4431/rq.git
3
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
1,700 Commits
2 Branches
0 Tags
3.3 MiB
main
master
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '83fa0adf15'
${ noResults }
Commit Graph

271 Commits (83fa0adf15daf718fac416214529072ac297293d)

Author SHA1 Message Date
Dustin Falgout 91a3aa7493 make args and kwargs explicit for enqueue_call function 
closes #466
 10 years ago
Selwin Ong edd139d86f Sync jobs should be cleaned up after execution. 10 years ago
Vincent Driessen 94258761ae Make string formatting consistent 10 years ago
RyanMTB 9a00b0eca6 Updated Worker API 10 years ago
Selwin Ong cff6fbff41 Merge pull request #477 from glaslos/ttl_tests_fixes 
TTL tests and fixes
 10 years ago
Tamir Duberstein 985a2664a4 Prevent `Queue#dequeue` from blowing the stack 
In the case of many sequential jobs having been deleted, a recursive
implementation of `Queue#dequeue` is prone to blowing the stack in the
absence of tail-recursion support. Change the implementation from
recursive to iterative to work around this issue in CPython.
 10 years ago
Vlad Pronsky 933e6697cd Fixed redis drivers bug 10 years ago
glaslos 24d5e08992 Merge branch 'master' into ttl_tests_fixes 
Conflicts:
	tests/test_job.py
 11 years ago
Taras Semenenko 3dfd044767 Add missed `multi` command after `watch` 
`watch` command should be used in conjunction with `multi` command
which was missed in enqueuing of job with dependencies.
Fix #487
 11 years ago
glaslos c155918d6d lrem only first occurrence of the job id (we expect only one) 11 years ago
Vincent Driessen b8d425b318 Various PEP8 fixes. 11 years ago
Vincent Driessen fac2b10309 Auto-sort imports using isort. 11 years ago
glaslos 703ab0e355 removed merged comment 11 years ago
glaslos 646f1cda22 Merge branch 'master' into ttl_tests_fixes 
Conflicts:
	rq/queue.py
 11 years ago
glaslos 5b5ab4860f passing on the ttl 11 years ago
Selwin Ong dac0be6cc7 Added DeferredJobsRegistry to keep track of deferred jobs. 11 years ago
Selwin Ong 3e674fbe6a queue.enqueue() should set job.origin. 11 years ago
Selwin Ong 9320496402 Simplify FailedQueue.quarantine and ensure that a deferred job's status is set to Queued when enqueued. 11 years ago
Selwin Ong 7fd2ac8ca6 Added "DEFERRED" Job status for jobs that have unsatisfied dependencies. 11 years ago
Selwin Ong 0a0a81da6d Merge pull request #463 from conslo/skip-queue 
Skip queue
 11 years ago
Jonathan Tushman 1ae5a12a81 merging master 11 years ago
Travis Johnson add4d625b6 Merge branch 'nvie' into skip-queue 
Conflicts:
	rq/queue.py
 11 years ago
Jonathan Tushman 82333d2ad5 triggering shutdown by setting a redis flag 11 years ago
Travis Johnson db75958ad2 use 'at_front' instead of 'skip_queue' 11 years ago
Selwin Ong e1801c5c44 Merge pull request #453 from glaslos/job_ttl 
TTL for jobs
 11 years ago
Travis Johnson ac61f502a1 skip_queue functionality 11 years ago
Travis Johnson f60e4884df a comment typo 11 years ago
glaslos 8a3fd91e2e renaming job.job_ttl to job.ttl 11 years ago
glaslos 55c541bc59 added job ttl to queue.enqueue() 11 years ago
Travis Johnson af4b0436c1 instantiate a job instead of fetch 
this removes the need for the depended on job.id already being saved
 11 years ago
Travis Johnson 14d1186241 use internal job_class for check 11 years ago
Travis Johnson 3467868f1f allow depends_on to be a job id or a job itself 
Other parts of the code (i.e.: the `.create()` method) allow the `depends_on` kwarg to be a `Job` object *or* a job id. This is an attempt to allow that same idea within the `.enqueue_call()` method for a queue. Since this part of the code requires actually knowing the precise redis key for the job that's depended on, my intuition is that a `.fetch()` is the only way to solve this.
 11 years ago
Selwin Ong 18c6ea64ef Merge pull request #419 from selwin/registry-len 
Added JobStartedRegistry.count
 11 years ago
Selwin Ong 202be75b21 Merge pull request #415 from foxx/feature/custom-job-ids 
Allow job ID to be set on enqueue/enqueue_call() - fixes #412
 11 years ago
Selwin Ong c1dc30eae3 Added __len__ method to Queue. 11 years ago
foxx 30ea76ddd6 Fixed silly typo 11 years ago
foxx 0466562a13 Fixes from comments in #415 
Added tox/vagrant tmp dir to gitignore
Removed unused import
Renamed id to job_id in enqueue_call() as per comments in #415
 11 years ago
Selwin Ong a28575088b Implemented WorkingQueue.cleanup(). 11 years ago
foxx f5779c194f Renamed job_id to id, and fixed py3 bug, per #415 11 years ago
Vincent Driessen fc803f5d0d Fix various flake8 complaints. 11 years ago
foxx 6bb8b26114 Allow job ID to be set on enqueue/enqueue_call() - fixes #412 11 years ago
Vincent Driessen 3de8a47f06 Merge pull request #390 from selwin/pipeline-dependency-registration 
job.register_dependency() should support pipelining
 11 years ago
Selwin Ong fc19a64670 job.register_dependency() should support pipelining. 11 years ago
Vincent Driessen 12932a433b Merge pull request #382 from selwin/cancel-should-remove-from-queue 
job.cancel() should remove itself from queue
 11 years ago
Selwin Ong 638211df20 job.cancel() should remove itself from queue. 11 years ago
Selwin Ong 141278bb42 Queue.__init__ supports job_class argument. 11 years ago
Vincent Driessen 83382f51d0 PEP8ify. 11 years ago
Vincent Driessen b048e23875 Merge pull request #342 from friedcell/master 
make job class dynamic
 11 years ago
Vincent Driessen ab9e6b852e Fix PEP8 complaints. 11 years ago
Vincent Driessen 38ec259b6e Enable the most modern Python syntax. 11 years ago
Marko Mrdjenovic 5cba260cca Job and Queue classes as variables (compatible with existing naming) 11 years ago
Marko Mrdjenovic 67dd1cbf34 make job class dynamic for more hackability 11 years ago
Vincent Driessen aef7af9c77 Merge pull request #334 from nvie/emptying-failed-queue 
Emptying failed queue
 11 years ago
Vincent Driessen 4d9c20d5d9 Reimplement Queue.empty() in a Lua script. 
This makes the .empty() function perform all the computing in Redis
itself, rather than in Python.  This is both atomic, and faster.
 11 years ago
Selwin Ong 4f918041e3 Rearranged how explicit kwargs are passed into queue.enqueue(). Fixes #322 11 years ago
Selwin Ong f5c3c9a6dd Deprecate job.status. 11 years ago
Malthe Borch ba0b39a43b Unpickle job data lazily. This fixes issue #294. 12 years ago
Vincent Driessen 888d771d4d Merge pull request #287 from malthe/shed-times-dependency 
Remove dependency on 'times' library (issue #286).
 12 years ago
Andy Yao 31830120f8 delte Unused variable 'prefix' 12 years ago
Malthe Borch c5a381fbe9 Remove dependency on 'times' library (see issue #286). 
Basically, for the functionality needed, a dependency on 'times' (which
in turn depends on 'python-dateutil') seem unnecessary.
 12 years ago
Joshua Chia ef2bbe762b Minor cleanup: Use Status enums instead of strings 12 years ago
Vincent Driessen 7fdd115e28 Merge branch 'jchia-master' 
Conflicts:
	rq/job.py
	rq/queue.py
 12 years ago
Vincent Driessen bb3dc5b0b2 Terminology change: waitlist -> dependents. 12 years ago
Joshua Chia b89d6c8990 Set timeout at Job creation instead of during enqueueing; made Queue.enqueue_call respect `timeout` when `after` is specified 12 years ago
Selwin Ong 15c4b562ba Reverse dependency mapping should be a set instead of a list. 12 years ago
Selwin Ong 93e5e552b7 Replaced "after" kwarg with "depends_on". 12 years ago
Mark Tozzi 5f6a1f0ac5 Empty the queue before deleteing the jobs, in case new jobs arrive 12 years ago
Mark Tozzi 0713ad03c2 Empty now deletes all jobs in the queue 12 years ago
Rob Harrigan 02c5d902ec Merge remote-tracking branch 'upstream/master' 
Conflicts:
	tests/test_queue.py
 12 years ago
Rob Harrigan 5e4d5f642e Only save queue to set on enqueue. 
Validate queue key still exists when all() is called
 12 years ago
Selwin Ong fd44ad39d4 Python 3 fixes for job dependency stuff. 12 years ago
Selwin Ong fcfe55fe13 Merge branch 'master' into job_dependency 
Conflicts:
	rq/job.py
	rq/queue.py
 12 years ago
Vincent Driessen 1b558704d3 Fix: COMPACT_QUEUE should be a unique key. 
This fixes #230.  Thanks, @sylvinus.
 12 years ago
Rob Harrigan c08b96e027 Use a redis set to store queue names, instead of relying on the redis "keys" method, 
which does not perform well on large databases.
 12 years ago
Vincent Driessen 90fcb6c9d0 PEP8ify. 12 years ago
Vincent Driessen 57ea6203d1 Let's not change positional argument order. 
This may break other people's programs.
 12 years ago
H. İbrahim Güngör 10bda9684d Pass description parameter to job constructor in order to distinguish job names in queue.jobs or in rq-dashboard. Add related test case. 12 years ago
Alex Morega 8d61d3bf26 port string handling to py3 
Redis uses byte values for everything. We save queue names and job
IDs as unicode. So we need to convert every time we get data from redis.
 12 years ago
Alex Morega a3b5ce5e46 accomodate py3 imports and builtins 12 years ago
Onilton Maciel 3afc32f08a End calculation in get_jobs_ids in fixed. Length is respected 12 years ago
Onilton Maciel 97de8ea3cc Fixed typos and errors found. Tests passing now 12 years ago
oniltonmaciel 5cfbae61a9 Replaced limit by length and start by offset 
Replaced limit by length and start by offset to remove a possible ambiguity
 12 years ago
Selwin Ong 6ee45597ca Don't fail if job dependency is modified during enqueue process. 12 years ago
Selwin Ong e7e8579888 Merge branch 'master' of git://github.com/nvie/rq into job_dependency 
Conflicts:
	rq/queue.py
 12 years ago
Selwin Ong 2e826e2b1f Internally renamed the term "parent" to "dependency". 12 years ago
Selwin Ong 0dfb041383 Simplify enqueue_waitlist by using lpop. 12 years ago
Selwin Ong 18ff57ef35 Avoid race conditions when enqueueing job with dependency. 12 years ago
Vincent Driessen 73b7453e40 Fix typo. 12 years ago
Vincent Driessen 2fb6e5ca1a Minor refactoring of the paging logic. 12 years ago
Alex Morega e3075ea6be get a page of jobs 12 years ago
Alex Morega ef0f04bff6 extract `safe_fetch_job` method 12 years ago
Selwin Ong 6550f86646 Don't enqueue waitlisted jobs on failed execution. 12 years ago
Selwin Ong eadc7db29f First stab at writing implementing job dependency. 12 years ago
Selwin Ong c987569650 Safe fetching a deleted job removes the deleted job from queue. 13 years ago
Vincent Driessen d39badb4cc Prevent the use of indefinite timeouts. 
Using them would really mess with the new expiring worker keys (they
would disappear, even though the workers aren't dead).
 13 years ago
Vincent Driessen e4d37332f0 Fix typos. 13 years ago
Vincent Driessen 640195d5e4 Merge branch 'master' into yaniv-aknin-worker_ttl 13 years ago
Vincent Driessen c0a594d47a Merge pull request #177 from selwin/master 
FailedQueue.requeue should set job status back to Queued
 13 years ago
Selwin Ong e5fa82aa5d Merge branch 'master' of github.com:selwin/rq 13 years ago
Selwin Ong 05d744c9af Requeuing a job should set its status back to Queued. 13 years ago
Vincent Driessen 4688498e2d Attach job ID when unpickling fails in .dequeue(). 
This makes the behaviour consistent with .dequeue_any().
 13 years ago
Vincent Driessen 54254f2271 Patch the connection instances. 
This patches the connection object (which is either a StrictRedis
instance or a Redis instance), to have alternative class methods that
behave exactly like their StrictRedis counterparts, no matter whether
which type the object is.  Only the ambiguous methods are patched.  The
exhaustive list:

- _zadd          (fixes argument order)
- _lrem          (fixes argument order)
- _setex         (fixes argument order)
- _pipeline      (always returns a StrictPipeline)
- _ttl           (fixes return value)
- _pttl          (fixes return value)

This makes it possible to call the methods reliably without polluting
the RQ code any further.
 13 years ago
Yaniv Aknin 74c2351232 Refactor dequeue_any to accept arbitrary timeouts 
The 'blocking' parameter was replaced with a 'timeout' parameter.

The timeout parameter is interpreted thus:
    0 - no timeout (block forever, equivalent to blocking=True)
    None - non-blocking (return value or None immediately, equivalent to
                         blocking=False)
    <integer> - maximum seconds to block

Upon timing out, a dequeue operation will raise DequeueTimeout.
 13 years ago
Vincent Driessen 90b15fd0b6 Don't silently fail when unpickling. 
When a pickled job string can't be unpickled because some required
module isn't loadable, this leads to an `UnpickleError` in the worker
(not in the horse).

Currently we just assume "garbage" in the job's data field, and silently
ignore it.

This is bad.

Really bad.

Because it avoids the normal exception handling mechanism that RQ has.

Historically, this "feature" was introduced to ignore any invalid pickle
data ("bad strings") on queues, and go on. However, we must assume data
inside `job.data` to be valid pickle data.

While an invalid _format_ of pickle data (e.g. the string "blablah"
isn't valid) leads to unpickle errors, unpickling errors will also occur
when the job can't be validly constructed in memory for other reasons,
like being unable to load a specific class.

Django is a good example of this: try submitting jobs that use
`django.conf.settings` while the `DJANGO_SETTINGS_MODULE` env var isn't
set. Currently, RQ workers will drop these jobs and dismiss them like
any non-valid pickle data. You won't be notified.

This patch changes RQ's behaviour to never ignore invalid string data on
any queue and _always_ handle these errors explicitly (but without
bringing the main loop down, of course).
 13 years ago
Selwin Ong f498de57b6 Always call job.save even on synchronous queues so get_current_job doesn't fail 13 years ago
Vincent Driessen 8581cd6463 Don't allow jobs without Redis connection. 
And other connection-related fixes.
 13 years ago
Vincent Driessen 4224304291 I like this implementation of an 'enum' better. 13 years ago
Selwin Ong 442b389b97 Job returning None as result are now persisted correctly. 
Job status can now be checked via ``status`` property which should
return either "queued", "finished" or "failed".
 13 years ago
Selwin Ong 06d75630e0 Execute job immediately if Queue(async=False) 13 years ago
Selwin Ong a5e6765990 Added "result_ttl" property on jobs that determines how long job results are persisted in Redis. 13 years ago
Vincent Driessen d697ddb93a Resolve connections early. 
Fixes #101.
 13 years ago
Vincent Driessen abac4a5f41 Since we only have the timeout option, don't be too generic. 13 years ago
Vincent Driessen f6374f2dfa Add new way of invoking .enqueue(), either implicitly or explicitly. 13 years ago
Vincent Driessen e6bb7de8c0 Get rid of the ambiguity when passing the timeout argument to .enqueue() calls. 13 years ago
Vincent Driessen f6e67431d7 Refactor the .enqueue() API to not gobble the timeout kwargs. 
This fixes #98.
 13 years ago
Vincent Driessen a032896453 Add means of specifying the job function using strings. 13 years ago
Goran Peretin 34d161eb11 requeueing preserves job timeout 13 years ago
Goran Peretin 317a58a3b5 quarantine preserves job timeout 13 years ago
Jonas 415a159ac3 Pass connection explicitly to Job.fetch 13 years ago
Vincent Driessen 697a4a89f8 Actually use any given default_timeout for queues. 13 years ago
Selwin Ong 5989228330 Queue.lpop should take connection as argument. 13 years ago
mattdennewitz 9f2f9e367c Class methods now use given "cls" 13 years ago
Vincent Driessen 4885458dde Move this hack to a separate file. 13 years ago
aiko1895 d26db69cbd added backport of total_ordering to work with 2.6 13 years ago
Vincent Driessen 2982486448 New connection management. 
Connections can now be set explicitly on Queues, Workers, and Jobs.
Jobs that are implicitly created by Queue or Worker API calls now
inherit the connection of their creator's.

For all RQ object instances that are created now holds that the
"current" connection is used if none is passed in explicitly.  The
"current" connection is thus hold on to at creation time and won't be
changed for the lifetime of the object.

Effectively, this means that, given a default Redis connection, say you
create a queue Q1, then push another Redis connection onto the
connection stack, then create Q2. In that case, Q1 means a queue on the
first connection and Q2 on the second connection.

This is way more clear than it used to be.

Also, I've removed the `use_redis()` call, which was named ugly.
Instead, some new alternatives for connection management now exist.

You can push/pop connections now:

    >>> my_conn = Redis()
    >>> push_connection(my_conn)
    >>> q = Queue()
    >>> q.connection == my_conn
    True
    >>> pop_connection() == my_conn

Also, you can stack them syntactically:

    >>> conn1 = Redis()
    >>> conn2 = Redis('example.org', 1234)
    >>> with Connection(conn1):
    ...     q = Queue()
    ...     with Connection(conn2):
    ...         q2 = Queue()
    ...     q3 = Queue()
    >>> q.connection == conn1
    True
    >>> q2.connection == conn2
    True
    >>> q3.connection == conn1
    True

Or, if you only require a single connection to Redis (for most uses):

    >>> use_connection(Redis())
 13 years ago
Vincent Driessen 6bc173f122 Bugfix. 13 years ago
Vincent Driessen c684949045 Don't expose the FailedQueue class at the top-level. 
This fixes #36.
 13 years ago
Vincent Driessen 8a856e79ea Initial attempt at job timeouts. 14 years ago
Vincent Driessen fb587297f6 Requeue should not error when called on a deleted job ID. 14 years ago
Vincent Driessen 5717a0ba15 Rename Job.for_call() -> Job.create(). 
This fixes #34.
 14 years ago
Vincent Driessen 7e0b843d06 Implement requeue() method on FailedQueue. 14 years ago
Vincent Driessen 240d2d941d Extracted method. 
This makes the act of moving failed jobs to the failed queue
responsibility of the FailedQueue itself, not of the Worker.

This fixes #32.
 14 years ago
Vincent Driessen d64ad225eb Make FailedQueue a full subclass of Queue. 
We will add special methods on it in the future.

This fixes #33.
 14 years ago
Vincent Driessen 06ce9622ea Add compact() method on Queues, to remove dead messages. 14 years ago
Vincent Driessen c49e564d3c Don't break the API when jobs get deleted. 14 years ago
Vincent Driessen e4055ca42f Allow queues to be emptied. 14 years ago
Vincent Driessen 0a0d9d1ceb Flake8 style fixes. 14 years ago
Vincent Driessen 1a8b80604d Minor refactoring to make the to-failed queue code a bit more readable. 14 years ago
Vincent Driessen e05acfedce Fix putting jobs on the failure queue when they fail. 14 years ago
Vincent Driessen bffe6cbbde Encapsulate internal function call representation. 
This means it's not allowed anymore to directly set func, args, and
kwargs.  Instead, use the for_call() constructor.
 14 years ago
Vincent Driessen 370399f8f7 CHECKPOINT: dequeue_any now returns the queue that was popped from. 14 years ago
Vincent Driessen f516f8df2e CHECKPOINT: Handle failing and unreadable jobs. 
Failing (or unreadable) jobs are correctly put on the failure queue by
the worker now.
 14 years ago
Vincent Driessen b1650cb9b9 CHECKPOINT: Second part of the big refactoring. 
Jobs are now stored in separate keys, and only job IDs are put on Redis
queues.  Much of the code has been hit by this change, but it is for the
good.

No really.
 14 years ago
Vincent Driessen 65105b44c3 CHECKPOINT: Initial part of the big refactor. 14 years ago
Vincent Driessen fcca48a9d7 Rename empty property -> is_empty() method. 14 years ago
Vincent Driessen 4d2f64d4b6 Mental note to self. 14 years ago
Vincent Driessen 7eb8d92605 Put unreadable tasks on the failure queue. 14 years ago
Vincent Driessen 0be1cb6ac0 Change the way jobs are pickled. 
There is no job tuple anymore, but instead Jobs are picklable by
themselves natively.  Furthermore, I've added a way to annotate Jobs
with created_at and enqueued_at timestamps, to drive any future Job
performance stats.  (And to enable requeueing, while keeping hold of the
queue that the Job originated from.)

This fixes #17.
 14 years ago
Vincent Driessen db5753b0d6 Put Job in its own file. 14 years ago
Vincent Driessen 210477c2ab Throw DequeueError when reading unprocessable data from queue. 14 years ago