* Added FailedJobRegistry.
* Added job.failure_ttl.
* queue.enqueue() now supports failure_ttl
* Added registry.get_queue().
* FailedJobRegistry.add() now assigns DEFAULT_FAILURE_TTL.
* StartedJobRegistry.cleanup() now moves expired jobs to FailedJobRegistry.
* Failed jobs are now added to FailedJobRegistry.
* Added FailedJobRegistry.requeue()
* Document the new `FailedJobRegistry` and changes in custom exception handler behavior.
* Added worker.disable_default_exception_handler.
* Document --disable-default-exception-handler option.
* Deleted worker.failed_queue.
* Deleted "move_to_failed_queue" exception handler.
* StartedJobRegistry should no longer move jobs to FailedQueue.
* Deleted requeue_job
* Fixed test error.
* Make requeue cli command work with FailedJobRegistry
* Added .pytest_cache to gitignore.
* Custom exception handlers are no longer run in reverse
* Restored requeue_job function
* Removed get_failed_queue
* Deleted FailedQueue
* Updated changelog.
* Document `failure_ttl`
* Updated docs.
* Remove job.status
* Fixed typo in test_registry.py
* Replaced _pipeline() with pipeline()
* FailedJobRegistry no longer fails on redis-py>=3
* Fixes test_clean_registries
* Worker names are now randomized
* Added a note about random worker names in CHANGES.md
* Worker will now stop working when encountering an unhandled exception.
* Worker should reraise SystemExit on cold shutdowns
* Added anchor.js to docs
* Support for Sentry-SDK (#1045)
* Updated RQ to support sentry-sdk
* Document Sentry integration
* Install sentry-sdk before running tests
* Improved rq info CLI command to be more efficient when displaying lar… (#1046)
* Improved rq info CLI command to be more efficient when displaying large number of workers
* Fixed an rq info --by-queue bug
* Fixed worker.total_working_time bug (#1047)
* queue.enqueue() no longer accepts `timeout` argument (#1055)
* Clean worker registry (#1056)
* queue.enqueue() no longer accepts `timeout` argument
* Added clean_worker_registry()
* Show worker hostname and PID on cli (#1058)
* Show worker hostname and PID on cli
* Improve test coverage
* Remove Redis version check when SSL is used
* Bump version to 1.0
* Removed pytest_cache/README.md
* Changed worker logging to use exc_info=True
* Removed unused queue.dequeue()
* Fixed typo in CHANGES.md
* setup_loghandlers() should always call logger.setLevel() if specified
* modify zadd calls for redis-py 3.0
redis-py 3.0 changes the zadd interface that accepts a single
mapping argument that is expected to be a dict.
https://github.com/andymccurdy/redis-py#mset-msetnx-and-zadd
* change FailedQueue.push_job_id to always push a str
redis-py 3.0 does not attempt to cast values to str and is left
to the user.
* remove Redis connection patching
Since in redis-py 3.0, Redis == StrictRedis class, we no longer
need to patch _zadd and other methods.
Ref: https://github.com/rq/rq/pull/1016#issuecomment-441010847
* Attempt to fix HerokuWorker test.
* Fixed test_shutdown_double_sigrtmin
* Another attempt to fix test
* Another attempt to fix test
* Another attempt to fix test
* When work horse runs into an exception, raise that exception before calling os._exit()
* Replaced async keyword with is_async in the Queue class to fix reserved keyword syntax errors in Python 3.7
* Updated tests to use is_async keyword when instantiating Queue objects
* Updated docs to reference is_async keyword for Queue objects
* Updated tox.ini, setup.py and .travis.yml with references to Python 3.7
* Define redis key prefix as class variable
Some prefixes were hardcoded in several places. This made it hard to
use custom prefixes via subclasses.
Resolves#920
* fixup! Define redis key prefix as class variable
* Initial take on delete_dependents
* Add tests including corner cases
* No need to canel dependents since they are not in a queue yet anyway
* The dependents keys can be deleted in all cases
* Update tests to included saved jobs in the deletion tests
* Correctly use pipeline in cancel method
* Unused connection
* Include dependents into dict format of job
* Add TODO
* Address comments from selwin
* Delete dependents key in redis if delete_dependents is called on its own
* Address recent comments from selwin
* Small change to trigger travis
* Remove TODO referring to canceled job state
* Remove dependent_ids from to_dict
* Address recent comments from selwin
After using `@job` decorator for a function that takes a long string, in RQ worker I got printed all the args/kwargs via `job.get_call_string()`.
To get this overridden, I added `description` argument to the decorator.
I decided not to put this in `delay` method because it's may be currently be used by end user.
* job.exc_info is now compressed.
* job.data is now stored in compressed format.
* Added worker_registration.unregister.
* Added worker_registration.get_keys().
* Modified Worker.all(), Worker.all_keys() and Worker.count() to accept "connection" and "queue" arguments.
* Fixed an issue where `birth` not present in Redis
Fixed an issue where worker.refresh() may fail if `birth` is not present in Redis
* added test coverage
* First stab at implementing worker statistics.
* Moved worker data restoration logic to worker.refresh().
* Failed and successfull job counts are now properly incremented.
* Worker now keeps track of total_working_time
* Ensure job.ended_at is set in the case of unhandled job failure.
* handle_job_failure shouldn't crash if job.started_at is not present.
* Added a custom exception for timeout transfer
* Added a util to transfer timeout to a united format
* Transfer timeout format when creating a queue or enqueue jobs
* Fixed typos
* Fixed bug in transfer_timeout function
* Added test for function transfer_timeout
* Updated transfer_timeout to allow uppercase unit
* Renamed function in utils
* Solve the UnicodeDecodeError while decode literal things.
* Add test case for when worker result is a unicode or str object that other than
pure ascii content.
Currently, the job is being performed inside `enqueue_call()`, which
means that `async=False` has no effect if `enqueue_job()` is called
directly. This commit fixes that.
Worker handle_exception and move_to_failed_queue couldn't handle a situation where the exception raised had non-ascii characters. This caused a UnicodeDecodeError when trying to format the exception strings.
If on Python 2, ensure strings get decoded before building the exception string.
Closes#482
This includes:
- a partial refactor of the CLI to organize the shared options
- extends the tests in areas where passing custom backend classes makes sense
- allow setting the core CLI options as env vars
- minor cosmetic changes here and there
request_force_stop_sigrtmin would fail for python3 because it would
try to read frame attributes that have been removed in python3
This patch fixes that by reading more fram attributes only for
python2
In order to solve issue 702 we have to check whether a work-horse
terminated unexpectedly (by inspecting the exit code of the work-horse
process). If it exited unexpectedly we check if the job has either been
marked as finished, failed or other valid states. If it's not in any
valid state we mark it as failed and move it to the failed queue.
Since the process was terminated unexpectedly (think OOM) we do not
have any exception context and we can't run any custom exception handlers.
There is still a chance that the job will finish successfully but the
work-horse process will be killed before the job is marked as finished
and we will erroneously mark it as failed. The users should take care
to write idempotent jobs.
Otherwise, there's no linear history between this old tag and any new,
future, tags. This is likely been a manual mistake when I released
0.6.0, so this is corrected now.
This allows jobs to use get_current_connection() with a resolvable connection.
And then these jobs can schedule new jobs for example (my use-case). Or attach
information to their job-object.
Also pop the pushed connection after running the jobs. This is needed for some
tests that check the _connection_stack afterwards;-) And also for use-cases
where the workers are used multiple times.
fixesnvie/rq#479
logger.setLevel() doesn't work when a string is passed in Python-2.6.
This patch checks if the level has been set and if not (in Python-2.6),
tries to set it again using appropriate level constants
Also unused dictConfig import is removed
This make it easy to write Python 2/3 compatible code.
e.g.
Worker(sys.argv[1:])
# Without from __future__ import unicode_literals
Worker(['high', 'normal', 'low'])
This has been discussed in #514, #282 and #88.
Using an explicit type check via `isinstance`, rather than duck typing, is typically considered unpythonic and breaks compatibility with mock objects such as FakeRedis. This patches removes the type check, and instead looks for a common method that should be present on the object as a hint on whether it's compatible or not.
syntax: assertEquals -> assertEqual, assertNotEquals -> assertNotEqual
usage: status of worker and job now will use get/set method instead of property method
For tracing job execution in a distributed system it is useful to tag log
entries with the worker id and job id. The current job is accessible
via get_current_job(), but that requires an extra redis connection. And
the current worker id (the worker id of the parent process) does not appear
to be available.
Rather than introducing an `rqworker` alternative or subclassing Worker, it
feels simple and efficient to make these contextual ids available as
environment variables. This should have no performance cost and no
API compatibility issues.
Some useful things to do with these values in the worker horse process:
+ include them in log messages
+ include them as 'x-' headers in HTTP requests made by workers
Current implementation is using the first job queue
to insert the dependent jobs. This makes RQ to ignore
the original queue where the jobs were enqueued.
With this change, we will use job origin attribute to
fetch the original queue, then insert the dependent
job to the fetched queue.
fixes issue #512
In the case of many sequential jobs having been deleted, a recursive
implementation of `Queue#dequeue` is prone to blowing the stack in the
absence of tail-recursion support. Change the implementation from
recursive to iterative to work around this issue in CPython.