36 Commits (master)

Author SHA1 Message Date
Rob Hudson ea063edf0a
Update linting configuration (#1915)
* Update linting configuration

This removes flake8 in favor of ruff, which also provides isort support, and
updates all files to be black, isort, and ruff compliant. This also adds black
and ruff checks to the tox and Github linting workflow.

* Tweak the code coverage config and calls
2 years ago
Rony Lutsky 04722339d7
Consolidate job callbacks execution (#1851)
* consolidate job failure callback execution

* fix

* success callback as well

* merge fix

* create Callback class and change how callbacks serde

* deprecation

* add tests

* string format

* pr fix

* revert serialization changes

* fix timeout typing

* add documentation

* add test

* fix a bug

* fix tests

* move job heartbeat call to worker and make sure there is always a callback timeout

* fix test

* fix test
2 years ago
Rony Lutsky 0ba3971d55
Add failure callback call to started job registry cleanup (#1824)
* Add started job registry cleanup job failure callback call

* WIP - need to fix test

* fix test

* rename, tests and docs

* better log message

* use class name

* Update registry.py
2 years ago
lowercase00 6813e5a2ba
Remove compatibility layer for < 3.5 (#1761)
* Remove unused code from compat module

* Remove unused dictconfig

* Remove total_ordering compat layer

* Remove compatibility layer

This completely removes the compat module. It moves utilities
functions (`as_text` and `decode_redis_hash`) to the `utils`
module, is eliminates the use of the proxies `text_type` and
`string_types`, using the `str` construct directly.

* Remove compat module

Finishes the cleaning of the compatibility module.
The last function being the `is_python_version` function
which was being used internally.

* Fix old import

* Fix Imports

* Remove Dummy (Force GH Actions)

* Fix Imports

* Organize Imports
2 years ago
Selwin Ong 0691b4d46e
Multiple results using Redis Streams (#1725)
* WIP job results

* Result can now be saved

* Successfully saved and restored result

* result.save() should accept pipeline

* Successful results are saved

* Failures are now saved properly too.

* Added test for Result.get_latest()

* Checkpoint

* Got Result.all() to work

* Added Result.count(), Result.delete()

* Backward compatibility for job.result and job.exc_info

* Added some typing

* More typing stuff

* Fixed typing in job.py

* More typing updates

* Only keep the last 10 results

* Documented job.results()

* Got results test to pass

* Don't run test_results.py on Redis server < 5.0

* Fixed mock import on some Python versions

* Remove Redis 3 from test matrix

* Jobs should never use the new Result implementation if server is < 5.0

* Results should only be created is Redis stream is supported.

* Added back Redis 3 to test matrix

* Fixed job.supports_redis_streams

* Fixed worker test

* Updated docs.
2 years ago
Yang Yang 9db728921d
Improve the lint situation (#1688)
* Move common flake8 options into config file

Currently --max-line-length being specified in two places. Just use the
existing value in the config file as the source of truth.

Move --count and --statistics to config file as well.

* Fix some lints
2 years ago
Hugo d5175c38da
Drop python2-specific syntax (#1674)
* Drop syntax required only for Python 2

* Drop python2-style super() calls

Co-authored-by: Selwin Ong <selwin.ong@gmail.com>
2 years ago
Josh Cohen a3fba1ca1f
Add missing functionality for CanceledJobRegistry (#1560) 3 years ago
Josh Cohen b80045d615
Respect serializer (#1538)
* Add serializer where missing in code

* Fix cli

* Pass option to command

* Add tests for serializer option

* Merge branch 'master' into respect-serializer
- Update enqueue cli to resp. serializer

* Address @selwin's review
3 years ago
Selwin Ong 246d52b977
job.cancel() puts job into CanceledJobRegistry. (#1546)
* job.cancel() puts job into CanceledJobRegistry.

* Improve test coverage
3 years ago
Selwin Ong f9ddf7dca5
Requeueing job should clear job.exc_info (#1543) 3 years ago
rpkak 9737ffa95f
Make unstable tests more stable (#1523)
Co-authored-by: rpkak <rpkak@users.noreply.github.com>
3 years ago
Selwin Ong 5b5cfdf9ab
Jobs that get cleaned up should also be retried (#1467) 4 years ago
Selwin Ong 49b156ecc7
Job retry feature. Docs WIP (#1299)
* Initial implementation of Retry class

* Fixes job.refresh() under Python 3.5

* Remove the use of text_type in job.py

* Retry can be scheduled

* monitor_work_horse() should call handle_job_failure() with queue argument.

* Flake8 fixes

* Added docs for job retries
4 years ago
Selwin Ong cfe389bd65
FailedJobRegistry.requeue() resets job.started_at and job.ended_at (#1227) 5 years ago
Selwin Ong 636d6d2f54
registry.cleanup() now writes information to job.exc_info (#1226) 5 years ago
Selwin Ong ccfd4a02cb
Failed jobs will now auto expire (#1182) 5 years ago
Selwin Ong baa0cc268a
Job scheduling (#1163)
* First RQScheduler prototype

* WIP job scheduling

* Fixed Python 2.7 tests

* Added ScheduledJobRegistry.get_scheduled_time(job)

* WIP on scheduler's threading mechanism

* Fixed test errors

* Changed scheduler.acquire_locks() to instance method

* Added scheduler.prepare_registries()

* Somewhat working implementation of RQ scheduler

* Only call stop_scheduler if there's a scheduler present

* Use OSError rather than ProcessLookupError for PyPy compatibility

* Added `auto_start` argument to scheduler.acquire_locks()

* Make RQScheduler play better with timezone

* Fixed test error

* Added --with-scheduler flag to rq worker CLI

* Fix tests on Python 2.x

* More Python 2 fixes

* Only call `scheduler.start` if worker is run in non burst mode

* Fixed an issue where running worker with scheduler would fail sometimes

* Make `worker.stop_scheduler()` more resilient to errors

* worker.dequeue_job_and_maintain_ttl() should also periodically run maintenance tasks

* Scheduler can now work with worker in both burst and non burst mode

* Fixed scheduler logging message

* Always log scheduler errors when running

* Improve scheduler error logging message

* Removed testing code

* Scheduler should periodically try to acquire locks for other queues it doesn't have

* Added tests for scheduler.should_reacquire_locks

* Added queue.enqueue_in()

* Fixes queue.enqueue_in() in Python 2.7

* First stab at documenting job scheduling

* Remove unused methods

* Remove Python 2.6 logging compatibility code

* Remove more unused imports

* Added convenience methods to access job registries from queue

* Added test for worker.run_maintenance_tasks()

* Simplify worker.queue_names() and worker.queue_keys()

* Updated changelog to mention RQ's new job scheduling mechanism.
5 years ago
Selwin Ong af678243e1
Added `delete_job` argument to registry.remove()` (#1161) 5 years ago
Selwin Ong c4cbb3af2f
RQ v1.0! (#1059)
* Added FailedJobRegistry.

* Added job.failure_ttl.

* queue.enqueue() now supports failure_ttl

* Added registry.get_queue().

* FailedJobRegistry.add() now assigns DEFAULT_FAILURE_TTL.

* StartedJobRegistry.cleanup() now moves expired jobs to FailedJobRegistry.

* Failed jobs are now added to FailedJobRegistry.

* Added FailedJobRegistry.requeue()

* Document the new `FailedJobRegistry` and changes in custom exception handler behavior.

* Added worker.disable_default_exception_handler.

* Document --disable-default-exception-handler option.

* Deleted worker.failed_queue.

* Deleted "move_to_failed_queue" exception handler.

* StartedJobRegistry should no longer move jobs to FailedQueue.

* Deleted requeue_job

* Fixed test error.

* Make requeue cli command work with FailedJobRegistry

* Added .pytest_cache to gitignore.

* Custom exception handlers are no longer run in reverse

* Restored requeue_job function

* Removed get_failed_queue

* Deleted FailedQueue

* Updated changelog.

* Document `failure_ttl`

* Updated docs.

* Remove job.status

* Fixed typo in test_registry.py

* Replaced _pipeline() with pipeline()

* FailedJobRegistry no longer fails on redis-py>=3

* Fixes test_clean_registries

* Worker names are now randomized

* Added a note about random worker names in CHANGES.md

* Worker will now stop working when encountering an unhandled exception.

* Worker should reraise SystemExit on cold shutdowns

* Added anchor.js to docs

* Support for Sentry-SDK (#1045)

* Updated RQ to support sentry-sdk

* Document Sentry integration

* Install sentry-sdk before running tests

* Improved rq info CLI command to be more efficient when displaying lar… (#1046)

* Improved rq info CLI command to be more efficient when displaying large number of workers

* Fixed an rq info --by-queue bug

* Fixed worker.total_working_time bug (#1047)

* queue.enqueue() no longer accepts `timeout` argument (#1055)

* Clean worker registry (#1056)

* queue.enqueue() no longer accepts `timeout` argument

* Added clean_worker_registry()

* Show worker hostname and PID on cli (#1058)

* Show worker hostname and PID on cli

* Improve test coverage

* Remove Redis version check when SSL is used

* Bump version to 1.0

* Removed pytest_cache/README.md

* Changed worker logging to use exc_info=True

* Removed unused queue.dequeue()

* Fixed typo in CHANGES.md

* setup_loghandlers() should always call logger.setLevel() if specified
6 years ago
Darshan Rai ada2ad03ca modify zadd calls for redis-py 3.0 (#1016)
* modify zadd calls for redis-py 3.0

redis-py 3.0 changes the zadd interface that accepts a single
mapping argument that is expected to be a dict.
https://github.com/andymccurdy/redis-py#mset-msetnx-and-zadd

* change FailedQueue.push_job_id to always push a str

redis-py 3.0 does not attempt to cast values to str and is left
to the user.

* remove Redis connection patching

Since in redis-py 3.0, Redis == StrictRedis class, we no longer
need to patch _zadd and other methods.
Ref: https://github.com/rq/rq/pull/1016#issuecomment-441010847
6 years ago
Theofanis Despoudis d6b12c2402 Issue 872 (#954)
* Fixes #872 - Use -1 to indicate infinite ttl

* Fixes #872 Restored comma

* #872 Code review fix
7 years ago
Selwin Ong c639018fb9 Registry objects can be instantiated by passing a queue object. 7 years ago
Selwin Ong f760fcb20f job.delete() should cleans itself from FailedQueue and various registries. 8 years ago
Jannis Leidel c019662430
Allow passing backend classes (job, queue, worker, connection) from CLI and other APIs
This includes:

- a partial refactor of the CLI to organize the shared options
- extends the tests in areas where passing custom backend classes makes sense
- allow setting the core CLI options as env vars
- minor cosmetic changes here and there
8 years ago
Jannis Leidel 27e4f3a768
Small refactor to simplify registry key setup. 8 years ago
Selwin Ong e9d227c3df Dependent jobs are now correctly enqueued even if their parent jobs have result_ttl=0. 9 years ago
ahxxm b06f112cb0 fix tests
syntax: assertEquals -> assertEqual, assertNotEquals -> assertNotEqual
usage: status of worker and job now will use get/set method instead of property method
9 years ago
Selwin Ong faf9d3e668 Added clean_registries(queue) function to clean job registries related to that queue. 10 years ago
Selwin Ong 70d5f971bd Jobs from FinishedJobRegistry that are moved to FailedQueue should have "failed" as status. 10 years ago
Selwin Ong 105b95e9b8 Fixed an error in super call. 10 years ago
Selwin Ong de1cd8a83c Fixed test error in Python 3. 10 years ago
Selwin Ong dac0be6cc7 Added DeferredJobsRegistry to keep track of deferred jobs. 10 years ago
Nic Cope 09cab7a90d Allow maximum job age to be specified when cleaning up or getting expired jobs from a registry.
This ensures that all jobs cleaned from a started registry end up in the failed queue.
10 years ago
Nic Cope 8fa184b86b Ensure the FinishedJobRegistry honors an 'infinite' result_ttl of -1 10 years ago
Selwin Ong 2307bc8253 Added FinishedJobRegistry to monitor finished jobs. 10 years ago