65 Commits (b50f1e2121171a2079282753ef699c07bcfe8918)

Author SHA1 Message Date
Ruslan Mullakhmetov ed264f08bb
feat: added job heartbeat to track whether job is actually executing (#1349)
* feat: added job heartbeat to track whether job is actually executing

heartbeat might be needed in cases when worker was hardkilled or the whole VM/docker was forcibly rebooted.

* fixed tests

* fixed test coverage issue

* chore: renamed job.heartbeat stuff according to review feedback

* chore: pipelined worker heartbeat and job heartbeat

* docs: documented job.heartbeat property

* fixes after review

* docs: updated last_heartbeat description

* chore: review

Co-authored-by: Ruslan Mullakhmetov <ruslan@twentythree.net>
4 years ago
Selwin Ong a721db34b1
Workers can listen to external commands via pubsub (#1363)
* Added a way to send shutdown command via pubsub

* Added kill-horse command

* Added kill horse command

* Added send_kill_horse_command() and send_shutdown_command()

* Document worker commands
4 years ago
Selwin Ong f5ce159b60 Remove alpha warning from scheduler docs 4 years ago
maces 48f5a68013
fixed minor import error in jobs docs (#1334) 4 years ago
Selwin Ong 57f286eac4 Minor doc fix 4 years ago
Selwin Ong 167ab96070 Slightly tweaked docs CSS 4 years ago
Selwin Ong dff7a09a1d Fixed Github Pages build 4 years ago
Selwin Ong f3c86c02c6 Bump version to 1.5.0 4 years ago
Zhicheng Cai a62b762200
Update index.md (#1306) 4 years ago
Selwin Ong 49b156ecc7
Job retry feature. Docs WIP (#1299)
* Initial implementation of Retry class

* Fixes job.refresh() under Python 3.5

* Remove the use of text_type in job.py

* Retry can be scheduled

* monitor_work_horse() should call handle_job_failure() with queue argument.

* Flake8 fixes

* Added docs for job retries
4 years ago
Paul Spooren 73506b26fc
Add get_job_position and get_position feature (#1271)
Fix #1197

Signed-off-by: Paul Spooren <mail@aparcar.org>
5 years ago
Vincent Jacques 9fb80424ef
Fix typo in scheduling doc (#1245) 5 years ago
Babatunde Olusola e1cbc3736c
Implement Customizable Serializer Support (#1219)
* Implement Customizable Serializer Support

* Refractor serializer instance methods

* Update tests with other serializers

* Edit function description

* Edit function description

* Raise appropriate exception

* Update tests for better code coverage

* Remove un-used imports and un-necessary code

* Refractor resolve_serializer

* Remove un-necessary alias from imports

* Add documentation

* Refractor tests, improve documentation
5 years ago
Selwin Ong 57f8fe6a0c Updated job.result docs. 5 years ago
Ryan Febriansyah 71b01ba1b6
fix code tag in connections.md (#1189)
* Update connections.md

* Update testing.md
5 years ago
Selwin Ong 43fb21b827 Show job scheduling section on docs website 5 years ago
Levin Rickert 0080b223fe
Fix typos in job_registries.md (#1185) 5 years ago
Selwin Ong baa0cc268a
Job scheduling (#1163)
* First RQScheduler prototype

* WIP job scheduling

* Fixed Python 2.7 tests

* Added ScheduledJobRegistry.get_scheduled_time(job)

* WIP on scheduler's threading mechanism

* Fixed test errors

* Changed scheduler.acquire_locks() to instance method

* Added scheduler.prepare_registries()

* Somewhat working implementation of RQ scheduler

* Only call stop_scheduler if there's a scheduler present

* Use OSError rather than ProcessLookupError for PyPy compatibility

* Added `auto_start` argument to scheduler.acquire_locks()

* Make RQScheduler play better with timezone

* Fixed test error

* Added --with-scheduler flag to rq worker CLI

* Fix tests on Python 2.x

* More Python 2 fixes

* Only call `scheduler.start` if worker is run in non burst mode

* Fixed an issue where running worker with scheduler would fail sometimes

* Make `worker.stop_scheduler()` more resilient to errors

* worker.dequeue_job_and_maintain_ttl() should also periodically run maintenance tasks

* Scheduler can now work with worker in both burst and non burst mode

* Fixed scheduler logging message

* Always log scheduler errors when running

* Improve scheduler error logging message

* Removed testing code

* Scheduler should periodically try to acquire locks for other queues it doesn't have

* Added tests for scheduler.should_reacquire_locks

* Added queue.enqueue_in()

* Fixes queue.enqueue_in() in Python 2.7

* First stab at documenting job scheduling

* Remove unused methods

* Remove Python 2.6 logging compatibility code

* Remove more unused imports

* Added convenience methods to access job registries from queue

* Added test for worker.run_maintenance_tasks()

* Simplify worker.queue_names() and worker.queue_keys()

* Updated changelog to mention RQ's new job scheduling mechanism.
5 years ago
goldstar611 b960ad53e6 Add example for unix:// (#1171) 5 years ago
Selwin Ong cf50151853 Added a note to indicate that `registry.remove(job, delete_job=True)` is new 5 years ago
Selwin Ong af678243e1
Added `delete_job` argument to registry.remove()` (#1161) 5 years ago
joncros 68276e7252 add doc page for job registries (#1140)
* add doc page for job registries

* Use consistent language in registry type descriptions

* Correct usage of 'connection' parameter in Job Registry doc
5 years ago
joncros 5328a6252b Edit jobs doc for clarity (#1136)
* add section "Job Creation" to jobs doc

* edit jobs doc for clarity
5 years ago
Selwin Ong ed15b9e031 Updated ttl argument docs 5 years ago
William Woodruff b020c7fa34 docs/jobs: Fix minor typo (#1128) 5 years ago
Joe 8df4e8ecfa Update worker document for setproctitle feature (#1109)
* Replace procname with setproctitle

* Document process title feature

* Fix typo
6 years ago
Paul Robertson e1c135d4de add the ability to have the worker stop executing after a max amount of jobs (#1094)
* add the ability to have the worker stop executing after a max amount of jobs

* rename to max-jobs

* updated logging messages
6 years ago
Christopher Pickering d2e06e4785 Update index.md (#1089)
added "import time"
6 years ago
Ted Summer eed51df14a Fix usage example of queue.enqueue() (#1082)
* Fix usage example of queue.enqueue()

* Refactor enqueue example
6 years ago
Selwin Ong f467e708f2 Updated leftover references to FailedQueue 6 years ago
Ted Summer da21a110b3 Fix docs on job attributes (#1083) 6 years ago
Selwin Ong 0d593f40e1 Clarify the use of "args" and "kwargs" in queue.enqueue() 6 years ago
Selwin Ong 2af3d829b0 Updated docs to change "normal" to "default" 6 years ago
Selwin Ong a33ed3a019 Updated docs to change "normal" to "default" 6 years ago
Selwin Ong 7021cedaf9
Implemented Job.fetch_many (#1072) 6 years ago
Mike Lissner 07317b62f3 Give better titles to documentation pages (#1064)
* Fix title on workers page

* Fix jobs title

* Fix results title

* Fix title on overview page
6 years ago
Selwin Ong b51c786e5d Updated systemd docs 6 years ago
Selwin Ong 065943f0b4 Added systemd example 6 years ago
Selwin Ong c66f202460 Added docs about running workers in production 6 years ago
Selwin Ong ba8c338654 Updated RQ docs 6 years ago
SSINGH a23e3736d8 Variable name mismatch (#1063)
Should be `redis = Redis()`
6 years ago
Selwin Ong c4cbb3af2f
RQ v1.0! (#1059)
* Added FailedJobRegistry.

* Added job.failure_ttl.

* queue.enqueue() now supports failure_ttl

* Added registry.get_queue().

* FailedJobRegistry.add() now assigns DEFAULT_FAILURE_TTL.

* StartedJobRegistry.cleanup() now moves expired jobs to FailedJobRegistry.

* Failed jobs are now added to FailedJobRegistry.

* Added FailedJobRegistry.requeue()

* Document the new `FailedJobRegistry` and changes in custom exception handler behavior.

* Added worker.disable_default_exception_handler.

* Document --disable-default-exception-handler option.

* Deleted worker.failed_queue.

* Deleted "move_to_failed_queue" exception handler.

* StartedJobRegistry should no longer move jobs to FailedQueue.

* Deleted requeue_job

* Fixed test error.

* Make requeue cli command work with FailedJobRegistry

* Added .pytest_cache to gitignore.

* Custom exception handlers are no longer run in reverse

* Restored requeue_job function

* Removed get_failed_queue

* Deleted FailedQueue

* Updated changelog.

* Document `failure_ttl`

* Updated docs.

* Remove job.status

* Fixed typo in test_registry.py

* Replaced _pipeline() with pipeline()

* FailedJobRegistry no longer fails on redis-py>=3

* Fixes test_clean_registries

* Worker names are now randomized

* Added a note about random worker names in CHANGES.md

* Worker will now stop working when encountering an unhandled exception.

* Worker should reraise SystemExit on cold shutdowns

* Added anchor.js to docs

* Support for Sentry-SDK (#1045)

* Updated RQ to support sentry-sdk

* Document Sentry integration

* Install sentry-sdk before running tests

* Improved rq info CLI command to be more efficient when displaying lar… (#1046)

* Improved rq info CLI command to be more efficient when displaying large number of workers

* Fixed an rq info --by-queue bug

* Fixed worker.total_working_time bug (#1047)

* queue.enqueue() no longer accepts `timeout` argument (#1055)

* Clean worker registry (#1056)

* queue.enqueue() no longer accepts `timeout` argument

* Added clean_worker_registry()

* Show worker hostname and PID on cli (#1058)

* Show worker hostname and PID on cli

* Improve test coverage

* Remove Redis version check when SSL is used

* Bump version to 1.0

* Removed pytest_cache/README.md

* Changed worker logging to use exc_info=True

* Removed unused queue.dequeue()

* Fixed typo in CHANGES.md

* setup_loghandlers() should always call logger.setLevel() if specified
6 years ago
William Setzer 1f3c9f6e59 Delete .DS_Store (#1043)
Remove OSX artifact from repo.
6 years ago
Selwin Ong 147efc54dd Added anchor.js to docs 6 years ago
Selwin Ong 4b8e615644 Changed docs to use Github compatible code block markup 6 years ago
Selwin Ong fc8dd95377 Fixed CSS styles for code blocks. 6 years ago
Selwin Ong 7d6cccb654 Try out an alternate way to indicate code blocks in docs 6 years ago
Finnci 14db0ecd26 Update/add flag for description logging (#991)
* test workers

* indent

* add docs and add option to the cli

* rename flag for cli

* logging
6 years ago
Selwin Ong 22f004f7d6
Updated requirements.txt on Heroku docs 6 years ago
Darshan Rai ada2ad03ca modify zadd calls for redis-py 3.0 (#1016)
* modify zadd calls for redis-py 3.0

redis-py 3.0 changes the zadd interface that accepts a single
mapping argument that is expected to be a dict.
https://github.com/andymccurdy/redis-py#mset-msetnx-and-zadd

* change FailedQueue.push_job_id to always push a str

redis-py 3.0 does not attempt to cast values to str and is left
to the user.

* remove Redis connection patching

Since in redis-py 3.0, Redis == StrictRedis class, we no longer
need to patch _zadd and other methods.
Ref: https://github.com/rq/rq/pull/1016#issuecomment-441010847
6 years ago