160 Commits (b960ad53e6486ae5c3d6ad8b83863fd8ec977bc5)

Author SHA1 Message Date
Vladimir Protasov 8c34e2b353 Store worker's RQ and Python versions (#1125)
* Store worker version to Redis

* Store worker's Python version to Redis

* Store worker version in __init__ body as suggested in review
5 years ago
Vladimir Protasov b62b9b0727 Fix unreliable test (#1126)
Also make error message more useful in case of future failures.
5 years ago
Selwin Ong d1813cdff9 Fixed test errors caused by _sentry_trace_headers 6 years ago
Selwin Ong f9d42e8a17
Added logging statements to handle_job_success and handle_job_failure (#1112) 6 years ago
Paul Robertson e1c135d4de add the ability to have the worker stop executing after a max amount of jobs (#1094)
* add the ability to have the worker stop executing after a max amount of jobs

* rename to max-jobs

* updated logging messages
6 years ago
Ted Summer 79a6fd7999 Fix timeout adding job to StartedJobRegistry (#1086)
* Fix timeout adding job to StartedJobRegistry

* Fix prepare_job_execution handling neg timeout

* Add test for inf job timeout in StartedJobRegistry

* refactor(worker): simplify checking neg timeout
6 years ago
Selwin Ong c4cbb3af2f
RQ v1.0! (#1059)
* Added FailedJobRegistry.

* Added job.failure_ttl.

* queue.enqueue() now supports failure_ttl

* Added registry.get_queue().

* FailedJobRegistry.add() now assigns DEFAULT_FAILURE_TTL.

* StartedJobRegistry.cleanup() now moves expired jobs to FailedJobRegistry.

* Failed jobs are now added to FailedJobRegistry.

* Added FailedJobRegistry.requeue()

* Document the new `FailedJobRegistry` and changes in custom exception handler behavior.

* Added worker.disable_default_exception_handler.

* Document --disable-default-exception-handler option.

* Deleted worker.failed_queue.

* Deleted "move_to_failed_queue" exception handler.

* StartedJobRegistry should no longer move jobs to FailedQueue.

* Deleted requeue_job

* Fixed test error.

* Make requeue cli command work with FailedJobRegistry

* Added .pytest_cache to gitignore.

* Custom exception handlers are no longer run in reverse

* Restored requeue_job function

* Removed get_failed_queue

* Deleted FailedQueue

* Updated changelog.

* Document `failure_ttl`

* Updated docs.

* Remove job.status

* Fixed typo in test_registry.py

* Replaced _pipeline() with pipeline()

* FailedJobRegistry no longer fails on redis-py>=3

* Fixes test_clean_registries

* Worker names are now randomized

* Added a note about random worker names in CHANGES.md

* Worker will now stop working when encountering an unhandled exception.

* Worker should reraise SystemExit on cold shutdowns

* Added anchor.js to docs

* Support for Sentry-SDK (#1045)

* Updated RQ to support sentry-sdk

* Document Sentry integration

* Install sentry-sdk before running tests

* Improved rq info CLI command to be more efficient when displaying lar… (#1046)

* Improved rq info CLI command to be more efficient when displaying large number of workers

* Fixed an rq info --by-queue bug

* Fixed worker.total_working_time bug (#1047)

* queue.enqueue() no longer accepts `timeout` argument (#1055)

* Clean worker registry (#1056)

* queue.enqueue() no longer accepts `timeout` argument

* Added clean_worker_registry()

* Show worker hostname and PID on cli (#1058)

* Show worker hostname and PID on cli

* Improve test coverage

* Remove Redis version check when SSL is used

* Bump version to 1.0

* Removed pytest_cache/README.md

* Changed worker logging to use exc_info=True

* Removed unused queue.dequeue()

* Fixed typo in CHANGES.md

* setup_loghandlers() should always call logger.setLevel() if specified
6 years ago
Wolfgang Langner 8fc987dc68 Make logging in worker consitent. (#1030)
Switch some messages from warn to info because it is normal requested bahavior.
6 years ago
Finnci 14db0ecd26 Update/add flag for description logging (#991)
* test workers

* indent

* add docs and add option to the cli

* rename flag for cli

* logging
6 years ago
Samuel Colvin 2f35222ddb skip test_1_sec_shutdown with pypy (#1020)
* skip test_1_sec_shutdown with pypy, fix #1019

* skip all HerokuWorkerShutdownTestCase with pypy
6 years ago
Darshan Rai ada2ad03ca modify zadd calls for redis-py 3.0 (#1016)
* modify zadd calls for redis-py 3.0

redis-py 3.0 changes the zadd interface that accepts a single
mapping argument that is expected to be a dict.
https://github.com/andymccurdy/redis-py#mset-msetnx-and-zadd

* change FailedQueue.push_job_id to always push a str

redis-py 3.0 does not attempt to cast values to str and is left
to the user.

* remove Redis connection patching

Since in redis-py 3.0, Redis == StrictRedis class, we no longer
need to patch _zadd and other methods.
Ref: https://github.com/rq/rq/pull/1016#issuecomment-441010847
6 years ago
Selwin Ong 6559b0ffd7
Replace "timeout" argument in queue.enqueue() with "job_timeout" (#1010) 6 years ago
Selwin Ong ad66d872f0 Fixed a unicode test. 6 years ago
Selwin Ong 47d291771f
SimpleWorker's ttl must always be longer than jobs. (#1002) 6 years ago
Selwin Ong 531fde8e3c worker.main_work_horse should always return 0 7 years ago
Thomas Kriechbaumer 3133d94b58 add periodic worker heartbeats (#945)
* add periodic worker heartbeats

fixes #944

* improve worker default option handling
7 years ago
Selwin Ong 7a3c85f185
Added the ability to fetch workers by queue (#911)
* job.exc_info is now compressed.

* job.data is now stored in compressed format.

* Added worker_registration.unregister.

* Added worker_registration.get_keys().

* Modified Worker.all(), Worker.all_keys() and Worker.count() to accept "connection" and "queue" arguments.
7 years ago
Samuel Colvin df571e14fd improve logging in worker.py (#902)
* improve logging in worker

* tests for log_result_lifespan
7 years ago
Selwin Ong f500186f3d
Job compression (#907)
job.exc_info and job.data is now stored in compressed format in Redis.

* job.data is now stored in compressed format.
7 years ago
vanife ff36e0656e Fixed an issue where `birth` not present in Redis (#901)
* Fixed an issue where `birth` not present in Redis

Fixed an issue where worker.refresh() may fail if `birth` is not present in Redis

* added test coverage
7 years ago
Selwin Ong 7b9c3b6b66 Fixed an issue where worker.refresh() may fail if last_heartbeat is not present in Redis. 7 years ago
Selwin Ong 1d7b5e834b
Worker statistics (#897)
* First stab at implementing worker statistics.

* Moved worker data restoration logic to worker.refresh().

* Failed and successfull job counts are now properly incremented.

* Worker now keeps track of total_working_time

* Ensure job.ended_at is set in the case of unhandled job failure.

* handle_job_failure shouldn't crash if job.started_at is not present.
7 years ago
Samuel Colvin 260fd84f51 add milliseconds into timestamps, fix #721 7 years ago
Theo 261f4ac3d5 Fixed #866 - Flak8 errors 7 years ago
Theo 096c5ad3c2 Fixed #866 - Flak8 errors 7 years ago
Samuel Colvin 423da3683c remove python 2.6 support 7 years ago
Selwin Ong dc45ab8799 Worker.find_by_key should use hmget instead of repeated hget calls. (#826) 8 years ago
Peng Liu b7d4b4ec1b Solve the UnicodeDecodeError while decode literal things. (#817)
* Solve the UnicodeDecodeError while decode literal things.

* Add test case for when worker result is a unicode or str object that other than
pure ascii content.
8 years ago
Selwin Ong f6b4c286c9 Merge pull request #757 from jaywink/fix-unicode-decode-error
Fix UnicodeDecodeError when failing jobs
8 years ago
Samuel Colvin fd9babe8ce correct heroku worker exit logic
as per @Chronial's comment on b4b99f3
8 years ago
Jason Robinson 213969742e Fix UnicodeDecodeError when failing jobs
Worker handle_exception and move_to_failed_queue couldn't handle a situation where the exception raised had non-ascii characters. This caused a UnicodeDecodeError when trying to format the exception strings.

If on Python 2, ensure strings get decoded before building the exception string.

Closes #482
8 years ago
Selwin Ong 83007b2074 Merge pull request #786 from jezdez/backend-class-overrides
Allow passing backend classes from CLI and other APIs
8 years ago
Benjamin Root 30a7ab4899 Add similar test for when the job fails 8 years ago
Benjamin Root f68aeff481 Added integration test for the metadata persistence feature 8 years ago
Jannis Leidel c019662430
Allow passing backend classes (job, queue, worker, connection) from CLI and other APIs
This includes:

- a partial refactor of the CLI to organize the shared options
- extends the tests in areas where passing custom backend classes makes sense
- allow setting the core CLI options as env vars
- minor cosmetic changes here and there
8 years ago
Samuel Colvin 763a304ec6 cope with python 2.6 yawwwwwwn 8 years ago
Samuel Colvin f50bdb31bf skip test_run_scheduled_access_self on pypy 8 years ago
Samuel Colvin b4b99f30b0 allow for already dead horse on heroku worker termination 8 years ago
Selwin Ong 2428370dfd Merge pull request #763 from jezdez/skip-heroku-macos
Stop running some Heroku tests on macOS since it’s missing the appropriate signals
8 years ago
Jannis Leidel fcec17c8e1
Stop running some Heroku tests on macOS since it’s missing the appropriate signals
Refs issue #584.
8 years ago
Stefan Hammer f9bff3d12b Merge branch 'master' into 739_fix_race_condition 8 years ago
Vincent Driessen af6ce54ea4 Fix PEP8 complaints 8 years ago
Stefan Hammer a0cee2d2a0 refactored worker code
Moved code into a new handle_job_success() method and reduced context of used
pipelines.
8 years ago
Stefan Hammer 44f98693c7 added a test for the race condition 8 years ago
Samuel Colvin bbf50f4893 use compat for PY2 test and add tests 9 years ago
Yannis Spiliopoulos 4a8aa0921f Set test timeout to original value 9 years ago
Yannis Spiliopoulos fbb29ec333 Investigating timeouts 9 years ago
Yannis Spiliopoulos 94d5caed15 Make clearer the puprose of test 9 years ago
Yannis Spiliopoulos 3362fe2ba5 Test killing work_horse after we start monitoring 9 years ago
Yannis Spiliopoulos c00d3681f9 Failing test to demonstrate issue #702
Test that demonstrates that if a work-horse process is terminated unexpectedly
the job being processed could be stuck at the "Started" state
(https://github.com/nvie/rq/issues/702)
9 years ago