154 Commits (2af3d829b09c6243148a561b0495965c95a27db0)

Author SHA1 Message Date
Selwin Ong c4cbb3af2f
RQ v1.0! (#1059)
* Added FailedJobRegistry.

* Added job.failure_ttl.

* queue.enqueue() now supports failure_ttl

* Added registry.get_queue().

* FailedJobRegistry.add() now assigns DEFAULT_FAILURE_TTL.

* StartedJobRegistry.cleanup() now moves expired jobs to FailedJobRegistry.

* Failed jobs are now added to FailedJobRegistry.

* Added FailedJobRegistry.requeue()

* Document the new `FailedJobRegistry` and changes in custom exception handler behavior.

* Added worker.disable_default_exception_handler.

* Document --disable-default-exception-handler option.

* Deleted worker.failed_queue.

* Deleted "move_to_failed_queue" exception handler.

* StartedJobRegistry should no longer move jobs to FailedQueue.

* Deleted requeue_job

* Fixed test error.

* Make requeue cli command work with FailedJobRegistry

* Added .pytest_cache to gitignore.

* Custom exception handlers are no longer run in reverse

* Restored requeue_job function

* Removed get_failed_queue

* Deleted FailedQueue

* Updated changelog.

* Document `failure_ttl`

* Updated docs.

* Remove job.status

* Fixed typo in test_registry.py

* Replaced _pipeline() with pipeline()

* FailedJobRegistry no longer fails on redis-py>=3

* Fixes test_clean_registries

* Worker names are now randomized

* Added a note about random worker names in CHANGES.md

* Worker will now stop working when encountering an unhandled exception.

* Worker should reraise SystemExit on cold shutdowns

* Added anchor.js to docs

* Support for Sentry-SDK (#1045)

* Updated RQ to support sentry-sdk

* Document Sentry integration

* Install sentry-sdk before running tests

* Improved rq info CLI command to be more efficient when displaying lar… (#1046)

* Improved rq info CLI command to be more efficient when displaying large number of workers

* Fixed an rq info --by-queue bug

* Fixed worker.total_working_time bug (#1047)

* queue.enqueue() no longer accepts `timeout` argument (#1055)

* Clean worker registry (#1056)

* queue.enqueue() no longer accepts `timeout` argument

* Added clean_worker_registry()

* Show worker hostname and PID on cli (#1058)

* Show worker hostname and PID on cli

* Improve test coverage

* Remove Redis version check when SSL is used

* Bump version to 1.0

* Removed pytest_cache/README.md

* Changed worker logging to use exc_info=True

* Removed unused queue.dequeue()

* Fixed typo in CHANGES.md

* setup_loghandlers() should always call logger.setLevel() if specified
6 years ago
Wolfgang Langner 8fc987dc68 Make logging in worker consitent. (#1030)
Switch some messages from warn to info because it is normal requested bahavior.
6 years ago
Finnci 14db0ecd26 Update/add flag for description logging (#991)
* test workers

* indent

* add docs and add option to the cli

* rename flag for cli

* logging
6 years ago
Samuel Colvin 2f35222ddb skip test_1_sec_shutdown with pypy (#1020)
* skip test_1_sec_shutdown with pypy, fix #1019

* skip all HerokuWorkerShutdownTestCase with pypy
6 years ago
Darshan Rai ada2ad03ca modify zadd calls for redis-py 3.0 (#1016)
* modify zadd calls for redis-py 3.0

redis-py 3.0 changes the zadd interface that accepts a single
mapping argument that is expected to be a dict.
https://github.com/andymccurdy/redis-py#mset-msetnx-and-zadd

* change FailedQueue.push_job_id to always push a str

redis-py 3.0 does not attempt to cast values to str and is left
to the user.

* remove Redis connection patching

Since in redis-py 3.0, Redis == StrictRedis class, we no longer
need to patch _zadd and other methods.
Ref: https://github.com/rq/rq/pull/1016#issuecomment-441010847
6 years ago
Selwin Ong 6559b0ffd7
Replace "timeout" argument in queue.enqueue() with "job_timeout" (#1010) 6 years ago
Selwin Ong ad66d872f0 Fixed a unicode test. 6 years ago
Selwin Ong 47d291771f
SimpleWorker's ttl must always be longer than jobs. (#1002) 6 years ago
Selwin Ong 531fde8e3c worker.main_work_horse should always return 0 7 years ago
Thomas Kriechbaumer 3133d94b58 add periodic worker heartbeats (#945)
* add periodic worker heartbeats

fixes #944

* improve worker default option handling
7 years ago
Selwin Ong 7a3c85f185
Added the ability to fetch workers by queue (#911)
* job.exc_info is now compressed.

* job.data is now stored in compressed format.

* Added worker_registration.unregister.

* Added worker_registration.get_keys().

* Modified Worker.all(), Worker.all_keys() and Worker.count() to accept "connection" and "queue" arguments.
7 years ago
Samuel Colvin df571e14fd improve logging in worker.py (#902)
* improve logging in worker

* tests for log_result_lifespan
7 years ago
Selwin Ong f500186f3d
Job compression (#907)
job.exc_info and job.data is now stored in compressed format in Redis.

* job.data is now stored in compressed format.
7 years ago
vanife ff36e0656e Fixed an issue where `birth` not present in Redis (#901)
* Fixed an issue where `birth` not present in Redis

Fixed an issue where worker.refresh() may fail if `birth` is not present in Redis

* added test coverage
7 years ago
Selwin Ong 7b9c3b6b66 Fixed an issue where worker.refresh() may fail if last_heartbeat is not present in Redis. 7 years ago
Selwin Ong 1d7b5e834b
Worker statistics (#897)
* First stab at implementing worker statistics.

* Moved worker data restoration logic to worker.refresh().

* Failed and successfull job counts are now properly incremented.

* Worker now keeps track of total_working_time

* Ensure job.ended_at is set in the case of unhandled job failure.

* handle_job_failure shouldn't crash if job.started_at is not present.
7 years ago
Samuel Colvin 260fd84f51 add milliseconds into timestamps, fix #721 7 years ago
Theo 261f4ac3d5 Fixed #866 - Flak8 errors 7 years ago
Theo 096c5ad3c2 Fixed #866 - Flak8 errors 7 years ago
Samuel Colvin 423da3683c remove python 2.6 support 7 years ago
Selwin Ong dc45ab8799 Worker.find_by_key should use hmget instead of repeated hget calls. (#826) 8 years ago
Peng Liu b7d4b4ec1b Solve the UnicodeDecodeError while decode literal things. (#817)
* Solve the UnicodeDecodeError while decode literal things.

* Add test case for when worker result is a unicode or str object that other than
pure ascii content.
8 years ago
Selwin Ong f6b4c286c9 Merge pull request #757 from jaywink/fix-unicode-decode-error
Fix UnicodeDecodeError when failing jobs
8 years ago
Samuel Colvin fd9babe8ce correct heroku worker exit logic
as per @Chronial's comment on b4b99f3
8 years ago
Jason Robinson 213969742e Fix UnicodeDecodeError when failing jobs
Worker handle_exception and move_to_failed_queue couldn't handle a situation where the exception raised had non-ascii characters. This caused a UnicodeDecodeError when trying to format the exception strings.

If on Python 2, ensure strings get decoded before building the exception string.

Closes #482
8 years ago
Selwin Ong 83007b2074 Merge pull request #786 from jezdez/backend-class-overrides
Allow passing backend classes from CLI and other APIs
8 years ago
Benjamin Root 30a7ab4899 Add similar test for when the job fails 8 years ago
Benjamin Root f68aeff481 Added integration test for the metadata persistence feature 8 years ago
Jannis Leidel c019662430
Allow passing backend classes (job, queue, worker, connection) from CLI and other APIs
This includes:

- a partial refactor of the CLI to organize the shared options
- extends the tests in areas where passing custom backend classes makes sense
- allow setting the core CLI options as env vars
- minor cosmetic changes here and there
8 years ago
Samuel Colvin 763a304ec6 cope with python 2.6 yawwwwwwn 8 years ago
Samuel Colvin f50bdb31bf skip test_run_scheduled_access_self on pypy 8 years ago
Samuel Colvin b4b99f30b0 allow for already dead horse on heroku worker termination 8 years ago
Selwin Ong 2428370dfd Merge pull request #763 from jezdez/skip-heroku-macos
Stop running some Heroku tests on macOS since it’s missing the appropriate signals
8 years ago
Jannis Leidel fcec17c8e1
Stop running some Heroku tests on macOS since it’s missing the appropriate signals
Refs issue #584.
8 years ago
Stefan Hammer f9bff3d12b Merge branch 'master' into 739_fix_race_condition 8 years ago
Vincent Driessen af6ce54ea4 Fix PEP8 complaints 8 years ago
Stefan Hammer a0cee2d2a0 refactored worker code
Moved code into a new handle_job_success() method and reduced context of used
pipelines.
8 years ago
Stefan Hammer 44f98693c7 added a test for the race condition 8 years ago
Samuel Colvin bbf50f4893 use compat for PY2 test and add tests 9 years ago
Yannis Spiliopoulos 4a8aa0921f Set test timeout to original value 9 years ago
Yannis Spiliopoulos fbb29ec333 Investigating timeouts 9 years ago
Yannis Spiliopoulos 94d5caed15 Make clearer the puprose of test 9 years ago
Yannis Spiliopoulos 3362fe2ba5 Test killing work_horse after we start monitoring 9 years ago
Yannis Spiliopoulos c00d3681f9 Failing test to demonstrate issue #702
Test that demonstrates that if a work-horse process is terminated unexpectedly
the job being processed could be stuck at the "Started" state
(https://github.com/nvie/rq/issues/702)
9 years ago
Samuel Colvin 0e26db9e08 correct wording in docstring and tests 9 years ago
Samuel Colvin 9f9c887645 better function names and process double SIGRTMIN 9 years ago
Samuel Colvin 2b544e5b17 add tests to HerokuWorker 9 years ago
Arnold Krille 8e99706b16 run python 3.5 on travis, adopt timeouts
Also
- Report the five slowest tests to watch for risk of timeout
- Double timeouts.
  Maybe that helps pypy on travis to finish successfully.
9 years ago
Arnold Krille df22f127eb Test the worker in its own subprocess
- run with an empty queue
- schedule one job (which uses get_current_connection and get_current_job) and
run `rqworker`
- schedule a job that itself schedules `access_self` and run `rqworker`
- Make sure the job didn't fail by assuring the failed queue is still empty
  afterwards.
- Install this package locally when running in travis.
  This actually unifies the behaviour of tox and travis as tox also builds the
  package and then installs it into each test environment.
- fix flake8 (as run by tox)
9 years ago
Selwin Ong 766bb60006 Merge branch 'dependent-jobs-with-result-ttl' 9 years ago