1767 Commits (ffacdcb67556050ed91023dae48e4c6be5e5b3ca)
 

Author SHA1 Message Date
mr-trouble 5f949f4cef Add a hard kill from the parent process with a 10% increased timeout … (#1169)
* Add a hard kill from the parent process with a 10% increased timeout in case the forked process gets stuck and cannot stop itself.

* Added test for the force kill of the parent process.

* Changed 10% to +1 second, and other misc changes based on review comments.
5 years ago
Florian Perucki 37a6304a4f fix: use correct logging handler for scheduler (#1178) 5 years ago
Selwin Ong baa0cc268a
Job scheduling (#1163)
* First RQScheduler prototype

* WIP job scheduling

* Fixed Python 2.7 tests

* Added ScheduledJobRegistry.get_scheduled_time(job)

* WIP on scheduler's threading mechanism

* Fixed test errors

* Changed scheduler.acquire_locks() to instance method

* Added scheduler.prepare_registries()

* Somewhat working implementation of RQ scheduler

* Only call stop_scheduler if there's a scheduler present

* Use OSError rather than ProcessLookupError for PyPy compatibility

* Added `auto_start` argument to scheduler.acquire_locks()

* Make RQScheduler play better with timezone

* Fixed test error

* Added --with-scheduler flag to rq worker CLI

* Fix tests on Python 2.x

* More Python 2 fixes

* Only call `scheduler.start` if worker is run in non burst mode

* Fixed an issue where running worker with scheduler would fail sometimes

* Make `worker.stop_scheduler()` more resilient to errors

* worker.dequeue_job_and_maintain_ttl() should also periodically run maintenance tasks

* Scheduler can now work with worker in both burst and non burst mode

* Fixed scheduler logging message

* Always log scheduler errors when running

* Improve scheduler error logging message

* Removed testing code

* Scheduler should periodically try to acquire locks for other queues it doesn't have

* Added tests for scheduler.should_reacquire_locks

* Added queue.enqueue_in()

* Fixes queue.enqueue_in() in Python 2.7

* First stab at documenting job scheduling

* Remove unused methods

* Remove Python 2.6 logging compatibility code

* Remove more unused imports

* Added convenience methods to access job registries from queue

* Added test for worker.run_maintenance_tasks()

* Simplify worker.queue_names() and worker.queue_keys()

* Updated changelog to mention RQ's new job scheduling mechanism.
5 years ago
Christian Clauss f09d4db080 Travis CI: sudo: deprecated and Xenial is default (#1175) 5 years ago
goldstar611 b960ad53e6 Add example for unix:// (#1171) 5 years ago
Selwin Ong a06421e125 Updated changelog 5 years ago
Selwin Ong a407bcfe36 Updated changelog 5 years ago
Yongtao Zhang 5bb5fda1a6 fix log level parameter override, Fixes #1166 (#1167) 5 years ago
Selwin Ong cf50151853 Added a note to indicate that `registry.remove(job, delete_job=True)` is new 5 years ago
Thomas Matecki 80c82f731f Multi Dependency Support - Registration & Enqueue Call (#1155)
* Multi Dependency Support - Registration & Enqueue Call

Internal API changes to support multiple dependencies.
* Store all of a job's _dependencies_ in a redis set. Delete that set when a job is deleted.
* Add Job#fetch_dependencies method - which return all jobs a job is dependent upon and optionally _WATCHES_ all dependency ids.
* Use Job#fetch_dependencies in Queue#call_enqueue. `fetch_dependencies` now sets WATCH and raises InvalidJobDependency, rather than call_enqueue.

`Queue` and `Job` public APIs still expect single ids of jobs for `depends_on` but internally register them in a way that could support multiple jobs being passed as dependencies.

Next up: need to update Queue#enqueue_dependents

* Use existing fetch_many method to get dependencies.

Modify fetch_dependencies to use fetch_many.

* Remove default value for fetch_many's connection parameter

* PR review housekeeping

* Remove a duplicate test
* Oneline something
* Fix missing colon in dependencies key
* Delete job key, dependents and dependencies at once

* More Fixes From Code Review

Updates to Job, Queue and associated tests.

* When Checking dependencies Avoid, trip to Redis

* When checking the status of a job, we have a 'clean' status of all dependencies(returned from Job#fetch_dependencies) and the job keys are WATCHed, so there's no reason to go back to Redis to get the status _again_.
* Looks as though, the `_status` set in `Job#restore` was bytes while it was converted to text(`as_text`) in `Job#get_status` - for consistency(and tests) converting to text in `restore` as well.
* In `Queue#enqueue_call`, moved WATCH of dependencies_key to before fetching dependencies. This doesn't really matter but seems more _correct_ - one can imagine some rogue API adding a dependency after they've been fetched but before they've been WATCHEed.

* Update Job#get_status to get _local_ status

* If refresh=False is passed, don't get status from Redis; return the value of _status. This is to avoid a trip to Redis if the caller can guarantee that the value of `_status` is _clean_.

* More Fixups

* Expire dependency keys in Job#cleanup
* Consistency in Job#fetch_dependencies
5 years ago
Selwin Ong af678243e1
Added `delete_job` argument to registry.remove()` (#1161) 5 years ago
Yongtao Zhang 5bb03b9c2c Fix rq info not found workers information error, Fixes #1139 (#1149)
* Fix rq info not found workers information error, Fixes #1139

* Add test(#1149)
5 years ago
Thomas Matecki 75644ba948 Multi Dependency Support [Internal API Changes] (#1147)
* Convert `_dependency_id` to `_dependency_ids`

Change `Job`s tracking from a single id of it's dependencies from a single _id_ to a list of _id_s. This change should be private to `Job` - especially leaving `Job#to_dict` and `Job#restore`s treatment of a single 'dependency_id' intact.

This change modifies existing tests.

* Remove reliance upon dependency property in tests

... use dependency.id not  `_dependency_id`

* Re-add assertions for Falsey Values

* Add _dependency_id property

For backwards compatibility with other libs such as django-rq and rq-scheduler
5 years ago
Ivan Piskunov cfc02816ea Fix dependent jobs enqueueing (#1145)
If Queue was subclassed and enqueue_job changed, these changes should affect on dependent jobs' enqueueing too.
5 years ago
Selwin Ong e43bce4467 worker.register_birth() should use hmset 5 years ago
Vladimir Protasov 8c34e2b353 Store worker's RQ and Python versions (#1125)
* Store worker version to Redis

* Store worker's Python version to Redis

* Store worker version in __init__ body as suggested in review
5 years ago
joncros 68276e7252 add doc page for job registries (#1140)
* add doc page for job registries

* Use consistent language in registry type descriptions

* Correct usage of 'connection' parameter in Job Registry doc
5 years ago
joncros 5328a6252b Edit jobs doc for clarity (#1136)
* add section "Job Creation" to jobs doc

* edit jobs doc for clarity
5 years ago
Selwin Ong ed15b9e031 Updated ttl argument docs 5 years ago
Vladimir Protasov b62b9b0727 Fix unreliable test (#1126)
Also make error message more useful in case of future failures.
5 years ago
Andy Lulham 1a382e761f Fix where -> were typo (#1121) 5 years ago
William Woodruff b020c7fa34 docs/jobs: Fix minor typo (#1128) 5 years ago
Bartłomiej Biernacki 51efc20371 Add failure_ttl on job decorator (#1130)
Without it you cannot specify custom timeout for failed jobs created using decorator
5 years ago
Igor Gnatenko 0a669f1f33 Include LICENSE in manifest (#1134) 5 years ago
Selwin Ong d1813cdff9 Fixed test errors caused by _sentry_trace_headers 6 years ago
Selwin Ong dc63b362e2 Bump version to 1.1.0 6 years ago
Selwin Ong f50e60d2f4 Updated CHANGES.md 6 years ago
Chyroc d0884be929 otp: reuse utcnow (#1115) 6 years ago
Selwin Ong f9d42e8a17
Added logging statements to handle_job_success and handle_job_failure (#1112) 6 years ago
Selwin Ong b14c4e288d
Added checks for 0 ttl (#1110) 6 years ago
Joe 8df4e8ecfa Update worker document for setproctitle feature (#1109)
* Replace procname with setproctitle

* Document process title feature

* Fix typo
6 years ago
Joe b52bad3ad0 Replace procname with setproctitle (#1108) 6 years ago
Selwin Ong 549648bd1b
rq info management command now cleans up registries when first run (#1107)
* rq info management command now cleans up registries when first run

* Deleted print statement

* Improve CLI test coverage

* Fixed CLI test on Linux
6 years ago
Selwin Ong 905824e8d8
Improve worker start/stop logging messages (#1105) 6 years ago
Paul Robertson e1c135d4de add the ability to have the worker stop executing after a max amount of jobs (#1094)
* add the ability to have the worker stop executing after a max amount of jobs

* rename to max-jobs

* updated logging messages
6 years ago
Selwin Ong 2baef02dbd
Update README.md 6 years ago
janierdavila d191adbeb9 Fixes disable-job-desc-logging flag on CLI (#1096)
PR #991 introduced this useful flag but it seems to have missed
passing this flag to the worker. This simple commit fixes that
issue.
6 years ago
Paul Robertson 077aa7d32e CI - test against py3.8 (#1095)
* test against py3.8

* switch to the matrix format
6 years ago
Selwin Ong 68c01bc537
Create FUNDING.yml 6 years ago
秋葉 e40fc393b2 remove unused code (#1092) 6 years ago
Selwin Ong 98fa81873b
Update README.md 6 years ago
Christopher Pickering d2e06e4785 Update index.md (#1089)
added "import time"
6 years ago
Ted Summer 79a6fd7999 Fix timeout adding job to StartedJobRegistry (#1086)
* Fix timeout adding job to StartedJobRegistry

* Fix prepare_job_execution handling neg timeout

* Add test for inf job timeout in StartedJobRegistry

* refactor(worker): simplify checking neg timeout
6 years ago
Ted Summer eed51df14a Fix usage example of queue.enqueue() (#1082)
* Fix usage example of queue.enqueue()

* Refactor enqueue example
6 years ago
Selwin Ong f467e708f2 Updated leftover references to FailedQueue 6 years ago
Ted Summer da21a110b3 Fix docs on job attributes (#1083) 6 years ago
Selwin Ong 0d593f40e1 Clarify the use of "args" and "kwargs" in queue.enqueue() 6 years ago
Selwin Ong 2af3d829b0 Updated docs to change "normal" to "default" 6 years ago
Selwin Ong a33ed3a019 Updated docs to change "normal" to "default" 6 years ago
Selwin Ong 7021cedaf9
Implemented Job.fetch_many (#1072) 6 years ago