488 Commits (fd116b33dbecd3acb39e33265834934aec4866b6)

Author SHA1 Message Date
Selwin Ong 8e3283dab3
Job should not be enqueued if dependency is canceled (#1695)
* Fix job.dependencies_are_met() if dependency is canceled

* Slightly better test coverage on dependencies_are_met()

* Fixed job.cancel(enqueue_dependent=True)
2 years ago
Yang Yang 9db728921d
Improve the lint situation (#1688)
* Move common flake8 options into config file

Currently --max-line-length being specified in two places. Just use the
existing value in the config file as the source of truth.

Move --count and --statistics to config file as well.

* Fix some lints
2 years ago
Selwin Ong bc5bb1fba2
Revert "Jobs that are run synchronously should always raise an exception" (#1682)
This reverts commit 0d21e714c33cfefba51a8c2b714a735bcad1264d.
2 years ago
Selwin Ong 5b95725dc4
Dependency with failures (#1681)
* added Dependency class with allow_failures

* Requested changes

* Check type before setting `job.dependency_allow_fail` within `Job.create`

* Set `job.dependency_allow_fail` within `Job.create`

* Added test to ensure persistence of `dependency_allow_fail`

* Removed typing and allow mixed list of ints and Job objects

* Convert dependency_allow_fail boolean to integer during serialization to avoid redis DataError

* Updated `test_multiple_dependencies_are_accepted_and_persisted` test to include `Dependency` cases

* Adding placeholder test to test actual behavior of new `Dependency` usage in `depends_on`

* Updated `test_job_dependency` to include cases using `Dependency`

* Added dependency_allow_fail logic to `Job.restore`

* Renamed `dependency_allow_fail` to a simpler `allow_failure`

* Update docs to add section about the new `Dependency` class and use-case

* Updated `Job.dependencies_are_met` logic to take `FAILED` and `STOPPED` jobs into account when `allow_failure=True`

* Updated `test_job_dependency` test. Still failing with `Dependency` case.

* Fix `allow_failure` type coercion in `Job.restore`

* Re-arrange tests, so default `Dependency.allow_failure` is before explicit `allow_failure=True`

* Fixed Dependency, so it works correctly when allow_failure=True

* Attempt to execute pipeline prior to queueing a failed job's dependents. test_create_and_cancel_job_enqueue_dependents_in_registry test now passes.

* Added `Depedency` test utilizing multiple dependencies

* Removed irrelevant on_success and on_failure keyword arguments in example

* Replaced use of long_running_job

* Add test to verify `Dependency.jobs` contraints

* Suppress connection error in handle_job_failure

* test_dependencies have passed

* All tests pass if enqueue_dependents called without pipeline.watch()

* All tests now pass

* Removed print statements

* Cleanup Dependency implementation

* Renamed job.allow_failure to job.allow_dependency_failures

Co-authored-by: mattchan <mattchan@tencent.com>
Co-authored-by: Mike Hill <mhilluniversal@gmail.com>
2 years ago
Tim Gates d82af1469f
docs: Fix a few typos (#1679)
There are small typos in:
- docs/docs/exceptions.md
- docs/docs/jobs.md
- rq/queue.py
- tests/fixtures.py
- tests/test_job.py

Fixes:
- Should read `slightly` rather than `slighty`.
- Should read `requeuing` rather than `requeueing`.
- Should read `implementers` rather than `implementors`.
- Should read `definition` rather than `defition`.
- Should read `canceled` rather than `canceld`.

Signed-off-by: Tim Gates <tim.gates@iress.com>
2 years ago
Selwin Ong 145884fcd5
When stopped jobs are deleted, they should also be removed from FailedJobRegistry. (#1677) 2 years ago
Hugo d5175c38da
Drop python2-specific syntax (#1674)
* Drop syntax required only for Python 2

* Drop python2-style super() calls

Co-authored-by: Selwin Ong <selwin.ong@gmail.com>
2 years ago
Hugo 61a4a1720b
Use unittest.mock instead of mock (#1673)
This module has been included in Python itself since 3.3.

Fixes: https://github.com/rq/rq/issues/1646
2 years ago
Selwin Ong 87bacb650e
Jobs that are run synchronously should always raise an exception (#1671) 3 years ago
Michael Hill f4602d30d5
Cross platform simple worker (#1629)
* Added CrossPlatformDeathPenalty that doesn't rely on signals

* Updated `SimpleWorker`'s `death_penalty_class` to utilize `CrossPlatformDeathPenalty` to allow use on Windows machines

* Changed `CrossPlatformDeathPenalty` to `TimerDeathPenalty`

* Removed overridden `death_penalty_class` in `SimpleWorker` until feature matures

* Added section in testing.md explaining how to utilize `SimpleWorker` on Windows OS

* Replaced usage of chatting with .format for python 3.5 compatibility

* Add tests for new timeout feature

* Explicitly set defaults.CALLBACK_TIMEOUT

* Implemented cross-thread method of raising errors by using ctypes

* Finished writing tests for new TimerDeathPenalty
3 years ago
Neptalí Piña 34e2427f67
job decorator allow on_failure and on_success parameters (#1626) (#1628)
Co-authored-by: neptali <neptali@plerk.io>
3 years ago
Xavier Fernandez cd17d17d71
rq.worker: remove useless set_state call in horse (#1618)
* rq.worker: remove useless set_state call in horse

The state should already have been set properly by the worker in
`execute_job`

`prepare_job_execution` is only called by `perform_job` which should only be
called by `main_work_horse`/`fork_work_horse` (themselves only called by `execute_job`).
Let `execute_job` do the bookkeeping.

* worker: update SimpleWorker's state in execute_job
3 years ago
th3hamm0r c5a1ef1734
Fixed wrong keys used to WATCH dependencies (#1605)
This bug has opened a lot of possible race-conditions, since the
watch-logic from redis did not fail anymore, if dependencies have been
changed in parallel.
3 years ago
Eric Feldman 93f34c796f
call callbacks and prepare job when running sync (#1599)
* call signals

* fix lines

* add tests and fix faliure status bug

* bump version
3 years ago
Selwin Ong 0147b30f2b
Fixes a bug that causes leftover job keys when result_ttl=0 (#1591)
* Fixes a bug that causes leftover job keys when result_ttl=0

* Fixed a buggy worker.maintain_heartbeats() behavior

* Fixed a bug in worker.maintain_heartbeats().
3 years ago
rpkak 6692587f74
Python 3.10 compatibility (#1563)
* use shutil.get_terminal_size instead of click.get_terminal_size()

resolves warning:

rq/cli/helpers.py:107: DeprecationWarning: 'click.get_terminal_size()' is deprecated and will be removed in Click 8.1. Use 'shutil.get_terminal_size()' instead.
  termwidth, _ = click.get_terminal_size()

* remove StrictVersion from rq

* asyncio.get_event_loop -> asyncio.new_event_loop()

resolves warning:

tests/test_job.py::TestJob::test_create_job_with_async
  rq/rq/job.py:839: DeprecationWarning: There is no current event loop
    loop = asyncio.get_event_loop()

* Add python 3.10 to matrix

Co-authored-by: rpkak <rpkak@users.noreply.github.com>
3 years ago
Josh Cohen e71fcb952e
Fix `job.cancel` to remove job from registries if not in queue (#1564)
* Fix `job.cancel` to remove job from registiries if not in queue

* Remove old queue remove call

* Block the ability to cancel job that are already canceled

* Fix py35 compat

* Rename helper method
3 years ago
Josh Cohen a3fba1ca1f
Add missing functionality for CanceledJobRegistry (#1560) 3 years ago
Selwin Ong cc70cacc1b
Add more tolerance to scheduler heartbeat (#1555) 3 years ago
Selwin Ong c556106a38
Main worker should use zadd(xx=True) to update heartbeat. (#1550) 3 years ago
Josh Cohen bac58f24ca
Add option to enqueue a job's dependents when canceling (#1549)
* Add option to enqueue a jobs dependents when canceling

* Address @selwin's review
3 years ago
Josh Cohen b80045d615
Respect serializer (#1538)
* Add serializer where missing in code

* Fix cli

* Pass option to command

* Add tests for serializer option

* Merge branch 'master' into respect-serializer
- Update enqueue cli to resp. serializer

* Address @selwin's review
3 years ago
Selwin Ong 246d52b977
job.cancel() puts job into CanceledJobRegistry. (#1546)
* job.cancel() puts job into CanceledJobRegistry.

* Improve test coverage
3 years ago
Selwin Ong e8ec07ed61
Minor changes (#1544)
* Added test for job timeout

* Added more debugging statements
3 years ago
Selwin Ong f9ddf7dca5
Requeueing job should clear job.exc_info (#1543) 3 years ago
Paul Spooren 63abea1522
job: add get_meta() function (#1536)
* job: add get_meta() function

The newly introduced function returns meta data stored for the job. This
is required since job.meta stays an empty dict until the job is
finished or failed.

With the new function it's possible to store arbiatraty states/stages of
the job and allow the user to track progress. A long running job may
return custom stages like `downloading_data`, `unpacking_data`,
`processing_data`, etc.

This may allow better interfaces since users can track progress.

Signed-off-by: Paul Spooren <mail@aparcar.org>

* docs: add missing `refresh` arg to get_status()

This was previously missing.

Signed-off-by: Paul Spooren <mail@aparcar.org>
3 years ago
rpkak d41f60b906
Allows enqueueing by the cli (#1466)
* Allows enqueueing by the cli

#372

* schedule support

* `_` to `-`

* fix flake8

* echo job-id

* Some improvements

 - Description as in python jobs
 - return result
 - quiet mode
 - allows `--boolean` and `--integer`
 - raises errors if not used correctly

* added tests

* add schedule tests

* add retry test

* use click exceptions

* add error test

* add job_func test

* change messages

https://github.com/rq/rq/pull/1466#discussion_r640211128
https://github.com/rq/rq/pull/1466#discussion_r640210850

* Use different format for arguments

View https://github.com/rq/rq/pull/1466#discussion_r650510889

* Add file support

Usage: @filename

* ast.literal_eval support with `#` instead of `:`

* func -> function

Makes error messages more readable

* click Error

* print function string

* add docs

* increase seconds in test

* Update `parse_function_arg`

Add `ParsingMode` enum (https://github.com/rq/rq/pull/1466#discussion_r656676114)
Change error messages (https://github.com/rq/rq/pull/1466#discussion_r656676800, https://github.com/rq/rq/pull/1466#discussion_r656677082)

* `#` to `%`

`#` is the letter for a comment in bash

* Add some tests

(https://github.com/rq/rq/pull/1466#discussion_r656674539, https://github.com/rq/rq/pull/1466#discussion_r656676543)

* Add some tests

* docs: Add some examples

* catch all literal_eval exceptions

There are some edge cases with other exceptions

* remove job_func
(https://github.com/rq/rq/pull/1466#pullrequestreview-690110118)

* edit docs

https://github.com/rq/rq/pull/1466#pullrequestreview-695758691

* format examples

* format examples

`queue.enqueue(path.to.func, args=['abc'])` to `queue.enqueue(path.to.func, 'abc')`

https://github.com/rq/rq/pull/1466#discussion_r673615464

* add examples

https://github.com/rq/rq/pull/1466#discussion_r673658933

* add doc test

https://github.com/rq/rq/pull/1466#discussion_r673659124

* Update index.md

* Update test_cli.py

* Update test_cli.py

* Add version info

Co-authored-by: rpkak <rpkak@users.noreply.github.com>
3 years ago
rpkak 9737ffa95f
Make unstable tests more stable (#1523)
Co-authored-by: rpkak <rpkak@users.noreply.github.com>
3 years ago
Bo Bayles adc03e8119
Use result_ttl for synchronous queues (#1510) 3 years ago
Adrian Sadłocha caa4efc35b
Extract `Job.get_call_string` logic to `utils.get_call_string` (#1499)
* Extract `Job.get_call_string` logic to `utils.get_call_string`

* Remove an outdaded comment

* Move `truncate_long_string` to `utils`

* Remove `truncate` parameter in `get_call_string`

* Add a test for `get_call_string`

* Move `truncate_long_string` to module's top level

* Add a test case for `truncate_long_string` suite
4 years ago
Selwin Ong 5590aab458
Success and failure callbacks (#1480)
* Support enqueueing with on_success_callback

* Got success callback to execute properly

* Added on_failure callback

* Document success and failure callbacks

* More Flake8 fixes

* Mention that options can also be passed in via environment variables

* Increase coverage on test_callbacks.py
4 years ago
Josh Cohen 591f11bcc3
Ensure pipeline in multi mode after dep setup (#1498) 4 years ago
Josh Cohen 456743b225
Make `Queue.enqueue`, `Queue.enqueue_call`, `Queue.enqueue_at``Queue.parse_args` accept `pipeline` arg, add `Queue.enqueue_many` method (#1485)
* Make `enqueue_*` and `Queue.parse_args` accept `pipeline` arg

* undo bad docs

* Fix lints in new code

* Implement enqueue_many, refactor dependency setup out to method

* Make name consistant

* Refactor enqueue_many, update tests,  add docs

* Refactor out enqueueing from dependency setup

* Move new docs to queue page

* Fix section header

* Fix section header again

* Update rq version to 1.9.0
4 years ago
rpkak d0cea42fbb
Docker (#1471)
* rqtest -> tests folder

* worker Dockerfile to repo folder

* Dockerfile: install rq from file next to it

* add docker workflow
4 years ago
Cyrille Lavigne 6fc9454675
Handle deserializing failures gracefully (#1428)
* adds unit test for a deserialization error

This tests that deserialization exceptions are properly logged, and fails in
the manner described in #1422 .

* Catch deserializing errors in Worker.handle_exception()

This fixes #1422 , and makes

tests/test_worker.py::TestWorker::test_deserializing_failure_is_handled

pass.

* made unit test less specific

This is required to get the test to pass under other serializers / other
python versions.

* Added generic DeserializationError

* switched ValueError to DeserializationError in a test

The changed test is creating an invalid job, which now raises
DeserializationError when data is accessed, as opposed to ValueError.
4 years ago
rpkak 73d0210d65
Add environment variables for connection (#1472)
* Add environment variables for connection

Use same args as in config file

#1342

* add test

* add suggestions

https://github.com/rq/rq/pull/1472#discussion_r640205865
https://github.com/rq/rq/pull/1472#discussion_r640206565

* remove unused import
4 years ago
Selwin Ong 5b5cfdf9ab
Jobs that get cleaned up should also be retried (#1467) 4 years ago
Ashoka Lella d333d20914
Make RQScheduler work with a serializer (#1455)
* Scheduler with a serializer

* test with metadata

Co-authored-by: alella <ashoka.lella@factset.com>
4 years ago
MyrikLD efb24161ab
Replace enum function with internal Enum class (#1459)
* Removed deprecated (object) inheritance
Add new py38,py39 versions to tox, removed deprecated py27,py34
Replace enum internal function with Enum class

* fix
4 years ago
pwws 40b90946a7
bugfix: Allow using staticmethods as jobs (#1458) 4 years ago
Omer Lachish 76ac0afbcd
Cleanup zombie worker leftovers as part of StartedJobRegistry's cleanup() (#1372)
* cleanup jobs that are not really running due to zombie workers

* remove registry entries for zombie jobs

* return only the job ids on cleanup

* test zombie job cleanup

* format code

* rename variable to explain that second element in tuple is expiry, not score

* remove worker_key

* detect zombie jobs using old heartbeats

* reuse get_expired_job_ids

* set score using current_timestamp

* test idle jobs using stale heartbeats

* extract timeout into variable

* move heartbeats into StartedJobRegistry

* use registry.heartbeat in tests

* remove heartbeats when job removed from StartedJobRegistry

* remove idle and expired jobs from both wip and heartbeats set

* send heartbeat_ttl to registry.add

* typo

* revert everything 😶

* only keep job heartbeats as score (and get rid of job timeouts as scores

* calculate heartbeat_ttl in an overrideable function + override it in SimpleWorker + move storing StartedJobRegistry scores to job.heartbeat()

* set heartbeat to monitoring interval for infinite timeouts

* track elapsed_execution_time as part of worker

* reset current job working time when work on a job is done

* persisting the job working time as part of monitoring
4 years ago
Biel Cardona 08ef54dcf4
Workers dequeuing jobs from queues using both Round-Robin and Random strategies (#1425)
* implemented round-robin and random access to queues

* added tests for RoundRobinQueue

* reverted change in gitignore

* removed linebreak

* added tests for random queues

* added documentation for round robin and random queues

* moved round robin strategy to worker

* reverted changes to queue.py

* reverted changes to workers.md

* reverted changes to test_queue

* added tests for RoundRobinWorker and RandomWorker

* added doc for round robin and random workers

* removed f-strings for backward compatibility

* corrected a mistake

* minor changes (code style)

* now using _ordered_queues instead of queues for reordering queues
4 years ago
Daniel Alley fc7940c77b
Add a "stopped" JobStatus and the machinery to properly handle it (#1394)
* Add a "stopped" JobStatus and the machinery to properly handle it

fixes #1389

* Apply requested changes
4 years ago
MyrikLD 14ca7881e4
Add runner for asyncio tasks (#1405)
* add asyncio runner

* add asyncio runner

* fix for old version

* fix tests
4 years ago
skieffer 59d1b40d14
Multidependencies (#1397)
* Also accept lists and tuples as value of `depends_on`.

* The elements of the lists/tuples may be either Jobs or Job IDs.
* A single Job / Job ID is still accepted as well.

* Represent _all_ job dependencies in `Job.to_dict()`.

We now represent the entire list, instead of just the first element.

* Fix some doctext regarding plurality of dependencies.

* Add unit tests for job dependencies.

* One unit test establishes a pattern for checking execution order as affected by dependencies.

* Another unit test applies this pattern to the new capability to name multiple dependencies.

* Add unit test for new `depends_on` input formats.

Also test that these are properly persisted.

* Repair `Job.restore()`.

Need to convert bytes back to strings when reloading `dependency_ids`.

* Maintain backwards compat. in `Job.to_dict()`.

Keep the old `dependency_id` (singular) key.

* Provide coverage for new test fixture.

* Simplify some code.

Cut some superfluous `as_text()` calls left over from an earlier commit.

* Check for `dependency_id` in `Job.restore()` for backwd. compat.

Also eliminate use of `as_text()` here, in favor of `.decode()`.

* Switch to snake case instead of camel case.

* Eliminate some usages of `as_text()`.

Also cut some `print` statements.

* Cleanup.

* Accept arbitrary iterables for `Job`'s `depends_on` kwarg.

Instead of requiring a list or tuple, we now make use of `ensure_list()`.

* Add test fixtures.

* Provide a system to get two workers working simultaneously, using `multiprocessing`.
* Define a simple job that just says whether its dependencies are met.
* In `rpush`, make an option to record the name of the worker.

* Improve unit tests on execution order with dependencies.

These now actually have two workers going, which makes a more thorough test.

* Add unit test examining `Job.dependencies_are_met()` at execution time.

* Redesign dependency execution order unit tests.

* Simplify assertions.

* Improve doctext and formatting.

* Move fixture tests to new, dedicated module `test_fixtures.py`.

* Use `enqueue` instead of `enqueue_call` in new unit tests.
4 years ago
f0cker efe703214e
Added --serializer option to cli, finishing off PR #1381 and fix #1357 (#1395)
* Added --serializer option to cli, finishing off PR #1381 and fix #1357

* Update documentation

* Update documentation

* Modified help message

Co-authored-by: f0cker <dturner@trustwave.com>
4 years ago
Adda Satya Ram 11c8631921
Add exception to catch redis connection failure to retry after wait time (#1387)
* add exception catch for redis connection failure

* Add test for connection recovery

* add exponential backoff

* limit worker max connection wait time to 60 seconds

* fix undefined class variable

* fix string formatting issue while printing error log

* cap max connection wait time:better code style

Co-authored-by: corynezin <cory.nezin@gmail.com>
4 years ago
AxeOfMen 709043989a
clean_worker_registry cleans in batches to prevent submitting too muc… (#1390)
* clean_worker_registry cleans in batches to prevent submitting too much data to redis at once when there are a large number of invalid keys

* Address code review comments

Rename MAX_REMOVABLE_KEYS to MAX_KEYS

* Fix tests

Co-authored-by: Joel Harris <combolations@gmail.com>
4 years ago
JackBoreczky 016da14723
Fix custom serializer in job fetches (#1381)
* Ensure that the custom serializer defined is passed into the job fetch calls

* add serializer as argument to fetch_many and dequeue_any methods

* add worker test for custom serializer

* move json serializer to serializers.py
4 years ago
BobReid 75a610bd4d
Fix RQScheduler when run with SSL connection (#1383)
* Quick and dirty set up of SSL

* copy connection kwargs in scheduler

* fix

* chmod the cert

* Skip SSL tests in CI
4 years ago