493 Commits (5f7ed6970d4eafeb047f30e61c2399cdd6559434)

Author SHA1 Message Date
eswolinsky3241 5119716911
Persist worker_name after job is finished (#1730)
* Persist worker_name after job is finished

Persisting the worker_name on the job object in Redis would allow for debugging and
analyzing logs from the worker

* Remove redundent job.save() method call

* Remove check for null worker

Now that worker name is persisted after job finishes or fails,
no need to assert that worker name is None

* Change github runner to Ubuntu 20.04

* Change github runner to Ubuntu 20.04
2 years ago
Selwin Ong f39554dbe4
Make test_commands pass on MacOS (#1735) 2 years ago
Selwin Ong 0691b4d46e
Multiple results using Redis Streams (#1725)
* WIP job results

* Result can now be saved

* Successfully saved and restored result

* result.save() should accept pipeline

* Successful results are saved

* Failures are now saved properly too.

* Added test for Result.get_latest()

* Checkpoint

* Got Result.all() to work

* Added Result.count(), Result.delete()

* Backward compatibility for job.result and job.exc_info

* Added some typing

* More typing stuff

* Fixed typing in job.py

* More typing updates

* Only keep the last 10 results

* Documented job.results()

* Got results test to pass

* Don't run test_results.py on Redis server < 5.0

* Fixed mock import on some Python versions

* Remove Redis 3 from test matrix

* Jobs should never use the new Result implementation if server is < 5.0

* Results should only be created is Redis stream is supported.

* Added back Redis 3 to test matrix

* Fixed job.supports_redis_streams

* Fixed worker test

* Updated docs.
2 years ago
lowercase00 375ace1747
Typing (#1698)
* Gitignore Venv + VScode

* Add Typings, Add Test to Makefile

* Fix, More typing, Redis Pipeline specific type

* More types

* Fix 3.7- Typing Compat, Add Tox Envs, Tests Dockerfile

* fix listindex error (#1700)

* More docstrings

* More Types

* Fix Typo on Dependency

* Last Types

Co-authored-by: Burak Yılmaz <46003469+yilmaz-burak@users.noreply.github.com>
2 years ago
Jahn Thomas Fidje 8404385592
Add feature to enqueue dependents at the front of queues (#1696)
* Add feature to enqueue dependents at the front of queues

* Add documentation for the Dependency(enqueue_at_front=...) parameter

* docs: Add `enqueue_at_front` to list of parameters for Dependency

* test: Update dependency test to not rely on Redis ordering

* refactor: Save enqueue_at_front boolean in job.meta instead of separate instance attr

* fix: Made enqueue_at_front an instance attribute instead of putting it inside meta
2 years ago
Selwin Ong 8e3283dab3
Job should not be enqueued if dependency is canceled (#1695)
* Fix job.dependencies_are_met() if dependency is canceled

* Slightly better test coverage on dependencies_are_met()

* Fixed job.cancel(enqueue_dependent=True)
2 years ago
Yang Yang 9db728921d
Improve the lint situation (#1688)
* Move common flake8 options into config file

Currently --max-line-length being specified in two places. Just use the
existing value in the config file as the source of truth.

Move --count and --statistics to config file as well.

* Fix some lints
2 years ago
Selwin Ong bc5bb1fba2
Revert "Jobs that are run synchronously should always raise an exception" (#1682)
This reverts commit 0d21e714c33cfefba51a8c2b714a735bcad1264d.
2 years ago
Selwin Ong 5b95725dc4
Dependency with failures (#1681)
* added Dependency class with allow_failures

* Requested changes

* Check type before setting `job.dependency_allow_fail` within `Job.create`

* Set `job.dependency_allow_fail` within `Job.create`

* Added test to ensure persistence of `dependency_allow_fail`

* Removed typing and allow mixed list of ints and Job objects

* Convert dependency_allow_fail boolean to integer during serialization to avoid redis DataError

* Updated `test_multiple_dependencies_are_accepted_and_persisted` test to include `Dependency` cases

* Adding placeholder test to test actual behavior of new `Dependency` usage in `depends_on`

* Updated `test_job_dependency` to include cases using `Dependency`

* Added dependency_allow_fail logic to `Job.restore`

* Renamed `dependency_allow_fail` to a simpler `allow_failure`

* Update docs to add section about the new `Dependency` class and use-case

* Updated `Job.dependencies_are_met` logic to take `FAILED` and `STOPPED` jobs into account when `allow_failure=True`

* Updated `test_job_dependency` test. Still failing with `Dependency` case.

* Fix `allow_failure` type coercion in `Job.restore`

* Re-arrange tests, so default `Dependency.allow_failure` is before explicit `allow_failure=True`

* Fixed Dependency, so it works correctly when allow_failure=True

* Attempt to execute pipeline prior to queueing a failed job's dependents. test_create_and_cancel_job_enqueue_dependents_in_registry test now passes.

* Added `Depedency` test utilizing multiple dependencies

* Removed irrelevant on_success and on_failure keyword arguments in example

* Replaced use of long_running_job

* Add test to verify `Dependency.jobs` contraints

* Suppress connection error in handle_job_failure

* test_dependencies have passed

* All tests pass if enqueue_dependents called without pipeline.watch()

* All tests now pass

* Removed print statements

* Cleanup Dependency implementation

* Renamed job.allow_failure to job.allow_dependency_failures

Co-authored-by: mattchan <mattchan@tencent.com>
Co-authored-by: Mike Hill <mhilluniversal@gmail.com>
2 years ago
Tim Gates d82af1469f
docs: Fix a few typos (#1679)
There are small typos in:
- docs/docs/exceptions.md
- docs/docs/jobs.md
- rq/queue.py
- tests/fixtures.py
- tests/test_job.py

Fixes:
- Should read `slightly` rather than `slighty`.
- Should read `requeuing` rather than `requeueing`.
- Should read `implementers` rather than `implementors`.
- Should read `definition` rather than `defition`.
- Should read `canceled` rather than `canceld`.

Signed-off-by: Tim Gates <tim.gates@iress.com>
2 years ago
Selwin Ong 145884fcd5
When stopped jobs are deleted, they should also be removed from FailedJobRegistry. (#1677) 2 years ago
Hugo d5175c38da
Drop python2-specific syntax (#1674)
* Drop syntax required only for Python 2

* Drop python2-style super() calls

Co-authored-by: Selwin Ong <selwin.ong@gmail.com>
2 years ago
Hugo 61a4a1720b
Use unittest.mock instead of mock (#1673)
This module has been included in Python itself since 3.3.

Fixes: https://github.com/rq/rq/issues/1646
2 years ago
Selwin Ong 87bacb650e
Jobs that are run synchronously should always raise an exception (#1671) 3 years ago
Michael Hill f4602d30d5
Cross platform simple worker (#1629)
* Added CrossPlatformDeathPenalty that doesn't rely on signals

* Updated `SimpleWorker`'s `death_penalty_class` to utilize `CrossPlatformDeathPenalty` to allow use on Windows machines

* Changed `CrossPlatformDeathPenalty` to `TimerDeathPenalty`

* Removed overridden `death_penalty_class` in `SimpleWorker` until feature matures

* Added section in testing.md explaining how to utilize `SimpleWorker` on Windows OS

* Replaced usage of chatting with .format for python 3.5 compatibility

* Add tests for new timeout feature

* Explicitly set defaults.CALLBACK_TIMEOUT

* Implemented cross-thread method of raising errors by using ctypes

* Finished writing tests for new TimerDeathPenalty
3 years ago
Neptalí Piña 34e2427f67
job decorator allow on_failure and on_success parameters (#1626) (#1628)
Co-authored-by: neptali <neptali@plerk.io>
3 years ago
Xavier Fernandez cd17d17d71
rq.worker: remove useless set_state call in horse (#1618)
* rq.worker: remove useless set_state call in horse

The state should already have been set properly by the worker in
`execute_job`

`prepare_job_execution` is only called by `perform_job` which should only be
called by `main_work_horse`/`fork_work_horse` (themselves only called by `execute_job`).
Let `execute_job` do the bookkeeping.

* worker: update SimpleWorker's state in execute_job
3 years ago
th3hamm0r c5a1ef1734
Fixed wrong keys used to WATCH dependencies (#1605)
This bug has opened a lot of possible race-conditions, since the
watch-logic from redis did not fail anymore, if dependencies have been
changed in parallel.
3 years ago
Eric Feldman 93f34c796f
call callbacks and prepare job when running sync (#1599)
* call signals

* fix lines

* add tests and fix faliure status bug

* bump version
3 years ago
Selwin Ong 0147b30f2b
Fixes a bug that causes leftover job keys when result_ttl=0 (#1591)
* Fixes a bug that causes leftover job keys when result_ttl=0

* Fixed a buggy worker.maintain_heartbeats() behavior

* Fixed a bug in worker.maintain_heartbeats().
3 years ago
rpkak 6692587f74
Python 3.10 compatibility (#1563)
* use shutil.get_terminal_size instead of click.get_terminal_size()

resolves warning:

rq/cli/helpers.py:107: DeprecationWarning: 'click.get_terminal_size()' is deprecated and will be removed in Click 8.1. Use 'shutil.get_terminal_size()' instead.
  termwidth, _ = click.get_terminal_size()

* remove StrictVersion from rq

* asyncio.get_event_loop -> asyncio.new_event_loop()

resolves warning:

tests/test_job.py::TestJob::test_create_job_with_async
  rq/rq/job.py:839: DeprecationWarning: There is no current event loop
    loop = asyncio.get_event_loop()

* Add python 3.10 to matrix

Co-authored-by: rpkak <rpkak@users.noreply.github.com>
3 years ago
Josh Cohen e71fcb952e
Fix `job.cancel` to remove job from registries if not in queue (#1564)
* Fix `job.cancel` to remove job from registiries if not in queue

* Remove old queue remove call

* Block the ability to cancel job that are already canceled

* Fix py35 compat

* Rename helper method
3 years ago
Josh Cohen a3fba1ca1f
Add missing functionality for CanceledJobRegistry (#1560) 3 years ago
Selwin Ong cc70cacc1b
Add more tolerance to scheduler heartbeat (#1555) 3 years ago
Selwin Ong c556106a38
Main worker should use zadd(xx=True) to update heartbeat. (#1550) 3 years ago
Josh Cohen bac58f24ca
Add option to enqueue a job's dependents when canceling (#1549)
* Add option to enqueue a jobs dependents when canceling

* Address @selwin's review
3 years ago
Josh Cohen b80045d615
Respect serializer (#1538)
* Add serializer where missing in code

* Fix cli

* Pass option to command

* Add tests for serializer option

* Merge branch 'master' into respect-serializer
- Update enqueue cli to resp. serializer

* Address @selwin's review
3 years ago
Selwin Ong 246d52b977
job.cancel() puts job into CanceledJobRegistry. (#1546)
* job.cancel() puts job into CanceledJobRegistry.

* Improve test coverage
3 years ago
Selwin Ong e8ec07ed61
Minor changes (#1544)
* Added test for job timeout

* Added more debugging statements
3 years ago
Selwin Ong f9ddf7dca5
Requeueing job should clear job.exc_info (#1543) 3 years ago
Paul Spooren 63abea1522
job: add get_meta() function (#1536)
* job: add get_meta() function

The newly introduced function returns meta data stored for the job. This
is required since job.meta stays an empty dict until the job is
finished or failed.

With the new function it's possible to store arbiatraty states/stages of
the job and allow the user to track progress. A long running job may
return custom stages like `downloading_data`, `unpacking_data`,
`processing_data`, etc.

This may allow better interfaces since users can track progress.

Signed-off-by: Paul Spooren <mail@aparcar.org>

* docs: add missing `refresh` arg to get_status()

This was previously missing.

Signed-off-by: Paul Spooren <mail@aparcar.org>
3 years ago
rpkak d41f60b906
Allows enqueueing by the cli (#1466)
* Allows enqueueing by the cli

#372

* schedule support

* `_` to `-`

* fix flake8

* echo job-id

* Some improvements

 - Description as in python jobs
 - return result
 - quiet mode
 - allows `--boolean` and `--integer`
 - raises errors if not used correctly

* added tests

* add schedule tests

* add retry test

* use click exceptions

* add error test

* add job_func test

* change messages

https://github.com/rq/rq/pull/1466#discussion_r640211128
https://github.com/rq/rq/pull/1466#discussion_r640210850

* Use different format for arguments

View https://github.com/rq/rq/pull/1466#discussion_r650510889

* Add file support

Usage: @filename

* ast.literal_eval support with `#` instead of `:`

* func -> function

Makes error messages more readable

* click Error

* print function string

* add docs

* increase seconds in test

* Update `parse_function_arg`

Add `ParsingMode` enum (https://github.com/rq/rq/pull/1466#discussion_r656676114)
Change error messages (https://github.com/rq/rq/pull/1466#discussion_r656676800, https://github.com/rq/rq/pull/1466#discussion_r656677082)

* `#` to `%`

`#` is the letter for a comment in bash

* Add some tests

(https://github.com/rq/rq/pull/1466#discussion_r656674539, https://github.com/rq/rq/pull/1466#discussion_r656676543)

* Add some tests

* docs: Add some examples

* catch all literal_eval exceptions

There are some edge cases with other exceptions

* remove job_func
(https://github.com/rq/rq/pull/1466#pullrequestreview-690110118)

* edit docs

https://github.com/rq/rq/pull/1466#pullrequestreview-695758691

* format examples

* format examples

`queue.enqueue(path.to.func, args=['abc'])` to `queue.enqueue(path.to.func, 'abc')`

https://github.com/rq/rq/pull/1466#discussion_r673615464

* add examples

https://github.com/rq/rq/pull/1466#discussion_r673658933

* add doc test

https://github.com/rq/rq/pull/1466#discussion_r673659124

* Update index.md

* Update test_cli.py

* Update test_cli.py

* Add version info

Co-authored-by: rpkak <rpkak@users.noreply.github.com>
3 years ago
rpkak 9737ffa95f
Make unstable tests more stable (#1523)
Co-authored-by: rpkak <rpkak@users.noreply.github.com>
3 years ago
Bo Bayles adc03e8119
Use result_ttl for synchronous queues (#1510) 3 years ago
Adrian Sadłocha caa4efc35b
Extract `Job.get_call_string` logic to `utils.get_call_string` (#1499)
* Extract `Job.get_call_string` logic to `utils.get_call_string`

* Remove an outdaded comment

* Move `truncate_long_string` to `utils`

* Remove `truncate` parameter in `get_call_string`

* Add a test for `get_call_string`

* Move `truncate_long_string` to module's top level

* Add a test case for `truncate_long_string` suite
4 years ago
Selwin Ong 5590aab458
Success and failure callbacks (#1480)
* Support enqueueing with on_success_callback

* Got success callback to execute properly

* Added on_failure callback

* Document success and failure callbacks

* More Flake8 fixes

* Mention that options can also be passed in via environment variables

* Increase coverage on test_callbacks.py
4 years ago
Josh Cohen 591f11bcc3
Ensure pipeline in multi mode after dep setup (#1498) 4 years ago
Josh Cohen 456743b225
Make `Queue.enqueue`, `Queue.enqueue_call`, `Queue.enqueue_at``Queue.parse_args` accept `pipeline` arg, add `Queue.enqueue_many` method (#1485)
* Make `enqueue_*` and `Queue.parse_args` accept `pipeline` arg

* undo bad docs

* Fix lints in new code

* Implement enqueue_many, refactor dependency setup out to method

* Make name consistant

* Refactor enqueue_many, update tests,  add docs

* Refactor out enqueueing from dependency setup

* Move new docs to queue page

* Fix section header

* Fix section header again

* Update rq version to 1.9.0
4 years ago
rpkak d0cea42fbb
Docker (#1471)
* rqtest -> tests folder

* worker Dockerfile to repo folder

* Dockerfile: install rq from file next to it

* add docker workflow
4 years ago
Cyrille Lavigne 6fc9454675
Handle deserializing failures gracefully (#1428)
* adds unit test for a deserialization error

This tests that deserialization exceptions are properly logged, and fails in
the manner described in #1422 .

* Catch deserializing errors in Worker.handle_exception()

This fixes #1422 , and makes

tests/test_worker.py::TestWorker::test_deserializing_failure_is_handled

pass.

* made unit test less specific

This is required to get the test to pass under other serializers / other
python versions.

* Added generic DeserializationError

* switched ValueError to DeserializationError in a test

The changed test is creating an invalid job, which now raises
DeserializationError when data is accessed, as opposed to ValueError.
4 years ago
rpkak 73d0210d65
Add environment variables for connection (#1472)
* Add environment variables for connection

Use same args as in config file

#1342

* add test

* add suggestions

https://github.com/rq/rq/pull/1472#discussion_r640205865
https://github.com/rq/rq/pull/1472#discussion_r640206565

* remove unused import
4 years ago
Selwin Ong 5b5cfdf9ab
Jobs that get cleaned up should also be retried (#1467) 4 years ago
Ashoka Lella d333d20914
Make RQScheduler work with a serializer (#1455)
* Scheduler with a serializer

* test with metadata

Co-authored-by: alella <ashoka.lella@factset.com>
4 years ago
MyrikLD efb24161ab
Replace enum function with internal Enum class (#1459)
* Removed deprecated (object) inheritance
Add new py38,py39 versions to tox, removed deprecated py27,py34
Replace enum internal function with Enum class

* fix
4 years ago
pwws 40b90946a7
bugfix: Allow using staticmethods as jobs (#1458) 4 years ago
Omer Lachish 76ac0afbcd
Cleanup zombie worker leftovers as part of StartedJobRegistry's cleanup() (#1372)
* cleanup jobs that are not really running due to zombie workers

* remove registry entries for zombie jobs

* return only the job ids on cleanup

* test zombie job cleanup

* format code

* rename variable to explain that second element in tuple is expiry, not score

* remove worker_key

* detect zombie jobs using old heartbeats

* reuse get_expired_job_ids

* set score using current_timestamp

* test idle jobs using stale heartbeats

* extract timeout into variable

* move heartbeats into StartedJobRegistry

* use registry.heartbeat in tests

* remove heartbeats when job removed from StartedJobRegistry

* remove idle and expired jobs from both wip and heartbeats set

* send heartbeat_ttl to registry.add

* typo

* revert everything 😶

* only keep job heartbeats as score (and get rid of job timeouts as scores

* calculate heartbeat_ttl in an overrideable function + override it in SimpleWorker + move storing StartedJobRegistry scores to job.heartbeat()

* set heartbeat to monitoring interval for infinite timeouts

* track elapsed_execution_time as part of worker

* reset current job working time when work on a job is done

* persisting the job working time as part of monitoring
4 years ago
Biel Cardona 08ef54dcf4
Workers dequeuing jobs from queues using both Round-Robin and Random strategies (#1425)
* implemented round-robin and random access to queues

* added tests for RoundRobinQueue

* reverted change in gitignore

* removed linebreak

* added tests for random queues

* added documentation for round robin and random queues

* moved round robin strategy to worker

* reverted changes to queue.py

* reverted changes to workers.md

* reverted changes to test_queue

* added tests for RoundRobinWorker and RandomWorker

* added doc for round robin and random workers

* removed f-strings for backward compatibility

* corrected a mistake

* minor changes (code style)

* now using _ordered_queues instead of queues for reordering queues
4 years ago
Daniel Alley fc7940c77b
Add a "stopped" JobStatus and the machinery to properly handle it (#1394)
* Add a "stopped" JobStatus and the machinery to properly handle it

fixes #1389

* Apply requested changes
4 years ago
MyrikLD 14ca7881e4
Add runner for asyncio tasks (#1405)
* add asyncio runner

* add asyncio runner

* fix for old version

* fix tests
4 years ago
skieffer 59d1b40d14
Multidependencies (#1397)
* Also accept lists and tuples as value of `depends_on`.

* The elements of the lists/tuples may be either Jobs or Job IDs.
* A single Job / Job ID is still accepted as well.

* Represent _all_ job dependencies in `Job.to_dict()`.

We now represent the entire list, instead of just the first element.

* Fix some doctext regarding plurality of dependencies.

* Add unit tests for job dependencies.

* One unit test establishes a pattern for checking execution order as affected by dependencies.

* Another unit test applies this pattern to the new capability to name multiple dependencies.

* Add unit test for new `depends_on` input formats.

Also test that these are properly persisted.

* Repair `Job.restore()`.

Need to convert bytes back to strings when reloading `dependency_ids`.

* Maintain backwards compat. in `Job.to_dict()`.

Keep the old `dependency_id` (singular) key.

* Provide coverage for new test fixture.

* Simplify some code.

Cut some superfluous `as_text()` calls left over from an earlier commit.

* Check for `dependency_id` in `Job.restore()` for backwd. compat.

Also eliminate use of `as_text()` here, in favor of `.decode()`.

* Switch to snake case instead of camel case.

* Eliminate some usages of `as_text()`.

Also cut some `print` statements.

* Cleanup.

* Accept arbitrary iterables for `Job`'s `depends_on` kwarg.

Instead of requiring a list or tuple, we now make use of `ensure_list()`.

* Add test fixtures.

* Provide a system to get two workers working simultaneously, using `multiprocessing`.
* Define a simple job that just says whether its dependencies are met.
* In `rpush`, make an option to record the name of the worker.

* Improve unit tests on execution order with dependencies.

These now actually have two workers going, which makes a more thorough test.

* Add unit test examining `Job.dependencies_are_met()` at execution time.

* Redesign dependency execution order unit tests.

* Simplify assertions.

* Improve doctext and formatting.

* Move fixture tests to new, dedicated module `test_fixtures.py`.

* Use `enqueue` instead of `enqueue_call` in new unit tests.
4 years ago