13 Commits (c2bec19b094bdd536d4c42f184983fabdabe7652)

Author SHA1 Message Date
Rob Hudson ea063edf0a
Update linting configuration (#1915)
* Update linting configuration

This removes flake8 in favor of ruff, which also provides isort support, and
updates all files to be black, isort, and ruff compliant. This also adds black
and ruff checks to the tox and Github linting workflow.

* Tweak the code coverage config and calls
2 years ago
Selwin Ong 64cb1a27b9
Worker pool (#1874)
* First stab at implementating worker pool

* Use process.is_alive() to check whether a process is still live

* Handle shutdown signal

* Check worker loop done

* First working version of `WorkerPool`.

* Added test for check_workers()

* Added test for pool.start()

* Better shutdown process

* Comment out test_start() to see if it fixes CI

* Make tests pass

* Make CI pass

* Comment out some tests

* Comment out more tests

* Re-enable a test

* Re-enable another test

* Uncomment check_workers test

* Added run_worker test

* Minor modification to dead worker detection

* More test cases

* Better process name for workers

* Added back pool.stop_workers() when signal is received

* Cleaned up cli.py

* WIP on worker-pool command

* Fix test

* Test that worker pool ignores consecutive shutdown signals

* Added test for worker-pool CLI command.

* Added timeout to CI jobs

* Fix worker pool test

* Comment out test_scheduler.py

* Fixed worker-pool in burst mode

* Increase test coverage

* Exclude tests directory from coverage.py

* Improve test coverage

* Renamed `Pool(num_workers=2) to `Pool(size=2)`

* Revert "Renamed `Pool(num_workers=2) to `Pool(size=2)`"

This reverts commit a1306f89ad0d8686c6bde447bff75e2f71f0733b.

* Renamed Pool to WorkerPool

* Added a new TestCase that doesn't use LocalStack

* Added job_class, worker_class and serializer arguments to WorkerPool

* Use parse_connection() in WorkerPool.__init__

* Added CLI arguments for worker-pool

* Minor WorkerPool and test fixes

* Fixed failing CLI test

* Document WorkerPool
2 years ago
Oscar Arbeláez-Echeverri 9aaceb22e6
Cache redis version in the connection object (#1742) 2 years ago
Yang Yang 9db728921d
Improve the lint situation (#1688)
* Move common flake8 options into config file

Currently --max-line-length being specified in two places. Just use the
existing value in the config file as the source of truth.

Move --count and --statistics to config file as well.

* Fix some lints
2 years ago
Hugo d5175c38da
Drop python2-specific syntax (#1674)
* Drop syntax required only for Python 2

* Drop python2-style super() calls

Co-authored-by: Selwin Ong <selwin.ong@gmail.com>
2 years ago
Hugo 61a4a1720b
Use unittest.mock instead of mock (#1673)
This module has been included in Python itself since 3.3.

Fixes: https://github.com/rq/rq/issues/1646
2 years ago
rpkak 6692587f74
Python 3.10 compatibility (#1563)
* use shutil.get_terminal_size instead of click.get_terminal_size()

resolves warning:

rq/cli/helpers.py:107: DeprecationWarning: 'click.get_terminal_size()' is deprecated and will be removed in Click 8.1. Use 'shutil.get_terminal_size()' instead.
  termwidth, _ = click.get_terminal_size()

* remove StrictVersion from rq

* asyncio.get_event_loop -> asyncio.new_event_loop()

resolves warning:

tests/test_job.py::TestJob::test_create_job_with_async
  rq/rq/job.py:839: DeprecationWarning: There is no current event loop
    loop = asyncio.get_event_loop()

* Add python 3.10 to matrix

Co-authored-by: rpkak <rpkak@users.noreply.github.com>
3 years ago
Adrian Sadłocha caa4efc35b
Extract `Job.get_call_string` logic to `utils.get_call_string` (#1499)
* Extract `Job.get_call_string` logic to `utils.get_call_string`

* Remove an outdaded comment

* Move `truncate_long_string` to `utils`

* Remove `truncate` parameter in `get_call_string`

* Add a test for `get_call_string`

* Move `truncate_long_string` to module's top level

* Add a test case for `truncate_long_string` suite
4 years ago
AxeOfMen 709043989a
clean_worker_registry cleans in batches to prevent submitting too muc… (#1390)
* clean_worker_registry cleans in batches to prevent submitting too much data to redis at once when there are a large number of invalid keys

* Address code review comments

Rename MAX_REMOVABLE_KEYS to MAX_KEYS

* Fix tests

Co-authored-by: Joel Harris <combolations@gmail.com>
4 years ago
Selwin Ong 39fb709c10
get_redis_server_version() should handle 4 digit version numbers (#1322) 4 years ago
Samuel Colvin 260fd84f51 add milliseconds into timestamps, fix #721 7 years ago
Theo ee64114e6e Fixed #870 Improved test coverage for connections.py and utils.py 7 years ago
luojiebin cd529d0ce1 Fixed issue#72 (#818)
* Added a custom exception for timeout transfer

* Added a util to transfer timeout to a united format

* Transfer timeout format when creating a queue or enqueue jobs

* Fixed typos

* Fixed bug in transfer_timeout function

* Added test for function transfer_timeout

* Updated transfer_timeout to allow uppercase unit

* Renamed function in utils
8 years ago