You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Selwin Ong 64cb1a27b9
Worker pool (#1874)
* First stab at implementating worker pool

* Use process.is_alive() to check whether a process is still live

* Handle shutdown signal

* Check worker loop done

* First working version of `WorkerPool`.

* Added test for check_workers()

* Added test for pool.start()

* Better shutdown process

* Comment out test_start() to see if it fixes CI

* Make tests pass

* Make CI pass

* Comment out some tests

* Comment out more tests

* Re-enable a test

* Re-enable another test

* Uncomment check_workers test

* Added run_worker test

* Minor modification to dead worker detection

* More test cases

* Better process name for workers

* Added back pool.stop_workers() when signal is received

* Cleaned up cli.py

* WIP on worker-pool command

* Fix test

* Test that worker pool ignores consecutive shutdown signals

* Added test for worker-pool CLI command.

* Added timeout to CI jobs

* Fix worker pool test

* Comment out test_scheduler.py

* Fixed worker-pool in burst mode

* Increase test coverage

* Exclude tests directory from coverage.py

* Improve test coverage

* Renamed `Pool(num_workers=2) to `Pool(size=2)`

* Revert "Renamed `Pool(num_workers=2) to `Pool(size=2)`"

This reverts commit a1306f89ad0d8686c6bde447bff75e2f71f0733b.

* Renamed Pool to WorkerPool

* Added a new TestCase that doesn't use LocalStack

* Added job_class, worker_class and serializer arguments to WorkerPool

* Use parse_connection() in WorkerPool.__init__

* Added CLI arguments for worker-pool

* Minor WorkerPool and test fixes

* Fixed failing CLI test

* Document WorkerPool
2 years ago
..
config_files Drop python2-specific syntax (#1674) 2 years ago
ssl_config Fix RQScheduler when run with SSL connection (#1383) 4 years ago
Dockerfile Typing (#1698) 2 years ago
__init__.py Worker pool (#1874) 2 years ago
fixtures.py Worker pool (#1874) 2 years ago
test.json Allows enqueueing by the cli (#1466) 3 years ago
test_callbacks.py Use job.return_value() instead of job.result when processing callbacks (#1801) 2 years ago
test_cli.py Worker pool (#1874) 2 years ago
test_commands.py Worker pool (#1874) 2 years ago
test_connection.py Worker pool (#1874) 2 years ago
test_decorator.py Improve the lint situation (#1688) 2 years ago
test_dependencies.py Check dependencies when enqueue via Queue.enqueue_job() (#1837) 2 years ago
test_fixtures.py Drop python2-specific syntax (#1674) 2 years ago
test_helpers.py Update arguments passed to the Sentinel Object when created from the settings (#1850) 2 years ago
test_job.py Worker pool (#1874) 2 years ago
test_queue.py Worker pool (#1874) 2 years ago
test_registry.py Consolidate job callbacks execution (#1851) 2 years ago
test_results.py [Results] Allow unserializable return values (#1888) 2 years ago
test_retry.py Make unstable tests more stable (#1523) 3 years ago
test_scheduler.py Worker pool (#1874) 2 years ago
test_sentry.py Worker pool (#1874) 2 years ago
test_serializers.py Drop python2-specific syntax (#1674) 2 years ago
test_timeouts.py Fix TimerDeathPenalty not properly handling negative/infinite timeout (#1845) 2 years ago
test_utils.py Worker pool (#1874) 2 years ago
test_worker.py Worker pool (#1874) 2 years ago
test_worker_pool.py Worker pool (#1874) 2 years ago
test_worker_registration.py Use unittest.mock instead of mock (#1673) 2 years ago