7 Commits (master)

Author SHA1 Message Date
Rob Hudson ea063edf0a
Update linting configuration (#1915)
* Update linting configuration

This removes flake8 in favor of ruff, which also provides isort support, and
updates all files to be black, isort, and ruff compliant. This also adds black
and ruff checks to the tox and Github linting workflow.

* Tweak the code coverage config and calls
2 years ago
Selwin Ong 64cb1a27b9
Worker pool (#1874)
* First stab at implementating worker pool

* Use process.is_alive() to check whether a process is still live

* Handle shutdown signal

* Check worker loop done

* First working version of `WorkerPool`.

* Added test for check_workers()

* Added test for pool.start()

* Better shutdown process

* Comment out test_start() to see if it fixes CI

* Make tests pass

* Make CI pass

* Comment out some tests

* Comment out more tests

* Re-enable a test

* Re-enable another test

* Uncomment check_workers test

* Added run_worker test

* Minor modification to dead worker detection

* More test cases

* Better process name for workers

* Added back pool.stop_workers() when signal is received

* Cleaned up cli.py

* WIP on worker-pool command

* Fix test

* Test that worker pool ignores consecutive shutdown signals

* Added test for worker-pool CLI command.

* Added timeout to CI jobs

* Fix worker pool test

* Comment out test_scheduler.py

* Fixed worker-pool in burst mode

* Increase test coverage

* Exclude tests directory from coverage.py

* Improve test coverage

* Renamed `Pool(num_workers=2) to `Pool(size=2)`

* Revert "Renamed `Pool(num_workers=2) to `Pool(size=2)`"

This reverts commit a1306f89ad0d8686c6bde447bff75e2f71f0733b.

* Renamed Pool to WorkerPool

* Added a new TestCase that doesn't use LocalStack

* Added job_class, worker_class and serializer arguments to WorkerPool

* Use parse_connection() in WorkerPool.__init__

* Added CLI arguments for worker-pool

* Minor WorkerPool and test fixes

* Fixed failing CLI test

* Document WorkerPool
2 years ago
Selwin Ong f39554dbe4
Make test_commands pass on MacOS (#1735) 2 years ago
Josh Cohen b80045d615
Respect serializer (#1538)
* Add serializer where missing in code

* Fix cli

* Pass option to command

* Add tests for serializer option

* Merge branch 'master' into respect-serializer
- Update enqueue cli to resp. serializer

* Address @selwin's review
3 years ago
Daniel Alley fc7940c77b
Add a "stopped" JobStatus and the machinery to properly handle it (#1394)
* Add a "stopped" JobStatus and the machinery to properly handle it

fixes #1389

* Apply requested changes
4 years ago
Selwin Ong 492e77d86d
send_stop_job_command (#1376)
* Added send_stop_job_command().

* send_stop_job_command now accepts just connection and job_id

* Document send_job_job_command

* Updated test coverage
4 years ago
Selwin Ong a721db34b1
Workers can listen to external commands via pubsub (#1363)
* Added a way to send shutdown command via pubsub

* Added kill-horse command

* Added kill horse command

* Added send_kill_horse_command() and send_shutdown_command()

* Document worker commands
4 years ago