* First stab at implementating worker pool
* Use process.is_alive() to check whether a process is still live
* Handle shutdown signal
* Check worker loop done
* First working version of `WorkerPool`.
* Added test for check_workers()
* Added test for pool.start()
* Better shutdown process
* Comment out test_start() to see if it fixes CI
* Make tests pass
* Make CI pass
* Comment out some tests
* Comment out more tests
* Re-enable a test
* Re-enable another test
* Uncomment check_workers test
* Added run_worker test
* Minor modification to dead worker detection
* More test cases
* Better process name for workers
* Added back pool.stop_workers() when signal is received
* Cleaned up cli.py
* WIP on worker-pool command
* Fix test
* Test that worker pool ignores consecutive shutdown signals
* Added test for worker-pool CLI command.
* Added timeout to CI jobs
* Fix worker pool test
* Comment out test_scheduler.py
* Fixed worker-pool in burst mode
* Increase test coverage
* Exclude tests directory from coverage.py
* Improve test coverage
* Renamed `Pool(num_workers=2) to `Pool(size=2)`
* Revert "Renamed `Pool(num_workers=2) to `Pool(size=2)`"
This reverts commit a1306f89ad0d8686c6bde447bff75e2f71f0733b.
* Renamed Pool to WorkerPool
* Added a new TestCase that doesn't use LocalStack
* Added job_class, worker_class and serializer arguments to WorkerPool
* Use parse_connection() in WorkerPool.__init__
* Added CLI arguments for worker-pool
* Minor WorkerPool and test fixes
* Fixed failing CLI test
* Document WorkerPool
* Added BaseWorker class
* Moved logging related functions to logutils
* Remove uneeded colorize function
* Updated worker.get_current_job_id() to not fail when job ID is None
* job.restore() shouldn't crash if is not present
* Fix various as_text() related crashes
* Added parse_connection function
* feat: allow custom connection pool class (#1885)
* Added test for SSL
---------
Co-authored-by: Cyril Chapellier <tchapi@users.noreply.github.com>
* consolidate job failure callback execution
* fix
* success callback as well
* merge fix
* create Callback class and change how callbacks serde
* deprecation
* add tests
* string format
* pr fix
* revert serialization changes
* fix timeout typing
* add documentation
* add test
* fix a bug
* fix tests
* move job heartbeat call to worker and make sure there is always a callback timeout
* fix test
* fix test
* test: check dependencies when enqueue via Queue.enqueue_job()
Signed-off-by: Simó Albert i Beltran <sim6@bona.gent>
* fix: check dependencies when enqueue via Queue.enqueue_job()
Signed-off-by: Simó Albert i Beltran <sim6@bona.gent>
Co-authored-by: Selwin Ong <selwin.ong@gmail.com>
---------
Signed-off-by: Simó Albert i Beltran <sim6@bona.gent>
Co-authored-by: Selwin Ong <selwin.ong@gmail.com>
* Updated arguments passed to the Sentinel Object when created from the settings
- added `USERNAME` key
- added `CONNECTION_KWARGS` key to allow passing additionals arguments to the Redis connections
- updated the documentation
* added missing comma
* tests(helpers): Added tests for Sentinel
---------
Co-authored-by: Joachim Burket <joachim.burket@hopitalvs.ch>
* Add started job registry cleanup job failure callback call
* WIP - need to fix test
* fix test
* rename, tests and docs
* better log message
* use class name
* Update registry.py
* Fix TimerDeathPenalty not properly handling negative/infinite timeout
* revert back to using exc_info
---------
Co-authored-by: Marcus <marcus@us2.ai>
* test: Dependency list in depends_on
Signed-off-by: Simó Albert i Beltran <sim6@bona.gent>
* fix: Dependency list in depends_on
Signed-off-by: Simó Albert i Beltran <sim6@bona.gent>
---------
Signed-off-by: Simó Albert i Beltran <sim6@bona.gent>
* fix accessing None when dequeued result is None (burst=True, or timeout=None)
* add a test
* implement + tests
* fix if
* adjust test
* merge
* test
* test
* merge master
* take max_idle_time into account for dequeue_timeout
* refactor a bit
* potential bug fix
* tests
* math.ceil
* buffer tests
* test: queue.result_ttl=-1
Signed-off-by: Simó Albert i Beltran <sim6@bona.gent>
* test: queue.result_ttl=0
Signed-off-by: Simó Albert i Beltran <sim6@bona.gent>
* fix: queue.result_ttl=-1
Signed-off-by: Simó Albert i Beltran <sim6@bona.gent>
---------
Signed-off-by: Simó Albert i Beltran <sim6@bona.gent>
* treewide: apply black style
This PR applied the black code style, adds it to the CI and README. The
changes look big, however no functional changed are applied at all.
The line length is set to 120, which is different from black
recommendation of 88. I would suggest to stick to the 88 recommendation
and adapt the flake8 check.
Add the `-S` parameter to `black` to keep single quotes.
Signed-off-by: Paul Spooren <mail@aparcar.org>
* README: add black badge
Help people to find the used code style.
Signed-off-by: Paul Spooren <mail@aparcar.org>
* CI: check codestyle via black
Automatically run the CI and check that the code style is still black.
Signed-off-by: Paul Spooren <mail@aparcar.org>
---------
Signed-off-by: Paul Spooren <mail@aparcar.org>
When running on debug mode, some debug logs were
using colorizers on lists, which would raise an unhandled exception
on the worker. The same happened for a debug log that was
using colors on a response from Redis (bytes).
* Enhanced Redis Connection Reliability
The Redis connection may fail for several reasons. As the connection can be
(1) explicitly passed to the worker or (2) implicity set, this will improve the
Connection configuration by setting a timeout to the socket, and adding
an ExponentialBackoff Retry logic.
* Simpler Connection logic
* Add simple retry logic to Redis Connection Error
* Make retry exponential, add keepalive & socket_connect_timeout
* Handles configuration on Redis' connection pool
* Simplifies timeout exception logic
* Fix burst bug, add test
* Add docs related to `socket_timeout`, improve compatibility with older RedisPy versions
* Fixes
* New timeout private method
* Fix timeout
* Improve docstrings on `connections`
* Enhanced Job methods docstrings & Serialization Protocol
This adds docstrings to all Job methods in a standard format.
It also implements a `serializer` protocol.
* Excludes `Protocol` (keeping compatibility with < 3.8)
* Add docstrings & type annotation to the `job` decorator
* Docstrings for the `defaults` vars
* Add deprecation warning to Connection context manager
* Fix Types
* Moves the `compact` helper to `utils`
There is a helper funcion that excludes `None` values
from a list. This was being declared both in the Queue
and in the worker. This centralizes this helper in the `utils`
Importing
* Fix Type Annotation
* Remove unused code from compat module
* Remove unused dictconfig
* Remove total_ordering compat layer
* Remove compatibility layer
This completely removes the compat module. It moves utilities
functions (`as_text` and `decode_redis_hash`) to the `utils`
module, is eliminates the use of the proxies `text_type` and
`string_types`, using the `str` construct directly.
* Remove compat module
Finishes the cleaning of the compatibility module.
The last function being the `is_python_version` function
which was being used internally.
* Fix old import
* Fix Imports
* Remove Dummy (Force GH Actions)
* Fix Imports
* Organize Imports
Selwin Ong
Cyril Chapellier
Rob Hudson
nhairs
Tianci Hu Marrero
Rony Lutsky
Jochen Kupperschmidt
Simó Albert i Beltran
Jesse P. Johnson
lowercase00
Joachim Burket
Marcus Ong
amks1
Daniel M
Rony Lutsky
Paul Spooren
Oleg