* treewide: apply black style
This PR applied the black code style, adds it to the CI and README. The
changes look big, however no functional changed are applied at all.
The line length is set to 120, which is different from black
recommendation of 88. I would suggest to stick to the 88 recommendation
and adapt the flake8 check.
Add the `-S` parameter to `black` to keep single quotes.
Signed-off-by: Paul Spooren <mail@aparcar.org>
* README: add black badge
Help people to find the used code style.
Signed-off-by: Paul Spooren <mail@aparcar.org>
* CI: check codestyle via black
Automatically run the CI and check that the code style is still black.
Signed-off-by: Paul Spooren <mail@aparcar.org>
---------
Signed-off-by: Paul Spooren <mail@aparcar.org>
When running on debug mode, some debug logs were
using colorizers on lists, which would raise an unhandled exception
on the worker. The same happened for a debug log that was
using colors on a response from Redis (bytes).
* Enhanced Redis Connection Reliability
The Redis connection may fail for several reasons. As the connection can be
(1) explicitly passed to the worker or (2) implicity set, this will improve the
Connection configuration by setting a timeout to the socket, and adding
an ExponentialBackoff Retry logic.
* Simpler Connection logic
* Add simple retry logic to Redis Connection Error
* Make retry exponential, add keepalive & socket_connect_timeout
* Handles configuration on Redis' connection pool
* Simplifies timeout exception logic
* Fix burst bug, add test
* Add docs related to `socket_timeout`, improve compatibility with older RedisPy versions
* Fixes
* New timeout private method
* Fix timeout
* Improve docstrings on `connections`
* Enhanced Job methods docstrings & Serialization Protocol
This adds docstrings to all Job methods in a standard format.
It also implements a `serializer` protocol.
* Excludes `Protocol` (keeping compatibility with < 3.8)
* Add docstrings & type annotation to the `job` decorator
* Docstrings for the `defaults` vars
* Add deprecation warning to Connection context manager
* Fix Types
* Moves the `compact` helper to `utils`
There is a helper funcion that excludes `None` values
from a list. This was being declared both in the Queue
and in the worker. This centralizes this helper in the `utils`
Importing
* Fix Type Annotation
* Remove unused code from compat module
* Remove unused dictconfig
* Remove total_ordering compat layer
* Remove compatibility layer
This completely removes the compat module. It moves utilities
functions (`as_text` and `decode_redis_hash`) to the `utils`
module, is eliminates the use of the proxies `text_type` and
`string_types`, using the `str` construct directly.
* Remove compat module
Finishes the cleaning of the compatibility module.
The last function being the `is_python_version` function
which was being used internally.
* Fix old import
* Fix Imports
* Remove Dummy (Force GH Actions)
* Fix Imports
* Organize Imports
* Add debug logs to dequeue
There were reports where a worker wouldn't fetch new jobs.
Since one possible cause is the infinite hanging of the `BLPOP` command,
this adds a couple of logs surrounding the `dequeue_any` function,
that interacts with Redis using `BLPOP`,
* Update worker.py
Fix Typo
* More logs
* Logs
* Update worker.py
* Add debug logs to the queue
* Reviewed logs
* Remove queue count before job push
* New log wording
* Remove logs
* Allow for configurable scheduler fallback period
This adds a parameter to the Scheduler called
`fallback_period`, which determines the period
before work fallsback to a new scheduler.
* Update defaults.py
Reduce Scheduler Fallback Period to 120 seconds.
* Update scheduler.py
Remove fallback period parameter.
Dummy tasks were probably used for load testing
in the past, but currently are not being imported from anywhere.
Also, similar task utilities are already present on the tests fixtures,
making the `dummy` module irrelevant.
* Persist worker_name after job is finished
Persisting the worker_name on the job object in Redis would allow for debugging and
analyzing logs from the worker
* Remove redundent job.save() method call
* Remove check for null worker
Now that worker name is persisted after job finishes or fails,
no need to assert that worker name is None
* Change github runner to Ubuntu 20.04
* Change github runner to Ubuntu 20.04
* WIP job results
* Result can now be saved
* Successfully saved and restored result
* result.save() should accept pipeline
* Successful results are saved
* Failures are now saved properly too.
* Added test for Result.get_latest()
* Checkpoint
* Got Result.all() to work
* Added Result.count(), Result.delete()
* Backward compatibility for job.result and job.exc_info
* Added some typing
* More typing stuff
* Fixed typing in job.py
* More typing updates
* Only keep the last 10 results
* Documented job.results()
* Got results test to pass
* Don't run test_results.py on Redis server < 5.0
* Fixed mock import on some Python versions
* Remove Redis 3 from test matrix
* Jobs should never use the new Result implementation if server is < 5.0
* Results should only be created is Redis stream is supported.
* Added back Redis 3 to test matrix
* Fixed job.supports_redis_streams
* Fixed worker test
* Updated docs.
* Add feature to enqueue dependents at the front of queues
* Add documentation for the Dependency(enqueue_at_front=...) parameter
* docs: Add `enqueue_at_front` to list of parameters for Dependency
* test: Update dependency test to not rely on Redis ordering
* refactor: Save enqueue_at_front boolean in job.meta instead of separate instance attr
* fix: Made enqueue_at_front an instance attribute instead of putting it inside meta
* doc: default queues order is a priority order
In introduction, we say that queue order matters, but we are not very specific.
Added a paragraph to explicit that.
* doc: fix typo
* Fix job.dependencies_are_met() if dependency is canceled
* Slightly better test coverage on dependencies_are_met()
* Fixed job.cancel(enqueue_dependent=True)
* Move common flake8 options into config file
Currently --max-line-length being specified in two places. Just use the
existing value in the config file as the source of truth.
Move --count and --statistics to config file as well.
* Fix some lints
The on_success and on_failure callbacks are currently supported in
Queue.enqueue but not Queue.enqueue_many, despite documentary indicating
the contrary.
This commit brings the bulk enqueuing API back into parity.
Fixes#1680
Paul Spooren
lowercase00
Oleg
Selwin Ong
dependabot[bot]
Oscar Arbeláez-Echeverri
gabriels1234
Ankush Menat
eswolinsky3241
Krzysztof Jeziorny
Jan Rüegg
Alex
Jahn Thomas Fidje
Miles Winter
Alex Garel
Rony Lutsky
Burak Yılmaz
Yang Yang