19 Commits (192fbc9c507bb5f3347f39304b5c2fb8ea8f7f43)

Author SHA1 Message Date
xzander bdbc9a4f9c
Scheduler should release and heartbeat only acquired locks (#1914)
* Scheduler should release and heartbeat only acquired locks.

* Added tests for heartbeat and release only acquired locks.

* Changed test description to correct one.
2 years ago
Rob Hudson ea063edf0a
Update linting configuration (#1915)
* Update linting configuration

This removes flake8 in favor of ruff, which also provides isort support, and
updates all files to be black, isort, and ruff compliant. This also adds black
and ruff checks to the tox and Github linting workflow.

* Tweak the code coverage config and calls
2 years ago
Selwin Ong 64cb1a27b9
Worker pool (#1874)
* First stab at implementating worker pool

* Use process.is_alive() to check whether a process is still live

* Handle shutdown signal

* Check worker loop done

* First working version of `WorkerPool`.

* Added test for check_workers()

* Added test for pool.start()

* Better shutdown process

* Comment out test_start() to see if it fixes CI

* Make tests pass

* Make CI pass

* Comment out some tests

* Comment out more tests

* Re-enable a test

* Re-enable another test

* Uncomment check_workers test

* Added run_worker test

* Minor modification to dead worker detection

* More test cases

* Better process name for workers

* Added back pool.stop_workers() when signal is received

* Cleaned up cli.py

* WIP on worker-pool command

* Fix test

* Test that worker pool ignores consecutive shutdown signals

* Added test for worker-pool CLI command.

* Added timeout to CI jobs

* Fix worker pool test

* Comment out test_scheduler.py

* Fixed worker-pool in burst mode

* Increase test coverage

* Exclude tests directory from coverage.py

* Improve test coverage

* Renamed `Pool(num_workers=2) to `Pool(size=2)`

* Revert "Renamed `Pool(num_workers=2) to `Pool(size=2)`"

This reverts commit a1306f89ad0d8686c6bde447bff75e2f71f0733b.

* Renamed Pool to WorkerPool

* Added a new TestCase that doesn't use LocalStack

* Added job_class, worker_class and serializer arguments to WorkerPool

* Use parse_connection() in WorkerPool.__init__

* Added CLI arguments for worker-pool

* Minor WorkerPool and test fixes

* Fixed failing CLI test

* Document WorkerPool
2 years ago
Selwin Ong 77e926c424
Added parse_connection function (#1884)
* Added parse_connection function

* feat: allow custom connection pool class (#1885)

* Added test for SSL

---------

Co-authored-by: Cyril Chapellier <tchapi@users.noreply.github.com>
2 years ago
Daniel M 3d840a79ad
Add to worker redis record scheduler info (#1787)
* add scheduler_pid property to queue

* Update return type

* Reformat code
2 years ago
Oleg 4bd0f12ec9
Restart scheduler process if the process is not alive (#1764) 2 years ago
gabriels1234 50d8d72928
Add at_front for scheduled/enqueue_at jobs (#1743)
* Add at_front for scheduled/enqueue_at jobs

* Add test to at_front
2 years ago
Yang Yang 9db728921d
Improve the lint situation (#1688)
* Move common flake8 options into config file

Currently --max-line-length being specified in two places. Just use the
existing value in the config file as the source of truth.

Move --count and --statistics to config file as well.

* Fix some lints
2 years ago
Hugo 61a4a1720b
Use unittest.mock instead of mock (#1673)
This module has been included in Python itself since 3.3.

Fixes: https://github.com/rq/rq/issues/1646
2 years ago
Selwin Ong cc70cacc1b
Add more tolerance to scheduler heartbeat (#1555) 3 years ago
Ashoka Lella d333d20914
Make RQScheduler work with a serializer (#1455)
* Scheduler with a serializer

* test with metadata

Co-authored-by: alella <ashoka.lella@factset.com>
4 years ago
BobReid 75a610bd4d
Fix RQScheduler when run with SSL connection (#1383)
* Quick and dirty set up of SSL

* copy connection kwargs in scheduler

* fix

* chmod the cert

* Skip SSL tests in CI
4 years ago
Selwin Ong f3e924cdd1
Added job.worker_name (#1375)
* Added job.worker_name

* Fix compatibility with Redis server 3.x

* Document job.worker_name

* Removed some Python 2 compatibility stuff.

* Remove unused codes
4 years ago
Selwin Ong d1528d776d
Release scheduler lock when running in burst mode (#1374)
* Fixed an issue where scheduler lock is not release when running worker in burst mode

* Remove unused import
4 years ago
Nikita Romaniuk 2da957a68d
scheduler: now operates with chunks of jobs (#1355)
* scheduler: now operates with chunks of jobs

* scheduler: set default chunk_size for ScheduledJobRegistry.get_jobs_to_schedule

* scheduler: fixed missing indent

* scheduler: added test for get_jobs_to_schedule() with chunk_size parameter

* scheduler: fixed test for passing python 3.5 (no f-strings)

* scheduler: fixed chunk_size in test make it lighter to run
4 years ago
Vladimir Ulupov 237e69123a
pass retry param to enqueue_at func (#1343) 4 years ago
Evan Ackmann bd0fcc1a07
Took into account DST when computing localtime zones. This wasn't ac… (#1258)
* Took into account DST when computing localtime zones.  This wasn't accounted for when using non-UTC datetimes with queue.enqueue_at()

* Updates tests with mocked timezones to test both scenarios
5 years ago
Ivan Kiryanov ed67de22c6 Add job status setting in enqueue_at (and in enqueue_in) methods (#1181)
* Add job status setting in enqueue_at (and in enqueue_in) methods

Update tests for this change
Closes: #1179

* Add status param to create_job func, rework enqueue_at status setting
5 years ago
Selwin Ong baa0cc268a
Job scheduling (#1163)
* First RQScheduler prototype

* WIP job scheduling

* Fixed Python 2.7 tests

* Added ScheduledJobRegistry.get_scheduled_time(job)

* WIP on scheduler's threading mechanism

* Fixed test errors

* Changed scheduler.acquire_locks() to instance method

* Added scheduler.prepare_registries()

* Somewhat working implementation of RQ scheduler

* Only call stop_scheduler if there's a scheduler present

* Use OSError rather than ProcessLookupError for PyPy compatibility

* Added `auto_start` argument to scheduler.acquire_locks()

* Make RQScheduler play better with timezone

* Fixed test error

* Added --with-scheduler flag to rq worker CLI

* Fix tests on Python 2.x

* More Python 2 fixes

* Only call `scheduler.start` if worker is run in non burst mode

* Fixed an issue where running worker with scheduler would fail sometimes

* Make `worker.stop_scheduler()` more resilient to errors

* worker.dequeue_job_and_maintain_ttl() should also periodically run maintenance tasks

* Scheduler can now work with worker in both burst and non burst mode

* Fixed scheduler logging message

* Always log scheduler errors when running

* Improve scheduler error logging message

* Removed testing code

* Scheduler should periodically try to acquire locks for other queues it doesn't have

* Added tests for scheduler.should_reacquire_locks

* Added queue.enqueue_in()

* Fixes queue.enqueue_in() in Python 2.7

* First stab at documenting job scheduling

* Remove unused methods

* Remove Python 2.6 logging compatibility code

* Remove more unused imports

* Added convenience methods to access job registries from queue

* Added test for worker.run_maintenance_tasks()

* Simplify worker.queue_names() and worker.queue_keys()

* Updated changelog to mention RQ's new job scheduling mechanism.
5 years ago