17 Commits (af2dfb14460db9d0c7883c6b9381bd06e3ab607d)

Author SHA1 Message Date
Selwin Ong 64cb1a27b9
Worker pool (#1874)
* First stab at implementating worker pool

* Use process.is_alive() to check whether a process is still live

* Handle shutdown signal

* Check worker loop done

* First working version of `WorkerPool`.

* Added test for check_workers()

* Added test for pool.start()

* Better shutdown process

* Comment out test_start() to see if it fixes CI

* Make tests pass

* Make CI pass

* Comment out some tests

* Comment out more tests

* Re-enable a test

* Re-enable another test

* Uncomment check_workers test

* Added run_worker test

* Minor modification to dead worker detection

* More test cases

* Better process name for workers

* Added back pool.stop_workers() when signal is received

* Cleaned up cli.py

* WIP on worker-pool command

* Fix test

* Test that worker pool ignores consecutive shutdown signals

* Added test for worker-pool CLI command.

* Added timeout to CI jobs

* Fix worker pool test

* Comment out test_scheduler.py

* Fixed worker-pool in burst mode

* Increase test coverage

* Exclude tests directory from coverage.py

* Improve test coverage

* Renamed `Pool(num_workers=2) to `Pool(size=2)`

* Revert "Renamed `Pool(num_workers=2) to `Pool(size=2)`"

This reverts commit a1306f89ad0d8686c6bde447bff75e2f71f0733b.

* Renamed Pool to WorkerPool

* Added a new TestCase that doesn't use LocalStack

* Added job_class, worker_class and serializer arguments to WorkerPool

* Use parse_connection() in WorkerPool.__init__

* Added CLI arguments for worker-pool

* Minor WorkerPool and test fixes

* Fixed failing CLI test

* Document WorkerPool
2 years ago
Selwin Ong 77e926c424
Added parse_connection function (#1884)
* Added parse_connection function

* feat: allow custom connection pool class (#1885)

* Added test for SSL

---------

Co-authored-by: Cyril Chapellier <tchapi@users.noreply.github.com>
2 years ago
Daniel M 3d840a79ad
Add to worker redis record scheduler info (#1787)
* add scheduler_pid property to queue

* Update return type

* Reformat code
2 years ago
Oleg 4bd0f12ec9
Restart scheduler process if the process is not alive (#1764) 2 years ago
gabriels1234 50d8d72928
Add at_front for scheduled/enqueue_at jobs (#1743)
* Add at_front for scheduled/enqueue_at jobs

* Add test to at_front
2 years ago
Yang Yang 9db728921d
Improve the lint situation (#1688)
* Move common flake8 options into config file

Currently --max-line-length being specified in two places. Just use the
existing value in the config file as the source of truth.

Move --count and --statistics to config file as well.

* Fix some lints
2 years ago
Hugo 61a4a1720b
Use unittest.mock instead of mock (#1673)
This module has been included in Python itself since 3.3.

Fixes: https://github.com/rq/rq/issues/1646
2 years ago
Selwin Ong cc70cacc1b
Add more tolerance to scheduler heartbeat (#1555) 3 years ago
Ashoka Lella d333d20914
Make RQScheduler work with a serializer (#1455)
* Scheduler with a serializer

* test with metadata

Co-authored-by: alella <ashoka.lella@factset.com>
4 years ago
BobReid 75a610bd4d
Fix RQScheduler when run with SSL connection (#1383)
* Quick and dirty set up of SSL

* copy connection kwargs in scheduler

* fix

* chmod the cert

* Skip SSL tests in CI
4 years ago
Selwin Ong f3e924cdd1
Added job.worker_name (#1375)
* Added job.worker_name

* Fix compatibility with Redis server 3.x

* Document job.worker_name

* Removed some Python 2 compatibility stuff.

* Remove unused codes
4 years ago
Selwin Ong d1528d776d
Release scheduler lock when running in burst mode (#1374)
* Fixed an issue where scheduler lock is not release when running worker in burst mode

* Remove unused import
4 years ago
Nikita Romaniuk 2da957a68d
scheduler: now operates with chunks of jobs (#1355)
* scheduler: now operates with chunks of jobs

* scheduler: set default chunk_size for ScheduledJobRegistry.get_jobs_to_schedule

* scheduler: fixed missing indent

* scheduler: added test for get_jobs_to_schedule() with chunk_size parameter

* scheduler: fixed test for passing python 3.5 (no f-strings)

* scheduler: fixed chunk_size in test make it lighter to run
4 years ago
Vladimir Ulupov 237e69123a
pass retry param to enqueue_at func (#1343) 4 years ago
Evan Ackmann bd0fcc1a07
Took into account DST when computing localtime zones. This wasn't ac… (#1258)
* Took into account DST when computing localtime zones.  This wasn't accounted for when using non-UTC datetimes with queue.enqueue_at()

* Updates tests with mocked timezones to test both scenarios
5 years ago
Ivan Kiryanov ed67de22c6 Add job status setting in enqueue_at (and in enqueue_in) methods (#1181)
* Add job status setting in enqueue_at (and in enqueue_in) methods

Update tests for this change
Closes: #1179

* Add status param to create_job func, rework enqueue_at status setting
5 years ago
Selwin Ong baa0cc268a
Job scheduling (#1163)
* First RQScheduler prototype

* WIP job scheduling

* Fixed Python 2.7 tests

* Added ScheduledJobRegistry.get_scheduled_time(job)

* WIP on scheduler's threading mechanism

* Fixed test errors

* Changed scheduler.acquire_locks() to instance method

* Added scheduler.prepare_registries()

* Somewhat working implementation of RQ scheduler

* Only call stop_scheduler if there's a scheduler present

* Use OSError rather than ProcessLookupError for PyPy compatibility

* Added `auto_start` argument to scheduler.acquire_locks()

* Make RQScheduler play better with timezone

* Fixed test error

* Added --with-scheduler flag to rq worker CLI

* Fix tests on Python 2.x

* More Python 2 fixes

* Only call `scheduler.start` if worker is run in non burst mode

* Fixed an issue where running worker with scheduler would fail sometimes

* Make `worker.stop_scheduler()` more resilient to errors

* worker.dequeue_job_and_maintain_ttl() should also periodically run maintenance tasks

* Scheduler can now work with worker in both burst and non burst mode

* Fixed scheduler logging message

* Always log scheduler errors when running

* Improve scheduler error logging message

* Removed testing code

* Scheduler should periodically try to acquire locks for other queues it doesn't have

* Added tests for scheduler.should_reacquire_locks

* Added queue.enqueue_in()

* Fixes queue.enqueue_in() in Python 2.7

* First stab at documenting job scheduling

* Remove unused methods

* Remove Python 2.6 logging compatibility code

* Remove more unused imports

* Added convenience methods to access job registries from queue

* Added test for worker.run_maintenance_tasks()

* Simplify worker.queue_names() and worker.queue_keys()

* Updated changelog to mention RQ's new job scheduling mechanism.
5 years ago