Version 6 (modified by 11 years ago) (diff) | ,
---|
jobq
Simple job queue management (a better 'batch')
- Install Debian package with source
deb http://deb.bearstech.com jobq/
(Etch, Lenny, Squeeze, Sid) - Download jobq-0.6.tar.gz
- Browse at source:/jobq/
- Checkout with
svn co svn://forge.bearstech.com/forge/jobq
jobq manages a simple batch queue, where one job is run after the other. Many queues can be run independently, and no specific privileges or daemon is required. Queue info and statistics are syslogged.
It is especially useful for complex crontabs, to force sequential execution of jobs that would otherwise run in parallel with unlimited instances. Jobs and load can be regulated by 'run queues' very easily with jobq.
jobq is licensed under the GNU GPL 3.0.
Example usage
Run two heavy jobs, first is run immediatly since queue is initially empty (shows in processes but not in queue), the second one is waiting (shows in queue).
hal:~$ jobq heavy_job arg1 hal:~$ jobq heavy_job arg2 param2 hal:~$ pgrep -lf heavy_job 10441 /usr/bin/heavy_job hal:~$ jobq ID Submission date Command 10447 2009-10-27 16:06:59 heavy_job arg2 param2 hal:~$ tail /var/log/user.log Oct 27 16:06:52 hal jobq: queue runner started for 'default' (pid 10421) Oct 27 16:06:52 hal job[10413]: start (delay:0 sec, todo:0): heavy_job arg1 Oct 27 16:07:22 hal job[10413]: done (time:30 sec, exit:0) Oct 27 16:07:22 hal job[10447]: start (delay:23 sec, todo:0): heavy_job arg2 param2 Oct 27 16:07:27 hal job[10447]: done (time:5 sec, exit:0)
Job 1 was scheduled and started at 16:06:52, and took 30 sec to complete.
Job 2 was scheduled 7 secondes later at 16:06:59 (not logged), started at 16:07:22 (23 sec after sheduling) and took 5 sec to complete.
Real life sample
The following picture is a 'load graph' from a server with a large crontab, launching many jobs at different frequency, from every minute to every month. Of course, sometimes jobs overlap badly, and if they compete for a scarce ressource (such as I/O on a local disk) they are slow and also slow down the whole server.
On Tuesday all jobs where run through a single queue by simply prepending 'jobq' to the existing commands in the crontab. The server was running smoothly again, and the problem was now a safer one where the job queue has to be monitored for not growing indefinitely.
Attachments (8)
- jobq-0.5.tar.gz (14.6 KB) - added by 11 years ago.
- jobq_effect.png (18.1 KB) - added by 11 years ago.
- jobq-0.6.tar.gz (15.2 KB) - added by 11 years ago.
- jobq-0.7.tar.gz (15.2 KB) - added by 11 years ago.
- jobq-0.7.1.tar.gz (15.3 KB) - added by 11 years ago.
- jobq-0.8.tar.gz (17.4 KB) - added by 11 years ago.
- jobq-0.8.1.tar.gz (17.7 KB) - added by 11 years ago.
- jobq-0.8.2.tar.gz (18.3 KB) - added by 10 years ago.
Download all attachments as: .zip