jobq

Simple job queue management (a better 'batch')

• Install Debian package with source deb http://deb.bearstech.com jobq/ (Etch, Lenny, Squeeze, Sid)
• Download jobq-0.6.tar.gz
• Browse at source:/jobq/
• Checkout with svn co svn://forge.bearstech.com/forge/jobq

jobq manages a simple batch queue, where one job is run after the other. Many queues can be run independently, and no specific privileges or daemon is required. Queue info and statistics are syslogged.

It is especially useful for complex crontabs, to force sequential execution of jobs that would otherwise run in parallel with unlimited instances. Jobs and load can be regulated by 'run queues' very easily with jobq.

jobq is licensed under the ​GNU GPL 3.0.

Example usage

Run two heavy jobs, first is run immediatly since queue is initially empty (shows in processes but not in queue), the second one is waiting (shows in queue).

hal:~$ jobq heavy_job arg1
hal:~$ jobq heavy_job arg2 param2

hal:~$ pgrep -lf heavy_job
10441 /usr/bin/heavy_job

hal:~$ jobq
ID     Submission date      Command
10447  2009-10-27 16:06:59  heavy_job arg2 param2

hal:~$ tail /var/log/user.log
Oct 27 16:06:52 hal jobq: queue runner started for 'default' (pid 10421)
Oct 27 16:06:52 hal job[10413]: start (delay:0 sec, todo:0): heavy_job arg1
Oct 27 16:07:22 hal job[10413]: done (time:30 sec, exit:0)
Oct 27 16:07:22 hal job[10447]: start (delay:23 sec, todo:0): heavy_job arg2 param2
Oct 27 16:07:27 hal job[10447]: done (time:5 sec, exit:0)

Job 1 was scheduled and started at 16:06:52, and took 30 sec to complete.

Job 2 was scheduled 7 secondes later at 16:06:59 (not logged), started at 16:07:22 (23 sec after sheduling) and took 5 sec to complete.

Real life sample

The following picture is a 'load graph' from a server with a large crontab, launching many jobs at different frequency, from every minute to every month. Of course, sometimes jobs overlap badly, and if they compete for a scarce ressource (such as I/O on a local disk) they are slow and also slow down the whole server.

On Tuesday all jobs where run through a single queue by simply prepending 'jobq' to the existing commands in the crontab. The server was running smoothly again, and the problem was now a safer one where the job queue has to be monitored for not growing indefinitely.