`Zaku` Task Queue｣

A fast, scalable task queue for machine learning workloads

Zaku is a light-weight task queue backed by redis for machine learning workloads. Zaku is built by researchers at MIT in fields including robotics, computer vision, and computer graphics.

light-weight and performant
scalable and versatile.
Open source, licensed under MIT

To Install:

pip install 'zaku[all]==0.0.15'

Here is an example of how to add and retrieve jobs from Zaku. For a more comprehensive list of examples, please refer to the examples page.

from zaku import TaskQ

app = TaskQ(uri="http://localhost:9000")

task_id, task = app.take()
try:
  print("do your things", task)
except Except as e:
  app.mark_reset(task_id)
  raise e
else:
  app.mark_done(task_id)

Life Cycle of a task

A task after being created and inserted into the queue, will be taken by a worker, processed, and then marked as done or reset. If the task is marked as done, it will be removed from the queue. If the task is marked as reset, it will be put back into the queue.

In the event of you forgetting to mark the task as done or reset, the task will be marked as reset after a certain amount of time. This is to prevent the task from being stuck in the queue forever. This time is called ttl or time-to-live.

`zaku` in Depth: Running the Tests¶

The best way to learn how to use zaku is by running the tests in the ./specs folder.

To do so, first download and setup redis-stack-server. Then start a zaku task queue server at local port 9000.

install zaku and look at the options:

pip install -U 'zaku[all,dev,examples]==0.0.15'
zaku -h

install and run the redis-stack-server:

brew install redis-stack-server
redis-stack-server

run the zaku task queue at port 9000:
```
zaku --port 9000 --verbose
```

Now you can run the tests by running

make test

In pycharm, you should see the following:

For a comprehensive documentation on the API, please refer to the API documentation on Components | zaku.

To Cite¶

You can use the following BibTex entry to cite zaku:

@online{yang@zaku,
  author = {Ge Yang},
  title = {Zaku Task Queue: A Fast, Scalable Task Queue for Machine Learning Workloads},
  year = {2024},
  url= {https://zaku.readthedocs.io/en/latest/},
  note = {Light-weight, scalable task queue for machine learning workloads. Accessed: 2024-10-15}
}

Zaku Task Queue｣

zaku in Depth: Running the Tests¶

To Cite¶

`Zaku` Task Queue｣

`zaku` in Depth: Running the Tests¶