luigi Features

luigi already has many nice and helpful features. All of them can be used with b2luigi as well. In the following is a collection of the most important features. Please note, that the following list is not exhaustive and check the luigi documentation for more information.

The Central Scheduler

When the number of tasks grows, it is sometimes hard to keep track of all of them (despite the summary in the end). For this, luigi (the parent project of b2luigi) brings a nice visualization and scheduling tool called the central scheduler. If you are submitting your jobs from a remote machine, you can forward the schedulers port to your local machine via ssh.

Parameters

luigi already has various Parameter types to pass to your tasks, like ListParameter, IntParameter, DateParameter and many more. Check the luigi parameter documentation for more information.

Logging

luigi has some logging features built in.

Even more settings

Many aspects of luigi’s scheduling mechanism can be controlled via the luigi configuration. Please be aware, that these settings is are a separate configuration from the settings in b2luigi. Yes, this is very confusing, but historically grown. In one point in the future, this might get changed.

Resource Management

Using the central scheduler, luigi can do some simple resource management. Just set the available resources in the luigi.cfg and set the amount used as a task property.