b2luigi Features

Once you have your first task tree setup, (b2)luigi has many features, that make your live easier. From leveraging batch systems, setting up different environments, keeping track over many parameters and files, there are many features to keep in mind. In the following sections some of the important features are explained in more detail. Also checkout our Examples and b2luigi Starter Kit for some working examples.

• luigi Features
  • The Central Scheduler
  • Parameters
  • Logging
  • Even more settings
  • Resource Management
• Run Modes
  • Start a Central Scheduler
• The b2luigi CLI
  • Subcommands overview
  • Project layout
  • b2luigi run
  • b2luigi show
  • b2luigi graph
  • b2luigi remove
  • b2luigi test
  • Utility commands
  • Shell auto-completion
  • Migration from the legacy interface
• Migrating to the New CLI
  • Starting point: a legacy project
  • Step 1 — install the CLI binary
  • Step 2 — nothing changes yet
  • Step 3 — swap the flags
  • What carries over unchanged
  • Stricter remove --keep
  • Step 4 — optional: adopt parameters.py
  • Step 5 — optional: replace a hand-written WrapperTask with ParameterGenerator
  • Coexistence during a gradual team migration
  • Troubleshooting
• Batch Processing
  • Choosing the batch system
  • Choosing the Environment
  • File System
  • Drawbacks of the batch mode
  • Implementing your own Batch Process
  • Batch System Specific Settings
  • Add your own batch system
• Parameters (Advanced)
  • Unique Task outputs
  • Exclude Parameters from the output name
  • Do not use the parameter name in the output name
  • Do not use “=” to separate the parameter name and value
  • Hashing List and Dict Parameters
• Parameter Grouping
  • Overview
  • Enabling Parameter Grouping
  • How a group is submitted
  • Failure Semantics and Resubmission
  • Advanced Usage: Custom Grouping Logic
  • Interaction with luigi Batching
• Handling Input and Outputs
  • Outputs
  • Inputs
• Settings
  • How to set settings
  • Available settings
  • Custom settings
• Remote Targets
  • XRootD
  • WebDAV
• General Usage
• Progress TUI
  • Installation
  • Usage
  • Key bindings
  • Log files