Configuration

Mara Configuration Values

The following configuration values are used by this module. They are defined as python functions in mara_db.config and can be changed with the monkey patch from Mara App. An example can be found here.

mara_pipelines.config.root_pipeline() → mara_pipelines.pipelines.Pipeline

A pipeline that contains all other pipelines of the project

mara_pipelines.config.data_dir() → str

Where to find local data files

mara_pipelines.config.default_db_alias() → str

The alias of the database that should be used when not specified otherwise

mara_pipelines.config.default_task_max_retries()

How many times a task is retried when it fails by default

mara_pipelines.config.first_date() → datetime.date

Ignore data before this date

mara_pipelines.config.last_date() → datetime.date

Ignore data after this date

mara_pipelines.config.max_number_of_parallel_tasks() → int

How many tasks can run in parallel at maximum

mara_pipelines.config.bash_command_string() → str

The command used for running a bash, should somehow include the pipefail option

mara_pipelines.config.system_statistics_collection_period() → Optional[float]

How often should system statistics be collected in seconds.

When zero or None the collecting of system statistics is disabled.

mara_pipelines.config.run_log_retention_in_days() → int

How many days to keep node run times, output logs and system statistics

mara_pipelines.config.allow_run_from_web_ui() → bool

When false, then it is not possible to run an ETL from the web UI

mara_pipelines.config.base_url() → str

External url of flask app, for linking nodes in slack messages

mara_pipelines.config.slack_token() → Optional[str]

Deprecated, use event_handlers function below instead.

When not None, then this slack webhook is notified of failed nodes. Slack channel’s token (i.e. THISIS/ASLACK/TOCKEN) can be retrieved from the channel’s app “Incoming WebHooks” configuration as part part of the Webhook URL

mara_pipelines.config.event_handlers() → List[mara_pipelines.events.EventHandler]

Configure additional event handlers that listen to pipeline events, e.g. chat bots that announce failed runs

Example

mara_pipelines.config.event_handlers = lambda: [mara_pipelines.notification.slack.Slack(‘123/ABC/cdef’)]

mara_pipelines.config.password_masks() → List[str]

Any passwords which should be masked in the UI or logs