Mara Pipelines documentation¶
Welcome to Mara Pipelines’s documentation. This is one of the core modules of the Mara Framework contains a lightweight data transformation framework with a focus on transparency and complexity reduction. It has a number of baked-in assumptions/ principles:
Data integration pipelines as code: pipelines, tasks and commands are created using declarative Python code.
PostgreSQL as a data processing engine.
Extensive web ui. The web browser as the main tool for inspecting, running and debugging pipelines.
GNU make semantics. Nodes depend on the completion of upstream nodes. No data dependencies or data flows.
No in-app data processing: command line tools as the main tool for interacting with databases and data.
Single machine pipeline execution based on Python’s multiprocessing. No need for distributed task queues. Easy debugging and output logging.
Cost based priority queues: nodes with higher cost (based on recorded run times) are run first.
User’s Guide¶
This part of the documentation focuses on step-by-step instructions how to use this module.
API Reference¶
If you are looking for information on a specific function, class or method, this part of the documentation is for you.
Additional Notes¶
Legal information and changelog are here for the interested.
- License
- Changelog
- 3.4.0 (2023-05-01)
- 3.3.0 (2022-09-22)
- 3.2.0 (2021-03-08)
- 3.1.1 (2020-07-31)
- 3.1.0 (2020-07-21)
- 3.0.0 (2020-06-11)
- 2.8.3 (2020-06-10)
- 2.8.2 (2020-05-04)
- 2.8.1 (2020-04-27)
- 2.8.0 (2020-03-25)
- 2.7.0 (2020-03-05)
- 2.6.1 (2020-02-20)
- 2.6.0 (2020-02-12)
- 2.5.1 (2019-08-01)
- 2.5.0 (2019-07-07)
- 2.4.0 - 2.4.2 (2019-07-04)
- 2.3.0 (2019-07-04)
- 2.2.0 (2019-07-02)
- 2.1.0 (2019-05-15)
- 2.0.0 - 2.0.1 (2019-04-12)
- 1.4.0 - 1.4.7 (2018-09-15)
- 1.3.0 (2018-07-17)
- 1.2.0 (2018-06-01)
- 1.1.0 (2018-05-23)
- 1.0.0 - 1.0.4 (2018-05-02)