This repository contains some experiments on what a modern data platform could look like. A strong emphasis lies on how observability can be achieved throughout all actions occurring on a data platform.

Currently the documentation is not published, but can be viewed in this repository.

mkdocs serve

• rust 1.58+
• python 3.8+
• poetry (for python development)
• just (for running workspace commands)

To run unit tests for the rust crates execute

cargo test

To build crates and services

cargo build

TODO explain mock framework

To set up the python environment run

poetry install

The python bindings for rust dependent crates have to be build and installed separately. Make sure you have the environment created by poetry activated and run.

make python-develop

To see the Dagster integrations in play, create two folders in the project root: .dagster and .fusion, then start an instance of the fusion service in a separate shell.

fusion server start --host 127.0.0.1 --port 50051 --log-level info

An example Dagster repository using most of the features from the dagster-fusion package is provided within examples/model-training. To inspect it and play with the configurations, run a local instance of dagster.

docker build -f docker/Dockerfile

docker run -d -p6831:6831/udp -p6832:6832/udp -p16686:16686 jaegertracing/all-in-one:latest

To generate proto definitions for use in python client.

make python-proto