use of dagster-dask on premis

[sephib]

Hi,
We are thinking of using dagster with a local (on-premis) yarn cluster with dask-yarn.
From the documentation we can see that the limitations is to :

must use S3 for intermediates and run storage.

How can we use dagster in our environment?

[schrockn]

Hey @sephib that sounds really exciting.

In order to use dagster in new or custom environment like this you must implement a SystemStorageDefinition (usually via the @system_storage decorator)

See python_modules/libraries/dagster-aws/dagster_aws/s3/system_storage.py for an example.

The system can operate on an arbitrary instance of SystemStorageDefinition as long as it is faithful to the APIs that it needs. The interesting question is what storage you want to use in your yarn cluster.

Please feel free to hop in our slack! We'd love to hear more about your use case.

[sephib]

The storage would be on hdfs with the output format being parquet or csv.
To access the hdfs I think we would try to work with pyarrow
Do you have any additional thoughts?
Do you know of a running dagster instance on Yarn?

[natekupp]

@sephib - makes sense! I do think it would be fairly straightforward to implement storage on HDFS instead of S3 using the system storage system that Nick mentioned above—I'm happy to work with you on this.

Re: YARN integration, we haven't deployed in that context yet but something we'd like to support. Definitely would love to hear more about your use case, if you are able to join our Slack (linked on the github page here: https://github.com/dagster-io/dagster) - would love to hear more about what your needs are!

[natekupp]

we have a prototype of HDFS system storage working now: https://dagster.phacility.com/D2259 - still have not explored dask-yarn. See also #2273

[sephib]

Thx for the update - however not yet sure when we will be able to check it out.