Add huggingface extension

[matthewmturner]

No description provided.

[matthewmturner]

@Xuanwo this is as far as i got today - im at the point where i need to now figure out how to map huggingface to object store semantics. I tried a quick create table statement and ended up with this error. the path looks okay to me but im not that familiar with huggingface and didnt get to look much into this.

I'll pick back up on this tomorrow but if you have any insight would it would be very helpful

[Xuanwo]

I'm guessing this is wrong.

[Xuanwo]

Hi, given the error output in your posted image, I assume we are trying to access an incorrect path. OpenDAL manages all paths internally, so we only need to provide the path relative to the repository root instead of trying to build the url.

---

I built a real example with the repo you are using:

use std::sync::Arc;
use opendal::Operator;
use opendal::Result;
use anyhow::Result;
use opendal::services::Huggingface;
use opendal::Operator;

#[tokio::main]
async fn main() -> Result<()> {
    // Create Huggingface backend builder
    let mut builder = Huggingface::default()
        // set the type of Huggingface repository
        .repo_type("dataset")
        // set the id of Huggingface repository
        .repo_id("HuggingFaceTB/finemath")
        // set the revision of Huggingface repository
        .revision("main")
        // set the root for Huggingface, all operations will happen under this root
        .root("/");

    let op: Operator = Operator::new(builder)?.finish();
    let entries = op.list("/").await?;
    println!("{:?}", entries.iter().map(|v| v.path()).collect::<Vec<_>>());

    let meta = op
        .stat("finemath-3plus/train-00000-of-00128.parquet")
        .await?;
    println!("{:?}", meta);

    Ok(())
}

The output will be:

["assets/", "finemath-3plus/", "finemath-4plus/", "infiwebmath-3plus/", "infiwebmath-4plus/", ".gitattributes", "README.md"]
Metadata { mode: FILE, is_current: None, is_deleted: false, cache_control: None, content_disposition: None, content_length: Some(507607173), content_md5: None, content_range: None, content_type: Some("application/json; charset=utf-8"), content_encoding: None, etag: Some("W/\"23e-Dio8lpah4iHFyrzOC8sgQMZGg8E\""), last_modified: Some(2024-12-19T09:49:58Z), version: None, user_metadata: None }

I hope this example effectively demonstrates how to properly configure the huggingface service here.

[Xuanwo]

I'm guessing we can register url as hf://datasets/<repo_id>/. And visit the file in the way hf://datasets/HuggingFaceTB/finemath/finemath-3plus/train-00000-of-00128.parquet.

There may be some tricks things on the url handling inside datafusion and oebjct_store.

[matthewmturner]

@Xuanwo thanks much for the feedback and apologies for the delay getting back - currently on vacation and not online as much.

For your information I ended up creating a separate repo to test out opendal with datafusion to get minimal working example independent from the context of dft. Once I have it working there I'll finish this branch.

[Xuanwo]

For your information I ended up creating a separate repo to test out opendal with datafusion to get minimal working example independent from the context of dft. Once I have it working there I'll finish this branch.

Hi, we have a great example for this: https://github.com/apache/opendal/blob/main/integrations/object_store/examples/datafusion.rs

[matthewmturner]

@Xuanwo got it working :) thanks for your help

[Xuanwo]

@Xuanwo got it working :) thanks for your help

Nice!