Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add read support for lazy http ndjson #998

Merged

Conversation

ceyhunkerti
Copy link
Contributor

iex(2)> frame = Explorer.DataFrame.from_ndjson("http://127.0.0.1:8080/demo.ndjson", lazy: true)
** (FunctionClauseError) no function clause matching in Explorer.PolarsBackend.LazyFrame.from_ndjson/3    
    
    The following arguments were given to Explorer.PolarsBackend.LazyFrame.from_ndjson/3:
    
        # 1
        %FSS.HTTP.Entry{
          url: "http://127.0.0.1:8080/demo.ndjson",
          config: %FSS.HTTP.Config{headers: []}
        }
    
        # 2
        1000
    
        # 3
        1000
    
    Attempted function clauses (showing 2 out of 2):
    
        def from_ndjson(%FSS.S3.Entry{}, _, _)
        def from_ndjson(%FSS.Local.Entry{} = entry, infer_schema_length, batch_size)
    
    (explorer 0.10.0-dev) lib/explorer/polars_backend/lazy_frame.ex:230: Explorer.PolarsBackend.LazyFrame.from_ndjson/3
    iex:2: (file)
  • This adds support for it.

Manual Test

  • to test locally you can download and extract the attached ndjson or simply create one like
  • demo.ndjson
{"a":1}
{"b":2}
  • run http-server . on the path containing demo.ndjson
  • Test after the change 👇🏼
iex(3)> frame = Explorer.DataFrame.from_ndjson("http://127.0.0.1:8080/demo.ndjson", lazy: true)
{:ok,
 #Explorer.DataFrame<
   LazyPolars[??? x 2]
   a s64 [1, nil]
   b s64 [nil, 2]
 >}

demo.ndjson.tar.gz

@philss philss merged commit 4ae1a27 into elixir-explorer:main Sep 30, 2024
3 checks passed
@philss
Copy link
Member

philss commented Sep 30, 2024

@ceyhunkerti thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants