Fix inspect between nodes #784

Jhonatannunessilva · 2023-12-22T01:41:43Z

This PR resolves #768

After this PR:

With DataFrame

iex(test_0@myhost)1> df = Explorer.Datasets.fossil_fuels()       
#Explorer.DataFrame<      
  Polars[1094 x 10]
  year integer [2010, 2010, 2010, 2010, 2010, ...]
  country string ["AFGHANISTAN", "ALBANIA", "ALGERIA", "ANDORRA",
   "ANGOLA", ...]
  total integer [2308, 1254, 32500, 141, 7924, ...]
  solid_fuel integer [627, 117, 332, 0, 0, ...]
  liquid_fuel integer [1601, 953, 12381, 141, 3649, ...]
  gas_fuel integer [74, 7, 14565, 0, 374, ...]
  cement integer [5, 177, 2598, 0, 204, ...]
  gas_flaring integer [0, 0, 2623, 0, 3697, ...]
  per_capita f64 [0.08, 0.43, 0.9, 1.68, 0.37, ...]
  bunker_fuels integer [9, 7, 663, 0, 321, ...]
>

iex(test_0@myhost)2> Node.spawn_link(:"test_1@myhost", fn -> IO.inspect(df) end)    
#PID<21907.295.0>
#Explorer.DataFrame<        
  Polars[node: test_0@myhost]
  year integer ???
  country string ???
  total integer ???
  solid_fuel integer ???
  liquid_fuel integer ???
  gas_fuel integer ???
  cement integer ???
  gas_flaring integer ???
  per_capita f64 ???
  bunker_fuels integer ???
>

iex(test_0@myhost)3> IO.inspect(df)
#Explorer.DataFrame<
  Polars[1094 x 10]
  year integer [2010, 2010, 2010, 2010, 2010, ...]
  country string ["AFGHANISTAN", "ALBANIA", "ALGERIA", "ANDORRA", "ANGOLA", ...]
  total integer [2308, 1254, 32500, 141, 7924, ...]
  solid_fuel integer [627, 117, 332, 0, 0, ...]
  liquid_fuel integer [1601, 953, 12381, 141, 3649, ...]
  gas_fuel integer [74, 7, 14565, 0, 374, ...]
  cement integer [5, 177, 2598, 0, 204, ...]
  gas_flaring integer [0, 0, 2623, 0, 3697, ...]
  per_capita f64 [0.08, 0.43, 0.9, 1.68, 0.37, ...]
  bunker_fuels integer [9, 7, 663, 0, 321, ...]
>
#Explorer.DataFrame<
  Polars[1094 x 10]
  year integer [2010, 2010, 2010, 2010, 2010, ...]
  country string ["AFGHANISTAN", "ALBANIA", "ALGERIA", "ANDORRA",
   "ANGOLA", ...]
  total integer [2308, 1254, 32500, 141, 7924, ...]
  solid_fuel integer [627, 117, 332, 0, 0, ...]
  liquid_fuel integer [1601, 953, 12381, 141, 3649, ...]
  gas_fuel integer [74, 7, 14565, 0, 374, ...]
  cement integer [5, 177, 2598, 0, 204, ...]
  gas_flaring integer [0, 0, 2623, 0, 3697, ...]
  per_capita f64 [0.08, 0.43, 0.9, 1.68, 0.37, ...]
  bunker_fuels integer [9, 7, 663, 0, 321, ...]
>

With Series

iex(test_0@myhost)1> series = Explorer.Series.from_list([1, 2, 3])                         
#Explorer.Series<
  Polars[3]
  integer [1, 2, 3]
>

iex(test_0@myhost)2> Node.spawn_link(:"test_1@myhost", fn -> IO.inspect(series) end)
#PID<21661.295.0>
#Explorer.Series<           
  Polars[node: test_0@myhost]
  integer ???
>

iex(test_0@myhost)3> IO.inspect(series)
#Explorer.Series<
  Polars[3]
  integer [1, 2, 3]
>
#Explorer.Series<
  Polars[3]
  integer [1, 2, 3]
>

Jhonatannunessilva · 2023-12-22T01:56:21Z

I had a problem when I tried to run the tests with the "mix ci" command... 6 tests are breaking because of this error:

  6) test from_query/3 queries database (Explorer.DataFrameTest)
     test/explorer/data_frame_test.exs:40
     ** (RuntimeError) failed to start child with the spec {Adbc.Database, [driver: :sqlite]}.
     Reason: an exception was raised:
         ** (ErlangError) Erlang error: :not_loaded
             :erlang.nif_error(:not_loaded)
             (adbc 0.2.2) lib/adbc_nif.ex:24: Adbc.Nif.adbc_database_new/0
             (adbc 0.2.2) lib/adbc_database.ex:51: Adbc.Database.start_link/1
             (stdlib 4.0.1) supervisor.erl:414: :supervisor.do_start_child_i/3
             (stdlib 4.0.1) supervisor.erl:400: :supervisor.do_start_child/2
             (stdlib 4.0.1) supervisor.erl:706: :supervisor.handle_start_child/2
             (stdlib 4.0.1) supervisor.erl:455: :supervisor.handle_call/3
             (stdlib 4.0.1) gen_server.erl:1146: :gen_server.try_handle_call/4
             (stdlib 4.0.1) gen_server.erl:1175: :gen_server.handle_msg/6
             (stdlib 4.0.1) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
     stacktrace:
       (ex_unit 1.14.0) lib/ex_unit/callbacks.ex:538: ExUnit.Callbacks.start_supervised!/2
       test/explorer/data_frame_test.exs:35: Explorer.DataFrameTest.__ex_unit_setup_0_0/1
       Explorer.DataFrameTest.__ex_unit_describe_0/1

I think I had this problem before and I had already solved it, but now that it's back I don't remember what I did... I've already tried to recompile, update the project, etc.

All tests work with the "mix test --only cloud_integration" command.

Versions I'm using:

rust nightly
elixir 1.14.0-otp-25
erlang 25.0.4
cmake version 3.27.0-rc4

I already tried with (which was the version I was on before):

rust nightly
elixir 1.14.0-otp-24
erlang 24.3.4.11

Would anyone know why?

philss

A minor detail :)

Would anyone know why? (About the Adbc exception)

Sorry, I don't remember either. But maybe José can help.

philss · 2023-12-22T05:42:30Z

lib/explorer/data_frame.ex

@@ -5785,6 +5785,12 @@ defmodule Explorer.DataFrame do
  defimpl Inspect do
    import Inspect.Algebra

+    def inspect(df, _opts) when node(df.data.resource) != node() do
+      raise RuntimeError.exception(
+              "It is not possible to inspect a DataFrame that belongs to another node"


We normally don't capitalize the message:

Suggested change

"It is not possible to inspect a DataFrame that belongs to another node"

"it is not possible to inspect a DataFrame that belongs to another node"

There are two issues here:

We probably should not raise, otherwise it makes impossible to see the value in logs, IEx, etc

df.data.resource is accessing an implementation detail of the backend. Ideally we want to move this to Explorer.DF.PolarsBackend or similar. And instead of raising, maybe we just print something like:

#Explorer.DataFrame< Polars[node: other@foobar] >

WDYT?

Yes, it makes sense

Just to add a little bit: we have the column names and types. We could also display them if we want (without the values). WDYT?

Ah, that's good. We can show them as:

Polars[node: ...] foo s64 ??? bar string ???

philss · 2023-12-22T05:42:43Z

lib/explorer/series.ex

@@ -6048,6 +6048,12 @@ defmodule Explorer.Series do
  defimpl Inspect do
    import Inspect.Algebra

+    def inspect(series, _opts) when node(series.data.resource) != node() do
+      raise RuntimeError.exception(
+              "It is not possible to inspect a Series that belongs to another node"


Suggested change

"It is not possible to inspect a Series that belongs to another node"

"it is not possible to inspect a Series that belongs to another node"

josevalim · 2023-12-26T09:01:49Z

lib/explorer/polars_backend/data_frame.ex

@@ -858,6 +858,10 @@ defmodule Explorer.PolarsBackend.DataFrame do
  # Inspect

  @impl true
+  def inspect(df, opts) when node(df.data.resource) != node() do
+    Explorer.Backend.DataFrame.inspect(df, "Polars", nil, opts, from_another_node: true)


Suggested change

Explorer.Backend.DataFrame.inspect(df, "Polars", nil, opts, from_another_node: true)

Explorer.Backend.DataFrame.inspect(df, "Polars", "node: #{df.data.resource}", opts, elide_columns: true)

This will generate this error:

iex(test_0@myhost)1> df = Explorer.Datasets.fossil_fuels() #Explorer.DataFrame< Polars[1094 x 10] year integer [2010, 2010, 2010, 2010, 2010, ...] country string ["AFGHANISTAN", "ALBANIA", "ALGERIA", "ANDORRA", "ANGOLA", ...] total integer [2308, 1254, 32500, 141, 7924, ...] solid_fuel integer [627, 117, 332, 0, 0, ...] liquid_fuel integer [1601, 953, 12381, 141, 3649, ...] gas_fuel integer [74, 7, 14565, 0, 374, ...] cement integer [5, 177, 2598, 0, 204, ...] gas_flaring integer [0, 0, 2623, 0, 3697, ...] per_capita f64 [0.08, 0.43, 0.9, 1.68, 0.37, ...] bunker_fuels integer [9, 7, 663, 0, 321, ...] > iex(test_0@myhost)2> "node: #{df.data.resource}" ** (Protocol.UndefinedError) protocol String.Chars not implemented for #Reference<0.641693014.1003356202.157361> of type Reference. This protocol is implemented for the following type(s): Atom, BitString, Complex, Date, DateTime, Explorer.Duration, Float, Hex.Solver.Assignment, Hex.Solver.Constraints.Empty, Hex.Solver.Constraints.Range, Hex.Solver.Constraints.Union, Hex.Solver.Incompatibility, Hex.Solver.PackageRange, Hex.Solver.Term, Integer, List, NaiveDateTime, Time, URI, Version, Version.Requirement (elixir 1.14.0) lib/string/chars.ex:3: String.Chars.impl_for!/1 (elixir 1.14.0) lib/string/chars.ex:22: String.Chars.to_string/1 iex:2: (file)

Sorry, we need to inspect the node, but the overall idea should be solid. :D

josevalim · 2023-12-29T08:21:32Z

💚 💙 💜 💛 ❤️

philss reviewed Dec 22, 2023

View reviewed changes

josevalim reviewed Dec 26, 2023

View reviewed changes

Jhonatannunessilva added 4 commits December 28, 2023 23:25

Fix inspect between nodes

4893877

Move and Update inspect function

09b0e1e

Refactor inspect function

c1fb046

Mix format

1247cf1

Jhonatannunessilva force-pushed the fix-inspect-between-nodes branch from 6511685 to 1247cf1 Compare December 29, 2023 02:29

josevalim merged commit 5c51c26 into elixir-explorer:main Dec 29, 2023
3 checks passed

This was referenced Jan 4, 2024

Update Polars to v0.36 #797

Closed

Ps bump polars to v0.36 #798

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix inspect between nodes #784

Fix inspect between nodes #784

Jhonatannunessilva commented Dec 22, 2023 •

edited

Loading

Jhonatannunessilva commented Dec 22, 2023

philss left a comment

philss Dec 22, 2023

josevalim Dec 22, 2023

Jhonatannunessilva Dec 22, 2023

philss Dec 22, 2023

josevalim Dec 22, 2023

Jhonatannunessilva Dec 26, 2023

philss Dec 22, 2023

josevalim Dec 26, 2023

Jhonatannunessilva Dec 28, 2023

josevalim Dec 28, 2023

Jhonatannunessilva Dec 29, 2023

josevalim commented Dec 29, 2023

	"It is not possible to inspect a DataFrame that belongs to another node"
	"it is not possible to inspect a DataFrame that belongs to another node"

	"It is not possible to inspect a Series that belongs to another node"
	"it is not possible to inspect a Series that belongs to another node"

	Explorer.Backend.DataFrame.inspect(df, "Polars", nil, opts, from_another_node: true)
	Explorer.Backend.DataFrame.inspect(df, "Polars", "node: #{df.data.resource}", opts, elide_columns: true)

Fix inspect between nodes #784

Fix inspect between nodes #784

Conversation

Jhonatannunessilva commented Dec 22, 2023 • edited Loading

Jhonatannunessilva commented Dec 22, 2023

philss left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

josevalim commented Dec 29, 2023

Jhonatannunessilva commented Dec 22, 2023 •

edited

Loading