Reading of a data source

[loleg]

In addition to the basic streamed loading of data sources into a Table as per #6, provide the ability to read through a Table with cast on iteration so that the individual cells of the table are formatted according to the Schema provided.

• A read API* should be accessible for a Table class.
• Standard library CSV loading as well as external loading supported.
• Headers are loaded and validated.
• Errors from loading are handled in a standard way.
• Tests cover the basic design and several reading scenarios.
• Tests cover cast on iteration, including exception handling.

[roll]

@loleg
Take a look on reading capabilities for the Table class - https://github.com/frictionlessdata/tableschema-py#table

It reads data source and cast data according to the schema (if provided). Also it doesn't load the whole data file into memory (but one row per time - streaming).

[loleg]

As per #4 I think we should continue discussion of streaming library choice here. In reviewing the implementation notes, I wondered:

iter(keyed/extended=False, cast=True, relations=False) -> (generator) (keyed/extended)row[]

Is this really a generator, or an iterator? In Julia, iterators are made by applying protocol interfaces to the type.

Since in our case generators are being used for streaming data, I am inclined to think that Channels would be more appropriate and performant.

But the most sensible course of action right now would probably be simply to implement what the approach we take to loading data recommends.

[roll]

@loleg
I think technically it's a generator - a function we need to call to get an iterator. That because we need an ability to read the data source multiple times. But TBH I don't think it's a good idea to rely to much onto this terminology because it could differ language to language.

rows = table.iter(keyed=True)
for row in rows:
  print(row)

Probably a good idea could be to implement the simplest table.read (read into memory all rows) function first. At least it will be working implementation in case of streaming problems which could block the whole work.

[loleg]

@roll I have updated this issue description with my current understanding, please confirm.

[roll]

@loleg
It sounds good. One question is still do we need to support both standard library CSV loading as well as external loading supported.

[loleg]

I've prototyped some approaches to include an external library, but don't like to have heavy dependencies or slow-to-run unit tests. Therefore I have made the Table source typeless and implemented and tested (see a1eca25) initial support to read the table from an IOBuffer or String, which can be used by any framework. For example, the headers and top rows of a DataFrame could be used for loading the Table, validation would happen through standard iteration.

[loleg]

Still a bit stuck on the way forward here, I've been considering implementing a DataStreams.jl interface or supporting IterableTables.jl.