Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for insert CSV #1067

Merged
merged 5 commits into from
Dec 16, 2022
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 10 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,15 +112,15 @@ important DataJoint schema or records.

### API docs

The API documentation can be built using sphinx by running
The API documentation can be built with mkdocs using the docker compose file in
`docs/` with the following command:

``` bash
pip install sphinx sphinx_rtd_theme
(cd docs-api/sphinx && make html)
MODE="LIVE" PACKAGE=datajoint UPSTREAM_REPO=https://github.com/datajoint/datajoint-python.git HOST_UID=$(id -u) docker compose -f docs/docker-compose.yaml up --build
```

Generated docs are written to `docs-api/docs/html/index.html`.
More details in [docs-api/README.md](docs-api/README.md).
The site will then be available at `http://localhost/`. When finished, be sure to run
the same command as above, but replace `up --build` with `down`.

## Running Tests Locally
<details>
Expand All @@ -141,11 +141,11 @@ HOST_GID=1000
* Add entry in `/etc/hosts` for `127.0.0.1 fakeservices.datajoint.io`
* Run desired tests. Some examples are as follows:

| Use Case | Shell Code |
| ---------------------------- | ------------------------------------------------------------------------------ |
| Run all tests | `nosetests -vsw tests --with-coverage --cover-package=datajoint` |
| Run one specific class test | `nosetests -vs --tests=tests.test_fetch:TestFetch.test_getattribute_for_fetch1` |
| Run one specific basic test | `nosetests -vs --tests=tests.test_external_class:test_insert_and_fetch` |
| Use Case | Shell Code |
| ---------------------------- | ------------------------------------------------------------------------------ |
| Run all tests | `nosetests -vsw tests --with-coverage --cover-package=datajoint` |
| Run one specific class test | `nosetests -vs --tests=tests.test_fetch:TestFetch.test_getattribute_for_fetch1` |
| Run one specific basic test | `nosetests -vs --tests=tests.test_external_class:test_insert_and_fetch` |


### Launch Docker Terminal
Expand Down
16 changes: 12 additions & 4 deletions datajoint/table.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
import pandas
import logging
import uuid
import csv
import re
from pathlib import Path
from .settings import config
Expand Down Expand Up @@ -345,13 +346,16 @@ def insert(
"""
Insert a collection of rows.

:param rows: An iterable where an element is a numpy record, a dict-like object, a
pandas.DataFrame, a sequence, or a query expression with the same heading as self.
:param rows: Either (a) an iterable where an element is a numpy record, a
dict-like object, a pandas.DataFrame, a sequence, or a query expression with
the same heading as self, or (b) a pathlib.Path object specifying a path
relative to the current directory with a CSV file, the contents of which
will be inserted.
:param replace: If True, replaces the existing tuple.
:param skip_duplicates: If True, silently skip duplicate inserts.
:param ignore_extra_fields: If False, fields that are not in the heading raise error.
:param allow_direct_insert: applies only in auto-populated tables. If False (default),
insert are allowed only from inside the make callback.
:param allow_direct_insert: Only applies in auto-populated tables. If False (default),
insert may only be called from inside the make callback.

Example:

Expand All @@ -366,6 +370,10 @@ def insert(
drop=len(rows.index.names) == 1 and not rows.index.names[0]
).to_records(index=False)

if isinstance(rows, Path):
with open(rows, newline="") as data_file:
rows = list(csv.DictReader(data_file, delimiter=","))
CBroz1 marked this conversation as resolved.
Show resolved Hide resolved

# prohibit direct inserts into auto-populated tables
if not allow_direct_insert and not getattr(self, "_allow_insert", True):
raise DataJointError(
Expand Down
6 changes: 2 additions & 4 deletions tests/test_university.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,11 +33,9 @@ def test_activate():
Enroll,
Grade,
):
import csv
from pathlib import Path

with open("./data/" + table.__name__ + ".csv") as f:
reader = csv.DictReader(f)
table().insert(reader)
table().insert(Path("./data/" + table.__name__ + ".csv"))


def test_fill():
Expand Down