Skip to content

Commit

Permalink
File connector quickstart (#179)
Browse files Browse the repository at this point in the history
* file connector quickstart

* update READNE.md
  • Loading branch information
Jeadie authored Sep 16, 2024
1 parent a5797ce commit e1e1be9
Showing 1 changed file with 79 additions and 0 deletions.
79 changes: 79 additions & 0 deletions file/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
# File Data Connector

Follow these steps to get started with using local files as a Data Connector.

## Parquet

1. Download or move a parquet file locally
```shell
wget https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2024-01.parquet -O yellow_tripdata_2024-01.parquet
```

2. Create the Spicepod.
```shell
cat <<EOF > spicepod.yaml
version: v1beta1
kind: Spicepod
name: file_quickstart
datasets:
- name: yellow_taxis
from: file:yellow_tripdata_2024-01.parquet
EOF
```

3. Run Spice runtime
```shell
spice run
```

4. Run SQL query
```sql
select avg(passenger_count) from taxi_trips
```

5. (Optional) Cleanup
```shell
rm spicepod.yaml
```

## Documents

1. Download markdown documents
```shell
base_url="https://raw.githubusercontent.com/spiceai/docs/refs/heads/trunk/spiceaidocs/docs/components/data-connectors"

files=(
"clickhouse.md"
"databricks.md"
"debezium.md"
"delta-lake.md"
)

for file in "${files[@]}"; do
wget "$base_url/$file"
done
```

2. Create the Spicepod.
```shell
cat <<EOF > spicepod.yaml
version: v1beta1
kind: Spicepod
name: file_quickstart
datasets:
- name: docs
from: file:./
params:
file_format: md
EOF
```

3. Run Spice runtime
```shell
spice run
```

4. Run SQL query
```sql
select location from docs
```

0 comments on commit e1e1be9

Please sign in to comment.