Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cleans up lifecycle calls #30

Merged
merged 6 commits into from
Oct 29, 2017
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 2 additions & 23 deletions README.md
Original file line number Diff line number Diff line change
@@ -62,13 +62,6 @@ Catalog entry:
:onyx/doc "Partitions a range of primary keys into subranges"}
```

Lifecycle entry:

```clojure
{:lifecycle/task :partition-keys
:lifecycle/calls :onyx.plugin.sql/partition-keys-calls}
```

`:sql/columns` supports restricting the select to only certain columns, e.g. `:sql/columns [:id :name]`.

`:sql/lower-bound` overrides `partition-key` calculation of min from the `:sql/id` column.
@@ -97,13 +90,6 @@ Catalog entry:
:onyx/doc "Reads rows of a SQL table bounded by a key range"}
```

Lifecycle entry:

```clojure
{:lifecycle/task :read-rows
:lifecycle/calls :onyx.plugin.sql/read-rows-calls}
```

##### write-rows

Writes segments to a SQL database. Expects segments with the same schema as the
@@ -160,13 +146,6 @@ Catalog entry:
:onyx/doc "Upserts segments from the :rows keys to the SQL database"}
```

Lifecycle entry:

```clojure
{:lifecycle/task :upsert-rows
:lifecycle/calls :onyx.plugin.sql/upsert-rows-calls}
```

#### Attributes

|key | type | description
@@ -190,8 +169,8 @@ Pull requests into the master branch are welcomed.

Running the tests can be easily performed by starting a mysql and postgres container with docker:
```
docker run -e MYSQL_ROOT_PASSWORD="password" -p 3306:3306 -d mysql:latest
docker run -e POSTGRES_PASSWORD=password -e POSTGRES_USER=postgresql -e POSTGRES_DB=onyx_input_test -p 5432:5432 library/postgres
docker run -e MYSQL_ROOT_PASSWORD="password" -e MYSQL_DATABASE="onyx_input_test" -p 3306:3306 -d mysql:latest
docker run -e POSTGRES_PASSWORD=password -e POSTGRES_USER=postgresql -e POSTGRES_DB=onyx_input_test -p 5432:5432 -d library/postgres
```

#### License
32 changes: 2 additions & 30 deletions src/onyx/plugin/sql.clj
Original file line number Diff line number Diff line change
@@ -141,6 +141,7 @@
this)

(stop [this event]
(.close (:datasource pool))
this)

p/BarrierSynchronization
@@ -189,6 +190,7 @@
this)

(stop [this event]
(.close (:datasource pool))
this)

p/BarrierSynchronization
@@ -225,33 +227,3 @@
table (:sql/table task-map)
pool (task->pool task-map)]
(->SqlUpserter pool table)))

(defn inject-write-rows
[event lifecycle]
{})

(defn close-write-rows
[event lifecycle]
{})

(defn inject-upsert-rows
[event lifecycle]
{})

(defn close-update-rows
[event lifecycle]
{})

(def partition-keys-calls
{})

(def partition-uuid-calls
{})

(def write-rows-calls
{:lifecycle/before-task-start inject-write-rows
:lifecycle/after-task-stop close-write-rows})

(def upsert-rows-calls
{:lifecycle/before-task-start inject-upsert-rows
:lifecycle/after-task-stop close-update-rows})
12 changes: 4 additions & 8 deletions src/onyx/sql/information_model.cljc
Original file line number Diff line number Diff line change
@@ -166,23 +166,19 @@
:lifecycles-entry
{:onyx.plugin.sql/partition-keys
{:model
[{:task.lifecycle/name :partition-keys
:lifecycle/calls :onyx.plugin.sql/partition-keys-calls}]}
[]}

:onyx.plugin.sql/read-rows
{:model
[{:task.lifecycle/name :read-rows
:lifecycle/calls :onyx.plugin.sql/read-rows-calls}]}
[]}

:onyx.plugin.sql/write-rows
{:model
[{:task.lifecycle/name :write-rows
:lifecycle/calls :onyx.plugin.sql/write-rows-calls}]}
[]}

:onyx.plugin.sql/upsert-rows
{:model
[{:task.lifecycle/name :upsert-rows
:lifecycle/calls :onyx.plugin.sql/upsert-rows-calls}]}}
[]}}

:display-order
{:onyx.plugin.sql/partition-keys
12 changes: 3 additions & 9 deletions src/onyx/tasks/sql.clj
Original file line number Diff line number Diff line change
@@ -28,9 +28,7 @@
:sql/rows-per-segment 500
:sql/read-buffer 1000
:onyx/doc "Partitions a range of primary keys into subranges"}
opts)
:lifecycles [{:lifecycle/task task-name
:lifecycle/calls :onyx.plugin.sql/partition-keys-calls}]}
opts)}
:schema {:task-map SqlPartitionKeysTaskMap}})
([task-name :- s/Keyword
classname :- s/Str
@@ -61,9 +59,7 @@
:sql/columns [:*]
:sql/rows-per-segment 500
:onyx/doc "Partitions a range of primary keys into subranges"}
opts)
:lifecycles [{:lifecycle/task task-name
:lifecycle/calls :onyx.plugin.sql/partition-uuid-calls}]}
opts)}
:schema {:task-map SqlPartitionKeysTaskMap}})
([task-name :- s/Keyword
classname :- s/Str
@@ -106,9 +102,7 @@
:onyx/fn :onyx.plugin.sql/read-rows
:onyx/type :function
:onyx/doc "Reads rows of a SQL table bounded by a key range"}
opts)
:lifecycles [{:lifecycle/task task-name
:lifecycle/calls :onyx.plugin.sql/read-rows-calls}]}
opts)}
:schema {:task-map SqlReaderTaskMap}})
([task-name :- s/Keyword
classname :- s/Str
2 changes: 0 additions & 2 deletions test/onyx/plugin/output_copy_test.clj
Original file line number Diff line number Diff line change
@@ -48,8 +48,6 @@
:onyx/type :function
:onyx/batch-size batch-size
:onyx/doc "Transforms a segment to prepare for SQL persistence"}]
:lifecycles [{:lifecycle/task :out
:lifecycle/calls :onyx.plugin.sql/write-rows-calls}]
:windows []
:triggers []
:flow-conditions []
2 changes: 0 additions & 2 deletions test/onyx/plugin/output_test.clj
Original file line number Diff line number Diff line change
@@ -46,8 +46,6 @@
:onyx/type :function
:onyx/batch-size batch-size
:onyx/doc "Transforms a segment to prepare for SQL persistence"}]
:lifecycles [{:lifecycle/task :out
:lifecycle/calls :onyx.plugin.sql/write-rows-calls}]
:windows []
:triggers []
:flow-conditions []
2 changes: 0 additions & 2 deletions test/onyx/plugin/output_update_test.clj
Original file line number Diff line number Diff line change
@@ -47,8 +47,6 @@
:onyx/type :function
:onyx/batch-size 1000
:onyx/doc "Transforms a segment to prepare for SQL persistence"}]
:lifecycles [{:lifecycle/task :out
:lifecycle/calls :onyx.plugin.sql/upsert-rows-calls}]
:windows []
:triggers []
:flow-conditions []