Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update dataset #241

Merged
merged 6 commits into from
Sep 22, 2016
Merged

Update dataset #241

merged 6 commits into from
Sep 22, 2016

Conversation

spalladino
Copy link
Contributor

Support running a reimport from resmap for a dataset. Sites and types insertion is now an upsert, and inserts/updates records as needed. New steps are added to the import job FSM to delete stale types and facilities.

An Update button has been added to the datasets list as well, to trigger an update operation.

Fixes #180

(vec)
(keep-indexed #(when (= id (:id %2)) %1))
(first))]
(assoc (vec coll) index replacement)))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not use a map or a reduce here?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And probably implement as (update-by-id coll (:id replacement) constantly replacement)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 7acb588

(vec)
(keep-indexed #(when (= id (:id %2)) %1))
(first))]
(apply update (vec coll) index update-fn args)))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above, use a map or reduce.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 7acb588

Methods for bulk insert first check if the facility and facility types exist, and insert/update as needed. Insert facilities now returns a hash instead of a vector, indicating the status for each of the facilities requested to be inserted.

Deletion methods now also accept an except key, to delete all facilities or types except those just upserted.
Add keys process-count and process-ids to import job to keep track of facilities to be processed, as only the new or moved facilities need reprocessing.

Add new fsm states for deleting old facilities and types.
@spalladino spalladino force-pushed the feature/update-dataset branch from 03c990f to c9e82f7 Compare September 22, 2016 19:16
(map (fn [facility]
(if-let [{id :id, :as existing} (some-> (filter #(= (:site-id %) (:site-id facility))
existing-facilities)
(first)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 0693beb

[:moved id])))
; Insert the new facility otherwise
(let [result (insert-facility! tx (assoc facility :dataset-id dataset-id))]
[:new (:id result)]))))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A cond instead of the nested ifs would be more readable here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 0693beb

(let [current-types (select-types-in-dataset tx {:dataset-id dataset-id})]
(->> types
(mapv (fn [type]
(if-let [existing-type (first (filter #(= (:name type) (:name %)) current-types))]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as before, implement a find-by somewhere and use here as well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 0693beb

@@ -42,7 +42,7 @@

(defn- server-status->string
[server-status]
(case (:status server-status)
(case (keyword (:status server-status))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missed the labels for the new server statuses.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 4fcb97a

@ggiraldez ggiraldez merged commit 60fe996 into master Sep 22, 2016
@ggiraldez ggiraldez deleted the feature/update-dataset branch September 22, 2016 20:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants