Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Iter on node and commit graphs #1077

Merged
merged 3 commits into from
Sep 15, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGES.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,8 @@
- Added a `clear` function for stores (#1071, @icristescu, @CraigFe)
- Requires digestif>=0.9 to use digestif's default variants
(#873, @pascutto, @samoht)
- Added `iter_commits` and `iter_nodes` functions to traverse the commits and
nodes graphs (#1077, @icristescu)

- **irmin-pack**:
- Added `index_throttle` option to `Irmin_pack.config`, which exposes the
Expand Down
23 changes: 23 additions & 0 deletions src/irmin/commit.ml
Original file line number Diff line number Diff line change
Expand Up @@ -185,6 +185,29 @@ module History (S : S.COMMIT_STORE) = struct
(fun acc -> function `Commit k -> k :: acc | _ -> acc)
[] (Graph.vertex g)

let pp_keys = Type.(pp_dump (list S.Key.t))

let ignore_lwt _ = Lwt.return_unit

let iter t ~min ~max ?(commit = ignore_lwt) ?(edge = fun _ -> ignore_lwt)
?(skip = fun _ -> Lwt.return_false) ?(rev = true) () =
Log.debug (fun f ->
f "iter on closure min=%a max=%a" pp_keys min pp_keys max);
let max = List.map (fun x -> `Commit x) max in
let min = List.map (fun x -> `Commit x) min in
let node = function `Commit x -> commit x | _ -> assert false in
let skip = function `Commit x -> skip x | _ -> assert false in
let pred = function
| `Commit k -> parents t k >|= List.map (fun x -> `Commit x)
| _ -> assert false
in
let edge n pred =
match (n, pred) with
| `Commit src, `Commit dst -> edge src dst
| _ -> assert false
in
Graph.iter ~pred ~min ~max ~node ~edge ~skip ~rev ()

module K = struct
type t = S.Key.t

Expand Down
21 changes: 17 additions & 4 deletions src/irmin/s.ml
Original file line number Diff line number Diff line change
Expand Up @@ -372,9 +372,9 @@ module type NODE_GRAPH = sig
?rev:bool ->
unit ->
unit Lwt.t
(** [iter min max node edge skip rev ()] iterates in topological order over
the closure of [t] as specified by {{!Private.Node.GRAPH.closure}
GRAPH.closure}.
(** [iter t min max node edge skip rev ()] iterates in topological order over
the closure of [t] as specified by {{!Irmin__Object_graph.S.closure}
Object_graph.closure}.

It applies three functions while traversing the graph: [node] on the
nodes; [edge n predecessor_of_n] on the directed edges and [skip n] to not
Expand Down Expand Up @@ -555,9 +555,22 @@ module type COMMIT_HISTORY = sig

val closure :
[> `Read ] t -> min:commit list -> max:commit list -> commit list Lwt.t
(** Same as {{!Private.Node.GRAPH.closure} GRAPH.closure} but for the history
(** Same as {{!NODE_GRAPH.closure} NODE_GRAPH.closure} but for the history
graph. *)

val iter :
[> `Read ] t ->
min:node list ->
max:node list ->
?commit:(commit -> unit Lwt.t) ->
?edge:(node -> node -> unit Lwt.t) ->
?skip:(node -> bool Lwt.t) ->
?rev:bool ->
unit ->
unit Lwt.t
(** Same as {{!NODE_GRAPH.iter} NODE_GRAPH.iter} but for traversing the
history graph. *)

(** {1 Value Types} *)

val commit_t : commit Type.t
Expand Down
4 changes: 4 additions & 0 deletions src/irmin/store.ml
Original file line number Diff line number Diff line change
Expand Up @@ -331,6 +331,10 @@ module Make (P : S.PRIVATE) = struct
(function
| Import_error e -> Lwt.return (Error (`Msg e))
| e -> Fmt.kstrf Lwt.fail_invalid_arg "impot error: %a" Fmt.exn e)

let iter_nodes t = Graph.iter (graph_t t)

let iter_commits t = H.iter (history_t t)
end

type t = {
Expand Down
27 changes: 27 additions & 0 deletions src/irmin/store_intf.ml
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,33 @@ module type S = sig
val import : t -> slice -> (unit, [ `Msg of string ]) result Lwt.t
(** [import t s] imports the contents of the slice [s] in [t]. Does not
modify branches. *)

val iter_nodes :
t ->
min:hash list ->
max:hash list ->
?node:(hash -> unit Lwt.t) ->
?contents:(hash * metadata -> unit Lwt.t) ->
?edge:(hash -> hash -> unit Lwt.t) ->
?skip:(hash -> bool Lwt.t) ->
?rev:bool ->
unit ->
unit Lwt.t
(** [iter t] iterates in reverse topological order over the closure graph of
[t] as specified by {{!Irmin__.S.NODE_GRAPH.iter} NODE_GRAPH.iter}. *)

val iter_commits :
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a feeling that this function is also going to iterate over the nodes and the contents referenced by the closure of commits given, it's just not going to do anything when it visits them; is that right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it just ignores them. So we can add ?node:(node -> unit Lwt.t) and ?contents:(contents * metadata -> unit Lwt.t). I didn't do it because I don't need it in the layered store, but maybe we should add them for future usage?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not having iterators for node and contents seems fine. I was mostly concerned about performance: iterating over all objects in the closure when you're only making progress for the commits feels like it might be quite inefficient.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, it actually only iterates over commits, as the predecessors of a commit are its parents. Unlike for nodes where predecessors can be both nodes and contents. So I was wrong above, we cannot iterate over nodes and contents with this function.

Copy link
Member

@craigfe craigfe Sep 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm... indeed. This iter function never seems to behave the way I expect it to 😛

Can you add something like `Node _ | `Contents _ -> assert false in the implementation, just to make that clear?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice suggestions, thanks!

t ->
min:hash list ->
max:hash list ->
?commit:(hash -> unit Lwt.t) ->
?edge:(hash -> hash -> unit Lwt.t) ->
?skip:(hash -> bool Lwt.t) ->
?rev:bool ->
unit ->
unit Lwt.t
(** [iter t] iterates over the closure graph of [t] as specified by
{{!Irmin__.S.COMMIT_HISTORY.iter} COMMIT_HISTORY.iter}. *)
end

val empty : repo -> t Lwt.t
Expand Down