You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@jthandy noticed that on a project with hundreds of models and hundreds of tests, parsing alone can take a long time. I asked him to profile the project in question, and it turns out that around 30-35 seconds is spent in parsing nodes alone. A very large component of that parsing is building and validating the macros-only Manifest in BaseParser.parse_node (at time of writing parsers/base.py line 107).
Profile result indicates each Manifest.__init__ call takes about 45ms. The vast majority (>90%) of that is validation. In a project with 750 models + tests, that adds up to about 34 seconds added to a run that we could strip off.
I can imagine some potential fixes:
we could cache the macros-only Manifest that gets re-used in parse_node
we could change parse_node to use the jinja AST to pull out refs and config values
we could skip validation when building the "fake" macros-only Manifest
Any one of these would result in an enormous speedup for the project in question.
The text was updated successfully, but these errors were encountered:
@jthandy noticed that on a project with hundreds of models and hundreds of tests, parsing alone can take a long time. I asked him to profile the project in question, and it turns out that around 30-35 seconds is spent in parsing nodes alone. A very large component of that parsing is building and validating the macros-only Manifest in
BaseParser.parse_node
(at time of writing parsers/base.py line 107).Profile result indicates each
Manifest.__init__
call takes about 45ms. The vast majority (>90%) of that is validation. In a project with 750 models + tests, that adds up to about 34 seconds added to a run that we could strip off.I can imagine some potential fixes:
Any one of these would result in an enormous speedup for the project in question.
The text was updated successfully, but these errors were encountered: