Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Rework relation field kinds #2961

Merged
merged 12 commits into from
Sep 7, 2024

Conversation

AndrewSisley
Copy link
Contributor

@AndrewSisley AndrewSisley commented Aug 26, 2024

Relevant issue(s)

Resolves #2619 #2493

Description

Reworks relation field kinds so that relations are not defined by names (which are mutable).

This PR replaces the old ObjectKind and ObjectArrayKinds with CollectionKind, SchemaKind and SelfKind. NamedKind has also been added - it allows users to interact with the definitions using their names (like before), before the kind gets translated into one of the other new kinds before storage.

Because Schemas must form a DAG (because of their CIDs (SchemaVersionID and Root), Schemas defined within the same action that form circular relations (e.g. User=>Dog=>User) have their CIDs grouped into a set with the same base id, plus an index relative to their lexographical order (name) within the set suffixed onto their CIDs. This change is not user visible, besides the -[index] suffix on their CIDs.

Quite a lot had to change to facilitate this, and only a handful of things have been broken out to separate commits. Most of the work is in commit Rework relation field kinds - when reviewing that commit, I suggest starting with tests/integration/collection_description/updates/replace/name_one_many_test.go to see the original bug, followed by a quick scan of client/schema_field_description.go, then a look at tests/integration/schema/self_ref_test.go before properly reviewing the changes. The bulk of the self reference logic is in internal/db/schema_id.go.

Please give me a shout if you have any questions! I think this should be the last structural change to collection/schema definitions :)

@AndrewSisley AndrewSisley added bug Something isn't working area/schema Related to the schema system area/collections Related to the collections system labels Aug 26, 2024
@AndrewSisley AndrewSisley added this to the DefraDB v0.14 milestone Aug 26, 2024
@AndrewSisley AndrewSisley requested a review from a team August 26, 2024 16:44
@AndrewSisley AndrewSisley self-assigned this Aug 26, 2024
@AndrewSisley AndrewSisley changed the title fix: Rework relation field kinds fix: Rework relation field kinds Aug 26, 2024
Copy link

codecov bot commented Aug 26, 2024

Codecov Report

Attention: Patch coverage is 80.75253% with 133 lines in your changes missing coverage. Please review.

Project coverage is 79.26%. Comparing base (f1489ef) to head (f0f1a10).
Report is 1 commits behind head on develop.

Files with missing lines Patch % Lines
client/definitions.go 61.45% 30 Missing and 2 partials ⚠️
client/schema_field_description.go 67.57% 23 Missing and 1 partial ⚠️
internal/db/description/collection.go 50.00% 9 Missing and 5 partials ⚠️
internal/db/collection_id.go 82.76% 5 Missing and 5 partials ⚠️
internal/core/key.go 72.41% 5 Missing and 3 partials ⚠️
internal/db/schema.go 92.79% 4 Missing and 4 partials ⚠️
internal/db/schema_id.go 95.15% 4 Missing and 4 partials ⚠️
internal/db/collection_define.go 60.00% 3 Missing and 3 partials ⚠️
client/errors.go 0.00% 5 Missing ⚠️
internal/db/errors.go 0.00% 5 Missing ⚠️
... and 5 more
Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff             @@
##           develop    #2961      +/-   ##
===========================================
- Coverage    79.28%   79.26%   -0.02%     
===========================================
  Files          326      328       +2     
  Lines        24778    25207     +429     
===========================================
+ Hits         19644    19980     +336     
- Misses        3715     3791      +76     
- Partials      1419     1436      +17     
Flag Coverage Δ
all-tests 79.26% <80.75%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
internal/db/collection_update.go 75.84% <100.00%> (ø)
internal/db/description/schema.go 66.18% <100.00%> (-0.95%) ⬇️
internal/request/graphql/schema/collection.go 90.41% <100.00%> (+0.06%) ⬆️
internal/request/graphql/schema/generate.go 87.92% <80.00%> (ø)
internal/db/backup.go 64.35% <71.43%> (+1.85%) ⬆️
internal/planner/mapper/mapper.go 88.61% <50.00%> (-0.18%) ⬇️
internal/db/collection.go 73.28% <60.00%> (-0.37%) ⬇️
internal/db/definition_validation.go 95.41% <94.74%> (+0.29%) ⬆️
client/errors.go 54.69% <0.00%> (-4.63%) ⬇️
internal/db/errors.go 64.61% <0.00%> (-1.07%) ⬇️
... and 8 more

... and 13 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f1489ef...f0f1a10. Read the comment docs.

Copy link
Collaborator

@fredcarle fredcarle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great work Andy! I like the complex schema being treated as a single schema ID with parts. The code is nice. There are only a few places where naming caused a bit of confusion on my end and I gave a few suggestion to alleviate it. Feel free to merge once the conversations are resolved.

return err
}

setIDs(setID, schemaSet)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thought: The close proximity of both set as a verb and set as a noun confused me a few time while reading this function. Maybe it would be worth changing the noun to something else. Maybe something like generateGroupID would be better?

Copy link
Contributor Author

@AndrewSisley AndrewSisley Sep 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lol - nice spot, yeah I'll change that to generateGroupID or similar.

  • rename setIDs

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed to assignIDs, as the func does not generate the ids

// GetDefinitionUncached returns the definition that the given [FieldKind] points to, if it is found in the given store.
//
// If the related definition is not found, or an error occurs, default and false will be returned.
func GetDefinitionUncached(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: GetDefinitionUncached is a confusing name. I would suggest instead being more explicit with GetDefinitionFromCache and GetDefinitionFromStore.

Copy link
Contributor Author

@AndrewSisley AndrewSisley Sep 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this, thanks.

  • Rename GetDefinitionUncached

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving GetDefinition as is, as it should be the default IMO. GetDefinitionUncached has been renamed to GetDefinitionFromStore

if _, ok := schemaVersionsAdded[schema.VersionID]; ok {
continue
}
// newDefinitionState creates a new definitionState object given the provided
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: The documentation for the function and the one above are too similar. This one should probably have the provided definition instead of description. Maybe more explicit names would help here as well: newDefinitionStateFromDef and newDefinitionStateFromDesc

Copy link
Contributor Author

@AndrewSisley AndrewSisley Sep 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a problem, I'll have a look and tweak them a bit.

  • Review newDefinitionStates plus docs

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Docs tweaked, renamed the first col-only func to newDefinitionStateFromCols (newDefinitionState should be the default func to use)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FromCols is not really clarifying anything though as they are both using collectionSomething.

ctx context.Context,
db *db,
newState *definitionState,
oldState *definitionState,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: You can do _ *definitionState istead of oldSate *definitionState. This way it shows that this is there to follow a given function signature but that the field isn't used within.

Copy link
Contributor Author

@AndrewSisley AndrewSisley Sep 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm probably going to leave this one as is, this func (and most other funcs in this file) satisfies an interface and they all have the same prop names and I think I prefer consistency over worrying about which funcs use which props (also aids copy-pasting the sig).

  • Have a quick look when looking at newDefinitionState

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving as-is

I stumbled across this accidentally, it is not relevant to this PR.
No tests have been modified.  a 'relations' file does not scale particularly well, and there is already a one_one file.
They have since been replaced by integration tests and are really quite annoying to maintain.
The value implies it has meaning and is checked, which it does not
Before this fix the test framework would panic when using things like DocIndexes for renamed collections.
These were broken by the previous commit, changes split out to this commit to avoid cluttering the main change
@AndrewSisley AndrewSisley merged commit 04531c3 into sourcenetwork:develop Sep 7, 2024
40 of 42 checks passed
@AndrewSisley AndrewSisley deleted the 2619-self-field-kind branch September 7, 2024 16:43
AndrewSisley added a commit that referenced this pull request Sep 26, 2024
## Relevant issue(s)

Resolves #3074

## Description

Correctly handles validation of nearby relation fields in schema patch.

Previously the equality check failed to account for `Kind` being a
pointer and thus always flagged relation fields as having mutated. This
was likely introduced in
#2961 when `Kind` became a
pointer.

Collections referenced by relation fields were also not included in the
validation set, causing the rules to think that the related object did
not exist.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/collections Related to the collections system area/schema Related to the schema system bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add Self field kind
2 participants