[Epic] Support schema updates

[AndrewSisley]

Currently schemas can only be created, never updated or deleted. We need to allow this mutation, and we need a mechanic (Lens) to handle version differences between nodes.

This epic has been converted to an almanac form available here: https://source.almanac.io/docs/N2o87ZyxzsruNHSlAErwHMVnqNBPOZFn

Watch out for time-traveling queries and absolutely ensure that they are tested.

WARNING: The P2P system will break in its current form if collections are allowed to be renamed, see discussion in discord thread dev-db/Collection vs Schema (search for "The second thing is related to the schema ID. The P2P system uses").

Children (very roughly ordered in descending order of todo):

Tasks

Preview Give feedback

1. Draft out schema update typed syntax #1013
   area/query area/schema +2
   
2. Draft out schema update DDL syntax #1014
   area/query area/schema +2
   
3. Rework CollectionKeys to allow versioning of schema #1022
   area/collections area/schema refactor +3
   
4. Couple commit to schema version #1023
   area/query area/schema feature +3
   
5. Add schema version identifier field to commit queries #1006
   area/query area/schema feature +3
   
6. Add schema version identifier field to time-travelling queries #1008
   area/query area/schema feature +3
   
7. Ensure transactional schema update protection #1015
   area/schema area/testing +2
8. Allow new non-relational fields to be added to existing schema locally #1004
   area/schema feature +2
   
9. Add P2P tests that cover new fields added to node(s) #1181
   area/p2p area/schema area/testing +3
   
10. Add GQL introspection tests for Schema Updates #1208
    area/schema area/testing +2
    
11. Allow string/named substitution of field Kind #1220
    area/schema feature +2
    
12. Add PatchSchema to CLI #1209
    area/cli area/schema feature +3
13. Add support for adding relational fields to schema #1747
    area/schema feature +2
    
14. Extend PatchSchema field Kind substitution magic to foriegn relations #1771
    area/schema feature +2
    
15. Add PatchSchema syntax sugar for RelationType in foreign relations #1772
    area/schema feature +2
    
16. Auto create foreign id fields in PatchSchema #1773
    area/schema feature +2
    
17. Auto define RelationName in PatchSchema foreign relations #1774
    area/schema feature +2
18. Allow field renaming via PatchSchema #1775
    area/schema +1
19. Allow string/named substitution of field indexes #1169
    area/schema feature +2
    
20. Allow getting collection by version id #1007
    area/collections area/schema feature +3
21. Allow non-relational schemas to be deleted locally #1003
    area/schema feature +2
22. Allow deletion of non-relational fields from local schema #1005
    area/schema feature +2
23. Add means to query deleted fields in time-travelling queries #1009
    area/query area/schema feature +3
24. Ensure deleted fields are targetable by commit queries #1010
    area/query area/schema area/testing feature +4
25. Ensure deleted schemas are targetable by commit queries #1011
    area/query area/schema area/testing feature +4
26. Add means to query deleted schema in time-travelling queries #1012
    area/query area/schema feature +3
27. Allow new schema versions without having to update the schema #1024
    area/p2p feature +2
28. Add schema migration system to Defra #1448
    area/schema epic priority/high +3
    
29. Track schema version history #1556
    area/schema feature +2
    
30. Cache schema history in the Lens Registery #1590
    area/schema perf +2
31. Simplify Fetcher interface #1589
    area/db-system refactor +2
    
32. Use Datastore version with Lens when going across P2P #1591
    area/p2p area/schema bug +3
    
33. LensRegistry does not allow registered items to be accessed within current txn #1592
    area/schema bug +2
    
34. Reset module state (not including params) on enumerable.Reset lens-vm/lens#46
    bug +1
35. Lens/registry.go requires unit tests #1596
    area/schema area/testing code quality +3
36. Handle Lens migration of indexed collections #1597
    area/db-system area/schema bug +3
37. Migrations assume no schema history branching #1598
    area/schema feature +2
38. Defra does not take advantage of Lens-Lens enumerable #1601
    area/schema perf +2
39. Reuse lens wasm modules #1640
    action/no-benchmark area/schema perf +3
    
40. Add migration system to CLI (and http) #1648
    area/api area/cli area/schema feature +4
    
41. Sort out client.LensRegistry transaction stuff #1649
    area/schema code quality +2
    
42. Handle downgrading of datastore versions of P2P synced docs #1719
    area/schema feature +2
    
43. Allow users to set default schema version #1884
    area/collections area/schema feature +3
    
44. Allow querying at explicit schema version id #1885
    area/query area/schema feature +3
45. Allow labelling of schema versions #1886
    area/query area/schema feature +3
46. Values in new fields are not visible if docs synced before receiving node updates schema #2255
    area/query area/schema bug +3
47. Clean up RelationType #2287
    area/schema feature +2
    

Loosely related:

• GetCollectionByName(ctx, "user") error should be better for inexistant name. #1016
• Ambiguity between use of 'Collection' and 'Schema' #1025
• P2P integration tests do not cover relational objects #1027
• Net/utils does not assert that all ReplicatorResults are present #985
• Add user-definable description property to collection #1386
• Add user-definable description property to fields #1385