feat: beta tree exports/import

NewTreeCommandsPlan.md

@@ -0,0 +1,58 @@
+# New Data Tree Commands Plan
+
+We're improving the `data export tree` and `data import tree` commands, but doing it in phases.
+
+## Phase 1: Beta the new commands. [Now]
+
+Our plan:
+
+1. Introduce the new `data import beta tree` and `data export beta tree` commands.
+1. Both commands use the new helper functions.
+1. Update the existing commands' output with a message that encourages users to try the new commands.
+1. Ensure that export files created by `data export beta tree` are compatible with both `data import tree` and `data import beta tree`.
+
+These are the breaking changes between the existing and beta commands:
+
+- `data import beta tree`: We removed the hidden and deprecated `--content-type` flag. The command supports only JSON files. Usage of the flag: ~5 per year.
+- `data import beta tree`: We removed the `--config-help` flag because the schema information is in the command help. Usage of the flag: ~1 per week.
+
+Other differences:
+
+- `data export tree --plan` uses the object names as the new file name. Previously it appended an `s` on the end, but the new one doesn't. So the filename is now `Account.json` and `Foo__c.json` instead of `Accounts.json` and the awful `Foo__cs.json`.
+- `data export beta tree` no longer writes empty child objects. Previously, you saw properties with `{records: []}` that had no effect when imported.
+- `data import beta tree --plan` doesn't care about `resolveRefs` and `saveRefs`.
+- `data import beta tree --plan` doesn't care about the order of files in your `plan` file. Rather, it defers unresolved references until they're resolved.
+- `data export beta tree --plan` now handles more than 2 levels of child objects in a query. It can handle up to 5 levels, which is the SOQL limit.
+- Both `data export beta tree` and `data import beta tree` handle objects that refer to objects of the same type. For example, an Account with ParentId that's an Account or a User with Manager that's a User.
+- `data import beta tree --plan` handles more than 200 records. It batches them into groups of 200 per object. The new record limit isn't documented; it most likely comes from your operating system call stack depth or other org limits, such as data storage or api call limits.
+- `data import tree` supported plan files where the `files` property could contain objects. It's unclear how those files were generated, but probably not from the `data export tree` command. The new `data import beta tree` command works only with strings and throws an error otherwise.
+- `data import beta tree --files` (and not `--plan`) imports the files in parallel. Files can only reference each other if you specify `--plan`.
+- `data import tree` outputs deprecation warnings for both `--content-type` and `--config-help` flags.
+
+## Phase 2: GA the new commands, put the old ones under the `legacy` sub-topic. [July 10, 2024]
+
+Our plan:
+
+1. Pin an issue when the change goes into the release candidate so if users run into problems they can quickly find the `legacy` commands.
+1. Move the "old" commands to `legacy` and mark them `hidden` and `deprecated` with the Phase 3 date.
+1. Move the `force:` aliases to the new commands.
+1. Remove the `beta` sub-topic from the new commands, but keep the `beta` alias so they will still work. Add the `deprecateAliases` property to encourage users to stop using the commands in the `beta` sub-topic.
+1. Update `data export tree --plan` to display a warning that the JSON output is going to change after the Phase 3 date. Specifically, the JSON output won't include `saveRefs` and `resolveRefs` information.
+
+## Phase 3: Retire the `legacy` commands and all their dependent code. [Nov 10 2024]
+
+Our plan:
+
+1. Update `data export tree --plan` to stop writing the unused `saveRefs` and `resolveRefs` properties on plan files, and stop returning them in JSON output.
+1. Tighten the schema to remove the `object` part of `files`, and remove `saveRefs` and `resolveRefs`.
+1. Check messages for any that aren't being used, then remove them.
+1. Remove the `beta` alias from `data import tree` and `data export tree`.
+1. Update the pinned issue to reflect these changes.
+
+## How Does This Beta->Legacy Thing Work?
+
+Salesforce CLI uses `beta` and `legacy` subtopics to safely introduce beta versions of existing commands and then GA them. This approach allows you to try out the beta, while continuing to use the existing command at the same time. Let's look at an example to see how this works.
+
+- We create and release the `data export beta tree` command, which is similar to the existing `data export tree` command, but with improvements and possibly breaking changes. The two separate commands co-exist, which means if you run `sf commands`, you see both the existing and `beta` command.
+- After the beta period is over, we GA the changes by moving the functionality we added to `data export beta tree` to the "official" `data export tree` command. We also move the functionality in the _old_ `data export tree` to a new command called `data export legacy tree`. We hide and deprecate both the `legacy` and `beta` versions of the command, but alias the `beta` version to the `data export tree` command because they're now the same. If you run `sf commands` you see only the `data export tree` command, although the `legacy` version is still available (but hidden) if you really need it.
+- At some point, we remove the `data export legacy tree` command; we'll warn you, don't worry. We also remove the `beta` alias.

command-snapshot.json

@@ -1,23 +1,25 @@
 [
   {
-    "command": "data:create:record",
-    "plugin": "@salesforce/plugin-data",
-    "flags": ["api-version", "json", "loglevel", "perflog", "sobject", "target-org", "use-tooling-api", "values"],
     "alias": ["force:data:record:create"],
+    "command": "data:create:record",
+    "flagAliases": ["apiversion", "sobjecttype", "targetusername", "u", "usetoolingapi"],
     "flagChars": ["o", "s", "t", "v"],
-    "flagAliases": ["apiversion", "sobjecttype", "targetusername", "u", "usetoolingapi"]
+    "flags": ["api-version", "json", "loglevel", "perflog", "sobject", "target-org", "use-tooling-api", "values"],
+    "plugin": "@salesforce/plugin-data"
   },
   {
-    "command": "data:delete:bulk",
-    "plugin": "@salesforce/plugin-data",
-    "flags": ["api-version", "async", "file", "json", "loglevel", "sobject", "target-org", "verbose", "wait"],
     "alias": [],
+    "command": "data:delete:bulk",
+    "flagAliases": ["apiversion", "csvfile", "sobjecttype", "targetusername", "u"],
     "flagChars": ["a", "f", "o", "s", "w"],
-    "flagAliases": ["apiversion", "csvfile", "sobjecttype", "targetusername", "u"]
+    "flags": ["api-version", "async", "file", "json", "loglevel", "sobject", "target-org", "verbose", "wait"],
+    "plugin": "@salesforce/plugin-data"
   },
   {
+    "alias": ["force:data:record:delete"],
     "command": "data:delete:record",
-    "plugin": "@salesforce/plugin-data",
+    "flagAliases": ["apiversion", "sobjectid", "sobjecttype", "targetusername", "u", "usetoolingapi"],
+    "flagChars": ["i", "o", "s", "t", "w"],
     "flags": [
       "api-version",
       "json",
@@ -29,29 +31,37 @@
       "use-tooling-api",
       "where"
     ],
-    "alias": ["force:data:record:delete"],
-    "flagChars": ["i", "o", "s", "t", "w"],
-    "flagAliases": ["apiversion", "sobjectid", "sobjecttype", "targetusername", "u", "usetoolingapi"]
+    "plugin": "@salesforce/plugin-data"
   },
   {
-    "command": "data:delete:resume",
-    "plugin": "@salesforce/plugin-data",
-    "flags": ["api-version", "job-id", "json", "loglevel", "target-org", "use-most-recent", "wait"],
     "alias": [],
+    "command": "data:delete:resume",
+    "flagAliases": ["jobid", "targetusername", "u"],
     "flagChars": ["i", "o"],
-    "flagAliases": ["jobid", "targetusername", "u"]
+    "flags": ["api-version", "job-id", "json", "loglevel", "target-org", "use-most-recent", "wait"],
+    "plugin": "@salesforce/plugin-data"
   },
   {
-    "command": "data:export:tree",
-    "plugin": "@salesforce/plugin-data",
+    "alias": [],
+    "command": "data:export:beta:tree",
+    "flagAliases": ["apiversion", "outputdir", "targetusername", "u"],
+    "flagChars": ["d", "o", "p", "q", "x"],
     "flags": ["api-version", "json", "loglevel", "output-dir", "plan", "prefix", "query", "target-org"],
+    "plugin": "@salesforce/plugin-data"
+  },
+  {
     "alias": ["force:data:tree:export"],
+    "command": "data:export:tree",
+    "flagAliases": ["apiversion", "outputdir", "targetusername", "u"],
     "flagChars": ["d", "o", "p", "q", "x"],
-    "flagAliases": ["apiversion", "outputdir", "targetusername", "u"]
+    "flags": ["api-version", "json", "loglevel", "output-dir", "plan", "prefix", "query", "target-org"],
+    "plugin": "@salesforce/plugin-data"
   },
   {
+    "alias": ["force:data:record:get"],
     "command": "data:get:record",
-    "plugin": "@salesforce/plugin-data",
+    "flagAliases": ["apiversion", "sobjectid", "sobjecttype", "targetusername", "u", "usetoolingapi"],
+    "flagChars": ["i", "o", "s", "t", "w"],
     "flags": [
       "api-version",
       "json",
@@ -63,21 +73,29 @@
       "use-tooling-api",
       "where"
     ],
-    "alias": ["force:data:record:get"],
-    "flagChars": ["i", "o", "s", "t", "w"],
-    "flagAliases": ["apiversion", "sobjectid", "sobjecttype", "targetusername", "u", "usetoolingapi"]
+    "plugin": "@salesforce/plugin-data"
+  },
+  {
+    "alias": [],
+    "command": "data:import:beta:tree",
+    "flagAliases": ["apiversion", "sobjecttreefiles", "targetusername", "u"],
+    "flagChars": ["f", "o", "p"],
+    "flags": ["api-version", "files", "json", "loglevel", "plan", "target-org"],
+    "plugin": "@salesforce/plugin-data"
   },
   {
-    "command": "data:import:tree",
-    "plugin": "@salesforce/plugin-data",
-    "flags": ["api-version", "config-help", "content-type", "files", "json", "loglevel", "plan", "target-org"],
     "alias": ["force:data:tree:import"],
+    "command": "data:import:tree",
+    "flagAliases": ["apiversion", "confighelp", "contenttype", "sobjecttreefiles", "targetusername", "u"],
     "flagChars": ["c", "f", "o", "p"],
-    "flagAliases": ["apiversion", "confighelp", "contenttype", "sobjecttreefiles", "targetusername", "u"]
+    "flags": ["api-version", "config-help", "content-type", "files", "json", "loglevel", "plan", "target-org"],
+    "plugin": "@salesforce/plugin-data"
   },
   {
+    "alias": ["force:data:soql:query"],
     "command": "data:query",
-    "plugin": "@salesforce/plugin-data",
+    "flagAliases": ["apiversion", "resultformat", "soqlqueryfile", "targetusername", "u", "usetoolingapi"],
+    "flagChars": ["b", "f", "o", "q", "r", "t", "w"],
     "flags": [
       "all-rows",
       "api-version",
@@ -93,29 +111,29 @@
       "use-tooling-api",
       "wait"
     ],
-    "alias": ["force:data:soql:query"],
-    "flagChars": ["b", "f", "o", "q", "r", "t", "w"],
-    "flagAliases": ["apiversion", "resultformat", "soqlqueryfile", "targetusername", "u", "usetoolingapi"]
+    "plugin": "@salesforce/plugin-data"
   },
   {
-    "command": "data:query:resume",
-    "plugin": "@salesforce/plugin-data",
-    "flags": ["api-version", "bulk-query-id", "json", "loglevel", "result-format", "target-org", "use-most-recent"],
     "alias": ["force:data:soql:bulk:report"],
+    "command": "data:query:resume",
+    "flagAliases": ["apiversion", "bulkqueryid", "resultformat", "targetusername", "u"],
     "flagChars": ["i", "o", "r"],
-    "flagAliases": ["apiversion", "bulkqueryid", "resultformat", "targetusername", "u"]
+    "flags": ["api-version", "bulk-query-id", "json", "loglevel", "result-format", "target-org", "use-most-recent"],
+    "plugin": "@salesforce/plugin-data"
   },
   {
-    "command": "data:resume",
-    "plugin": "@salesforce/plugin-data",
-    "flags": ["api-version", "batch-id", "job-id", "json", "loglevel", "target-org"],
     "alias": [],
+    "command": "data:resume",
+    "flagAliases": ["apiversion", "batchid", "jobid", "targetusername", "u"],
     "flagChars": ["b", "i", "o"],
-    "flagAliases": ["apiversion", "batchid", "jobid", "targetusername", "u"]
+    "flags": ["api-version", "batch-id", "job-id", "json", "loglevel", "target-org"],
+    "plugin": "@salesforce/plugin-data"
   },
   {
+    "alias": ["force:data:record:update"],
     "command": "data:update:record",
-    "plugin": "@salesforce/plugin-data",
+    "flagAliases": ["apiversion", "sobjectid", "sobjecttype", "targetusername", "u", "usetoolingapi"],
+    "flagChars": ["i", "o", "s", "t", "v", "w"],
     "flags": [
       "api-version",
       "json",
@@ -128,13 +146,13 @@
       "values",
       "where"
     ],
-    "alias": ["force:data:record:update"],
-    "flagChars": ["i", "o", "s", "t", "v", "w"],
-    "flagAliases": ["apiversion", "sobjectid", "sobjecttype", "targetusername", "u", "usetoolingapi"]
+    "plugin": "@salesforce/plugin-data"
   },
   {
+    "alias": [],
     "command": "data:upsert:bulk",
-    "plugin": "@salesforce/plugin-data",
+    "flagAliases": ["apiversion", "csvfile", "externalid", "sobjecttype", "targetusername", "u"],
+    "flagChars": ["a", "f", "i", "o", "s", "w"],
     "flags": [
       "api-version",
       "async",
@@ -147,40 +165,38 @@
       "verbose",
       "wait"
     ],
-    "alias": [],
-    "flagChars": ["a", "f", "i", "o", "s", "w"],
-    "flagAliases": ["apiversion", "csvfile", "externalid", "sobjecttype", "targetusername", "u"]
+    "plugin": "@salesforce/plugin-data"
   },
   {
-    "command": "data:upsert:resume",
-    "plugin": "@salesforce/plugin-data",
-    "flags": ["api-version", "job-id", "json", "loglevel", "target-org", "use-most-recent", "wait"],
     "alias": [],
+    "command": "data:upsert:resume",
+    "flagAliases": ["jobid", "targetusername", "u"],
     "flagChars": ["i", "o"],
-    "flagAliases": ["jobid", "targetusername", "u"]
+    "flags": ["api-version", "job-id", "json", "loglevel", "target-org", "use-most-recent", "wait"],
+    "plugin": "@salesforce/plugin-data"
   },
   {
-    "command": "force:data:bulk:delete",
-    "plugin": "@salesforce/plugin-data",
-    "flags": ["api-version", "file", "json", "loglevel", "sobject", "target-org", "wait"],
     "alias": [],
+    "command": "force:data:bulk:delete",
+    "flagAliases": ["apiversion", "csvfile", "sobjecttype", "targetusername", "u"],
     "flagChars": ["f", "o", "s", "w"],
-    "flagAliases": ["apiversion", "csvfile", "sobjecttype", "targetusername", "u"]
+    "flags": ["api-version", "file", "json", "loglevel", "sobject", "target-org", "wait"],
+    "plugin": "@salesforce/plugin-data"
   },
   {
-    "command": "force:data:bulk:status",
-    "plugin": "@salesforce/plugin-data",
-    "flags": ["api-version", "batch-id", "job-id", "json", "loglevel", "target-org"],
     "alias": [],
+    "command": "force:data:bulk:status",
+    "flagAliases": ["apiversion", "batchid", "jobid", "targetusername", "u"],
     "flagChars": ["b", "i", "o"],
-    "flagAliases": ["apiversion", "batchid", "jobid", "targetusername", "u"]
+    "flags": ["api-version", "batch-id", "job-id", "json", "loglevel", "target-org"],
+    "plugin": "@salesforce/plugin-data"
   },
   {
-    "command": "force:data:bulk:upsert",
-    "plugin": "@salesforce/plugin-data",
-    "flags": ["api-version", "external-id", "file", "json", "loglevel", "serial", "sobject", "target-org", "wait"],
     "alias": [],
+    "command": "force:data:bulk:upsert",
+    "flagAliases": ["apiversion", "csvfile", "externalid", "sobjecttype", "targetusername", "u"],
     "flagChars": ["f", "i", "o", "r", "s", "w"],
-    "flagAliases": ["apiversion", "csvfile", "externalid", "sobjecttype", "targetusername", "u"]
+    "flags": ["api-version", "external-id", "file", "json", "loglevel", "serial", "sobject", "target-org", "wait"],
+    "plugin": "@salesforce/plugin-data"
   }
 ]

messages/importApi.md

@@ -49,3 +49,35 @@ Data plan file %s did not validate against the schema. Errors: %s.
 # FlsError
 
 We couldn't process your request because you don't have access to %s on %s. To learn more about field-level security, visit Tips and Hints for Page Layouts and Field-Level Security in our Developer Documentation.
+
+# error.InvalidDataImport
+
+Data plan file %s did not validate against the schema. Errors: %s.
+
+# error.InvalidDataImport.actions
+
+- Did you run the "sf data export tree" command with the --plan flag?
+
+- Make sure you're importing a plan definition file.
+
+- Get help with the import plan schema by running "sf data import beta tree --help".
+
+# saveResolveRefsIgnored
+
+The plan contains the 'saveRefs' and/or 'resolveRefs' properties.
+These properties will be ignored and can be removed.
+In the future, the `tree export` command will not produce them.
+
+# error.NonStringFiles
+
+The `files` property of the plan objects must contain only strings
+
+# error.UnresolvableRefs
+
+There are references in a data file %s that can't be resolved:
+
+%s
+
+# error.RefsInFiles
+
+The file %s includes references (ex: '@AccountRef1'). Those are only supported with --plan, not --files.`

messages/tree.export.md

@@ -30,7 +30,7 @@ Directory in which to generate the JSON files; default is current directory.
 
 - Export records retrieved with the specified SOQL query into a single JSON file in the current directory; the command uses your default org:
 
-  <%= config.bin %> <%= command.id %> --query "SELECT Id, Name, (SELECT Name, Address__c FROM Properties__r) FROM Broker__c"
+  <%= config.bin %> <%= command.id %> --query "SELECT Id, Name, (SELECT Name, Address\_\_c FROM Properties\_\_r) FROM Broker\_\_c"
 
 - Export data using a SOQL query in the "query.txt" file and generate JSON files for each object and a plan that aggregates them:
 
@@ -39,3 +39,7 @@ Directory in which to generate the JSON files; default is current directory.
 - Prepend "export-demo" before each generated file and generate the files in the "export-out" directory; run the command on the org with alias "my-scratch":
 
   <%= config.bin %> <%= command.id %> --query query.txt --plan --prefix export-demo --output-dir export-out --target-org my-scratch
+
+# PrefixSlashError
+
+`--prefix` cannot contain a forward slash or backslash.