diff --git a/NewTreeCommandsPlan.md b/NewTreeCommandsPlan.md new file mode 100644 index 00000000..d4c5cb3f --- /dev/null +++ b/NewTreeCommandsPlan.md @@ -0,0 +1,58 @@ +# New Data Tree Commands Plan + +We're improving the `data export tree` and `data import tree` commands, but doing it in phases. + +## Phase 1: Beta the new commands. [Now] + +Our plan: + +1. Introduce the new `data import beta tree` and `data export beta tree` commands. +1. Both commands use the new helper functions. +1. Update the existing commands' output with a message that encourages users to try the new commands. +1. Ensure that export files created by `data export beta tree` are compatible with both `data import tree` and `data import beta tree`. + +These are the breaking changes between the existing and beta commands: + +- `data import beta tree`: We removed the hidden and deprecated `--content-type` flag. The command supports only JSON files. Usage of the flag: ~5 per year. +- `data import beta tree`: We removed the `--config-help` flag because the schema information is in the command help. Usage of the flag: ~1 per week. + +Other differences: + +- `data export tree --plan` uses the object names as the new file name. Previously it appended an `s` on the end, but the new one doesn't. So the filename is now `Account.json` and `Foo__c.json` instead of `Accounts.json` and the awful `Foo__cs.json`. +- `data export beta tree` no longer writes empty child objects. Previously, you saw properties with `{records: []}` that had no effect when imported. +- `data import beta tree --plan` doesn't care about `resolveRefs` and `saveRefs`. +- `data import beta tree --plan` doesn't care about the order of files in your `plan` file. Rather, it defers unresolved references until they're resolved. +- `data export beta tree --plan` now handles more than 2 levels of child objects in a query. It can handle up to 5 levels, which is the SOQL limit. +- Both `data export beta tree` and `data import beta tree` handle objects that refer to objects of the same type. For example, an Account with ParentId that's an Account or a User with Manager that's a User. +- `data import beta tree --plan` handles more than 200 records. It batches them into groups of 200 per object. The new record limit isn't documented; it most likely comes from your operating system call stack depth or other org limits, such as data storage or api call limits. +- `data import tree` supported plan files where the `files` property could contain objects. It's unclear how those files were generated, but probably not from the `data export tree` command. The new `data import beta tree` command works only with strings and throws an error otherwise. +- `data import beta tree --files` (and not `--plan`) imports the files in parallel. Files can only reference each other if you specify `--plan`. +- `data import tree` outputs deprecation warnings for both `--content-type` and `--config-help` flags. + +## Phase 2: GA the new commands, put the old ones under the `legacy` sub-topic. [July 10, 2024] + +Our plan: + +1. Pin an issue when the change goes into the release candidate so if users run into problems they can quickly find the `legacy` commands. +1. Move the "old" commands to `legacy` and mark them `hidden` and `deprecated` with the Phase 3 date. +1. Move the `force:` aliases to the new commands. +1. Remove the `beta` sub-topic from the new commands, but keep the `beta` alias so they will still work. Add the `deprecateAliases` property to encourage users to stop using the commands in the `beta` sub-topic. +1. Update `data export tree --plan` to display a warning that the JSON output is going to change after the Phase 3 date. Specifically, the JSON output won't include `saveRefs` and `resolveRefs` information. + +## Phase 3: Retire the `legacy` commands and all their dependent code. [Nov 10 2024] + +Our plan: + +1. Update `data export tree --plan` to stop writing the unused `saveRefs` and `resolveRefs` properties on plan files, and stop returning them in JSON output. +1. Tighten the schema to remove the `object` part of `files`, and remove `saveRefs` and `resolveRefs`. +1. Check messages for any that aren't being used, then remove them. +1. Remove the `beta` alias from `data import tree` and `data export tree`. +1. Update the pinned issue to reflect these changes. + +## How Does This Beta->Legacy Thing Work? + +Salesforce CLI uses `beta` and `legacy` subtopics to safely introduce beta versions of existing commands and then GA them. This approach allows you to try out the beta, while continuing to use the existing command at the same time. Let's look at an example to see how this works. + +- We create and release the `data export beta tree` command, which is similar to the existing `data export tree` command, but with improvements and possibly breaking changes. The two separate commands co-exist, which means if you run `sf commands`, you see both the existing and `beta` command. +- After the beta period is over, we GA the changes by moving the functionality we added to `data export beta tree` to the "official" `data export tree` command. We also move the functionality in the _old_ `data export tree` to a new command called `data export legacy tree`. We hide and deprecate both the `legacy` and `beta` versions of the command, but alias the `beta` version to the `data export tree` command because they're now the same. If you run `sf commands` you see only the `data export tree` command, although the `legacy` version is still available (but hidden) if you really need it. +- At some point, we remove the `data export legacy tree` command; we'll warn you, don't worry. We also remove the `beta` alias. diff --git a/command-snapshot.json b/command-snapshot.json index c96842d3..0dfe90c5 100644 --- a/command-snapshot.json +++ b/command-snapshot.json @@ -1,23 +1,25 @@ [ { - "command": "data:create:record", - "plugin": "@salesforce/plugin-data", - "flags": ["api-version", "json", "loglevel", "perflog", "sobject", "target-org", "use-tooling-api", "values"], "alias": ["force:data:record:create"], + "command": "data:create:record", + "flagAliases": ["apiversion", "sobjecttype", "targetusername", "u", "usetoolingapi"], "flagChars": ["o", "s", "t", "v"], - "flagAliases": ["apiversion", "sobjecttype", "targetusername", "u", "usetoolingapi"] + "flags": ["api-version", "json", "loglevel", "perflog", "sobject", "target-org", "use-tooling-api", "values"], + "plugin": "@salesforce/plugin-data" }, { - "command": "data:delete:bulk", - "plugin": "@salesforce/plugin-data", - "flags": ["api-version", "async", "file", "json", "loglevel", "sobject", "target-org", "verbose", "wait"], "alias": [], + "command": "data:delete:bulk", + "flagAliases": ["apiversion", "csvfile", "sobjecttype", "targetusername", "u"], "flagChars": ["a", "f", "o", "s", "w"], - "flagAliases": ["apiversion", "csvfile", "sobjecttype", "targetusername", "u"] + "flags": ["api-version", "async", "file", "json", "loglevel", "sobject", "target-org", "verbose", "wait"], + "plugin": "@salesforce/plugin-data" }, { + "alias": ["force:data:record:delete"], "command": "data:delete:record", - "plugin": "@salesforce/plugin-data", + "flagAliases": ["apiversion", "sobjectid", "sobjecttype", "targetusername", "u", "usetoolingapi"], + "flagChars": ["i", "o", "s", "t", "w"], "flags": [ "api-version", "json", @@ -29,29 +31,37 @@ "use-tooling-api", "where" ], - "alias": ["force:data:record:delete"], - "flagChars": ["i", "o", "s", "t", "w"], - "flagAliases": ["apiversion", "sobjectid", "sobjecttype", "targetusername", "u", "usetoolingapi"] + "plugin": "@salesforce/plugin-data" }, { - "command": "data:delete:resume", - "plugin": "@salesforce/plugin-data", - "flags": ["api-version", "job-id", "json", "loglevel", "target-org", "use-most-recent", "wait"], "alias": [], + "command": "data:delete:resume", + "flagAliases": ["jobid", "targetusername", "u"], "flagChars": ["i", "o"], - "flagAliases": ["jobid", "targetusername", "u"] + "flags": ["api-version", "job-id", "json", "loglevel", "target-org", "use-most-recent", "wait"], + "plugin": "@salesforce/plugin-data" }, { - "command": "data:export:tree", - "plugin": "@salesforce/plugin-data", + "alias": [], + "command": "data:export:beta:tree", + "flagAliases": ["apiversion", "outputdir", "targetusername", "u"], + "flagChars": ["d", "o", "p", "q", "x"], "flags": ["api-version", "json", "loglevel", "output-dir", "plan", "prefix", "query", "target-org"], + "plugin": "@salesforce/plugin-data" + }, + { "alias": ["force:data:tree:export"], + "command": "data:export:tree", + "flagAliases": ["apiversion", "outputdir", "targetusername", "u"], "flagChars": ["d", "o", "p", "q", "x"], - "flagAliases": ["apiversion", "outputdir", "targetusername", "u"] + "flags": ["api-version", "json", "loglevel", "output-dir", "plan", "prefix", "query", "target-org"], + "plugin": "@salesforce/plugin-data" }, { + "alias": ["force:data:record:get"], "command": "data:get:record", - "plugin": "@salesforce/plugin-data", + "flagAliases": ["apiversion", "sobjectid", "sobjecttype", "targetusername", "u", "usetoolingapi"], + "flagChars": ["i", "o", "s", "t", "w"], "flags": [ "api-version", "json", @@ -63,21 +73,29 @@ "use-tooling-api", "where" ], - "alias": ["force:data:record:get"], - "flagChars": ["i", "o", "s", "t", "w"], - "flagAliases": ["apiversion", "sobjectid", "sobjecttype", "targetusername", "u", "usetoolingapi"] + "plugin": "@salesforce/plugin-data" + }, + { + "alias": [], + "command": "data:import:beta:tree", + "flagAliases": ["apiversion", "sobjecttreefiles", "targetusername", "u"], + "flagChars": ["f", "o", "p"], + "flags": ["api-version", "files", "json", "loglevel", "plan", "target-org"], + "plugin": "@salesforce/plugin-data" }, { - "command": "data:import:tree", - "plugin": "@salesforce/plugin-data", - "flags": ["api-version", "config-help", "content-type", "files", "json", "loglevel", "plan", "target-org"], "alias": ["force:data:tree:import"], + "command": "data:import:tree", + "flagAliases": ["apiversion", "confighelp", "contenttype", "sobjecttreefiles", "targetusername", "u"], "flagChars": ["c", "f", "o", "p"], - "flagAliases": ["apiversion", "confighelp", "contenttype", "sobjecttreefiles", "targetusername", "u"] + "flags": ["api-version", "config-help", "content-type", "files", "json", "loglevel", "plan", "target-org"], + "plugin": "@salesforce/plugin-data" }, { + "alias": ["force:data:soql:query"], "command": "data:query", - "plugin": "@salesforce/plugin-data", + "flagAliases": ["apiversion", "resultformat", "soqlqueryfile", "targetusername", "u", "usetoolingapi"], + "flagChars": ["b", "f", "o", "q", "r", "t", "w"], "flags": [ "all-rows", "api-version", @@ -93,29 +111,29 @@ "use-tooling-api", "wait" ], - "alias": ["force:data:soql:query"], - "flagChars": ["b", "f", "o", "q", "r", "t", "w"], - "flagAliases": ["apiversion", "resultformat", "soqlqueryfile", "targetusername", "u", "usetoolingapi"] + "plugin": "@salesforce/plugin-data" }, { - "command": "data:query:resume", - "plugin": "@salesforce/plugin-data", - "flags": ["api-version", "bulk-query-id", "json", "loglevel", "result-format", "target-org", "use-most-recent"], "alias": ["force:data:soql:bulk:report"], + "command": "data:query:resume", + "flagAliases": ["apiversion", "bulkqueryid", "resultformat", "targetusername", "u"], "flagChars": ["i", "o", "r"], - "flagAliases": ["apiversion", "bulkqueryid", "resultformat", "targetusername", "u"] + "flags": ["api-version", "bulk-query-id", "json", "loglevel", "result-format", "target-org", "use-most-recent"], + "plugin": "@salesforce/plugin-data" }, { - "command": "data:resume", - "plugin": "@salesforce/plugin-data", - "flags": ["api-version", "batch-id", "job-id", "json", "loglevel", "target-org"], "alias": [], + "command": "data:resume", + "flagAliases": ["apiversion", "batchid", "jobid", "targetusername", "u"], "flagChars": ["b", "i", "o"], - "flagAliases": ["apiversion", "batchid", "jobid", "targetusername", "u"] + "flags": ["api-version", "batch-id", "job-id", "json", "loglevel", "target-org"], + "plugin": "@salesforce/plugin-data" }, { + "alias": ["force:data:record:update"], "command": "data:update:record", - "plugin": "@salesforce/plugin-data", + "flagAliases": ["apiversion", "sobjectid", "sobjecttype", "targetusername", "u", "usetoolingapi"], + "flagChars": ["i", "o", "s", "t", "v", "w"], "flags": [ "api-version", "json", @@ -128,13 +146,13 @@ "values", "where" ], - "alias": ["force:data:record:update"], - "flagChars": ["i", "o", "s", "t", "v", "w"], - "flagAliases": ["apiversion", "sobjectid", "sobjecttype", "targetusername", "u", "usetoolingapi"] + "plugin": "@salesforce/plugin-data" }, { + "alias": [], "command": "data:upsert:bulk", - "plugin": "@salesforce/plugin-data", + "flagAliases": ["apiversion", "csvfile", "externalid", "sobjecttype", "targetusername", "u"], + "flagChars": ["a", "f", "i", "o", "s", "w"], "flags": [ "api-version", "async", @@ -147,40 +165,38 @@ "verbose", "wait" ], - "alias": [], - "flagChars": ["a", "f", "i", "o", "s", "w"], - "flagAliases": ["apiversion", "csvfile", "externalid", "sobjecttype", "targetusername", "u"] + "plugin": "@salesforce/plugin-data" }, { - "command": "data:upsert:resume", - "plugin": "@salesforce/plugin-data", - "flags": ["api-version", "job-id", "json", "loglevel", "target-org", "use-most-recent", "wait"], "alias": [], + "command": "data:upsert:resume", + "flagAliases": ["jobid", "targetusername", "u"], "flagChars": ["i", "o"], - "flagAliases": ["jobid", "targetusername", "u"] + "flags": ["api-version", "job-id", "json", "loglevel", "target-org", "use-most-recent", "wait"], + "plugin": "@salesforce/plugin-data" }, { - "command": "force:data:bulk:delete", - "plugin": "@salesforce/plugin-data", - "flags": ["api-version", "file", "json", "loglevel", "sobject", "target-org", "wait"], "alias": [], + "command": "force:data:bulk:delete", + "flagAliases": ["apiversion", "csvfile", "sobjecttype", "targetusername", "u"], "flagChars": ["f", "o", "s", "w"], - "flagAliases": ["apiversion", "csvfile", "sobjecttype", "targetusername", "u"] + "flags": ["api-version", "file", "json", "loglevel", "sobject", "target-org", "wait"], + "plugin": "@salesforce/plugin-data" }, { - "command": "force:data:bulk:status", - "plugin": "@salesforce/plugin-data", - "flags": ["api-version", "batch-id", "job-id", "json", "loglevel", "target-org"], "alias": [], + "command": "force:data:bulk:status", + "flagAliases": ["apiversion", "batchid", "jobid", "targetusername", "u"], "flagChars": ["b", "i", "o"], - "flagAliases": ["apiversion", "batchid", "jobid", "targetusername", "u"] + "flags": ["api-version", "batch-id", "job-id", "json", "loglevel", "target-org"], + "plugin": "@salesforce/plugin-data" }, { - "command": "force:data:bulk:upsert", - "plugin": "@salesforce/plugin-data", - "flags": ["api-version", "external-id", "file", "json", "loglevel", "serial", "sobject", "target-org", "wait"], "alias": [], + "command": "force:data:bulk:upsert", + "flagAliases": ["apiversion", "csvfile", "externalid", "sobjecttype", "targetusername", "u"], "flagChars": ["f", "i", "o", "r", "s", "w"], - "flagAliases": ["apiversion", "csvfile", "externalid", "sobjecttype", "targetusername", "u"] + "flags": ["api-version", "external-id", "file", "json", "loglevel", "serial", "sobject", "target-org", "wait"], + "plugin": "@salesforce/plugin-data" } ] diff --git a/messages/importApi.md b/messages/importApi.md index bfbc840c..08591a7b 100644 --- a/messages/importApi.md +++ b/messages/importApi.md @@ -49,3 +49,35 @@ Data plan file %s did not validate against the schema. Errors: %s. # FlsError We couldn't process your request because you don't have access to %s on %s. To learn more about field-level security, visit Tips and Hints for Page Layouts and Field-Level Security in our Developer Documentation. + +# error.InvalidDataImport + +Data plan file %s did not validate against the schema. Errors: %s. + +# error.InvalidDataImport.actions + +- Did you run the "sf data export tree" command with the --plan flag? + +- Make sure you're importing a plan definition file. + +- Get help with the import plan schema by running "sf data import beta tree --help". + +# saveResolveRefsIgnored + +The plan contains the 'saveRefs' and/or 'resolveRefs' properties. +These properties will be ignored and can be removed. +In the future, the `tree export` command will not produce them. + +# error.NonStringFiles + +The `files` property of the plan objects must contain only strings + +# error.UnresolvableRefs + +There are references in a data file %s that can't be resolved: + +%s + +# error.RefsInFiles + +The file %s includes references (ex: '@AccountRef1'). Those are only supported with --plan, not --files.` diff --git a/messages/tree.export.md b/messages/tree.export.md index 07aa1460..9ab8d577 100644 --- a/messages/tree.export.md +++ b/messages/tree.export.md @@ -30,7 +30,7 @@ Directory in which to generate the JSON files; default is current directory. - Export records retrieved with the specified SOQL query into a single JSON file in the current directory; the command uses your default org: - <%= config.bin %> <%= command.id %> --query "SELECT Id, Name, (SELECT Name, Address__c FROM Properties__r) FROM Broker__c" + <%= config.bin %> <%= command.id %> --query "SELECT Id, Name, (SELECT Name, Address\_\_c FROM Properties\_\_r) FROM Broker\_\_c" - Export data using a SOQL query in the "query.txt" file and generate JSON files for each object and a plan that aggregates them: @@ -39,3 +39,7 @@ Directory in which to generate the JSON files; default is current directory. - Prepend "export-demo" before each generated file and generate the files in the "export-out" directory; run the command on the org with alias "my-scratch": <%= config.bin %> <%= command.id %> --query query.txt --plan --prefix export-demo --output-dir export-out --target-org my-scratch + +# PrefixSlashError + +`--prefix` cannot contain a forward slash or backslash. diff --git a/messages/tree.import.beta.md b/messages/tree.import.beta.md new file mode 100644 index 00000000..92748eb1 --- /dev/null +++ b/messages/tree.import.beta.md @@ -0,0 +1,43 @@ +# summary + +Import data from one or more JSON files into an org. + +# description + +The JSON files that contain the data are in sObject tree format, which is a collection of nested, parent-child records with a single root record. Use the "<%= config.bin %> data export tree" command to generate these JSON files. + +If you used the --plan flag when exporting the data to generate a plan definition file, use the --plan flag to reference the file when you import. If you're not using a plan, use the --files flag to list the files. If you specify multiple JSON files that depend on each other in a parent-child relationship, be sure you list them in the correct order. + +# flags.files.summary + +Comma-separated and in-order JSON files that contain the records, in sObject tree format, that you want to insert. + +# flag.files.description + +Each file can contain up to 200 total records. For more information, see the REST API Developer Guide. (https://developer.salesforce.com/docs/atlas.en-us.api_rest.meta/api_rest/resources_composite_sobject_tree.htm) + +# flags.plan.summary + +Plan definition file to insert multiple data files. + +# flags.plan.description + +Unlike the `--files` flag, the files listed in the plan definition file **can** contain more then 200 records. These will be automatically batched to comply with that limit. + +The file order matters--records with lookups to records in another file should be listed AFTER that file Example: you're loading Account and Contact records, and the contacts have references to those Accounts. The Accounts file should come before the Contacts file. + +The plan definition file has the following schema: + +- items(object) - SObject Type: Definition of records to be insert per SObject Type + - sobject(string) - Name of SObject: Child file references must have SObject roots of this type + - files(array) - Files: An array of files paths to load + +# examples + +- Import the records contained in two JSON files into the org with alias "my-scratch": + + <%= config.bin %> <%= command.id %> --files Contact.json,Account.json --target-org my-scratch + +- Import records using a plan definition file into your default org: + + <%= config.bin %> <%= command.id %> --plan Account-Contact-plan.json diff --git a/messages/tree.import.md b/messages/tree.import.md index 43bab30c..8a7d3167 100644 --- a/messages/tree.import.md +++ b/messages/tree.import.md @@ -22,10 +22,18 @@ Plan definition file to insert multiple data files. Content type of import files if their extention is not .json. +# flags.content-type.deprecation + +The `config-type` flag is deprecated and will be moved to a `legacy` command after July 10, 2024. It will be completely removed after Nov 10, 2024. Use the new `data tree beta import` command. + # flags.config-help.summary Display schema information for the --plan configuration file to stdout; if you specify this flag, all other flags except --json are ignored. +# flags.config-help.deprecation + +The `config-help` flag is deprecated and will be moved to a `legacy` command after July 10, 2024. It will be completely removed after Nov 10, 2024. Use the new `data tree beta import` command. + # examples - Import the records contained in two JSON files into the org with alias "my-scratch": diff --git a/src/api/data/tree/exportApi.ts b/src/api/data/tree/exportApi.ts index 597d5c54..61540382 100644 --- a/src/api/data/tree/exportApi.ts +++ b/src/api/data/tree/exportApi.ts @@ -5,7 +5,7 @@ * For full license text, see LICENSE.txt file in the repo root or https://opensource.org/licenses/BSD-3-Clause */ -import path from 'node:path' +import path from 'node:path'; import fs from 'node:fs'; import { Logger, Messages, Org, SfError, Lifecycle } from '@salesforce/core'; @@ -19,7 +19,7 @@ import { SObjectTreeInput, } from '../../../dataSoqlQueryTypes.js'; -Messages.importMessagesDirectoryFromMetaUrl(import.meta.url) +Messages.importMessagesDirectoryFromMetaUrl(import.meta.url); const messages = Messages.loadMessages('@salesforce/plugin-data', 'exportApi'); const DATA_PLAN_FILENAME_PART = '-plan.json'; diff --git a/src/api/data/tree/functions.ts b/src/api/data/tree/functions.ts new file mode 100644 index 00000000..159ffd4d --- /dev/null +++ b/src/api/data/tree/functions.ts @@ -0,0 +1,17 @@ +/* + * Copyright (c) 2023, salesforce.com, inc. + * All rights reserved. + * Licensed under the BSD 3-Clause license. + * For full license text, see LICENSE.txt file in the repo root or https://opensource.org/licenses/BSD-3-Clause + */ + +import { SObjectTreeInput } from '../../../dataSoqlQueryTypes.js'; + +/** This is the format for references created by the export command */ +const genericRefRegex = new RegExp('^@\\w+Ref\\d+$'); + +export const isUnresolvedRef = (v: unknown): boolean => typeof v === 'string' && genericRefRegex.test(v); + +/** at least record in the array has at least one property value that matches the regex */ +export const hasUnresolvedRefs = (records: SObjectTreeInput[]): boolean => + records.some((r) => Object.values(r).some(isUnresolvedRef)); diff --git a/src/api/data/tree/importApi.ts b/src/api/data/tree/importApi.ts index 4dcd6131..b844c4d7 100644 --- a/src/api/data/tree/importApi.ts +++ b/src/api/data/tree/importApi.ts @@ -12,8 +12,11 @@ import { fileURLToPath } from 'node:url'; import { AnyJson, Dictionary, getString, JsonMap } from '@salesforce/ts-types'; import { Logger, Messages, Org, SchemaValidator, SfError } from '@salesforce/core'; import { DataPlanPart, hasNestedRecords, isAttributesElement, SObjectTreeInput } from '../../../dataSoqlQueryTypes.js'; +import { TreeResponse } from './importTypes.js'; +import { ResponseRefs } from './importTypes.js'; +import { ImportResults } from './importTypes.js'; -Messages.importMessagesDirectoryFromMetaUrl(import.meta.url) +Messages.importMessagesDirectoryFromMetaUrl(import.meta.url); const messages = Messages.loadMessages('@salesforce/plugin-data', 'importApi'); const importPlanSchemaFile = path.join( @@ -34,26 +37,6 @@ const xmlRefRegex = /[.]*<[A-Z0-9_]*>@([A-Z0-9_]*)<\/[A-Z0-9_]*[ID]>[.]*/gim; const INVALID_DATA_IMPORT_ERR_NAME = 'InvalidDataImport'; -type TreeResponse = - | { - hasErrors: false; - results: Array<{ - referenceId: string; - id: string; - }>; - } - | { - hasErrors: true; - results: Array<{ - referenceId: string; - errors: Array<{ - statusCode: string; - message: string; - fields: string[]; - }>; - }>; - }; - interface DataImportComponents { instanceUrl: string; saveRefs?: boolean; @@ -69,17 +52,6 @@ export interface ImportConfig { plan?: string; } -interface ResponseRefs { - referenceId: string; - id: string; -} - -export interface ImportResults { - responseRefs?: ResponseRefs[]; - sobjectTypes?: Dictionary; - errors?: string[]; -} - interface RequestMeta { refRegex: RegExp; isJson: boolean; diff --git a/src/api/data/tree/importCommon.ts b/src/api/data/tree/importCommon.ts new file mode 100644 index 00000000..cdd3deef --- /dev/null +++ b/src/api/data/tree/importCommon.ts @@ -0,0 +1,59 @@ +/* + * Copyright (c) 2023, salesforce.com, inc. + * All rights reserved. + * Licensed under the BSD 3-Clause license. + * For full license text, see LICENSE.txt file in the repo root or https://opensource.org/licenses/BSD-3-Clause + */ +import { Connection, SfError, Messages } from '@salesforce/core'; +import { SObjectTreeInput, SObjectTreeFileContents } from '../../../dataSoqlQueryTypes.js'; +import { ResponseRefs, TreeResponse } from './importTypes.js'; + +Messages.importMessagesDirectoryFromMetaUrl(import.meta.url); +const messages = Messages.loadMessages('@salesforce/plugin-data', 'importApi'); + +/** makes the API request */ +export const sendSObjectTreeRequest = + (conn: Connection) => + (sobject: string) => + (rawContents: string): Promise => + // post request with to-be-insert sobject tree content + conn.request({ + method: 'POST', + url: `/composite/tree/${sobject}`, + body: rawContents, + headers: { + 'content-type': 'application/json', + }, + }); + +/** handle an error throw by sendSObjectTreeRequest. Always throws */ +export const treeSaveErrorHandler = (error: unknown): never => { + if (error instanceof Error && 'errorCode' in error && error.errorCode === 'INVALID_FIELD') { + const field = error.message.split("'")[1]; + const object = error.message.slice(error.message.lastIndexOf(' ') + 1, error.message.length); + throw messages.createError('FlsError', [field, object]); + } + if (error instanceof Error) { + throw SfError.wrap(error); + } + throw error; +}; + +export const parseDataFileContents = + (filePath: string) => + (contents: string): SObjectTreeInput[] => { + if (!contents) { + throw messages.createError('dataFileEmpty', [filePath]); + } + return (JSON.parse(contents) as SObjectTreeFileContents).records; + }; + +/** look inside the response. If the hasErrors property is true, throw a formatted error. Otherwise, extract the results property */ +export const getResultsIfNoError = + (filePath: string) => + (response: TreeResponse): ResponseRefs[] => { + if (response.hasErrors === true) { + throw messages.createError('dataImportFailed', [filePath, JSON.stringify(response.results, null, 4)]); + } + return response.results; + }; diff --git a/src/api/data/tree/importFiles.ts b/src/api/data/tree/importFiles.ts new file mode 100644 index 00000000..b516aff9 --- /dev/null +++ b/src/api/data/tree/importFiles.ts @@ -0,0 +1,90 @@ +/* + * Copyright (c) 2023, salesforce.com, inc. + * All rights reserved. + * Licensed under the BSD 3-Clause license. + * For full license text, see LICENSE.txt file in the repo root or https://opensource.org/licenses/BSD-3-Clause + */ +import fs from 'node:fs'; +import { Logger, Connection, Messages } from '@salesforce/core'; +import { isFulfilled } from '@salesforce/kit'; +import { flattenNestedRecords } from '../../../export.js'; +import { SObjectTreeInput, isAttributesEntry } from '../../../dataSoqlQueryTypes.js'; +import { + sendSObjectTreeRequest, + treeSaveErrorHandler, + parseDataFileContents, + getResultsIfNoError, +} from './importCommon.js'; +import { ImportResult, ResponseRefs, TreeResponse } from './importTypes.js'; +import { hasUnresolvedRefs } from './functions.js'; + +export type FileInfo = { + rawContents: string; + records: SObjectTreeInput[]; + filePath: string; + sobject: string; +}; + +export const importFromFiles = async (conn: Connection, dataFilePaths: string[]): Promise => { + const logger = Logger.childFromRoot('data:import:tree:importSObjectTreeFile'); + const fileInfos = (await Promise.all(dataFilePaths.map(parseFile))).map(logFileInfo(logger)).map(validateNoRefs); + const refMap = createSObjectTypeMap(fileInfos.flatMap((fi) => fi.records)); + const results = await Promise.allSettled( + fileInfos.map((fi) => sendSObjectTreeRequest(conn)(fi.sobject)(fi.rawContents)) + ); + return results.map(getSuccessOrThrow).flatMap(getValueOrThrow(fileInfos)).map(addObjectTypes(refMap)); +}; + +const getSuccessOrThrow = (result: PromiseSettledResult): PromiseFulfilledResult => + isFulfilled(result) ? result : treeSaveErrorHandler(result.reason); + +const getValueOrThrow = + (fi: FileInfo[]) => + (response: PromiseFulfilledResult, index: number): ResponseRefs[] => + getResultsIfNoError(fi[index].filePath)(response.value); + +const addObjectTypes = + (refMap: Map) => + (result: ResponseRefs): ImportResult => ({ + refId: result.referenceId, + type: refMap.get(result.referenceId) ?? 'Unknown', + id: result.id, + }); + +const contentsToSobjectType = (records: SObjectTreeInput[]): string => records[0].attributes.type; + +const logFileInfo = + (logger: Logger) => + (fileInfo: FileInfo): FileInfo => { + logger.debug(`Parsed file ${fileInfo.filePath} for sobject type ${fileInfo.sobject}`); + return fileInfo; + }; + +/** check the tree files for references, throw error telling user they are only supported with `--plan */ +export const validateNoRefs = (fileInfo: FileInfo): FileInfo => { + if (hasUnresolvedRefs(fileInfo.records)) { + Messages.importMessagesDirectoryFromMetaUrl(import.meta.url); + const messages = Messages.loadMessages('@salesforce/plugin-data', 'importApi'); + + throw new Error(messages.getMessage('error.RefsInFiles', [fileInfo.filePath])); + } + return fileInfo; +}; + +/** gets information about the file, including the sobject, contents, parsed contents */ +const parseFile = async (filePath: string): Promise => { + const rawContents = await fs.promises.readFile(filePath, 'utf8'); + const records = parseDataFileContents(filePath)(rawContents); + const sobjectType = contentsToSobjectType(records); + return { rawContents, records, filePath, sobject: sobjectType }; +}; + +/** Create a hash of sobject { ReferenceId: Type }. */ +export const createSObjectTypeMap = (records: SObjectTreeInput[]): Map => + new Map( + records + .flatMap(flattenNestedRecords) + .flatMap(Object.entries) + .filter(isAttributesEntry) + .map(([, val]) => [val.referenceId, val.type]) + ); diff --git a/src/api/data/tree/importPlan.ts b/src/api/data/tree/importPlan.ts new file mode 100644 index 00000000..2e14cb7b --- /dev/null +++ b/src/api/data/tree/importPlan.ts @@ -0,0 +1,253 @@ +/* + * Copyright (c) 2023, salesforce.com, inc. + * All rights reserved. + * Licensed under the BSD 3-Clause license. + * For full license text, see LICENSE.txt file in the repo root or https://opensource.org/licenses/BSD-3-Clause + */ +import path from 'node:path'; +import { EOL } from 'node:os'; +import { fileURLToPath } from 'node:url'; +import fs from 'node:fs'; +import { createHash } from 'node:crypto'; + +import { AnyJson, isString } from '@salesforce/ts-types'; +import { Logger, SchemaValidator, SfError, Connection, Messages } from '@salesforce/core'; +import { SObjectTreeInput } from '../../../dataSoqlQueryTypes.js'; +import { DataPlanPartFilesOnly, ImportResult } from './importTypes.js'; +import { + getResultsIfNoError, + parseDataFileContents, + sendSObjectTreeRequest, + treeSaveErrorHandler, +} from './importCommon.js'; +import { isUnresolvedRef } from './functions.js'; +import { hasUnresolvedRefs } from './functions.js'; + +Messages.importMessagesDirectoryFromMetaUrl(import.meta.url); +const messages = Messages.loadMessages('@salesforce/plugin-data', 'importApi'); + +// the "new" type for these. We're ignoring saveRefs/resolveRefs +export type EnrichedPlanPart = Omit & { + filePath: string; + sobject: string; + records: SObjectTreeInput[]; +}; +/** an accumulator for api results. Fingerprints exist to break recursion */ +type ResultsSoFar = { + results: ImportResult[]; + fingerprints: Set; +}; + +const TREE_API_LIMIT = 200; + +const refRegex = (object: string): RegExp => new RegExp(`^@${object}Ref\\d+$`); +export const importFromPlan = async (conn: Connection, planFilePath: string): Promise => { + const resolvedPlanPath = path.resolve(process.cwd(), planFilePath); + const logger = Logger.childFromRoot('data:import:tree:importFromPlan'); + + const planContents = await Promise.all( + ( + await validatePlanContents(logger)( + resolvedPlanPath, + (await JSON.parse(fs.readFileSync(resolvedPlanPath, 'utf8'))) as DataPlanPartFilesOnly[] + ) + ) + // there *shouldn't* be multiple files for the same sobject in a plan, but the legacy code allows that + .flatMap((dpp) => dpp.files.map((f) => ({ ...dpp, filePath: path.resolve(path.dirname(resolvedPlanPath), f) }))) + .map(async (i) => ({ + ...i, + records: parseDataFileContents(i.filePath)(await fs.promises.readFile(i.filePath, 'utf-8')), + })) + ); + // using recursion to sequentially send the requests so we get refs back from each round + const { results } = await getResults(conn)(logger)({ results: [], fingerprints: new Set() })(planContents); + + return results; +}; + +/** recursively splits files (for size or unresolved refs) and makes API calls, storing results for subsequent calls */ +const getResults = + (conn: Connection) => + (logger: Logger) => + (resultsSoFar: ResultsSoFar) => + async (planParts: EnrichedPlanPart[]): Promise => { + const newResultWithFingerPrints = addFingerprint(resultsSoFar)(planParts); + const [head, ...tail] = planParts; + if (!head.records.length) { + return tail.length ? getResults(conn)(logger)(newResultWithFingerPrints)(tail) : resultsSoFar; + } + const partWithRefsReplaced = { ...head, records: replaceRefs(resultsSoFar.results)(head.records) }; + const { ready, notReady } = replaceRefsInTheSameFile(partWithRefsReplaced); + if (notReady) { + logger.debug(`Not all refs are resolved yet. Splitting ${partWithRefsReplaced.filePath} into two`); + + // Do the ones with all refs resolved to ID, then the rest, then the other files. Essentially, we split the file into 2 parts and start over + return getResults(conn)(logger)(newResultWithFingerPrints)([ready, notReady, ...tail]); + } + + // We could have refs to records in a file we haven't loaded yet. + const { resolved, unresolved } = filterUnresolved(partWithRefsReplaced.records); + if (unresolved.length) { + logger.debug( + `Not all refs are resolved yet. Splitting ${partWithRefsReplaced.filePath} into two with the unresolved refs last` + ); + + return getResults(conn)(logger)(newResultWithFingerPrints)([ + { ...head, records: resolved }, + ...tail, + { ...head, records: unresolved, filePath: `${head.filePath}` }, + ]); + } + + if (partWithRefsReplaced.records.length > TREE_API_LIMIT) { + logger.debug( + `There are more than ${TREE_API_LIMIT} records in ${partWithRefsReplaced.filePath}. Will split into multiple requests.` + ); + return getResults(conn)(logger)(newResultWithFingerPrints)([...fileSplitter(partWithRefsReplaced), ...tail]); + } + logger.debug( + `Sending ${partWithRefsReplaced.filePath} (${partWithRefsReplaced.records.length} records for ${partWithRefsReplaced.sobject}) to the API` + ); + try { + const contents = JSON.stringify({ records: partWithRefsReplaced.records }); + const newResults = getResultsIfNoError(partWithRefsReplaced.filePath)( + await sendSObjectTreeRequest(conn)(partWithRefsReplaced.sobject)(contents) + ); + const output = { + ...newResultWithFingerPrints, + results: [ + ...newResultWithFingerPrints.results, + ...newResults.map((r) => ({ refId: r.referenceId, type: partWithRefsReplaced.sobject, id: r.id })), + ], + }; + return tail.length ? await getResults(conn)(logger)(output)(tail) : output; + } catch (e) { + return treeSaveErrorHandler(e); + } + }; + +/** if the file has more than TREE_API_LIMIT records, split it into multiple files */ +export const fileSplitter = (planPart: EnrichedPlanPart): EnrichedPlanPart[] => { + const head = planPart.records.slice(0, TREE_API_LIMIT); + const tail = planPart.records.slice(TREE_API_LIMIT); + return tail.length ? [{ ...planPart, records: head }, ...fileSplitter({ ...planPart, records: tail })] : [planPart]; +}; + +/** + * it's possible that a file has records that refer to each other (ex: account/parentId). So they're not in refs yet. + * so we'll parse the JSON and split the records into 2 sets: those with refs and those without, if necessary + */ +export const replaceRefsInTheSameFile = ( + planPart: EnrichedPlanPart +): { ready: EnrichedPlanPart; notReady?: EnrichedPlanPart } => { + const unresolvedRefRegex = refRegex(planPart.sobject); + + const refRecords = planPart.records.filter((r) => Object.values(r).some(matchesRefFilter(unresolvedRefRegex))); + return refRecords.length + ? { + ready: { + ...planPart, + // have no refs, so they can go in immediately + records: planPart.records.filter((r) => !Object.values(r).some(matchesRefFilter(unresolvedRefRegex))), + }, + notReady: { ...planPart, records: refRecords }, + } + : { ready: planPart }; +}; + +/** recursively replace the `@ref` with the id, using the accumulated results objects */ +export const replaceRefs = + (results: ImportResult[]) => + (records: SObjectTreeInput[]): SObjectTreeInput[] => { + if (results.length === 0) return records; + const [head, ...tail] = results; + const updatedRecords = records.map(replaceRefWithId(head)); + return tail.length ? replaceRefs(tail)(updatedRecords) : updatedRecords; + }; + +/** replace 1 record with 1 ref for all of its fields */ +const replaceRefWithId = + (ref: ImportResult) => + (record: SObjectTreeInput): SObjectTreeInput => + Object.fromEntries( + Object.entries(record).map(([k, v]) => [k, v === `@${ref.refId}` ? ref.id : v]) + ) as SObjectTreeInput; + +export const validatePlanContents = + (logger: Logger) => + async (planPath: string, planContents: unknown): Promise => { + const childLogger = logger.child('validatePlanContents'); + const planSchema = path.join( + path.dirname(fileURLToPath(import.meta.url)), + '..', + '..', + '..', + '..', + 'schema', + 'dataImportPlanSchema.json' + ); + + const val = new SchemaValidator(childLogger, planSchema); + try { + await val.validate(planContents as AnyJson); + const output = planContents as DataPlanPartFilesOnly[]; + if (hasRefs(output)) { + childLogger.warn( + "The plan contains the 'saveRefs' and/or 'resolveRefs' properties. These properties will be ignored and can be removed." + ); + } + if (!hasOnlySimpleFiles(output)) { + throw messages.createError('error.NonStringFiles'); + } + return planContents as DataPlanPartFilesOnly[]; + } catch (err) { + if (err instanceof Error && err.name === 'ValidationSchemaFieldError') { + throw messages.createError('error.InvalidDataImport', [planPath, err.message]); + } else if (err instanceof Error) { + throw SfError.wrap(err); + } + throw err; + } + }; + +const matchesRefFilter = + (unresolvedRefRegex: RegExp) => + (v: unknown): boolean => + typeof v === 'string' && unresolvedRefRegex.test(v); + +const hasOnlySimpleFiles = (planParts: DataPlanPartFilesOnly[]): boolean => + planParts.every((p) => p.files.every((f) => typeof f === 'string')); + +const hasRefs = (planParts: DataPlanPartFilesOnly[]): boolean => + planParts.some((p) => p.saveRefs !== undefined || p.resolveRefs !== undefined); + +// TODO: change this implementation to use Object.groupBy when it's on all supported node versions +const filterUnresolved = ( + records: SObjectTreeInput[] +): { resolved: SObjectTreeInput[]; unresolved: SObjectTreeInput[] } => ({ + resolved: records.filter((r) => !hasUnresolvedRefs([r])), + unresolved: records.filter((r) => hasUnresolvedRefs([r])), +}); + +/** given the 2 parameters that can change, break the recursion if asked to do an operation that's already been done */ +const addFingerprint = + (resultsSoFar: ResultsSoFar) => + (planParts: EnrichedPlanPart[]): ResultsSoFar => { + const fingerprint = hashObject({ resultsSoFar, planParts }); + + if (resultsSoFar.fingerprints.has(fingerprint)) { + const unresolved = planParts[0].records.map(Object.values).flat().filter(isString).filter(isUnresolvedRef); + const e = messages.createError('error.UnresolvableRefs', [ + planParts[0].filePath, + unresolved.map((s) => `- ${s}`).join(EOL), + ]); + e.setData(resultsSoFar.results); + throw e; + } + return { ...resultsSoFar, fingerprints: resultsSoFar.fingerprints.add(fingerprint) }; + }; + +const hashObject = (obj: Record): string => + createHash('sha256') + .update(Buffer.from(JSON.stringify(obj))) + .digest('hex'); diff --git a/src/api/data/tree/importTypes.ts b/src/api/data/tree/importTypes.ts new file mode 100644 index 00000000..32cadd59 --- /dev/null +++ b/src/api/data/tree/importTypes.ts @@ -0,0 +1,56 @@ +/* + * Copyright (c) 2023, salesforce.com, inc. + * All rights reserved. + * Licensed under the BSD 3-Clause license. + * For full license text, see LICENSE.txt file in the repo root or https://opensource.org/licenses/BSD-3-Clause + */ +import { Dictionary } from '@salesforce/ts-types'; +import { DataPlanPart } from '../../../dataSoqlQueryTypes.js'; + +/* + * Copyright (c) 2023, salesforce.com, inc. + * All rights reserved. + * Licensed under the BSD 3-Clause license. + * For full license text, see LICENSE.txt file in the repo root or https://opensource.org/licenses/BSD-3-Clause + */ +export type TreeResponse = TreeResponseSuccess | TreeResponseError; + +type TreeResponseSuccess = { + hasErrors: false; + results: Array<{ + referenceId: string; + id: string; + }>; +}; + +type TreeResponseError = { + hasErrors: true; + results: Array<{ + referenceId: string; + errors: Array<{ + statusCode: string; + message: string; + fields: string[]; + }>; + }>; +}; + +export interface ResponseRefs { + referenceId: string; + id: string; +} +export interface ImportResults { + responseRefs?: ResponseRefs[]; + sobjectTypes?: Dictionary; + errors?: string[]; +} + +export type ImportResult = { + refId: string; + type: string; + id: string; +}; /** like the original DataPlanPart but without the non-string options inside files */ + +export type DataPlanPartFilesOnly = { sobject: string; files: string[] } & Partial< + Pick +>; diff --git a/src/bulkDataRequestCache.ts b/src/bulkDataRequestCache.ts index ab3cee32..41278525 100644 --- a/src/bulkDataRequestCache.ts +++ b/src/bulkDataRequestCache.ts @@ -5,13 +5,11 @@ * For full license text, see LICENSE.txt file in the repo root or https://opensource.org/licenses/BSD-3-Clause */ - import { TTLConfig, Global, Logger, Messages, Org } from '@salesforce/core'; import { Duration } from '@salesforce/kit'; -import { QueryOperation } from 'jsforce/lib/api/bulk.js'; import { ResumeOptions } from './types.js'; -Messages.importMessagesDirectoryFromMetaUrl(import.meta.url) +Messages.importMessagesDirectoryFromMetaUrl(import.meta.url); const messages = Messages.loadMessages('@salesforce/plugin-data', 'messages'); export type BulkDataCacheConfig = { @@ -63,37 +61,45 @@ export abstract class BulkDataRequestCache extends TTLConfig; if (useMostRecent) { const key = this.getLatestKey(); if (key) { - const entry = this.get(key); - resumeOptions.options.connection = (await Org.create({ aliasOrUsername: entry.username })).getConnection( - apiVersion - ); - resumeOptions.jobInfo = { id: entry.jobId }; - return resumeOptions; + // key definitely exists because it came from the cache + const entry = this.get(key)!; + + return { + jobInfo: { id: entry.jobId }, + options: { + ...resumeOptionsOptions, + connection: (await Org.create({ aliasOrUsername: entry.username })).getConnection(apiVersion), + }, + }; } } if (bulkJobId) { const entry = this.get(bulkJobId); if (entry) { - resumeOptions.options.connection = (await Org.create({ aliasOrUsername: entry.username })).getConnection( - apiVersion - ); - resumeOptions.jobInfo = { id: entry.jobId }; - return resumeOptions; + return { + jobInfo: { id: entry.jobId }, + options: { + ...resumeOptionsOptions, + connection: (await Org.create({ aliasOrUsername: entry.username })).getConnection(apiVersion), + }, + }; } else if (org) { - resumeOptions.options.connection = org.getConnection(apiVersion); - resumeOptions.jobInfo = { id: bulkJobId }; - return resumeOptions; + return { + jobInfo: { id: bulkJobId }, + options: { + ...resumeOptionsOptions, + connection: org.getConnection(apiVersion), + }, + }; } else { throw messages.createError('cannotCreateResumeOptionsWithoutAnOrg'); } diff --git a/src/bulkUtils.ts b/src/bulkUtils.ts index 642b4f40..bc70f95c 100644 --- a/src/bulkUtils.ts +++ b/src/bulkUtils.ts @@ -5,22 +5,19 @@ * For full license text, see LICENSE.txt file in the repo root or https://opensource.org/licenses/BSD-3-Clause */ - import { IngestJobV2, IngestJobV2Results, IngestOperation, JobInfoV2 } from 'jsforce/lib/api/bulk.js'; import { Schema } from 'jsforce'; import { Connection, Messages } from '@salesforce/core'; import { BulkProcessedRecordV2, BulkRecordsV2 } from './types.js'; -Messages.importMessagesDirectoryFromMetaUrl(import.meta.url) +Messages.importMessagesDirectoryFromMetaUrl(import.meta.url); const messages = Messages.loadMessages('@salesforce/plugin-data', 'messages'); -export const POLL_FREQUENCY_MS = 5000; +const POLL_FREQUENCY_MS = 5000; export const isBulkV2RequestDone = (jobInfo: JobInfoV2): boolean => ['Aborted', 'Failed', 'JobComplete'].includes(jobInfo.state); -export const didBulkV2RequestJobFail = (jobInfo: JobInfoV2): boolean => ['Aborted', 'Failed'].includes(jobInfo.state); - export const transformResults = (results: IngestJobV2Results): BulkRecordsV2 => ({ successfulResults: results.successfulResults.map((record) => record as unknown as BulkProcessedRecordV2), failedResults: results.failedResults.map((record) => record as unknown as BulkProcessedRecordV2), diff --git a/src/commands/data/export/beta/tree.ts b/src/commands/data/export/beta/tree.ts new file mode 100644 index 00000000..069660ba --- /dev/null +++ b/src/commands/data/export/beta/tree.ts @@ -0,0 +1,60 @@ +/* + * Copyright (c) 2020, salesforce.com, inc. + * All rights reserved. + * Licensed under the BSD 3-Clause license. + * For full license text, see LICENSE.txt file in the repo root or https://opensource.org/licenses/BSD-3-Clause + */ + +import { Messages } from '@salesforce/core'; +import { SfCommand, Flags, Ux } from '@salesforce/sf-plugins-core'; +import { orgFlags, prefixValidation } from '../../../../flags.js'; +import { ExportConfig, runExport } from '../../../../export.js'; +import { DataPlanPart, SObjectTreeFileContents } from '../../../../dataSoqlQueryTypes.js'; + +Messages.importMessagesDirectoryFromMetaUrl(import.meta.url); +const messages = Messages.loadMessages('@salesforce/plugin-data', 'tree.export'); + +export default class Export extends SfCommand { + public static readonly summary = messages.getMessage('summary'); + public static readonly description = messages.getMessage('description'); + public static readonly examples = messages.getMessages('examples'); + // TODO: when you remove the beta state, put the force: aliases back in + public static readonly state = 'beta'; + + public static readonly flags = { + ...orgFlags, + query: Flags.string({ + char: 'q', + summary: messages.getMessage('flags.query.summary'), + required: true, + }), + plan: Flags.boolean({ + char: 'p', + summary: messages.getMessage('flags.plan.summary'), + }), + prefix: Flags.string({ + char: 'x', + summary: messages.getMessage('flags.prefix.summary'), + parse: prefixValidation, + }), + 'output-dir': Flags.directory({ + char: 'd', + summary: messages.getMessage('flags.output-dir.summary'), + aliases: ['outputdir'], + deprecateAliases: true, + }), + }; + + public async run(): Promise { + const { flags } = await this.parse(Export); + const exportConfig: ExportConfig = { + outputDir: flags['output-dir'], + plan: flags.plan, + prefix: flags.prefix, + query: flags.query, + conn: flags['target-org'].getConnection(flags['api-version']), + ux: new Ux({ jsonEnabled: this.jsonEnabled() }), + }; + return runExport(exportConfig); + } +} diff --git a/src/commands/data/export/tree.ts b/src/commands/data/export/tree.ts index 9183b607..f3f47bd1 100644 --- a/src/commands/data/export/tree.ts +++ b/src/commands/data/export/tree.ts @@ -5,15 +5,13 @@ * For full license text, see LICENSE.txt file in the repo root or https://opensource.org/licenses/BSD-3-Clause */ - - import { Messages } from '@salesforce/core'; import { SfCommand, Flags, Ux } from '@salesforce/sf-plugins-core'; import { orgFlags } from '../../../flags.js'; import { ExportApi, ExportConfig } from '../../../api/data/tree/exportApi.js'; import { DataPlanPart, SObjectTreeFileContents } from '../../../dataSoqlQueryTypes.js'; -Messages.importMessagesDirectoryFromMetaUrl(import.meta.url) +Messages.importMessagesDirectoryFromMetaUrl(import.meta.url); const messages = Messages.loadMessages('@salesforce/plugin-data', 'tree.export'); export default class Export extends SfCommand { @@ -47,6 +45,10 @@ export default class Export extends SfCommand { + this.info( + 'Try the the new "sf data export beta tree" command. It support SOQL queries with up to 5 levels of objects!' + ); + const { flags } = await this.parse(Export); const ux = new Ux({ jsonEnabled: this.jsonEnabled() }); const exportApi = new ExportApi(flags['target-org'], ux); diff --git a/src/commands/data/import/beta/tree.ts b/src/commands/data/import/beta/tree.ts new file mode 100644 index 00000000..8529dec4 --- /dev/null +++ b/src/commands/data/import/beta/tree.ts @@ -0,0 +1,60 @@ +/* + * Copyright (c) 2020, salesforce.com, inc. + * All rights reserved. + * Licensed under the BSD 3-Clause license. + * For full license text, see LICENSE.txt file in the repo root or https://opensource.org/licenses/BSD-3-Clause + */ + +import { Messages } from '@salesforce/core'; +import { SfCommand, Flags, arrayWithDeprecation } from '@salesforce/sf-plugins-core'; +import { importFromPlan } from '../../../../api/data/tree/importPlan.js'; +import { importFromFiles } from '../../../../api/data/tree/importFiles.js'; +import { orgFlags } from '../../../../flags.js'; +import { ImportResult } from '../../../../api/data/tree/importTypes.js'; + +Messages.importMessagesDirectoryFromMetaUrl(import.meta.url); +const messages = Messages.loadMessages('@salesforce/plugin-data', 'tree.import.beta'); + +/** + * Command that provides data import capability via the SObject Tree Save API. + */ +export default class Import extends SfCommand { + public static readonly summary = messages.getMessage('summary'); + public static readonly description = messages.getMessage('description'); + public static readonly examples = messages.getMessages('examples'); + + // TODO: when you remove the beta state, put the force: aliases back in + public static readonly state = 'beta'; + + public static readonly flags = { + ...orgFlags, + files: arrayWithDeprecation({ + char: 'f', + summary: messages.getMessage('flags.files.summary'), + exactlyOne: ['files', 'plan'], + aliases: ['sobjecttreefiles'], + deprecateAliases: true, + }), + plan: Flags.file({ + char: 'p', + summary: messages.getMessage('flags.plan.summary'), + exactlyOne: ['files', 'plan'], + exists: true, + }), + }; + + public async run(): Promise { + const { flags } = await this.parse(Import); + + const conn = flags['target-org'].getConnection(flags['api-version']); + const results = flags.plan ? await importFromPlan(conn, flags.plan) : await importFromFiles(conn, flags.files); + + this.styledHeader('Import Results'); + this.table(results, { + refId: { header: 'Reference ID' }, + type: { header: 'Type' }, + id: { header: 'ID' }, + }); + return results; + } +} diff --git a/src/commands/data/import/tree.ts b/src/commands/data/import/tree.ts index 5815f21f..c5ae56cf 100644 --- a/src/commands/data/import/tree.ts +++ b/src/commands/data/import/tree.ts @@ -5,23 +5,16 @@ * For full license text, see LICENSE.txt file in the repo root or https://opensource.org/licenses/BSD-3-Clause */ - - import { Messages } from '@salesforce/core'; import { getString, JsonMap } from '@salesforce/ts-types'; import { SfCommand, Flags, arrayWithDeprecation } from '@salesforce/sf-plugins-core'; +import { ImportResult } from '../../../api/data/tree/importTypes.js'; import { ImportApi, ImportConfig } from '../../../api/data/tree/importApi.js'; import { orgFlags } from '../../../flags.js'; -Messages.importMessagesDirectoryFromMetaUrl(import.meta.url) +Messages.importMessagesDirectoryFromMetaUrl(import.meta.url); const messages = Messages.loadMessages('@salesforce/plugin-data', 'tree.import'); -type ImportResult = { - refId: string; - type: string; - id: string; -}; - /** * Command that provides data import capability via the SObject Tree Save API. */ @@ -52,12 +45,15 @@ export default class Import extends SfCommand { hidden: true, aliases: ['contenttype'], deprecateAliases: true, + deprecated: { message: messages.getMessage('flags.content-type.deprecation') }, }), // displays the schema for a data import plan 'config-help': Flags.boolean({ summary: messages.getMessage('flags.config-help.summary'), aliases: ['confighelp'], deprecateAliases: true, + hidden: true, + deprecated: { message: messages.getMessage('flags.config-help.deprecation') }, }), }; @@ -96,6 +92,11 @@ export default class Import extends SfCommand { type: { header: 'Type' }, id: { header: 'ID' }, }); + + this.info( + 'Be sure to check out the new "sf data import beta tree". It handles more records and objects with lookups to the same object (ex: parent account)' + ); + return processedResult; } } diff --git a/src/dataSoqlQueryTypes.ts b/src/dataSoqlQueryTypes.ts index 55be455f..9a778bc4 100644 --- a/src/dataSoqlQueryTypes.ts +++ b/src/dataSoqlQueryTypes.ts @@ -44,6 +44,11 @@ export type BasicRecord = { }; }; +export type SObjectTreeInput = Omit & { + attributes: Omit & { + referenceId: string; + }; +}; export interface DataPlanPart { sobject: string; saveRefs: boolean; @@ -51,21 +56,27 @@ export interface DataPlanPart { files: Array; } -export type SObjectTreeInput = { - attributes: { - type: string; - referenceId: string; - }; -} & { - [index: string]: unknown; -}; - export type SObjectTreeFileContents = { records: SObjectTreeInput[]; }; -export const hasNestedRecords = (element: unknown): element is { records: T[] } => - Array.isArray((element as { records: T[] })?.records); +type ElementWithRecords = { records: T[] }; + +/** element: a field value from the sobject. Empty arrays return true */ +export const hasNestedRecords = (element: unknown): element is ElementWithRecords => + Array.isArray((element as ElementWithRecords)?.records); + +/** element: a field value from the sobject. Empty arrays return false */ +export const hasNonEmptyNestedRecords = (element: unknown): element is ElementWithRecords => + hasNestedRecords(element) && element.records.length > 0; + +/** convenience method for filtering Object.entries array */ +export const hasNestedRecordsFilter = (entry: [string, unknown]): entry is [string, ElementWithRecords] => + typeof entry[0] === 'string' && hasNonEmptyNestedRecords(entry[1]); export const isAttributesElement = (element: unknown): element is SObjectTreeInput['attributes'] => !!(element as SObjectTreeInput['attributes']).referenceId && !!(element as SObjectTreeInput['attributes']).type; + +/** convenience method for filtering Object.entries array */ +export const isAttributesEntry = (entry: [string, unknown]): entry is ['attributes', SObjectTreeInput['attributes']] => + entry[0] === 'attributes' && isAttributesElement(entry[1]); diff --git a/src/export.ts b/src/export.ts new file mode 100644 index 00000000..f5471513 --- /dev/null +++ b/src/export.ts @@ -0,0 +1,396 @@ +/* + * Copyright (c) 2020, salesforce.com, inc. + * All rights reserved. + * Licensed under the BSD 3-Clause license. + * For full license text, see LICENSE.txt file in the repo root or https://opensource.org/licenses/BSD-3-Clause + */ + +import path from 'node:path'; +import fs from 'node:fs'; + +import { Logger, Messages, SfError, Lifecycle, Connection } from '@salesforce/core'; +import type { DescribeSObjectResult, QueryResult } from 'jsforce'; +import { Ux } from '@salesforce/sf-plugins-core'; +import { ensure } from '@salesforce/ts-types'; +import { + BasicRecord, + DataPlanPart, + hasNonEmptyNestedRecords, + hasNestedRecordsFilter, + SObjectTreeFileContents, + SObjectTreeInput, + hasNestedRecords, +} from './dataSoqlQueryTypes.js'; +import { hasUnresolvedRefs } from './api/data/tree/functions.js'; + +Messages.importMessagesDirectoryFromMetaUrl(import.meta.url); +const messages = Messages.loadMessages('@salesforce/plugin-data', 'exportApi'); + +const DATA_PLAN_FILENAME_PART = 'plan.json'; + +export interface ExportConfig { + query: string; + outputDir?: string; + plan?: boolean; + prefix?: string; + conn: Connection; + ux: Ux; +} + +/** refFromIdByType.get('account').get(someAccountId) => AccountRef1 */ +export type RefFromIdByType = Map>; + +/** only used internally, but a more useful structure than the original */ +type PlanFile = Omit & { contents: SObjectTreeFileContents; file: string; dir: string }; + +export const runExport = async (configInput: ExportConfig): Promise => { + const { outputDir, plan, query, conn, prefix, ux } = validate(configInput); + const logger = Logger.childFromRoot('runExport'); + const { records: recordsFromQuery } = await queryRecords(conn)(query); + const describe = await cacheAllMetadata(conn)(recordsFromQuery); + + const refFromIdByType: RefFromIdByType = new Map(); + const flatRecords = recordsFromQuery.flatMap(flattenNestedRecords); + flatRecords.map(buildRefMap(refFromIdByType)); // get a complete map of ID<->Ref + + logger.debug(messages.getMessage('dataExportRecordCount', [flatRecords.length, query])); + + if (outputDir) { + await fs.promises.mkdir(outputDir, { recursive: true }); + } + + if (plan) { + const planMap = reduceByType( + recordsFromQuery + .flatMap(flattenWithChildRelationships(describe)()) + .map(addReferenceIdToAttributes(refFromIdByType)) + .map(removeChildren) + .map(replaceParentReferences(describe)(refFromIdByType)) + .map(removeNonPlanProperties) + ); + + const planFiles = [...planMap.entries()].map( + ([sobject, records]): PlanFile => ({ + sobject, + contents: { records }, + saveRefs: shouldSaveRefs(records, [...planMap.values()].flat()), + resolveRefs: hasUnresolvedRefs(records), + file: `${getPrefixedFileName(sobject, prefix)}.json`, + dir: outputDir ?? '', + }) + ); + const output = planFiles.map(planFileToDataPartPlan); + const planName = getPrefixedFileName([...describe.keys(), DATA_PLAN_FILENAME_PART].join('-'), prefix); + await Promise.all([ + ...planFiles.map(writePlanDataFile(ux)), + fs.promises.writeFile(path.join(outputDir ?? '', planName), JSON.stringify(output, null, 4)), + ]); + return output; + } else { + if (flatRecords.length > 200) { + // use lifecycle so warnings show up in stdout and in the json + await Lifecycle.getInstance().emitWarning( + messages.getMessage('dataExportRecordCountWarning', [flatRecords.length, query]) + ); + } + const contents = { records: processRecordsForNonPlan(describe)(refFromIdByType)(recordsFromQuery) }; + const filename = path.join(outputDir ?? '', getPrefixedFileName(`${[...describe.keys()].join('-')}.json`, prefix)); + ux.log(`wrote ${flatRecords.length} records to ${filename}`); + fs.writeFileSync(filename, JSON.stringify(contents, null, 4)); + return contents; + } +}; + +// TODO: remove the saveRefs/resolveRefs from the types and all associated code. It's not used by the updated `import` command +/** for records of an object type, at least one record has a ref to it */ +const shouldSaveRefs = (recordsOfType: SObjectTreeInput[], allRecords: SObjectTreeInput[]): boolean => { + const refs = new Set(recordsOfType.map((r) => `@${r.attributes.referenceId}`)); + return allRecords.some((r) => Object.values(r).some((v) => typeof v === 'string' && refs.has(v))); +}; + +/** convert between types. DataPlanPart is exported and part of the command's return type and file structure so we're stuck with it */ +const planFileToDataPartPlan = (p: PlanFile): DataPlanPart => ({ + sobject: p.sobject, + saveRefs: p.saveRefs, + resolveRefs: p.resolveRefs, + files: [p.file], +}); + +const writePlanDataFile = + (ux: Ux) => + async (p: PlanFile): Promise => { + await fs.promises.writeFile(path.join(p.dir, p.file), JSON.stringify(p.contents, null, 4)); + ux.log(`wrote ${p.contents.records.length} records to ${p.file}`); + }; + +// future: use Map.groupBy() when it's available +// https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Map/groupBy +const reduceByType = (records: SObjectTreeInput[]): Map => + records.reduce>((acc, curr) => { + acc.set(curr.attributes.type, (acc.get(curr.attributes.type) ?? []).concat([curr])); + return acc; + }, new Map()); + +const processRecordsForNonPlan = + (describe: Map) => + (refFromIdByType: RefFromIdByType) => + (records: BasicRecord[]): SObjectTreeInput[] => + records + .map(recurseNestedValues(describe)(refFromIdByType)) + .map(addReferenceIdToAttributes(refFromIdByType)) + .map(replaceParentReferences(describe)(refFromIdByType)) + .map(removeNonPlanProperties); + +const recurseNestedValues = + (describe: Map) => + (refFromIdByType: RefFromIdByType) => + (record: BasicRecord): BasicRecord => + Object.fromEntries( + Object.entries(record).map(([key, value]) => + hasNonEmptyNestedRecords(value) + ? [key, { records: processRecordsForNonPlan(describe)(refFromIdByType)(value.records) }] + : [key, value] + ) + ) as BasicRecord; + +/** removing nulls, Ids, and objects who only property is records: [] */ +const removeNonPlanProperties = (record: SObjectTreeInput): SObjectTreeInput => + Object.fromEntries( + Object.entries(record).filter(nonPlanPropertiesFilter).map(removeUrlIfAttributes) + ) as SObjectTreeInput; + +/** the url doesn't exist in the non-plan tree output. Whether that matters or not, this results in cleaner files */ +const removeUrlIfAttributes = ([key, value]: T): T => + [ + key, + key === 'attributes' && typeof value === 'object' && value && 'type' in value + ? { + type: (value as SObjectTreeInput['attributes']).type, + referenceId: (value as SObjectTreeInput['attributes']).referenceId, + } + : value, + ] as T; + +/** remove Id, nulls and empty records arrays */ +const nonPlanPropertiesFilter = ([key, value]: [string, unknown]): boolean => + key !== 'Id' && value !== null && !isEmptyRecordsArray(value); + +const isEmptyRecordsArray = (value: unknown): value is { records: [] } => + typeof value === 'object' && + value !== null && + 'records' in value && + Object.keys(value).length === 1 && + Array.isArray(value.records) && + value.records.length === 0; + +/** + * pass in a type if you have one to make the search more efficient. + * otherwise, it'll have to search ALL the refs in all the types + */ +export const maybeConvertIdToRef = + (refFromIdByType: RefFromIdByType) => + ([id, type]: [id: string, type?: string]): string => { + const ref = type + ? // we have a type, so we can easily get the ID + refFromIdByType.get(type)?.get(id) + : // it's polymorphic (ex: whatId on event/task) so we gotta check ALL of them :( + [...refFromIdByType.values()].find((map) => map.has(id))?.get(id); + return ref ? `@${ref}` : id; + }; + +/** replace parent references, converting IDs into ref */ +export const replaceParentReferences = + (describe: Map) => + (refFromIdByType: RefFromIdByType) => + (record: SObjectTreeInput | BasicRecord): SObjectTreeInput => { + const typeDescribe = ensure(describe.get(record.attributes.type), `Missing describe for ${record.attributes.type}`); + const replacedReferences = Object.fromEntries( + Object.entries(record) + .filter(isRelationshipFieldFilter(typeDescribe)) // only look at the fields that are references + // We can check describe to see what the type could be. + // If it narrows to only 1 type, pass that to refFromId. + // If it's polymorphic, don't pass a type because refFromId will need to check all the types. + .map(([key, value]) => [ + key, + maybeConvertIdToRef(refFromIdByType)([value, getRelatedToWithMetadata(typeDescribe, key)]), + ]) + ); + + return { ...record, ...replacedReferences } as SObjectTreeInput; + }; + +/** + * Side effect: set it in the refFromIdByType Map if it wasn't already. + * */ +export const buildRefMap = + (refFromIdByType: RefFromIdByType) => + (obj: BasicRecord): RefFromIdByType => { + const [id, type] = [idFromRecord(obj), typeFromRecord(obj)]; + + if (!refFromIdByType.get(type)?.get(id)) { + // we don't know about this ID yet + refFromIdByType.set( + type, + (refFromIdByType.get(type) ?? new Map()).set( + id, + /** calculate the next ref number based on the existing ones for that type. Start with 1 */ + `${type}Ref${(refFromIdByType.get(type)?.size ?? 0) + 1}` + ) + ); + } + return refFromIdByType; + }; + +/** * if there is an ID, we'll turn it into a ref in our mapping and add it to the records's attributes. */ +const addReferenceIdToAttributes = + (refFromIdByType: RefFromIdByType) => + (obj: BasicRecord): SObjectTreeInput => ({ + ...obj, + attributes: { + type: obj.attributes.type, + referenceId: ensure(refFromIdByType.get(typeFromRecord(obj))?.get(idFromRecord(obj))), + }, + }); + +/** + * Ensures a valid query is defined in the export configuration, + * which can be either a soql query or a path to a file containing + * a soql query. + * + * @param config - The export configuration. + */ +const validate = (config: ExportConfig): ExportConfig => { + if (!config.query) { + throw new SfError(messages.getMessage('queryNotProvided'), 'queryNotProvided'); + } + + const filepath = path.resolve(process.cwd(), config.query); + if (fs.existsSync(filepath)) { + config.query = fs.readFileSync(filepath, 'utf8'); + + if (!config.query) { + throw messages.createError('queryNotProvided'); + } + } + + config.query = config.query.trim(); + if (!config.query.toLowerCase().startsWith('select')) { + throw messages.createError('soqlInvalid', [config.query]); + } + + return config; +}; + +const isRelationshipWithMetadata = + (metadata: DescribeSObjectResult) => + (fieldName: string): boolean => + metadata.fields.some( + (f) => f.name.toLowerCase() === fieldName.toLowerCase() && f.type.toLowerCase() === 'reference' + ); + +const getRelatedToWithMetadata = (metadata: DescribeSObjectResult, fieldName: string): string | undefined => { + const result = metadata.fields.find((field) => field.name === fieldName && field.referenceTo?.length); + if (!result?.referenceTo) { + throw new SfError(`Unable to find relation for ${metadata.name}`); + } + + // if there is one type, we know what it is. If there are multiple (polymorphic), we don't know what it is. + return result.referenceTo.length === 1 ? result.referenceTo[0] : undefined; +}; + +/** turn a record into an array of records, recursively extracting its children if any */ +export const flattenNestedRecords = (record: T): T[] => + [record].concat( + Object.entries(record) + .filter(hasNestedRecordsFilter) + .flatMap(([, value]) => value.records) + .flatMap(flattenNestedRecords) + ); + +/** return a record without the properties that have nested records */ +export const removeChildren = (record: T): T => + Object.fromEntries(Object.entries(record).filter(([, value]) => !hasNestedRecords(value))) as T; + +type ParentRef = { + type: string; + id: string; + relationshipName: string; +}; + +/** while flattening, pass the parent information in + * so that related objects from the parent (ex: Account.Cases) + * can be converted to a lookup (ex: Case.AccountId) */ +export const flattenWithChildRelationships = + (describe: Map) => + (parent?: ParentRef) => + (record: BasicRecord): BasicRecord[] => + [setLookupId(describe)(parent)(record)].concat( + Object.entries(record) + .filter(hasNestedRecordsFilter) + .flatMap(([k, v]) => + v.records.flatMap( + flattenWithChildRelationships(describe)({ + type: typeFromRecord(record), + id: idFromRecord(record), + relationshipName: k, + }) + ) + ) + ); + +const setLookupId = + (describe: Map) => + (parent?: ParentRef) => + (record: BasicRecord): BasicRecord => { + if (!parent) return record; + + const field = describe + .get(parent.type) + ?.childRelationships.find((cr) => cr.relationshipName === parent.relationshipName)?.field; + + if (!field) { + void Lifecycle.getInstance().emitWarning( + `no matching field found on ${parent.type} for ${parent.relationshipName}` + ); + return record; + } + + return { ...record, [field]: parent.id }; + }; + +const getPrefixedFileName = (fileName: string, prefix?: string): string => + prefix ? `${prefix}-${fileName}` : fileName; + +/** get all the object types in one pass, and return their describes */ +const cacheAllMetadata = + (conn: Connection) => + async (records: BasicRecord[]): Promise> => { + const uniqueTypes = [...new Set(records.flatMap(flattenNestedRecords).map((r) => r.attributes.type))]; + const describes = await Promise.all(uniqueTypes.map((t) => conn.sobject(t).describe())); + return new Map(describes.map((d) => [d.name, d])); + }; + +const queryRecords = + (conn: Connection) => + async (query: string): Promise> => { + try { + return (await conn.autoFetchQuery(query)) as unknown as QueryResult; + } catch (err) { + if (err instanceof Error && err.name === 'MALFORMED_QUERY') { + const errMsg = messages.getMessage('soqlMalformed', [query]); + const errMsgAction = messages.getMessage('soqlMalformedAction'); + throw new SfError(errMsg, 'MalformedQuery', [errMsgAction]); + } else { + throw err; + } + } + }; + +/** return only fields that, based on metadata, are lookups/master-details AND have a string value (id will be a string) */ +const isRelationshipFieldFilter = + (typeDescribe: DescribeSObjectResult) => + (tuple: [string, unknown]): tuple is [string, string] => + isRelationshipWithMetadata(typeDescribe)(tuple[0]) && typeof tuple[1] === 'string'; + +const idFromRecord = (record: BasicRecord): string => path.basename(record.attributes.url); +const typeFromRecord = (record: BasicRecord): string => record.attributes.type; diff --git a/src/flags.ts b/src/flags.ts index b5340806..d7f13ec5 100644 --- a/src/flags.ts +++ b/src/flags.ts @@ -5,7 +5,6 @@ * For full license text, see LICENSE.txt file in the repo root or https://opensource.org/licenses/BSD-3-Clause */ - import { Messages } from '@salesforce/core'; import { Flags, @@ -14,7 +13,7 @@ import { requiredOrgFlagWithDeprecations, } from '@salesforce/sf-plugins-core'; -Messages.importMessagesDirectoryFromMetaUrl(import.meta.url) +Messages.importMessagesDirectoryFromMetaUrl(import.meta.url); const messages = Messages.loadMessages('@salesforce/plugin-data', 'messages'); export const perflogFlag = Flags.boolean({ @@ -40,3 +39,11 @@ export const resultFormatFlag = Flags.string({ aliases: ['resultformat'], deprecateAliases: true, }); + +export const prefixValidation = (i: string): Promise => { + if (i.includes('/') || i.includes('\\')) { + const treeExportMsgs = Messages.loadMessages('@salesforce/plugin-data', 'tree.export'); + throw new Error(treeExportMsgs.getMessage('PrefixSlashError')); + } + return Promise.resolve(i); +}; diff --git a/src/reporters.ts b/src/reporters.ts index d91b795a..02d09588 100644 --- a/src/reporters.ts +++ b/src/reporters.ts @@ -6,17 +6,15 @@ */ import { EOL } from 'node:os'; - import { Logger, Messages } from '@salesforce/core'; import { ux } from '@oclif/core'; import chalk from 'chalk'; import { get, getArray, getNumber, isString, Optional } from '@salesforce/ts-types'; -import { BatchInfo, BatchState, IngestJobV2Results, JobInfoV2 } from 'jsforce/lib/api/bulk.js'; -import { Schema } from 'jsforce'; +import { JobInfoV2 } from 'jsforce/lib/api/bulk.js'; import { capitalCase } from 'change-case'; import { Field, FieldType, SoqlQueryResult } from './dataSoqlQueryTypes.js'; -Messages.importMessagesDirectoryFromMetaUrl(import.meta.url) +Messages.importMessagesDirectoryFromMetaUrl(import.meta.url); const messages = Messages.loadMessages('@salesforce/plugin-data', 'soql.query'); const reporterMessages = Messages.loadMessages('@salesforce/plugin-data', 'reporter'); @@ -379,53 +377,6 @@ export const escape = (value: string): string => { return value; }; -export const BatchInfoColumns = { - id: { header: 'Batch Id' }, - state: { header: 'State' }, - failed: { header: 'Failed' }, - stateMessage: { header: 'Message' }, -}; - -export const getBatchTotals = (batches: BatchInfo[]): { total: number; failed: number; success: number } => - batches.reduce( - (acc: { total: number; failed: number; success: number }, batch) => { - acc.total += parseInt(batch.numberRecordsProcessed, 10); - acc.failed += parseInt(batch.numberRecordsFailed, 10); - acc.success = acc.total - acc.failed; - return acc; - }, - { total: 0, failed: 0, success: 0 } - ); - -export function getBulk2JobTotals( - results: IngestJobV2Results -): { total: number; failed: number; success: number; unprocessed: number } { - const ttls = { total: 0, failed: 0, success: 0, unprocessed: 0 }; - ttls.total = - Object.keys(results.successfulResults).length + - Object.keys(results.failedResults).length + - Object.keys(results.unprocessedRecords).length; - ttls.failed = Object.keys(results.failedResults).length; - ttls.success = Object.keys(results.successfulResults).length; - ttls.unprocessed = Object.keys(results.unprocessedRecords).length; - return ttls; -} - -export const getFailedBatchesForDisplay = ( - batches: BatchInfo[] -): Array<{ id: string; state: BatchState; failed: string; stateMessage: string }> => { - const failedBatches = batches.filter((batch) => parseInt(batch.numberRecordsFailed, 10) > 0); - const batchesForTable = failedBatches.map((batch) => ({ - id: batch.id, - state: batch.state, - failed: `${batch.numberRecordsFailed.toString().padStart(5, ' ')}/${batch.numberRecordsProcessed - .toString() - .padStart(5, ' ')}`, - stateMessage: batch.stateMessage, - })); - return batchesForTable; -}; - export const getResultMessage = (jobInfo: JobInfoV2): string => reporterMessages.getMessage('bulkV2Result', [ jobInfo.id, diff --git a/test/api/data/tree/export.test.ts b/test/api/data/tree/export.test.ts new file mode 100644 index 00000000..ed296ab4 --- /dev/null +++ b/test/api/data/tree/export.test.ts @@ -0,0 +1,444 @@ +/* + * Copyright (c) 2023, salesforce.com, inc. + * All rights reserved. + * Licensed under the BSD 3-Clause license. + * For full license text, see LICENSE.txt file in the repo root or https://opensource.org/licenses/BSD-3-Clause + */ + +import { expect, config } from 'chai'; +import { DescribeSObjectResult } from 'jsforce'; +import { + RefFromIdByType, + buildRefMap, + flattenNestedRecords, + flattenWithChildRelationships, + maybeConvertIdToRef, + removeChildren, + replaceParentReferences, +} from '../../../../src/export.js'; + +config.truncateThreshold = 0; + +const describeMetadata = new Map([ + [ + 'Account', + { + name: 'Account', + childRelationships: [ + { childSObject: 'Case', field: 'AccountId', relationshipName: 'Cases' }, + { + childSObject: 'Contact', + field: 'AccountId', + relationshipName: 'Contacts', + }, + ], + fields: [ + { name: 'Name', referenceTo: [], type: 'string' }, + { name: 'Type', referenceTo: [], type: 'picklist' }, + { name: 'Industry', referenceTo: [], type: 'picklist' }, + ], + } as unknown as DescribeSObjectResult, + ], + [ + 'Case', + { + name: 'Case', + childRelationships: [], + fields: [ + { name: 'AccountId', referenceTo: ['Account'], type: 'reference' }, + { name: 'Status', referenceTo: [], type: 'picklist' }, + { name: 'Origin', referenceTo: [], type: 'picklist' }, + { name: 'Subject', referenceTo: [], type: 'string' }, + ], + } as unknown as DescribeSObjectResult, + ], + [ + 'Contact', + { + name: 'Contact', + childRelationships: [], + fields: [ + { name: 'AccountId', referenceTo: ['Account'], type: 'reference' }, + { name: 'LastName', referenceTo: [], type: 'string' }, + { name: 'FirstName', referenceTo: [], type: 'string' }, + { name: 'Phone', referenceTo: [], type: 'phone' }, + { name: 'Email', referenceTo: [], type: 'email' }, + ], + } as unknown as DescribeSObjectResult, + ], +]); + +const testRecordList = { + totalSize: 2, + done: true, + records: [ + { + attributes: { + type: 'Account', + url: '/services/data/v39.0/sobjects/Account/001xx000003DHzvAAG', + }, + Id: '001xx000003DHzvAAG', + Name: 'BigDogs', + Industry: 'Construction', + Cases: { + totalSize: 1, + done: true, + records: [ + { + attributes: { + type: 'Case', + url: '/services/data/v39.0/sobjects/Case/500xx000000Yn2uAAC', + }, + Status: 'New', + Origin: 'Web', + Subject: 'I never read the instructions', + AccountId: '001xx000003DHzvAAG', + }, + ], + }, + Contacts: { + totalSize: 1, + done: true, + records: [ + { + attributes: { + type: 'Contact', + url: '/services/data/v39.0/sobjects/Contact/003xx000004TpUeAAK', + }, + FirstName: 'John', + LastName: 'Doe', + Email: 'john@doe.gov', + Phone: '123-456-7890', + }, + ], + }, + }, + { + attributes: { + type: 'Account', + url: '/services/data/v39.0/sobjects/Account/001xx000003DHzvBAG', + }, + Id: '001xx000003DHzvBAG', + Name: 'HotDogs', + Industry: 'Fine Dining', + Cases: null, + Contacts: null, + }, + ], +}; + +describe('flatten', () => { + it('single record', () => { + const oneRecord = { + attributes: { + type: 'Account', + url: '/services/data/v39.0/sobjects/Account/001xx000003DHzvAAG', + }, + Id: '001xx000003DHzvAAG', + Name: 'BigDogs', + Industry: 'Construction', + }; + const queryResult = { + totalSize: 1, + done: true, + records: [oneRecord], + }; + expect(queryResult.records.flatMap(flattenNestedRecords)).to.deep.equal([oneRecord]); + }); + it('no records yield an empty array', () => { + const queryResult = { + totalSize: 0, + done: true, + records: [], + }; + expect(queryResult.records.flatMap(flattenNestedRecords)).to.deep.equal([]); + }); + it('with children', () => { + expect(testRecordList.records.flatMap(flattenNestedRecords)).to.deep.equal([ + { + attributes: { + type: 'Account', + url: '/services/data/v39.0/sobjects/Account/001xx000003DHzvAAG', + }, + Id: '001xx000003DHzvAAG', + Name: 'BigDogs', + Industry: 'Construction', + Cases: { + totalSize: 1, + done: true, + records: [ + { + attributes: { + type: 'Case', + url: '/services/data/v39.0/sobjects/Case/500xx000000Yn2uAAC', + }, + Status: 'New', + Origin: 'Web', + Subject: 'I never read the instructions', + AccountId: '001xx000003DHzvAAG', + }, + ], + }, + Contacts: { + totalSize: 1, + done: true, + records: [ + { + attributes: { + type: 'Contact', + url: '/services/data/v39.0/sobjects/Contact/003xx000004TpUeAAK', + }, + FirstName: 'John', + LastName: 'Doe', + Email: 'john@doe.gov', + Phone: '123-456-7890', + }, + ], + }, + }, + { + attributes: { + type: 'Case', + url: '/services/data/v39.0/sobjects/Case/500xx000000Yn2uAAC', + }, + Status: 'New', + Origin: 'Web', + Subject: 'I never read the instructions', + AccountId: '001xx000003DHzvAAG', + }, + { + attributes: { + type: 'Contact', + url: '/services/data/v39.0/sobjects/Contact/003xx000004TpUeAAK', + }, + FirstName: 'John', + LastName: 'Doe', + Email: 'john@doe.gov', + Phone: '123-456-7890', + }, + { + attributes: { + type: 'Account', + url: '/services/data/v39.0/sobjects/Account/001xx000003DHzvBAG', + }, + Id: '001xx000003DHzvBAG', + Name: 'HotDogs', + Industry: 'Fine Dining', + Cases: null, + Contacts: null, + }, + ]); + }); +}); + +describe('flattenWithChildRelationships', () => { + const fnWithContext = flattenWithChildRelationships(describeMetadata)(undefined); + it('single record', () => { + const oneRecord = { + attributes: { + type: 'Account', + url: '/services/data/v39.0/sobjects/Account/001xx000003DHzvAAG', + }, + Id: '001xx000003DHzvAAG', + Name: 'BigDogs', + Industry: 'Construction', + }; + const queryResult = { + totalSize: 1, + done: true, + records: [oneRecord], + }; + expect(queryResult.records.flatMap(fnWithContext)).to.deep.equal([oneRecord]); + }); + it('no records yield an empty array', () => { + const queryResult = { + totalSize: 0, + done: true, + records: [], + }; + expect(queryResult.records.flatMap(fnWithContext)).to.deep.equal([]); + }); + it('with children', () => { + expect(testRecordList.records.flatMap(fnWithContext)).to.deep.equal([ + { + attributes: { + type: 'Account', + url: '/services/data/v39.0/sobjects/Account/001xx000003DHzvAAG', + }, + Id: '001xx000003DHzvAAG', + Name: 'BigDogs', + Industry: 'Construction', + Cases: { + totalSize: 1, + done: true, + records: [ + { + attributes: { + type: 'Case', + url: '/services/data/v39.0/sobjects/Case/500xx000000Yn2uAAC', + }, + Status: 'New', + Origin: 'Web', + Subject: 'I never read the instructions', + AccountId: '001xx000003DHzvAAG', + }, + ], + }, + Contacts: { + totalSize: 1, + done: true, + records: [ + { + attributes: { + type: 'Contact', + url: '/services/data/v39.0/sobjects/Contact/003xx000004TpUeAAK', + }, + FirstName: 'John', + LastName: 'Doe', + Email: 'john@doe.gov', + Phone: '123-456-7890', + }, + ], + }, + }, + { + attributes: { + type: 'Case', + url: '/services/data/v39.0/sobjects/Case/500xx000000Yn2uAAC', + }, + Status: 'New', + Origin: 'Web', + Subject: 'I never read the instructions', + AccountId: '001xx000003DHzvAAG', + }, + { + attributes: { + type: 'Contact', + url: '/services/data/v39.0/sobjects/Contact/003xx000004TpUeAAK', + }, + FirstName: 'John', + LastName: 'Doe', + Email: 'john@doe.gov', + Phone: '123-456-7890', + // lookup is added to the child record because of the relationship + AccountId: '001xx000003DHzvAAG', + }, + { + attributes: { + type: 'Account', + url: '/services/data/v39.0/sobjects/Account/001xx000003DHzvBAG', + }, + Id: '001xx000003DHzvBAG', + Name: 'HotDogs', + Industry: 'Fine Dining', + Cases: null, + Contacts: null, + }, + ]); + }); +}); + +describe('buildRefMap', () => { + const record = { + attributes: { + type: 'Account', + url: '/services/data/v39.0/sobjects/Account/001xx000003DHzvAAG', + }, + Id: '001xx000003DHzvAAG', + Name: 'BigDogs', + Industry: 'Construction', + }; + it('does not have object, sets correct id/ref#', () => { + const refMap: RefFromIdByType = new Map(); + const result = buildRefMap(refMap)(record); + expect(result.get('Account')).to.deep.equal(new Map([['001xx000003DHzvAAG', 'AccountRef1']])); + }); + + it('has object with an entry already present, so adds a new entry with correct id/ref#', () => { + const refMap: RefFromIdByType = new Map([['Account', new Map([['001xx000004DHzvAAG', 'AccountRef1']])]]); + const result = buildRefMap(refMap)(record); + expect(result.get('Account')?.size).to.equal(2); + expect(result.get('Account')?.get('001xx000004DHzvAAG')).to.equal('AccountRef1'); + expect(result.get('Account')?.get(record.Id)).to.equal('AccountRef2'); + }); +}); + +describe('replaceParentReferences', () => { + const caseRecord = { + attributes: { + type: 'Case', + url: '/services/data/v39.0/sobjects/Case/500xx000000Yn2uAAC', + }, + Status: 'New', + Origin: 'Web', + Subject: 'I never read the instructions', + AccountId: '001xx000003DHzvAAG', + }; + const refMap: RefFromIdByType = new Map([['Account', new Map([['001xx000003DHzvAAG', 'AccountRef1']])]]); + const fnToTest = replaceParentReferences(describeMetadata)(refMap); + + it('replaces parent references with ref#', () => { + const result = fnToTest(caseRecord); + expect(result.AccountId).to.equal('@AccountRef1'); + }); + it('no changes when object is not in refMap', () => { + const emptyMap = new Map>(); + expect(replaceParentReferences(describeMetadata)(emptyMap)(caseRecord)).to.deep.equal(caseRecord); + }); + + it('no changes when id is not in refMap for object', () => { + const modifiedRecord = { ...caseRecord, AccountId: '001xx000003DHzvBAG' }; + expect(fnToTest(modifiedRecord)).to.deep.equal(modifiedRecord); + }); + + it('no changes when there is not parent Id field on the record', () => { + const { AccountId, ...caseWithNoParent } = caseRecord; + const result = fnToTest(caseWithNoParent); + expect(result).to.deep.equal(caseWithNoParent); + }); +}); + +describe('maybeConvertIdToRef', () => { + const refMap: RefFromIdByType = new Map([['Account', new Map([['001xx000004DHzvAAG', 'AccountRef1']])]]); + const fnToTest = maybeConvertIdToRef(refMap); + describe('has type', () => { + it('converts id to ref#', () => { + expect(fnToTest(['001xx000004DHzvAAG', 'Account'])).to.equal('@AccountRef1'); + }); + it('leaves other IDs along', () => { + expect(fnToTest(['001xx000004DHzvBAG', 'Account'])).to.equal('001xx000004DHzvBAG'); + }); + }); + describe('no type', () => { + it('finds id and converts to ref# (for polymorphic fields', () => { + expect(fnToTest(['001xx000004DHzvAAG'])).to.equal('@AccountRef1'); + }); + it('leaves other IDs along', () => { + expect(fnToTest(['001xx000004DHzvBAG'])).to.equal('001xx000004DHzvBAG'); + }); + }); +}); + +describe('removeChildren', () => { + it('removes 2 children of different types', () => { + const record = testRecordList.records[0]; + expect(record).to.have.property('Cases'); + expect(record).to.have.property('Contacts'); + const result = removeChildren(record); + expect(result).to.not.have.property('Cases'); + expect(result).to.not.have.property('Contacts'); + + const { Cases, Contacts, ...originalForComparison } = record; + expect(result).to.deep.equal(originalForComparison); + }); + + it('removes empty record arrays', () => { + const record = { ...testRecordList.records[0], Bars: { records: [] } }; + expect(record).to.have.property('Bars'); + const result = removeChildren(record); + expect(result).to.not.have.property('Bars'); + + const { Bars, Cases, Contacts, ...originalForComparison } = record; + expect(result).to.deep.equal(originalForComparison); + }); +}); diff --git a/test/api/data/tree/importFiles.test.ts b/test/api/data/tree/importFiles.test.ts new file mode 100644 index 00000000..a75caf4e --- /dev/null +++ b/test/api/data/tree/importFiles.test.ts @@ -0,0 +1,67 @@ +/* + * Copyright (c) 2023, salesforce.com, inc. + * All rights reserved. + * Licensed under the BSD 3-Clause license. + * For full license text, see LICENSE.txt file in the repo root or https://opensource.org/licenses/BSD-3-Clause + */ +import fs from 'node:fs'; +import { Messages } from '@salesforce/core'; + +import { expect, assert } from 'chai'; +import { SObjectTreeFileContents } from '../../../../src/dataSoqlQueryTypes.js'; +import { FileInfo, createSObjectTypeMap, validateNoRefs } from '../../../../src/api/data/tree/importFiles.js'; + +Messages.importMessagesDirectoryFromMetaUrl(import.meta.url); +const messages = Messages.loadMessages('@salesforce/plugin-data', 'importApi'); + +describe('importFiles', () => { + describe('validateNoRefs', () => { + const good: FileInfo = { + rawContents: '', + records: [ + { + attributes: { + type: 'Account', + referenceId: 'ref1', + }, + }, + ], + filePath: 'testPath', + sobject: 'Account', + }; + it('return a good FileInfo', () => { + expect(validateNoRefs(good)).to.deep.equal(good); + }); + it('throws for a bad ref', () => { + const bad = { + ...good, + // eslint-disable-next-line camelcase + records: [...good.records, { attributes: { type: 'Account', referenceId: 'ref2' }, Field__c: '@ContactRef46' }], + }; + const expectedError = messages.getMessage('error.RefsInFiles', [bad.filePath]); + try { + validateNoRefs(bad); + throw new Error('Expected an error'); + } catch (e) { + assert(e instanceof Error); + expect(e.message).to.equal(expectedError); + } + }); + }); + describe('createSobjectTypeMap', () => { + it('works with a 2-level tree file with nested records', () => { + const accountsContactsTreeJSON = JSON.parse( + fs.readFileSync('test/api/data/tree/test-files/accounts-contacts-tree.json', 'utf-8') + ) as SObjectTreeFileContents; + + expect(createSObjectTypeMap(accountsContactsTreeJSON.records)).to.deep.equal( + new Map([ + ['SampleAccountRef', 'Account'], + ['PresidentSmithRef', 'Contact'], + ['VPEvansRef', 'Contact'], + ['SampleAcct2Ref', 'Account'], + ]) + ); + }); + }); +}); diff --git a/test/api/data/tree/importPlan.test.ts b/test/api/data/tree/importPlan.test.ts new file mode 100644 index 00000000..075cf4dc --- /dev/null +++ b/test/api/data/tree/importPlan.test.ts @@ -0,0 +1,187 @@ +/* + * Copyright (c) 2023, salesforce.com, inc. + * All rights reserved. + * Licensed under the BSD 3-Clause license. + * For full license text, see LICENSE.txt file in the repo root or https://opensource.org/licenses/BSD-3-Clause + */ +/* eslint-disable camelcase */ // for salesforce __c style fields + +import { expect, assert } from 'chai'; +import { shouldThrow } from '@salesforce/core/lib/testSetup.js'; +import { Logger } from '@salesforce/core'; +import { + replaceRefsInTheSameFile, + EnrichedPlanPart, + replaceRefs, + fileSplitter, + validatePlanContents, +} from '../../../../src/api/data/tree/importPlan.js'; + +describe('importPlan', () => { + describe('replaceRefsInTheSameFile', () => { + it('returns the ref when there are no unresolved refs', () => { + const planPart = { + filePath: 'somePath', + sobject: 'Foo__c', + records: [{ attributes: { referenceId: 'FooRef1', type: 'Foo__c' } }], + files: [], + } satisfies EnrichedPlanPart; + + expect(replaceRefsInTheSameFile(planPart).ready).to.deep.equal(planPart); + expect(replaceRefsInTheSameFile(planPart).notReady).to.be.undefined; + }); + it('splits the ref into two plan "files" if there are unresolved', () => { + const planPart = { + filePath: 'somePath', + sobject: 'Foo__c', + records: [ + { attributes: { referenceId: 'Foo__cRef1', type: 'Foo__c' } }, + { attributes: { referenceId: 'Foo__cRef2', type: 'Foo__c' }, lookup__c: '@Foo__cRef1' }, + { attributes: { referenceId: 'Foo__cRef3', type: 'Foo__c' }, lookup__c: '@Foo__cRef2' }, + ], + files: [], + } satisfies EnrichedPlanPart; + + const result = replaceRefsInTheSameFile(planPart); + expect(result).to.deep.equal({ + ready: { + filePath: 'somePath', + sobject: 'Foo__c', + records: [{ attributes: { referenceId: 'Foo__cRef1', type: 'Foo__c' } }], + files: [], + }, + notReady: { + filePath: 'somePath', + sobject: 'Foo__c', + records: [ + { attributes: { referenceId: 'Foo__cRef2', type: 'Foo__c' }, lookup__c: '@Foo__cRef1' }, + { attributes: { referenceId: 'Foo__cRef3', type: 'Foo__c' }, lookup__c: '@Foo__cRef2' }, + ], + files: [], + }, + }); + }); + }); + describe('replaceRefs', () => { + it('replaces refs in a record', () => { + const records = [ + { attributes: { referenceId: 'Foo__cRef1', type: 'Foo__c' } }, + { attributes: { referenceId: 'Foo__cRef2', type: 'Foo__c' }, lookup__c: '@Foo__cRef1' }, + { attributes: { referenceId: 'Foo__cRef3', type: 'Foo__c' }, lookup__c: '@Foo__cRef2' }, + ]; + const resultsSoFar = [ + { refId: 'Foo__cRef1', type: 'Foo__c', id: '001000000000001' }, + { refId: 'Foo__cRef2', type: 'Foo__c', id: '001000000000002' }, + ]; + expect(replaceRefs(resultsSoFar)(records)).to.deep.equal([ + { attributes: { referenceId: 'Foo__cRef1', type: 'Foo__c' } }, + { attributes: { referenceId: 'Foo__cRef2', type: 'Foo__c' }, lookup__c: '001000000000001' }, + { attributes: { referenceId: 'Foo__cRef3', type: 'Foo__c' }, lookup__c: '001000000000002' }, + ]); + }); + }); + describe('fileSplitter', () => { + const planPartBase = { + filePath: 'somePath', + sobject: 'Foo__c', + records: [], + files: [], + } satisfies EnrichedPlanPart; + + it('returns the same file if it has less than 200 records', () => { + const records = new Array(40).fill({ attributes: { referenceId: 'FooRef1', type: 'Foo__c' } }); + const result = fileSplitter({ ...planPartBase, records }); + expect(result).to.have.length(1); + expect(result[0].records).to.have.length(40); + }); + it('splits a bigger file into multiple files', () => { + const records = new Array(500).fill({ attributes: { referenceId: 'FooRef1', type: 'Foo__c' } }); + + const result = fileSplitter({ ...planPartBase, records }); + expect(result).to.have.length(3); + expect(result[0].records).to.have.length(200); + expect(result[1].records).to.have.length(200); + expect(result[2].records).to.have.length(100); + }); + }); + describe('plan validation', () => { + const logger = new Logger({ name: 'importPlanTest', useMemoryLogger: true }); + afterEach(() => { + // @ts-expect-error private stuff + // eslint-disable-next-line @typescript-eslint/no-unsafe-member-access + logger.memoryLogger.loggedData = []; + }); + const validator = validatePlanContents(logger); + it('good plan in classic format', async () => { + const plan = [ + { + sobject: 'Account', + saveRefs: true, + resolveRefs: true, + files: ['Account.json'], + }, + ]; + expect(await validator('some/path', plan)).to.equal(plan); + expect(getLogMessages(logger)).to.include('saveRefs'); + }); + it('good plan in classic format', async () => { + const plan = [ + { + sobject: 'Account', + saveRefs: true, + resolveRefs: true, + files: ['Account.json', 'Account2.json'], + }, + ]; + expect(await validator('some/path', plan)).to.equal(plan); + }); + it('throws on bad plan (missing the object)', async () => { + const plan = [ + { + saveRefs: true, + resolveRefs: true, + files: ['Account.json', 'Account2.json'], + }, + ]; + try { + await shouldThrow(validator('some/path', plan)); + } catch (e) { + assert(e instanceof Error); + expect(e.name).to.equal('InvalidDataImportError'); + } + }); + // TODO: remove this test when schema moves to simple files only + it('throws when files are an object that meets current schema', async () => { + const plan = [ + { + sobject: 'Account', + saveRefs: true, + resolveRefs: true, + files: [{ file: 'foo', contentType: 'application/json', saveRefs: true, resolveRefs: true }], + }, + ]; + try { + await shouldThrow(validator('some/path', plan)); + } catch (e) { + assert(e instanceof Error); + expect(e.name, JSON.stringify(e)).to.equal('NonStringFilesError'); + } + }); + it('good plan in new format is valid and produces no warnings', async () => { + const plan = [ + { + sobject: 'Account', + files: ['Account.json'], + }, + ]; + expect(await validator('some/path', plan)).to.equal(plan); + expect(getLogMessages(logger)).to.not.include('saveRefs'); + }); + }); +}); + +const getLogMessages = (logger: Logger): string => + logger + .getBufferedRecords() + .map((i) => i.msg) + .join('/n'); diff --git a/test/commands/data/tree/dataTreeCommonChild.nut.ts b/test/commands/data/tree/dataTreeCommonChild.nut.ts new file mode 100644 index 00000000..6ae77ac8 --- /dev/null +++ b/test/commands/data/tree/dataTreeCommonChild.nut.ts @@ -0,0 +1,87 @@ +/* + * Copyright (c) 2020, salesforce.com, inc. + * All rights reserved. + * Licensed under the BSD 3-Clause license. + * For full license text, see LICENSE.txt file in the repo root or https://opensource.org/licenses/BSD-3-Clause + */ +import path from 'node:path'; +import { expect } from 'chai'; +import { execCmd, TestSession } from '@salesforce/cli-plugins-testkit'; +import { ImportResult } from '../../../../src/api/data/tree/importTypes.js'; +import { QueryResult } from '../dataSoqlQuery.nut.js'; + +describe('data:tree commands with a polymorphic whatId (on tasks) shared between multiple parents', () => { + let testSession: TestSession; + const importAlias = 'commonChild'; + const prefix = 'CC'; + + before(async () => { + testSession = await TestSession.create({ + scratchOrgs: [ + { + config: 'config/project-scratch-def.json', + setDefault: true, + }, + { + config: 'config/project-scratch-def.json', + setDefault: false, + alias: importAlias, + }, + ], + project: { sourceDir: path.join('test', 'test-files', 'data-project') }, + devhubAuthStrategy: 'AUTO', + }); + }); + + after(async () => { + await testSession?.clean(); + }); + + it('import -> export -> import round trip should succeed', () => { + const query = + "SELECT Id, Name, (SELECT Id, Name, StageName, CloseDate, (SELECT Id, Subject FROM Tasks) FROM Opportunities), (SELECT Id, Subject, Status, (SELECT Id, Subject FROM Tasks) FROM Cases) FROM Account where name != 'Sample Account for Entitlements'"; + + // Import data to the default org. + execCmd( + `data:import:beta:tree --plan ${path.join( + '.', + 'data', + 'commonChild', + 'Account-Opportunity-Task-Case-plan.json' + )} --json`, + { + ensureExitCode: 0, + } + ); + + execCmd( + `data:export:beta:tree --query "${query}" --prefix ${prefix} --outputdir ${path.join( + '.', + 'export_data' + )} --plan --json`, + { ensureExitCode: 0 } + ); + + // Import data to the 2nd org org. + execCmd( + `data:import:beta:tree --target-org ${importAlias} --plan ${path.join( + '.', + 'export_data', + `${prefix}-Account-Opportunity-Task-Case-plan.json` + )} --json`, + { + ensureExitCode: 0, + } + ); + + // query the new org for import verification + const queryResults = execCmd(`data:query --target-org ${importAlias} --query "${query}" --json`, { + ensureExitCode: 0, + }).jsonOutput; + + expect(queryResults?.result.totalSize).to.equal( + 2, + `Expected 2 Account objects returned by the query to org: ${importAlias}` + ); + }); +}); diff --git a/test/commands/data/tree/dataTreeDeep.nut.ts b/test/commands/data/tree/dataTreeDeep.nut.ts new file mode 100644 index 00000000..8a3fdd43 --- /dev/null +++ b/test/commands/data/tree/dataTreeDeep.nut.ts @@ -0,0 +1,101 @@ +/* + * Copyright (c) 2020, salesforce.com, inc. + * All rights reserved. + * Licensed under the BSD 3-Clause license. + * For full license text, see LICENSE.txt file in the repo root or https://opensource.org/licenses/BSD-3-Clause + */ +import path from 'node:path'; +import { expect } from 'chai'; +import { execCmd, TestSession } from '@salesforce/cli-plugins-testkit'; +import { Dictionary, get, getString } from '@salesforce/ts-types'; +import { QueryResult } from '../dataSoqlQuery.nut.js'; + +describe('data:tree commands with more than 2 levels', () => { + let testSession: TestSession; + + before(async () => { + testSession = await TestSession.create({ + scratchOrgs: [ + { + config: 'config/project-scratch-def.json', + setDefault: true, + }, + { + config: 'config/project-scratch-def.json', + setDefault: false, + alias: 'importOrg', + }, + ], + project: { sourceDir: path.join('test', 'test-files', 'data-project') }, + devhubAuthStrategy: 'AUTO', + }); + }); + + after(async () => { + await testSession?.clean(); + }); + + it('should error with invalid soql', () => { + const result = execCmd( + `data:export:tree --query 'SELECT' --prefix INT --outputdir ${path.join('.', 'export_data')}` + ); + const stdError = getString(result, 'shellOutput.stderr', '').toLowerCase(); + const errorKeywords = ['malformed', 'check the soql', 'invalid soql query']; + expect(errorKeywords.some((keyWord) => stdError.includes(keyWord))).to.be.true; + }); + + it('import -> export -> import round trip should succeed', () => { + const query = + "SELECT Id, Name, Phone, Website, NumberOfEmployees, Industry, (SELECT Lastname, Title, Email FROM Contacts) FROM Account WHERE Name LIKE 'SampleAccount%'"; + + // Import data to the default org. + execCmd(`data:import:tree --plan ${path.join('.', 'data', 'accounts-contacts-plan.json')} --json`, { + ensureExitCode: 0, + }); + + execCmd( + `data:export:tree --query "${query}" --prefix INT --outputdir ${path.join('.', 'export_data')} --plan --json`, + { ensureExitCode: 0 } + ); + + // Import data to the default org. + execCmd( + `data:import:tree --target-org importOrg --plan ${path.join( + '.', + 'export_data', + 'INT-Account-Contact-plan.json' + )} --json`, + { + ensureExitCode: 0, + } + ); + + // query the new org for import verification + const queryResults = execCmd(`data:query --target-org importOrg --query "${query}" --json`, { + ensureExitCode: 0, + }).jsonOutput; + + expect(queryResults?.result.totalSize).to.equal( + 2, + 'Expected 2 Account objects returned by the query to org: importOrg' + ); + + const records = queryResults?.result.records ?? []; + const sampleAccountRecord = records.find((account) => account.Name === 'SampleAccount'); + const sampleAccount2Record = records.find((account) => account.Name === 'SampleAccount2'); + + // verify data is imported + expect(sampleAccountRecord).to.have.property('Phone', '1234567890'); + expect(sampleAccountRecord).to.have.property('Website', 'www.salesforce.com'); + expect(sampleAccountRecord).to.have.property('NumberOfEmployees', 100); + expect(sampleAccountRecord).to.have.property('Industry', 'Banking'); + expect(sampleAccountRecord?.Contacts).to.have.property('totalSize', 3); + + expect(sampleAccount2Record).to.have.property('Phone', '1234567890'); + expect(sampleAccount2Record).to.have.property('Website', 'www.salesforce2.com'); + expect(sampleAccount2Record).to.have.property('NumberOfEmployees', 100); + expect(sampleAccount2Record).to.have.property('Industry', 'Banking'); + const contactRecords = get(sampleAccount2Record, 'Contacts.records') as Dictionary[]; + expect(contactRecords[0]).to.have.property('LastName', 'Woods'); + }); +}); diff --git a/test/commands/data/tree/dataTreeDeepBeta.nut.ts b/test/commands/data/tree/dataTreeDeepBeta.nut.ts new file mode 100644 index 00000000..a1c76098 --- /dev/null +++ b/test/commands/data/tree/dataTreeDeepBeta.nut.ts @@ -0,0 +1,105 @@ +/* + * Copyright (c) 2020, salesforce.com, inc. + * All rights reserved. + * Licensed under the BSD 3-Clause license. + * For full license text, see LICENSE.txt file in the repo root or https://opensource.org/licenses/BSD-3-Clause + */ +import path from 'node:path'; +import { expect } from 'chai'; +import { execCmd, TestSession } from '@salesforce/cli-plugins-testkit'; +import { Dictionary, get, getString } from '@salesforce/ts-types'; +import { QueryResult } from '../dataSoqlQuery.nut.js'; + +describe('data:tree beta commands with more than 2 levels', () => { + const prefix = 'DEEP'; + let testSession: TestSession; + + before(async () => { + testSession = await TestSession.create({ + scratchOrgs: [ + { + config: 'config/project-scratch-def.json', + setDefault: true, + }, + { + config: 'config/project-scratch-def.json', + setDefault: false, + alias: 'importOrg', + }, + ], + project: { sourceDir: path.join('test', 'test-files', 'data-project') }, + devhubAuthStrategy: 'AUTO', + }); + }); + + after(async () => { + await testSession?.clean(); + }); + + it('should error with invalid soql', () => { + const result = execCmd( + `data:export:beta:tree --query 'SELECT' --prefix ${prefix} --outputdir ${path.join('.', 'export_data')}` + ); + const stdError = getString(result, 'shellOutput.stderr', '').toLowerCase(); + const errorKeywords = ['malformed', 'check the soql', 'invalid soql query']; + expect(errorKeywords.some((keyWord) => stdError.includes(keyWord))).to.be.true; + }); + + it('import -> export -> import round trip should succeed', () => { + const query = + "SELECT Id, Name, Phone, Website, NumberOfEmployees, Industry, (SELECT Lastname, Title, Email FROM Contacts) FROM Account WHERE Name LIKE 'SampleAccount%'"; + + // Import data to the default org. + execCmd(`data:import:beta:tree --plan ${path.join('.', 'data', 'deep', 'accounts-contacts-plan.json')} --json`, { + ensureExitCode: 0, + }); + + execCmd( + `data:export:beta:tree --query "${query}" --prefix ${prefix} --outputdir ${path.join( + '.', + 'export_data' + )} --plan --json`, + { ensureExitCode: 0 } + ); + + // Import data to the default org. + execCmd( + `data:import:beta:tree --target-org importOrg --plan ${path.join( + '.', + 'export_data', + `${prefix}-Account-Contact-plan.json` + )} --json`, + { + ensureExitCode: 0, + } + ); + + // query the new org for import verification + const queryResults = execCmd(`data:query --target-org importOrg --query "${query}" --json`, { + ensureExitCode: 0, + }).jsonOutput; + + expect(queryResults?.result.totalSize).to.equal( + 2, + 'Expected 2 Account objects returned by the query to org: importOrg' + ); + + const records = queryResults?.result.records ?? []; + const sampleAccountRecord = records.find((account) => account.Name === 'SampleAccount'); + const sampleAccount2Record = records.find((account) => account.Name === 'SampleAccount2'); + + // verify data is imported + expect(sampleAccountRecord).to.have.property('Phone', '1234567890'); + expect(sampleAccountRecord).to.have.property('Website', 'www.salesforce.com'); + expect(sampleAccountRecord).to.have.property('NumberOfEmployees', 100); + expect(sampleAccountRecord).to.have.property('Industry', 'Banking'); + expect(sampleAccountRecord?.Contacts).to.have.property('totalSize', 3); + + expect(sampleAccount2Record).to.have.property('Phone', '1234567890'); + expect(sampleAccount2Record).to.have.property('Website', 'www.salesforce2.com'); + expect(sampleAccount2Record).to.have.property('NumberOfEmployees', 100); + expect(sampleAccount2Record).to.have.property('Industry', 'Banking'); + const contactRecords = get(sampleAccount2Record, 'Contacts.records') as Dictionary[]; + expect(contactRecords[0]).to.have.property('LastName', 'Woods'); + }); +}); diff --git a/test/commands/data/tree/dataTreeMissingRef.nut.ts b/test/commands/data/tree/dataTreeMissingRef.nut.ts new file mode 100644 index 00000000..5a5afcb8 --- /dev/null +++ b/test/commands/data/tree/dataTreeMissingRef.nut.ts @@ -0,0 +1,56 @@ +/* + * Copyright (c) 2020, salesforce.com, inc. + * All rights reserved. + * Licensed under the BSD 3-Clause license. + * For full license text, see LICENSE.txt file in the repo root or https://opensource.org/licenses/BSD-3-Clause + */ +import path from 'node:path'; +import { expect } from 'chai'; +import { execCmd, TestSession } from '@salesforce/cli-plugins-testkit'; +import { ImportResult } from '../../../../src/api/data/tree/importTypes.js'; + +describe('data:tree beta commands with a missing reference', () => { + let testSession: TestSession; + + before(async () => { + testSession = await TestSession.create({ + scratchOrgs: [ + { + config: 'config/project-scratch-def.json', + setDefault: true, + }, + ], + project: { sourceDir: path.join('test', 'test-files', 'data-project') }, + devhubAuthStrategy: 'AUTO', + }); + }); + + after(async () => { + await testSession?.clean(); + }); + + it('import breaks recursion and fails with good error when a ref is missing', () => { + const failResult = execCmd( + `data:import:beta:tree --plan ${path.join('.', 'data', 'missingRef', 'Account-Opportunity-plan.json')} --json`, + { + ensureExitCode: 'nonZero', + } + ); + expect(failResult.jsonOutput?.name).to.equal('UnresolvableRefsError'); + expect(failResult.jsonOutput?.message).to.include('@AccountRef2000'); // includes the missing ref + expect(failResult.jsonOutput?.message).to.includes('Opportunity.json'); // includes the filename where the ref is + expect(failResult.jsonOutput?.data).to.have.length(2); // data contains results that have already succeeded in import + }); + + it('import breaks recursion and fails with good error when a ref is missing', () => { + const failResult = execCmd( + `data:import:beta:tree --plan ${path.join('.', 'data', 'missingSelfRef', 'Account-plan.json')} --json`, + { + ensureExitCode: 'nonZero', + } + ); + expect(failResult.jsonOutput?.name).to.equal('UnresolvableRefsError'); + expect(failResult.jsonOutput?.message).to.include('@AccountRef2000'); // includes the missing ref + expect(failResult.jsonOutput?.message).to.includes('Account.json'); // includes the filename where the ref is + }); +}); diff --git a/test/commands/data/tree/dataTreeMoreThan200.nut.ts b/test/commands/data/tree/dataTreeMoreThan200.nut.ts new file mode 100644 index 00000000..491b3110 --- /dev/null +++ b/test/commands/data/tree/dataTreeMoreThan200.nut.ts @@ -0,0 +1,82 @@ +/* + * Copyright (c) 2020, salesforce.com, inc. + * All rights reserved. + * Licensed under the BSD 3-Clause license. + * For full license text, see LICENSE.txt file in the repo root or https://opensource.org/licenses/BSD-3-Clause + */ +import path from 'node:path'; +import { expect } from 'chai'; +import { execCmd, TestSession } from '@salesforce/cli-plugins-testkit'; +import { ImportResult } from '../../../../src/api/data/tree/importTypes.js'; +import { QueryResult } from '../dataSoqlQuery.nut.js'; + +describe('data:tree commands with more than 200 records are batches in safe groups', () => { + let testSession: TestSession; + const importAlias = 'importOrgMoreThan200'; + const prefix = '200'; + + before(async () => { + testSession = await TestSession.create({ + scratchOrgs: [ + { + config: 'config/project-scratch-def.json', + setDefault: true, + }, + { + config: 'config/project-scratch-def.json', + setDefault: false, + alias: importAlias, + }, + ], + project: { sourceDir: path.join('test', 'test-files', 'data-project') }, + devhubAuthStrategy: 'AUTO', + }); + }); + + after(async () => { + await testSession?.clean(); + }); + + it('import -> export -> import round trip should succeed', () => { + const query = "SELECT Id, Name, ParentId FROM Account where name != 'Sample Account for Entitlements'"; + + // Import data to the default org. + const importResult = execCmd( + `data:import:beta:tree --plan ${path.join('.', 'data', 'moreThan200', 'Account-plan.json')} --json`, + { + ensureExitCode: 0, + } + ); + expect(importResult.jsonOutput?.result.length).to.equal(265, 'Expected 265 records to be imported'); + + execCmd( + `data:export:beta:tree --query "${query}" --prefix ${prefix} --outputdir ${path.join( + '.', + 'export_data' + )} --plan --json`, + { ensureExitCode: 0 } + ); + + // Import data to the 2nd org org. + execCmd( + `data:import:beta:tree --target-org ${importAlias} --plan ${path.join( + '.', + 'export_data', + `${prefix}-Account-plan.json` + )} --json`, + { + ensureExitCode: 0, + } + ); + + // query the new org for import verification + const queryResults = execCmd(`data:query --target-org ${importAlias} --query "${query}" --json`, { + ensureExitCode: 0, + }).jsonOutput; + + expect(queryResults?.result.totalSize).to.equal( + 265, + `Expected 265 Account objects returned by the query to org: ${importAlias}` + ); + }); +}); diff --git a/test/commands/data/tree/dataTreeSelfReferencing.nut.ts b/test/commands/data/tree/dataTreeSelfReferencing.nut.ts new file mode 100644 index 00000000..c5978a05 --- /dev/null +++ b/test/commands/data/tree/dataTreeSelfReferencing.nut.ts @@ -0,0 +1,76 @@ +/* + * Copyright (c) 2020, salesforce.com, inc. + * All rights reserved. + * Licensed under the BSD 3-Clause license. + * For full license text, see LICENSE.txt file in the repo root or https://opensource.org/licenses/BSD-3-Clause + */ +import path from 'node:path'; +import { expect } from 'chai'; +import { execCmd, TestSession } from '@salesforce/cli-plugins-testkit'; +import { QueryResult } from '../dataSoqlQuery.nut.js'; + +describe('data:tree commands with records that refer to other records of the same type in the same file', () => { + let testSession: TestSession; + + before(async () => { + testSession = await TestSession.create({ + scratchOrgs: [ + { + config: 'config/project-scratch-def.json', + setDefault: true, + }, + { + config: 'config/project-scratch-def.json', + setDefault: false, + alias: 'importOrg', + }, + ], + project: { sourceDir: path.join('test', 'test-files', 'data-project') }, + devhubAuthStrategy: 'AUTO', + }); + }); + + after(async () => { + await testSession?.clean(); + }); + + it('import -> export -> import round trip should succeed', () => { + // exclude an account that occurs in many scratch orgs + const query = "SELECT Id, Name, ParentId FROM Account where name != 'Sample Account for Entitlements'"; + + // Import data to the default org. + execCmd(`data:import:beta:tree --plan ${path.join('.', 'data', 'self-referencing', 'Account-plan.json')} --json`, { + ensureExitCode: 0, + }); + + execCmd( + `data:export:beta:tree --query "${query}" --prefix INT --outputdir ${path.join( + '.', + 'export_data' + )} --plan --json`, + { ensureExitCode: 0 } + ); + + // Import data to the 2nd org org. + execCmd( + `data:import:beta:tree --target-org importOrg --plan ${path.join( + '.', + 'export_data', + 'INT-Account-plan.json' + )} --json`, + { + ensureExitCode: 0, + } + ); + + // query the new org for import verification + const queryResults = execCmd(`data:query --target-org importOrg --query "${query}" --json`, { + ensureExitCode: 0, + }).jsonOutput; + + expect(queryResults?.result.totalSize).to.equal( + 12, + 'Expected 12 Account objects returned by the query to org: importOrg' + ); + }); +}); diff --git a/test/test-files/data-project/data/commonChild/Account-Opportunity-Task-Case-plan.json b/test/test-files/data-project/data/commonChild/Account-Opportunity-Task-Case-plan.json new file mode 100644 index 00000000..39f5fd72 --- /dev/null +++ b/test/test-files/data-project/data/commonChild/Account-Opportunity-Task-Case-plan.json @@ -0,0 +1,26 @@ +[ + { + "sobject": "Account", + "saveRefs": true, + "resolveRefs": false, + "files": ["Account.json"] + }, + { + "sobject": "Opportunity", + "saveRefs": true, + "resolveRefs": true, + "files": ["Opportunity.json"] + }, + { + "sobject": "Task", + "saveRefs": false, + "resolveRefs": true, + "files": ["Task.json"] + }, + { + "sobject": "Case", + "saveRefs": true, + "resolveRefs": true, + "files": ["Case.json"] + } +] diff --git a/test/test-files/data-project/data/commonChild/Account.json b/test/test-files/data-project/data/commonChild/Account.json new file mode 100644 index 00000000..dbdad593 --- /dev/null +++ b/test/test-files/data-project/data/commonChild/Account.json @@ -0,0 +1,18 @@ +{ + "records": [ + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef1" + }, + "Name": "Test" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef2" + }, + "Name": "Other Test Account" + } + ] +} diff --git a/test/test-files/data-project/data/commonChild/Case.json b/test/test-files/data-project/data/commonChild/Case.json new file mode 100644 index 00000000..3eb16767 --- /dev/null +++ b/test/test-files/data-project/data/commonChild/Case.json @@ -0,0 +1,12 @@ +{ + "records": [ + { + "attributes": { + "type": "Case", + "referenceId": "CaseRef1" + }, + "Status": "New", + "AccountId": "@AccountRef1" + } + ] +} diff --git a/test/test-files/data-project/data/commonChild/Opportunity.json b/test/test-files/data-project/data/commonChild/Opportunity.json new file mode 100644 index 00000000..a060c009 --- /dev/null +++ b/test/test-files/data-project/data/commonChild/Opportunity.json @@ -0,0 +1,14 @@ +{ + "records": [ + { + "attributes": { + "type": "Opportunity", + "referenceId": "OpportunityRef1" + }, + "Name": "Test Oppty", + "StageName": "Qualification", + "CloseDate": "2024-02-29", + "AccountId": "@AccountRef1" + } + ] +} diff --git a/test/test-files/data-project/data/commonChild/Task.json b/test/test-files/data-project/data/commonChild/Task.json new file mode 100644 index 00000000..047e6cb2 --- /dev/null +++ b/test/test-files/data-project/data/commonChild/Task.json @@ -0,0 +1,20 @@ +{ + "records": [ + { + "attributes": { + "type": "Task", + "referenceId": "TaskRef1" + }, + "Subject": "Do something", + "WhatId": "@OpportunityRef1" + }, + { + "attributes": { + "type": "Task", + "referenceId": "TaskRef2" + }, + "Subject": "Do a case thing", + "WhatId": "@CaseRef1" + } + ] +} diff --git a/test/test-files/data-project/data/deep/accounts-contacts-plan.json b/test/test-files/data-project/data/deep/accounts-contacts-plan.json new file mode 100644 index 00000000..6dd9c4da --- /dev/null +++ b/test/test-files/data-project/data/deep/accounts-contacts-plan.json @@ -0,0 +1,10 @@ +[ + { + "sobject": "Account", + "files": ["accounts-only.json"] + }, + { + "sobject": "Contact", + "files": ["contacts-only-1.json", "contacts-only-2.json"] + } +] diff --git a/test/test-files/data-project/data/deep/accounts-only.json b/test/test-files/data-project/data/deep/accounts-only.json new file mode 100644 index 00000000..d8b5c302 --- /dev/null +++ b/test/test-files/data-project/data/deep/accounts-only.json @@ -0,0 +1,34 @@ +{ + "records": [ + { + "attributes": { "type": "Account", "referenceId": "SampleAccountRef" }, + "name": "SampleAccount", + "phone": "1234567890", + "website": "www.salesforce.com", + "numberOfEmployees": "100", + "industry": "Banking", + "Contacts": { + "records": [ + { + "attributes": { "type": "Contact", "referenceId": "PresidentSmithRef" }, + "lastname": "Smith", + "title": "President" + }, + { + "attributes": { "type": "Contact", "referenceId": "VPEvansRef" }, + "lastname": "Evans", + "title": "Vice President" + } + ] + } + }, + { + "attributes": { "type": "Account", "referenceId": "SampleAcct2Ref" }, + "name": "SampleAccount2", + "phone": "1234567890", + "website": "www.salesforce2.com", + "numberOfEmployees": "100", + "industry": "Banking" + } + ] +} diff --git a/test/test-files/data-project/data/deep/contacts-only-1.json b/test/test-files/data-project/data/deep/contacts-only-1.json new file mode 100644 index 00000000..a123cd71 --- /dev/null +++ b/test/test-files/data-project/data/deep/contacts-only-1.json @@ -0,0 +1,16 @@ +{ + "records": [ + { + "attributes": { "type": "Contact", "referenceId": "FrontDeskRef" }, + "lastname": "Washington", + "title": "President", + "AccountId": "@SampleAccountRef" + }, + { + "attributes": { "type": "Contact", "referenceId": "ManagerRef" }, + "lastname": "Woods", + "title": "Vice President", + "AccountId": "@SampleAcct2Ref" + } + ] +} diff --git a/test/test-files/data-project/data/deep/contacts-only-2.json b/test/test-files/data-project/data/deep/contacts-only-2.json new file mode 100644 index 00000000..fc07ca5c --- /dev/null +++ b/test/test-files/data-project/data/deep/contacts-only-2.json @@ -0,0 +1,14 @@ +{ + "records": [ + { + "attributes": { "type": "Contact", "referenceId": "JanitorRef" }, + "lastname": "Williams", + "title": "President" + }, + { + "attributes": { "type": "Contact", "referenceId": "DeveloperRef" }, + "lastname": "Davenport", + "title": "Vice President" + } + ] +} diff --git a/test/test-files/data-project/data/missingRef/Account-Opportunity-plan.json b/test/test-files/data-project/data/missingRef/Account-Opportunity-plan.json new file mode 100644 index 00000000..00492ecb --- /dev/null +++ b/test/test-files/data-project/data/missingRef/Account-Opportunity-plan.json @@ -0,0 +1,14 @@ +[ + { + "sobject": "Account", + "saveRefs": true, + "resolveRefs": false, + "files": ["Account.json"] + }, + { + "sobject": "Opportunity", + "saveRefs": true, + "resolveRefs": true, + "files": ["Opportunity.json"] + } +] diff --git a/test/test-files/data-project/data/missingRef/Account.json b/test/test-files/data-project/data/missingRef/Account.json new file mode 100644 index 00000000..dbdad593 --- /dev/null +++ b/test/test-files/data-project/data/missingRef/Account.json @@ -0,0 +1,18 @@ +{ + "records": [ + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef1" + }, + "Name": "Test" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef2" + }, + "Name": "Other Test Account" + } + ] +} diff --git a/test/test-files/data-project/data/missingRef/Opportunity.json b/test/test-files/data-project/data/missingRef/Opportunity.json new file mode 100644 index 00000000..51cf7f78 --- /dev/null +++ b/test/test-files/data-project/data/missingRef/Opportunity.json @@ -0,0 +1,14 @@ +{ + "records": [ + { + "attributes": { + "type": "Opportunity", + "referenceId": "OpportunityRef1" + }, + "Name": "Oppty With Bad Ref", + "StageName": "Qualification", + "CloseDate": "2024-02-29", + "AccountId": "@AccountRef2000" + } + ] +} diff --git a/test/test-files/data-project/data/missingSelfRef/Account-plan.json b/test/test-files/data-project/data/missingSelfRef/Account-plan.json new file mode 100644 index 00000000..0eb4a460 --- /dev/null +++ b/test/test-files/data-project/data/missingSelfRef/Account-plan.json @@ -0,0 +1,8 @@ +[ + { + "sobject": "Account", + "saveRefs": true, + "resolveRefs": true, + "files": ["Account.json"] + } +] diff --git a/test/test-files/data-project/data/missingSelfRef/Account.json b/test/test-files/data-project/data/missingSelfRef/Account.json new file mode 100644 index 00000000..1abf2e52 --- /dev/null +++ b/test/test-files/data-project/data/missingSelfRef/Account.json @@ -0,0 +1,90 @@ +{ + "records": [ + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef1" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef2" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef3" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef4" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef5" + }, + "Name": "Grandchild", + "ParentId": "@AccountRef2000" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef6" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef7" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef8" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef9" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef10" + }, + "Name": "Child", + "ParentId": "@AccountRef11" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef11" + }, + "Name": "Original" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef12" + }, + "Name": "Another Sample Account" + } + ] +} diff --git a/test/test-files/data-project/data/moreThan200/Account-plan.json b/test/test-files/data-project/data/moreThan200/Account-plan.json new file mode 100644 index 00000000..a1088d38 --- /dev/null +++ b/test/test-files/data-project/data/moreThan200/Account-plan.json @@ -0,0 +1,8 @@ +[ + { + "sobject": "Account", + "saveRefs": false, + "resolveRefs": false, + "files": ["Account.json"] + } +] diff --git a/test/test-files/data-project/data/moreThan200/Account.json b/test/test-files/data-project/data/moreThan200/Account.json new file mode 100644 index 00000000..e154c38a --- /dev/null +++ b/test/test-files/data-project/data/moreThan200/Account.json @@ -0,0 +1,1859 @@ +{ + "records": [ + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef1" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef2" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef3" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef4" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef5" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef6" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef7" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef8" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef9" + }, + "Name": "Original" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef10" + }, + "Name": "Another Test Account" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef11" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef12" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef13" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef14" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef15" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef16" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef17" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef18" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef19" + }, + "Name": "Original" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef20" + }, + "Name": "Another Test Account" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef21" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef22" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef23" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef24" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef25" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef26" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef27" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef28" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef29" + }, + "Name": "Original" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef30" + }, + "Name": "Another Test Account" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef31" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef32" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef33" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef34" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef35" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef36" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef37" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef38" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef39" + }, + "Name": "Original" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef40" + }, + "Name": "Another Test Account" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef41" + }, + "Name": "Child" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef42" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef43" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef44" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef45" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef46" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef47" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef48" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef49" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef50" + }, + "Name": "Original" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef51" + }, + "Name": "Another Test Account" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef52" + }, + "Name": "Child" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef53" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef54" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef55" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef56" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef57" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef58" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef59" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef60" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef61" + }, + "Name": "Original" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef62" + }, + "Name": "Another Test Account" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef63" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef64" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef65" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef66" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef67" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef68" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef69" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef70" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef71" + }, + "Name": "Original" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef72" + }, + "Name": "Another Test Account" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef73" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef74" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef75" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef76" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef77" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef78" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef79" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef80" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef81" + }, + "Name": "Original" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef82" + }, + "Name": "Another Test Account" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef83" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef84" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef85" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef86" + }, + "Name": "Child" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef87" + }, + "Name": "Grandchild" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef88" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef89" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef90" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef91" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef92" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef93" + }, + "Name": "Original" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef94" + }, + "Name": "Another Test Account" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef95" + }, + "Name": "Another Test Account" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef96" + }, + "Name": "Child" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef97" + }, + "Name": "Child" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef98" + }, + "Name": "Grandchild" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef99" + }, + "Name": "Grandchild" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef100" + }, + "Name": "Child" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef101" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef102" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef103" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef104" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef105" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef106" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef107" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef108" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef109" + }, + "Name": "Original" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef110" + }, + "Name": "Another Test Account" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef111" + }, + "Name": "Grandchild" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef112" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef113" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef114" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef115" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef116" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef117" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef118" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef119" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef120" + }, + "Name": "Original" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef121" + }, + "Name": "Another Test Account" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef122" + }, + "Name": "Child" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef123" + }, + "Name": "Child" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef124" + }, + "Name": "Grandchild" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef125" + }, + "Name": "Grandchild" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef126" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef127" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef128" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef129" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef130" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef131" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef132" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef133" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef134" + }, + "Name": "Original" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef135" + }, + "Name": "Another Test Account" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef136" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef137" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef138" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef139" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef140" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef141" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef142" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef143" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef144" + }, + "Name": "Original" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef145" + }, + "Name": "Another Test Account" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef146" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef147" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef148" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef149" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef150" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef151" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef152" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef153" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef154" + }, + "Name": "Original" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef155" + }, + "Name": "Another Test Account" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef156" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef157" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef158" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef159" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef160" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef161" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef162" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef163" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef164" + }, + "Name": "Original" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef165" + }, + "Name": "Another Test Account" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef166" + }, + "Name": "Child" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef167" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef168" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef169" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef170" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef171" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef172" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef173" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef174" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef175" + }, + "Name": "Original" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef176" + }, + "Name": "Another Test Account" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef177" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef178" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef179" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef180" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef181" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef182" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef183" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef184" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef185" + }, + "Name": "Original" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef186" + }, + "Name": "Another Test Account" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef187" + }, + "Name": "Grandchild" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef188" + }, + "Name": "Grandchild" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef189" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef190" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef191" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef192" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef193" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef194" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef195" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef196" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef197" + }, + "Name": "Original" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef198" + }, + "Name": "Another Test Account" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef199" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef200" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef201" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef202" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef203" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef204" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef205" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef206" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef207" + }, + "Name": "Original" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef208" + }, + "Name": "Another Test Account" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef209" + }, + "Name": "Grandchild" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef210" + }, + "Name": "Child" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef211" + }, + "Name": "Grandchild" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef212" + }, + "Name": "Child" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef213" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef214" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef215" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef216" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef217" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef218" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef219" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef220" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef221" + }, + "Name": "Original" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef222" + }, + "Name": "Another Test Account" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef223" + }, + "Name": "Grandchild" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef224" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef225" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef226" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef227" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef228" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef229" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef230" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef231" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef232" + }, + "Name": "Original" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef233" + }, + "Name": "Another Test Account" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef234" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef235" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef236" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef237" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef238" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef239" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef240" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef241" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef242" + }, + "Name": "Original" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef243" + }, + "Name": "Another Test Account" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef244" + }, + "Name": "Child" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef245" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef246" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef247" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef248" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef249" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef250" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef251" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef252" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef253" + }, + "Name": "Original" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef254" + }, + "Name": "Another Test Account" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef255" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef256" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef257" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef258" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef259" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef260" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef261" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef262" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef263" + }, + "Name": "Original" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef264" + }, + "Name": "Another Test Account" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef265" + }, + "Name": "Child" + } + ] +} diff --git a/test/test-files/data-project/data/self-referencing/Account-plan.json b/test/test-files/data-project/data/self-referencing/Account-plan.json new file mode 100644 index 00000000..0eb4a460 --- /dev/null +++ b/test/test-files/data-project/data/self-referencing/Account-plan.json @@ -0,0 +1,8 @@ +[ + { + "sobject": "Account", + "saveRefs": true, + "resolveRefs": true, + "files": ["Account.json"] + } +] diff --git a/test/test-files/data-project/data/self-referencing/Account.json b/test/test-files/data-project/data/self-referencing/Account.json new file mode 100644 index 00000000..03130a80 --- /dev/null +++ b/test/test-files/data-project/data/self-referencing/Account.json @@ -0,0 +1,90 @@ +{ + "records": [ + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef1" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef2" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef3" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef4" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef5" + }, + "Name": "Grandchild", + "ParentId": "@AccountRef10" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef6" + }, + "Name": "Global Media" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef7" + }, + "Name": "Acme" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef8" + }, + "Name": "salesforce.com" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef9" + }, + "Name": "sample" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef10" + }, + "Name": "Child", + "ParentId": "@AccountRef11" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef11" + }, + "Name": "Original" + }, + { + "attributes": { + "type": "Account", + "referenceId": "AccountRef12" + }, + "Name": "Another Sample Account" + } + ] +} diff --git a/test/tsconfig.json b/test/tsconfig.json index 29b16d50..b20e8302 100644 --- a/test/tsconfig.json +++ b/test/tsconfig.json @@ -1,16 +1,12 @@ { "extends": "@salesforce/dev-config/tsconfig-test-strict-esm", - "include": [ - "./**/*.ts" - ], + "include": ["./**/*.ts"], "compilerOptions": { "skipLibCheck": true, "sourceMap": true, "baseUrl": "..", "paths": { - "@salesforce/kit": [ - "node_modules/@salesforce/kit" - ] + "@salesforce/kit": ["node_modules/@salesforce/kit"] } } -} \ No newline at end of file +}