Deploy/retrieve to support metadata API format

[keirbowden]

Is your feature request related to a problem? Please describe.
The new force:source:retrieve and force:source:deploy are great, but they require the code to be structured in the new source format. Our existing (4+ years worth of history) project is in metadata API format, so we are in the position of needing to convert our code into source format to interact with the org via VS Code and then back again when we need to pull changes from version control, which isn't practical for day to day development.

Describe the solution you'd like
A mechanism to interact with an org at the file level that doesn't require us to switch to source format.

Describe alternatives you've considered
I'll be sticking with my script that carries out deployments using the mdapi subcommands.

Additional context
Those of us with mature orgs/codebases seem to be caught between a rock and a hard place at the moment. The Force.com IDE supports our existing formats but isn't developed any more, while VS Code is tightly coupled with the SalesforceDX way of doing things, so we either have an out of date IDE or one where a bunch of stuff doesn't work because our code is not in the expected format.

[wadewegner]

Note: we will be scrubbing our docs so that "Salesforce DX source format" simply becomes "source format" and "metadata source format" becomes "metadata packaging format". This will help to express the purpose of these different formats.

We fully appreciate the challenges that have existed while our tooling hasn't provided you with all the capabilities against the new source format that have existed against the metadata packaging format. Our goal is to update all our tooling to work with the source format so that there's no longer a need to switch back and forth -- you will simply be able to use the source format and benefit from the significant advantages it brings. The release of force:source:retrieve/deploy is a step in that direction.

Once the tooling is in place, what benefits do you see with continuing to use the metadata packaging format?

[keirbowden]

@wadewegner There's a few reasons why we'll likely hang on to the metadata packaging format for some of our mature projects:

1. Several years of git history. Not a huge deal as we will be able to take most of it with us I think, but some projects/customers will be a little sensitive about losing anything.

2. Retraining the developers. Again varies by project and customer, but switching over to a new format will require training for a bunch of people.

3. Custom tooling. We've written our own CLI over the last 3-4 years, building on first the Force CLI and then the Salesforce CLI, which allows us to deploy to multiple "endpoints" with differences in configurations and contents from a single codebase. This is reasonably sophisticated in places, for example being able to merge packaging metadata and override specific nodes in XML when generating files. This would require rework to enable us to switch over to the new format, but given 1 and 2 I'd actually need to be able to support both the source and packaging metadata formats. This is also integrated with VS Code and allows me, among other things, to deploy all the files that have changed since my last deployment, the differences between two commit ids. This means that I can carry on with the old format without too many problems, which in turn makes it a little more difficult to justify the investment in switching!

For new projects this becomes a pretty straightforward decision once the tooling is in place, and we'll likely move over on a case by case basis, it's the existing pieces that are the headache at the moment.

[ntotten]

I am particularly curious about 1. If we had an automated tool to help you convert from metadata format to source format while maintaining a clean git history would that be enough to overcome the issue? I did write a blog post describing how to do this, obviously, we could make this a lot easier though. https://ntotten.com/2018/05/11/convert-metadata-to-source-format-while-maintain-git-history/

2 and 3 are valid concerns. I'm going to think on this for a while. It would be great to chat more as well. If you would like to feel free to schedule some time with me: https://calendly.com/totten/office-hours (Note, I am probably fully booked the rest of this week and I am on vacation next week, but happy to chat any time after that.

[keirbowden]

@ntotten yes, that would remove any objections from the point 1 perspective.

When you come back I go on vacation for two weeks, then briefly back before Dreamforce. I'll be attending so maybe I could chat to you then (although I suspect we'll both have busy weeks then too!).

[ntotten]

I'd be happy to meet up at dreamforce as well. I will be busy, but I'm sure we can find some time. email me [email protected] and we can put something on the schedule.

[ntotten]

I am going to close this as we currently have no plans to support metadata source format in Visual Studio Code. All new work is being done on the source format and we are encouraging everyone to move to that format. Please reopen if you would like to continue the discussion.

[chulai]

It's unfortunate that this request won't be properly addressed. All concerns of OP are valid points for any big project using metadata format. We have an Org that is 5+years old, a 4+ year old repository history, around 20 dev teams working in different projects for a total of 80 to 90 developers actively working 24/7. Our CI is a custom solution of ANT + bash scripts. If we want to switch to Vs Code, DX CLI, Scratch Orgs and source format is something we need to do incrementally. How are we supposed to re-train developers in new CI, tools, and IDE overnight? Also, considering this is a 24/7 environment we cannot expect all developers to switch to new IDE and push code/metadata with the new source format in the blink of an eye. We can certainly start migrating our builds to use DX CLI instead of ANT (still using metadata api retrieve/deploy with sandbox instead of scratch orgs) but developers need to be able to use their current IDE (Sublime/Atom.io + MavensMate/ANT) to retrieve/deploy for a while until they are ready to migrate to VS Code (still using metadata retrieve/deploy for a while).
Hope this decision can be reconsidered. Otherwise, It would be even harder for us to migrate to SFDX and Vs Code. Thanks!

[ntotten]

What we will be doing instead is providing more tooling and guidance around how you can do the migration incrementally. Unfortunately, it is a major effort for us to support both formats and it’s something that will be a continued cost to maintain if we support both formats. I understand that migrations may be difficult or take time, but we believe in the long run it will be better for everyone to move to source format. There are significant productivity, readability, and source tracking benefits from the new format that simply aren’t possible with the old format. Additionally, we will continue to improve the new format by exposing all metadata types in ways that can be edited in VSCode and stored in source control systems. You can expect much more guidance on how to make these types of transitions in the coming year.

[dcarroll]

I think it's important to understand that the difference between source format and mdapi format is not that actual code itself. Rather it's the organization of the metadata files in the local file system. Having said that, we are preserving commands that allow you to retain the mdapi format. It may take a command or two extra and some management of your local files, but you will be able to still have mdapi as your gold code.

This is really just a matter of how the metadata is distributed within your local project, the way you work with Apex, VF and LC/LWC doesn't change with the switch to source format.

Hope this helps. In the next few weeks we will be able to socialize the changes that we are making and the are committed to allowing you to switch at such time as makes sense to you.

[chulai]

@ntotten I understand the benefits of switching to source format (one large object file vs smaller files for fields, listViews et al.) and changing the file extension of files that store XML files to -meta.xml (so we can get better support from XML tools). They all make perfect sense. I'd love to see something like that could be done with Profiles/Permission Sets that currently are a hassle to maintain. I also understand your reasons for deciding to not support the old metadata format for new tools. All I was looking for is basic support to retrieve/deploy using mdAPI by right-clicking a file or folder in Vs Code so developers could start using and learning their ways in Vs Code so when everybody was comfortable with the new tools to convert to new source format. And I guess I was minimizing the effort to just adding the menu options in Vs Code. I found that none of the SFDX commands for metadata api (mdapi namespace) allows specifying a folder to deploy/retrieve while this is possible with commands such as source:deploy/source:retrieve albeit with new source format. Any reason for this limitation? Without this limitation, it would be easier to customize the salesforce dx vs code extension to support mdapi deploy/retrieve.

@dcarroll yes, it's clear to me this is a reorganization of the metadata files. But I'm more concerned about developers unintentionally pushing files with the old format/folder structure after we've done the conversion because of still using legacy tools/IDEs (ANT, MavensMate). It's going to take a while and it seems during that transition, code reviewers on each workstream/project would have to be extra diligent in catching these issues before they get into the release branch. Unfortunately, our version control cloud service, BitBucket, does not allow to create server-side hooks to check and prevent this commits/push.

Hope this makes sense and I'm looking forward to tools, articles that could help us with this conversion/transition.