DynamicDependency - Skeletal outline

[DrusTheAxe]

This is not complete yet but it builds cleanly and has the right structure. Sharing as other devs working on similar things coming up right behind me are stumbling over some of the same pains (or don't realize yet how ProjectReunion.sln needs to be refined)

What's new:

\dev

• Detours - builds Detours as a static library as Detours provides no NuGet or like reasonably usable prebuilt library and compiling it requires very different settings than the rest of ProjectReunion_DLL. This gives us a 'firewall' so we can build Detours from source but with the same compiler settings as their makefiles do
• DynamicDependency - directory containing shared project providing Flat-C API, WinRT API and Detoured package graph APIs
• Flat-C API - Header and .def exports per spec. Source is skeletal and mostly not implemented yet; calls return E_NOTIMPL.
• WinRT API - IDL per spec. Implemented over Flat-C API.
• Detoured Package Graph APIs --
• ProjectReunion_BoostrapDLL - redist needed by non-packaged apps providing API to find and load Microsoft.ProjectReunion.dll from the ProjectReunion Framework package, so the DynamicDependency API can be called.
• ProjectReunion_DLL - References to DynamicDependency and Detours projects. DllMain() to detour package graph APIs and bootstrap access to the DynamicDependency API

\test

• DynamicDependency - root of DynamicDependency test code and data
• data - root for packages needed by DynamicDependency's tests
• Framework.Math.Add - Framework package providing a dll with an export for testing DynamicDependency API
• Framework.Math.Multiply - Framework package providing a dll with an export for testing DynamicDependency API
• DynamicDependency_Test_Win32 - test project to exercise DynamicDependency Flat-C API

Plus some other random things (fixed up ARM[64] project snippets etc)

What's not present:

\dev

• DynamicDependency Flat-C implementation
• DynamicDepedency\pch.h - should not exist but won't build without. Added this as a temporary stopgap

\test

• Bootstrap tests
• DynamicDependency Flat-C API tests
• DynamicDependency WinRT API tests
• More test packages
• data\makemsix.cmd - creates .msix but doesn't sign it. Needs to sign. Reviewing microsoft-ui-xaml's scripts to steal shamelessly from :P

These will be addressed in upcoming PRs

Observations:

• CppShared is too coarse One shared project per feature under \dev, but 1 shared project for all features under \test? Test should follow dev's model. As I've done for DynamicDepednency
• makemsix.cmd is crude - Low tech but effective. Still, feels like it should be better. We probably need a top level "tools" directory where we can put such things (like windows-ui-xaml has). After moving it for use by others we can refactor it to something better (as opposed to entirely blocking until I created this, crude as it may be)
• SampleFlatC project should go away - readme.md should be kept (with some updates) but moved elsewhere (out of the solution) as directions to future devs how to extend the solution
• SampleWinRT project should go away - same, readme.md is good (with some tweaks) but should move and rest goes away
• CppTest_UWP - delete RSN. Useful for a short while more as a template reference for new projects to spin up doing similar.
• CppTest_Win32 - delete RSN. Useful for a short while more as a template reference for new projects to spin up doing similar.
• CppShared - delete RSN. Obsolete once CppTest_UWP+Win32 are gone
• *Learn to edit .vcxproj - I'm finding it's all too common to need to edit projects, regardless if you create a new one via VS or clone from an existing similar one. Learn how to do both, it's a useful and all too necessary skill
• ** .vcproj docs could be better* - VS docs are a bit thin and confusing when it comes to this level of hackery. microsoft-ui-xaml project's useful to help reverse engineer how to make VS do what you need but it's still time a lot of surfing the web and piecing together the magic
• New VC Projects lack ARM|ARM64 - Manual editing *.vcproj required to add these
• MakeMSIX Via UI or edit .vcproj - you can add this via the UI but it's a lot easier to directly hack the .vcproj file adding the <Target> snippet. Defined generically via macros so simple copy/paste job
• ProjectReunion_DLL OnePchToRuleThemAll - this is likely to bloat as the project grows, impacting everyone's inner loop as we get ever more features cohabitating in the same project. Not a blocker yet but an area of concern to keep our eye on

[jonwis]

Can we pull in detours as a submodule instead? What were the changes made to support Reunion in Detours?

[jevansaks]

Can we pull in detours as a submodule instead? What were the changes made to support Reunion in Detours?

I'd like to avoid submodules, they're kind of a pain. I would have suggested nupkg of source or vcpkg -- looks like @DrusTheAxe had a conversation with detours folks about this (microsoft/Detours#124). Sounds like someone made a vcpkg of detours but it requires that you have vcpkg available. A copy of the source until vcpkg is integrated into the VS install seems ok.

[wjk]

I have a comment to make on the source code organization discipline of this project. Having one vcxitems file in a subfolder of the dev directory per feature does not scale IMHO, and as #157 shows, this repo is going to get a whole lot bigger given enough time. While I understand the benefits of this approach (fewer files for apps to have to deploy), I would recommend that we use static libraries and the /WHOLEARCHIVE and /OPT:ICF linker flags instead. Do we really want component X to be able to include private headers from component Y? With vcxitems, that behavior is unavoidable. (Furthermore, I cannot see how the current repository organization in WinUI 3 will possibly survive the vast code dump that is the currently closed-source components of the WinUI NuGet package.) Were any other approaches considered?

[DrusTheAxe]

I have a comment to make on the source code organization discipline of this project... Were any other approaches considered?

A few approaches were considered but this seemed a good start. We're still learning which choices are effective and which seemed like a good idea at the time but... One of the tenets of the Project Reunion effort is to be more flexible than traditional development of Windows, which sometimes means rewriting things (like, say, project structure...). But we need to get that first hand experience to evaluate what's not working and pick a better path.

I share your reservations with the current structure (and I've got the bloody wounds to prove it...). I don't see some of the current lasting long term. I've already made some changes and I expect more to come (by myself and others), but...you gotta start somewhere. In the crawl/walk/run lifecycle we're still in the delivery room.

Your static lib suggestion is interesting. I can see a few other options, though the tradeoffs are a bit fuzzy yet. Maybe I need more bloodletting (or maybe I'm woozy from current blood loss ).

I'm aiming to get DynamicDependencies stood up and in master this month. After that, reviewing pain points and proposing changes, including project structure and other technical details, as well as non-technical process issues. I've made a note to include your comment in that list of things to re-evaluate.

The journey of a thousand miles starts with a single step. We're far closer to that 1st step than the 1,000th but we're working on it. Please keep the feedback coming, but also be patient as we work to build a better world. R(ome)union won't be built in a day :-)

-Howard

P.S. Release configs should already have good optimizations enabled (I thought /OPT:ICF was already enabled, for one). That should be true regardless of the source structure how we combine the bits into end files. If you find oversights or inconsistencies please speak up. I'm hoping we get to a more automated structure but short term there's a lot of manual effort as we bootstrap the system. I'm playing whack-a-mole as I spot things but suggestions are (always) welcome

[jonwis]

How much of this is real product work vs. ES work? I'm leery of merging something that looks done (your product work) but isn't. Could some of the ES work be moved out to another branch temporarily while that finishes up?

[jonwis]

Is there a "user's manual" for how another app would be able to generate their own lifetime manager etc.?

Feels like there's some common code we could do here, like "put this markup in your sxs application manifest and we'll find it" ... could even use a form of <dependency><packageDependency><package familyName="...."/> then the DLL that they carry with them (the mdd seperate DLL) knows how to find and process this node to do all the dependency stuff. That could be optional, of course, so the app just calls MddAddDefaultDependencies() if they've done the configuration.

[DrusTheAxe]

How much of this is real product work vs. ES work? I'm leery of merging something that looks done (your product work) but isn't. Could some of the ES work be moved out to another branch temporarily while that finishes up?

All of my work is real product work, as in product code, or test code to verify product code. That happens to include some ES-y things e.g. make new projects for my tests, with the right compiler settings etc. But those are means to my ends, only done out of necessity to finish my work.

Anything that smacks of straight ES I've largely been looking to see the fruits of @EHO-makai 's work :-) We'll eventually have some things we can refine, polish or simplify, and we are cross pollinating as we find problems the other's found solutions for. But so far we're pretty parallel/independent and focused on our lanes.

[DrusTheAxe]

PR is complete for review. Several places are marked with //TODO; will be addressed in upcoming PR.

Would like to get signoff and complete this PR in the 'commit early and commit often' approach. Future PRs should be more bite-size given this foundation (easier to review, easier to change, faster code velocity, yadda yadda)

[wjk]

You stated that this PR is "ready for review", but if so, why isn’t this implemented? Inquiring minds want to know. 😄

[wjk]

Ditto.

[wjk]

Ditto.

[DrusTheAxe]

Branch is wonky given the shift to main. I'm going to abandon this PR and make a new one rebased against main