Allow specifying an Event Hub to use as the default EntityPath for namespace connection string

[oising]

Description

• Add new extension method WithDefaultEntity(string)

Only one EventHub can be specified as the default entity path. The entity path can still be overridden by settings in integrations. This method can be called multiple times with the same name.

• Add test to validate EntityPath is in connection string, and is well-formed
• Add test to validate setting multiple default entities is invalid
• Updated playground to remove settings usage (but left setting usage in other tests)
• Updated documentation in README.md for integrations to mention new functionality

An exception is thrown early if WithDefaultEntity sees that a hub has already been flagged.

Manual testing (playground project):

• Tested with emulator
• Tested with local dev, deploy remote resources
• Tested via azd up full deployment to azure

Notes

I cleaned up the validation logic and tried to simplify it in the base client shared component EventHubsComponent.cs in the method EnsureConnectionStringOrNamespaceProvided. I also fixed an edge case bug that already existed for parsing. I added #define clauses to the playground for disabling the emulator to make it easier to test. There are also some #define conditionals in the projects to allow using AzureCli credential source, but functionally the playground project as it stands is unchanged.

As the FQNS & token credential method doesn't work with docker and the emulator, I couldn't add local unit testing. That said, all scenarios were fully tested, and the code that extracts the event hub name hint for the FQNS method is shared among all five client types, so I'm confident it's solid.

For those not super familiar with the quirks of event hub connection strings, this PR ensures that the SAS-key style connection string has the EntityPath keyword set if WithDefaultEntity(string hubName) is called:

i.e. Endpoint=sb://localhost:57195;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=SAS_KEY_VALUE;UseDevelopmentEmulator=true;EntityPath=hub

For the FQNS (Uri) & Token Credential connection, this PR stashes a hint in the FQNS (fully qualified namespace) Uri via a QueryString parameter:

https://foobar.servicebus.azure.net:443/?EntityPath=hub

The client integration packages will extract this hint from the Url and assign it to the client's settings.EventHubName if it is found to be unset, thus using the prior tested code path of the settings callback.

Fixes #7093

Checklist

• Is this feature complete?
  • Yes. Ready to ship.
  • No. Follow-up changes expected.
• Are you including unit tests for the changes and scenario tests if relevant?
  • Yes
  • No
• Did you add public API?
  • Yes
    • If yes, did you have an API Review for it?
      • Yes
      • No
    • Did you add <remarks /> and <code /> elements on your triple slash comments?
      • Yes
      • No
  • No
• Does the change make any security assumptions or guarantees?
  • Yes
    • If yes, have you done a threat model and had a security review?
      • Yes
      • No
  • No
• Does the change require an update in our Aspire docs?
  • Yes
    • Is this introducing a breaking change?
      • Yes
        • Link to aspire-docs issue (please use this breaking-change template):
      • No
        • Link to aspire-docs issue (please use this doc-idea template):
  • No

[oising]

If we can agree on tihs, I'll send a PR for the same thing in Azure Service Bus for queue/topic in EntityPath, or I can put it in this one

[davidfowl]

Does this also work when deployed?

[oising]

Does this also work when deployed?

It's funny you should ask -- I had just deployed the playground project to test, and I found a bug. I'll report back when fixed.

[oising]

Well, sh*t.

It turns out that there's no way to officially pass the event hub name when using the FQNS + TokenCredential for a real Event Hub. It's fine for the emulator, because the non-tokencredential connection string allows you to pass an EntityPath. I had mistakely believed you could include the hub name as part of the Uri in the non-emulator case.

One way to do it would be to add Aspire-specific handling of a FQNS that includes the hub name as a query string, e.g. https://foobar.servicebus.windows.net?EntityPath=hubName. But this would be setting a new precedent for Aspire I think, i.e. special casing parsing of connection strings in the Aspire clients. As for my thoughts on the matter, at first it felt off (and it does a little still) but it does not break compatibility in the connection strings for non-Aspire wrapped clients.

The other way might be to emit an environment variable, but then this needs even more thought for scoping/namespacing it.
Thoughts?

@davidfowl @mitchdenny @eerhardt

[davidfowl]

We can use our own connection string format that the client understands. This is what openai does

[sebastienros]

We can use our own connection string format that the client understands. This is what openai does

What about setting the ENV that will be bound to the EntityPath property in the settings on the client instead? I am doing this in a PR for openai to split the connection string from the models/deployments.

Another comment on this PR, late one sorry, I personally prefer calling a helper like WithEntityPath(string?) instead of setting a flag on any hub. I assume this way you could even set it if you don't create the hubs at the host level too?

[oising]

We can use our own connection string format that the client understands. This is what openai does

What about setting the ENV that will be bound to the EntityPath property in the settings on the client instead? I am doing this in a PR for openai to split the connection string from the models/deployments.

Another comment on this PR, late one sorry, I personally prefer calling a helper like WithEntityPath(string?) instead of setting a flag on any hub. I assume this way you could even set it if you don't create the hubs at the host level too?

Hi @sebastienros - I had thought of that in an earlier point in time, but given that there are five different clients for eventhub, we'd have to inject five env vars to bind to all client settings. Also, they would take precedence over client appsettings which might be at best surprising and at worst possibly breaking, right? If we use a custom connection string then it's just a hint and the regular configuration should take precedence, if provided.

As for the WithEntityPath idea, I guess so? What does everyone else think?

[davidfowl]

I prefer a single connection string… This brings back our connection info discussion 🙂

[oising]

@davidfowl:

I prefer a single connection string… This brings back our connection info discussion 🙂

Alrighty -- so I'll add a query path hint in the FQNS. And are we good with having WithEntityPath(string) ? It would just set IsDefaultEntity on the model, and I'd drop the visibility to internal on the field. I take it we don't want to start down the "two ways to do everything" path.

[davidfowl]

before making the change, can we compare both API samples?

[oising]

@davidfowl

before making the change, can we compare both API samples?

A

var eventHub = builder.AddAzureEventHubs("eventhubns")
    .RunAsEmulator()
    .WithHub("hub1")
    .WithHub("hub2", h => h.IsDefaultEntity = true);

vs

B

var eventHub = builder.AddAzureEventHubs("eventhubns")
    .RunAsEmulator()
    .WithHub("hub1")
    .WithHub("hub2")
    .WithDefaultEntity("hub2"); // WithEntityPath ?

At first you might think why not WithDefaultHub? And the reasoning is if we bring the same API to service bus, you'd need WithDefaultQueue and WithDefaultTopic, which would be mutually exclusive. Using Entity is clearer, IMO. I prefer WithDefaultEntity over WithEntityPath because the former is about the resource and the latter is about connection strings, but the implication should be clear.

[sebastienros]

I am fine with both approaches, my preference being an extension method (whatever name makes more sense), or an argument on the AddAzureEventHubs(). The value could then be stored in the event hubs resource. We may don't even have to deal with invalid state, in case you want to set the value with a hub name that doesn't need to be defined on the resource (like you can do today).

Does my suggestion to generate a separate ENV to flow to the settings make sense? It would be prefixed with the resource name to bind to the correct settings.

[oising]

@sebastienros

Does my suggestion to generate a separate ENV to flow to the settings make sense? It would be prefixed with the resource name to bind to the correct settings.

Yes, I thought I had mentioned this above. It does make sense from a mechanical perspective, but like I had said, there are five different clients that ship in the integration and you'd have to inject an ENV var for all of them since you cannot know in the hosting side which clients will be used. Personally, I prefer to inline a hint in the connectionstring. It keeps the scope narrower. Although injecting environment variables means we could probably get away without modifying the clients, it might cause unexpected behaviour when someone tries to specify/override the event hub in appsetting.json -- the env var will always win for configuration.

[sebastienros]

when someone tries to specify/override the event hub in appsetting.json

Isn't it of higher priority than ENVs? And then the configure callback has the most priority.

there are five different clients that ship in the integration

I checked the code as I am not familiar with it, I see the issue is each component has its own settings class, but the EventHubName is set on the base one.

Would something like this work:

    protected override void BindSettingsToConfiguration(AzureMessagingEventHubsBufferedProducerSettings settings,
        IConfiguration configuration)
    {
        configuration.Bind((AzureMessagingEventHubsSettings)settings);
        configuration.Bind(settings);
    }

If not set in component specific section, it would take the one from the apphost config. configureSettings would still work. Bonus, now you can even define the defaults for any component.

[oising]

when someone tries to specify/override the event hub in appsetting.json

Isn't it of higher priority than ENVs? And then the configure callback has the most priority.

Nope. AppSettings are loaded first, then environment. Last one wins. Priority is effectively:

https://learn.microsoft.com/en-us/dotnet/core/extensions/configuration

[oising]

Ooops, I accidentally checked in overrides in the playground to use AzureCliCredential for my local testing. I'll remove them in the next commit.

[oising]

Ready for review @davidfowl

I've updated the PR description with more details to help with the review.

[sebastienros]

Is there a reason we'd want to use this without having WithHub? Example: connecting to an existing azure resource without the need to call WithHub for each of the existing ones.

After all WithHub("foo") doesn't ensure it actually exists on the resource either. Almost like we are asking users twice in that case.

And related to this, as a user it might be nice to be able to call it twice, the second being one winning. That would mean store the hub name in the resource itself, instead of having a bool on each resource to verify the integrity (might be simpler too).

Is "Default" in DefaultEntity something concrete? Why not called WithEntityPath()? (ignore me if I already asked ;) )

[oising]

Is there a reason we'd want to use this without having WithHub? Example: connecting to an existing azure resource without the need to call WithHub for each of the existing ones.

When you mentioned this first, I had to think about it and I don't believe there's a scenario where we'd want to set a default entity path without having any associated hubs in the resource. If we want to use a pre-existing resource that already has hubs, we would use AddConnectionString. Anything that uses AddAzureEventHubs needs to provide at least one hub, or health checks will fail. I understand it is possible to want to create a namespace without hubs and have hubs created dynamically (i.e. some dapr patterns encourage dthis but you have to grant it management permissions.)

After all WithHub("foo") doesn't ensure it actually exists on the resource either. Almost like we are asking users twice in that case.

The resource itself doesn't neccessarily exist either -- all this does is mutate the model. At runtime, it's either projected into the emulator config, or emitted in the manifest. I don't think this is our concern?

And related to this, as a user it might be nice to be able to call it twice, the second being one winning. That would mean store the hub name in the resource itself, instead of having a bool on each resource to verify the integrity (might be simpler too).

I'm not sure how this scenario ("nice to be able to call it twice") would arise beyond a mistake?

Is "Default" in DefaultEntity something concrete? Why not called WithEntityPath()? (ignore me if I already asked ;) )

I say "default" because you can override it in the client settings.

[oising]

@sebastienros -- you're right -- it does make sense to allow someone to call it multiple times with different hubs, with last one winning. I've had a head cold all week and for some reason I didn't quite get it, lol. I'll fix that now.

[oising]

I'm renaming WithDefaultEntity to WithEntityPath since this now opens up:

var ns = builder.AddAzureEventHubs("eventhubns")
    .RunAsEmulator()
    .WithHub("hub")
    .WithHub("hub2");
    
builder.AddProject<Projects.EventHubsConsumerA>("consumera")
    .WithReference(ns.WithEntityPath("hub"));
    
builder.AddProject<Projects.EventHubsConsumerB>("consumerb")
    .WithReference(ns.WithEntityPath("hub2"));
 
 // ...

@davidfowl @eerhardt ^ ?

[oising]

Although ... this approach could be troublesome due to it mutating the parent. Example:

ar ns = builder.AddAzureEventHubs("eventhubns")
    .RunAsEmulator()
    .WithHub("hub")
    .WithHub("hub2");

var hub = ns.WithEntityPath("hub");
var hub2 = ns.WithEntityPath("hub2");

builder.AddProject<Projects.EventHubsConsumerA>("consumera")
    .WithReference(hub);
    
builder.AddProject<Projects.EventHubsConsumerB>("consumerb")
    .WithReference(hub2);

So here, both get pointed at hub2. Oops. What are your thoughts? I like that With vs Add has well defined semantics. Is it clear enough though to dissuade people doing things like this? Maybe this isn't the only API that could fall foul to this, so it's an accepted tradeoff?

[oising]

@eerhardt And this will still allow someone to override this in the client part via settings - either configuration or the settings callback? The thing I don't particularly like about this approach is that it renders the connection string incompatible for the pure Azure SDK client, but I guess you already discussed this. This will be a breaking change for 9.0 clients too.

[oising]

simply put, if you use https://myeventhub.servicebus.windows.net:443/?EntityPath=hub&ConsumerGroup=foo as a format, you won't break pure azure sdk, and you can still deconstruct into settings.

[oising]

Urgh -- I'm getting embrace, extend, extinguish vibes here but Microsoft own both sides of the contract, so maybe it's not so bad lol. On a completely different tangent -- if we're prepared to accept Aspire lock-in on both sides of the contract, why not make it a choice and have resources implement an additional "enhanced" binding interface, e.g. IResourceWithConnectionProperties and leave IResourceWithConnectionString alone? The latter would be a downlevel binding for using the Azure SDK natively, or for components that don't require out of band properties. The former could do whatever the hell it likes. /cc @davidfowl @eerhardt @ReubenBond

[eerhardt]

And this will still allow someone to override this in the client part via settings - either configuration or the settings callback?

You can override it in the settings callback. For configuration, the "client integrations" rules of order are:

• bind settings to configuration
• read connection string information from ConnectionStrings:<name>
• settings callback

So if the info is in both config and ConnectionsStrings, ConnectionStrings wins.

The thing I don't particularly like about this approach is that it renders the connection string incompatible for the pure Azure SDK client

It was already incompatible with the pure Azure SDK client. You can't just pass a URL string (with no shared keys) into the "connectionString" parameter of the client ctor. You already had to check if it wasn't a connection string and pass it into the "fullyQualifiedNamespace" parameter (and fill out "credential") and not the "connectionString" parameter.

This will be a breaking change for 9.0 clients too.

It isn't technically a breaking change because in 9.0 you could never .WithReference on the child EventHub. You could only .WithReference on the top-level EH namespace.

if you use https://myeventhub.servicebus.windows.net:443/?EntityPath=hub&ConsumerGroup=foo as a format, you won't break pure azure sdk, and you can still deconstruct into settings.

This doesn't work for the local emulator. You'd still need to parse out ConsumerGroup of the local emulator connection string. The thinking is to "go all in" on the connection string format instead of one-off formats for each azure service. We already build up our own connection strings for things like Azure OpenAI and Search, as well as Milvus, Qdrant, Elastic. A similar change was made for CosmosDB's Database and Containers. So if we need to encode multiple pieces of information into a single string, we prefer to put it in a connection string format.

have resources implement an additional "enhanced" binding interface

I agree this isn't perfect. I've been pushing for Add support for Connection Properties (dotnet/aspire#1765), but we haven't prioritized it. So single strings is what we have today.

My hope is that in the future the Azure SDK supports IConfiguration natively and we can just bind to a configuration section where the Azure SDK loads whatever information is necessary to make the connection.

[oising]

This doesn't work for the local emulator. You'd still need to parse out ConsumerGroup of the local emulator connection string.

I'm very aware of this - this entire PR that is being abandoned implements all this functionality for both SAS connectionstrings and FQNS/tokencredential.

[davidfowl]

The thing that feels strange about this to be is that fact that we would mutate the original resource to have a default entity instead of making a child resource (like we had before), which could be a way to "deep link" (to use @mitchdenny's words) into an event hub instance. That way you aren't limited to one and there's no strange default behavior to implement.

[oising]

@davidfowl

The thing that feels strange about this to be is that fact that we would mutate the original resource to have a default entity instead of making a child resource (like we had before), which could be a way to "deep link" (to use @mitchdenny's words) into an event hub instance. That way you aren't limited to one and there's no strange default behavior to implement.

I already started that PR before you guys got entangled in internal debate and switched to this model, so then I spent my time on this instead. This is frustrating as a contributor who isn't paid to do this.

This is just a hint to the model which gets projected to emulator config, or into bicep via the manifest. I don't see a problem with the namespace (the primary resource) having a "default entity path" hint, since it can be overridden later. In your own words, "With" mutates the resource model, and "Add" creates new child resources. Just go with this now, you're already deprecated v1 of the AddEventHub calls, remove them in v10, then add them back in v11 with child resources for event hubs. Deep linking can still happen without interfering with having a default entity path on the namespace resource.

update:

"...making a child resource (like we had before)" -- you guys never had child resources for azure event hub (nor service bus.) You had AddEventHub calls that mutated the namespace. This (welcome) distinction between With and Add is good, but it's new.

[davidfowl]

We had an offline discussion about "what is a resource" and decided to make these resources. @eerhardt is going to follow up with a spike.

[davidfowl]

See #7453