-
Notifications
You must be signed in to change notification settings - Fork 9.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Added OSGi metadata via build #6282
Conversation
okhttp-tls/build.gradle
Outdated
attributes('Automatic-Module-Name': 'okhttp3.tls') | ||
} | ||
// modify these lines for MANIFEST.MF properties or for specific bnd instructions | ||
// are okhttp3.tls.internal.* packages used in dependent libraries? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Lets talk about this. Can OkHttp libraries use each other’s internal packages?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No. In OSGi there is no concept that allows "special" modules to access packages other modules can not. Therefore if any other module needs access to the okhttp3.tls.internal.* packages we need to export them. If we do not export them, OSGi will detect at resolution time that the package okhttp3.tls.internal is imported from module x but module y does not export it.
There is the Fragment specification which might allow that but I think it will introduce more trouble than it solves.
okhttp-urlconnection/build.gradle
Outdated
} | ||
// modify these lines for MANIFEST.MF properties or for specific bnd instructions | ||
// because this project uses the same package name as the default okhttp project (which is mostly "invalid" for OSGi) | ||
// we have to remove okhttp from the bnd classpath and import it's packages manually |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what does OSGi want us to do in this case? Should we just say this package doesn’t exist for OSGi users? It’s pretty negligible in terms of value.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are 2 solutions to this:
- Rename the package name (OSGi preferred way - breaks backwards compatibility)
- Make okttp-urlconnection a fragment bundle which will at runtime basically mean that the root okhttp bundle and the okhttp-urlconnection bundle use the same ClassLoader. This means that externally when both bundles are present it's like "a single bundle". Fragment bundles are not really best practices and should generally be avoided if possible but it might work for this case.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we make all of OkHttp’s libraries a fragment bundle and get the internal
visibility we crave?
Aside from architectural purity, what’s the drawback of fragment bundles?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fragment bundles are kind of niche and behave differently based on the environment on some cases. PDE for example the OSGi build system from eclipse (the eclipse version of "bnd") will fail to honor packages added by a fragment module which is not defined on the host module. To make things work for PDE we would additionally need the "Eclipse-ExtensibleAPI: true" attribute in the manifest.
For the okhttp use-case fragment bundles might work to not expose the internal packages because every sibling module depends on the okhttp core library anyway. Do you know if any popular thirdparty library depend on one or more internal packages? (e.g. retrofit?) If that's the case we would need to export these packages anyway.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OkHttp’s internal
stuff is OkHttp-only.
Another option – we can stop reaching across submodules to do internal stuff? That might be simpler.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
... turns out that’s a nonstarter for MockWebServer.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Getting rid of the internal stuff would be indeed the best solution. For okhttp-urlconnection we would still need a fragment because of the package name issue.
Btw. MockWebServer - I have missed that module, I have to modify the build.gradle for that one.
Maybe a hybrid solution would be possible? For those modules still requiring internal apis we use the fragment approach and for the other modules we create a "normal" OSGi bundle.
okhttp/build.gradle
Outdated
attributes('Automatic-Module-Name': 'okhttp3') | ||
} | ||
// modify these lines for MANIFEST.MF properties or for specific bnd instructions | ||
// internal package needs to be exported too because interceptors use internal packages |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
interceptors? Or sibling libraries?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah I meant the sibling libraries.
okhttp/build.gradle
Outdated
bnd ''' | ||
Export-Package: \ | ||
okhttp3,\ | ||
okhttp3.internal.* |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a concept of a friend package?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See okhttp-tls comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
in zipkin we do things like this..
export your internal package..
https://github.com/openzipkin/brave/blob/master/brave/bnd.bnd#L9
explicitly import when internal is used
https://github.com/openzipkin/brave/blob/master/context/slf4j/bnd.bnd#L1
block import when something is source retention..
https://github.com/openzipkin/brave/blob/master/instrumentation/http/bnd.bnd#L2
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just to be sure I understand the approach you set the "braveinternal" attribute mandatory* which enforces that the "braveinternal" attribute is set for every Import-Package statement? Therefore if someone wants to Import one of the internal packages the "braveinternal" attribute must explicitly set otherwise the bundle would not resolve?
So for my understanding this would be like a "I agree that what I'm importing is not public API and can change at will". I kind of like that approach because it prevents accidently importing a package which you shouldn't because it's internal.
* For Reference: https://docs.osgi.org/specification/osgi.core/7.0.0/framework.module.html#i2515263
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you understand it, yes. We have had bugs where we forgot to explicitly import the internal thing, but I think it is a better optimization anyway.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I did hope that bnd does not specify mandatory attributes automatically but I have just tried it and bnd does in fact automatically infer mandatory attributes for Import-Package statements. Therefore if someone is using bnd as a build system he wouldn't even notice that there is an mandatory attribute called "okhttpinternal"/"braveinternal"...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
trying to understand the impact of this.. third parties outside this repo shouldn't be using the internal code anyway, so should be not impacted by ignorance of them. Only here, we have to make sure the bnds are coherent.. am I missing something?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We want to ensure that only the okhttp sibling libraries are able to use the internal package of the main okhttp project. Therefore we add a mandatory "okhttpinternal" directive which ensures that for each Import-Package: okhttp3.internal statement the attribute must be explicitly defined. For us it's easy to define that attribute for the siblings okhttp3 libraries which use those internal packages. If a 3rdparty consumer would also import one of the internal packages without defining the attribute the bundle would resolve with an error according to OSGi spec because the attribute is not defined.
Unfortunately if the 3rdparty is also using bnd as a build system (like we in our internal project for example) we could freely use the okhttp3.internal package (even though we shouldn't) and we wouldn't event notice that we should define a mandatory attribute because bnd figures that out automatically.
As an example: I tried to add that attribute to the root okhttp3 project and I didn't have to specify the attribute in the sibling okhttp-sse library manually because bnd automatically inferred the mandatory attribute which kind of defeats the whole purpose of the attribute.
Manifest-Version: 1.0 Automatic-Module-Name: okhttp3.sse Bnd-LastModified: 1601454505191 Bundle-ManifestVersion: 2 Bundle-Name: com.squareup.okhttp3.sse Bundle-SymbolicName: com.squareup.okhttp3.sse Bundle-Version: 4.10.0.SNAPSHOT Created-By: 14.0.2 (AdoptOpenJDK) Export-Package: okhttp3.sse;uses:="kotlin,kotlin.jvm,okhttp3";version= "4.10.0" Import-Package: kotlin;version="[1.4,2)",kotlin.io;version="[1.4,2)",k otlin.jvm;version="[1.4,2)",kotlin.jvm.internal;version="[1.4,2)",okh ttp3;version="[4.10,5)",okhttp3.internal;version="[4.10,5)";okhttpint ernal=true,okhttp3.internal.connection;version="[4.10,5)";okhttpinter nal=true,okio Private-Package: okhttp3.internal.sse Require-Capability: osgi.ee;filter:="(&(osgi.ee=JavaSE)(version=1.8))" Tool: Bnd-5.1.2.202007211702
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If this automatically sucks in the internal stuff, it would seem to be a behavioral difference in manifest generation here vs for example maven-bundle-plugin. This might be a different interpretation of what is sane behavior, or simply an accident in one or the other.
In any case, would you agree that the internal attribution is important to annotate even the gradle plugin or default configuration thwarts that guard rail?
cc also @helgoboss for input as it is sometimes hard for project teams to come to a sane approach for things like this if they aren't actively using the tooling.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have added the attribute although when bnd is used the benefit is negligible. I did not have to explicitly define the import in the sibling libraries because bnd adds the imports with the property set to true automatically...
dalvik.system;resolution:=optional,\ | ||
org.conscrypt;resolution:=optional,\ | ||
org.bouncycastle.*;resolution:=optional,\ | ||
org.openjsse.*;resolution:=optional,\ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we introduce new dependencies, is there a test that’ll fail to remind us to update this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If new dependencies are introduced you have to decide on a case by case basis if it's an optional dependency or not. If you forget to add the library in the list the worst case will be that a OSGi user of that library has to put that library on the classpath to ensure OSGi can resolve all imports. The resolution optional attribute basically means that those packages are not required at runtime.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe a move the comment about optional (ex why not Import-Package: *
) here?
Platform
or similar might need an update to mention when adding optional library support, to not forget to update this and wherever the test cases are! It could also be useful to use those paths in the OSGi test
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we add a simple "*" here all the android/ssl packages would be required on the classpath for the bundle to resolve therefore I explicitly define the packages which are not required in every case at runtime.
Hmm... maybe we should add the comment to the build.gradle dependency block instead of the platform class because the dependency block must be touched to add new dependencies. I kind of dislike adding comments to code which are build related.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I suppose I meant whatever the package-info equiv of kotlin. I suspect any new member of this directory would possibly taint here. https://github.com/square/okhttp/tree/master/okhttp/src/main/kotlin/okhttp3/internal/platform
Not too fussed where the comment goes, it can be in the CONTRIBUTING file.
thx for the clarification on not '*' that part could be a comment inline here in the gradle.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added documentation to the CONTRIBUTING file
Is there a way to run our JUnit tests with OSGi somehow, so our tests fail if we break OSGi’s rules? I’d hate to introduce an |
I could add a bnd Launchpad test that creates a OSGi environment with every okhttp module and tries to resolve it (resolving is the OSGi phase where OSGi tries to find a Export-Package for every Import-Package). Most OSGi issues will show in the resolving phase. When resolving works the OSGi metadata should be fine. |
So I’m currently struggling with If OSGi doesn’t add encapsulation, what does it add? |
From the Okio bug:
What are enterprises getting for those hours? Is it encapsulation, just not in this case? |
For the MockWebServer we should consider using a fragment, it allows us to keep the internal package private and still give the MockWebServer module the possibility to access those packages. OSGi enforces encapsulation, the issue is getting libraries not written with OSGi in mind to fit into the OSGi corset. When a project is fully engaging in the OSGi world it gives you encapsulation no other ecosystem is giving you but it comes with a cost. OSGi also provides a very fine granular permission model (as I said only if you fully engage in the OSGi world) with the permission admin service where you can define exact permissions who can access which service and much more. |
Yep, good point about the classloading features. Can we make |
Sure - later removing the package would "require" a major version bump though because OSGi bundles normally adhere semantic versioning and removing a previously exported package would break backwards compatibility. Of course providing OSGi metadata would still be better than providing nothing and later removing the package would be collateral damage 😅 |
The internal package is already exposed to jar and java module system users and we break its binary API constantly. I don't think we'd treat removal of it from OSGi as anything but a bug fix. |
Wish list:
I think this means we publish |
I have added a test to verify that okhttp with all sibling modules can be resolved properly. The test fails currently because the dependent okio library used is not OSGi ready yet. Therefore the okio library can't be resolved. I have also changed the bnd classpath to use the OSGi ready kotlin stdlib with that in place bnd generates Import-Package statements for a specific kotlin stdlib version e.g. "kotlin;version="[1.4,2)"" instead of a unversioned Import-package statement. This ensures that if there are multiple kotlin stdlib versions on the OSGi classpath it always uses the correct version. I would recommend the same thing for okio too but I just want to hear back from you what you think before I create a pull request. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the help with this. cc @reta if you don't mind helping review
import java.util.stream.Stream; | ||
|
||
/** | ||
* Tries to resolve the OSGi metadata specified in the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
someone died writing? :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hah - fixed it 🤷♂️
@@ -0,0 +1,13 @@ | |||
-runfw: org.eclipse.osgi |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it feels weird to split responsibility between code and text like this. I'm not sure it is bad per se, but I think sometimes of testcontainers and enjoy seeing the setup logic in one markup (even if that is compiled :) )
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have moved the bnd instructions from the bndrun, cnf/build.bnd into code so everything should be in one place now.
okhttp/build.gradle
Outdated
bnd ''' | ||
Export-Package: \ | ||
okhttp3,\ | ||
okhttp3.internal.* |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
in zipkin we do things like this..
export your internal package..
https://github.com/openzipkin/brave/blob/master/brave/bnd.bnd#L9
explicitly import when internal is used
https://github.com/openzipkin/brave/blob/master/context/slf4j/bnd.bnd#L1
block import when something is source retention..
https://github.com/openzipkin/brave/blob/master/instrumentation/http/bnd.bnd#L2
dalvik.system;resolution:=optional,\ | ||
org.conscrypt;resolution:=optional,\ | ||
org.bouncycastle.*;resolution:=optional,\ | ||
org.openjsse.*;resolution:=optional,\ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe a move the comment about optional (ex why not Import-Package: *
) here?
Platform
or similar might need an update to mention when adding optional library support, to not forget to update this and wherever the test cases are! It could also be useful to use those paths in the OSGi test
@adriancole thanks for helping out with this |
@adriancole @swankjesse is there anything still missing for this PR? @swankjesse is there a date when we can expect that a new okio version is released which we can use in okhttp so the tests get back to green? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me, disclaimer being I'm not an OSGi person.
In projects I've worked on, we used servicemix bundles of okhttp, so it is probably worth pinging someone from there for a second set of eyes cc @jbonofre @WillemJiang @iocanel
For example zipkin almost always uses okhttp for client reporting. With this in, we'd likely switch out re-bundling
@swimmesberger bumping okio deps here #6307, I believe Jesse's plan is RC of okhttp to test this and graal specifically. |
@@ -46,6 +46,14 @@ Running Android Tests | |||
|
|||
$ ANDROID_SDK_ROOT=PATH_TO_ANDROID_HOME/sdk ./gradlew :android-test:connectedCheck | |||
|
|||
OSGi | |||
--------------------- |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for adding this
'bnd': "biz.aQute.bnd:biz.aQute.bnd.gradle:${versions.bnd}", | ||
'bndResolve': "biz.aQute.bnd:biz.aQute.resolve:${versions.bnd}", | ||
'equinox': "org.eclipse.platform:org.eclipse.osgi:${versions.equinox}", | ||
'kotlinStdlibOsgi': "org.jetbrains.kotlin:kotlin-osgi-bundle:${versions.kotlin}" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is a 4.2M .jar, compared to 1.4M for kotlin-stdlib, presumably because this includes kotlin-reflect.
} | ||
|
||
project.dependencies { | ||
// when we provide the OSGi version for the kotlin standard library BND can infer the bundle version for the import package statements |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
tricky. Presumably this doesn’t show up elsewhere. Whew.
I attempted to run this locally, and it failed like this:
|
Okay, figured it out! I needed to run tests with Gradle, and I needed to add |
Merged manually, including some small style changes. 74f9781 Thanks for your patience and endurance on this! |
Thanks everyone involved! @swankjesse FYI: In the final commit the CONTRIBUTING.md stuff seems to be missing. I can also understand the concerns about using "kotlin-osgi-bundle" but we only use it in the osgiApi configuration which will only be added to the bnd classpath. Therefore bnd could potentially use classes/packages (to calculate stuff) not available in the "default" jar but in practice it simply leads to proper versioned packages which is always a win for OSGi. |
@swimmesberger the CONTRIBUTING change is deliberate. We've never had an external contributor introduce a new dependency and I think that documenting how to do it might imply that it was a thing we would expect or accept. OkHttp is very prudent with its dependencies (2!) so I don't think we lose much here. |
Added bnd plugin to the gradle build so that META-INF/MANIFEST.MF file contains OSGi metadata.
This is analogous to the implementation in okio: square/okio#777
I added some comments for those modules where it was difficult for me to know which packages should be exported or not. There is also the okhttp-urlconnection module which uses the same package name as the root okhttp project which BND/OSGi does not really like. In OSGi each module should export only a uniquely identifiable package. Therefore for this package I had to find a "creative" way around it. Maybe the better solution would be to publish a OSGi specific artifact for that module with a changed package name (e.g. okhttp3.urlconnection). I don't really know how much that module is used and why it has the same package name.