Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update main with the java17 edits from 4.0.X #226

Merged
merged 2 commits into from
Oct 11, 2024
Merged

update main with the java17 edits from 4.0.X #226

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c08c2c8
update main with the java17 edits from 4.0.X
leerho Oct 6, 2024
2a08f94
make NOTICE, pom and README in main the same as 4.0.X
leerho Oct 6, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/manual-coverage.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ jobs:
strategy:
fail-fast: false
matrix:
jdk: [ 8 ]
jdk: [ 17 ]
os: [ ubuntu-latest ]
include:
# - os: windows-latest
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/manual-os-matrix.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ jobs:
fail-fast: false

matrix:
jdk: [ 8, 11 ]
jdk: [ 17 ]
os: [ windows-latest, ubuntu-latest, macos-latest ]
include:
- os: windows-latest
Expand Down
70 changes: 1 addition & 69 deletions LICENSE
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -215,24 +215,6 @@ APPENDIX B: Additional licenses relevant to this product:
conditions of the following licenses.



=============================================================
Apache License version 2.0 (see above)
=============================================================
DirectMemory
Copyright 2015 Odnoklassniki Ltd, Mail.Ru Group

Code locations:
-------------------------------------------------------------
This product contains a method to create a dummy read-only ByteBuffer using unsafe:
* datasketches-memory-java8/src/main/java/org/apache/datasketches/memory/internal/AccessByteBuffer.java,
Method: getDummyReadOnlyDirectByteBuffer(...)
and adapted from Java source code located at:
* src/one/nio/mem/DirectMemory.java,
Method: wrap(...).



=============================================================
Apache License version 2.0 (see above)
=============================================================
Expand All @@ -251,55 +233,6 @@ APPENDIX B: Additional licenses relevant to this product:
* https://github.com/OpenHFT/Zero-Allocation-Hashing



=============================================================
Google Protobuf License:
=============================================================
Copyright 2008 Google Inc.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:

* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the name of Google Inc. nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Code generated by the Protocol Buffer compiler is owned by the owner
of the input file used when generating it. This code is not
standalone and requires a support library to be linked with it. This
support library is itself covered by the above license.

Code locations:
-------------------------------------------------------------
This product contains code for encoding, decoding and testing UTF8:
* datasketches-memory-java8/src/main/java/org/apache/datasketches/memory/internal/Utf8.java
* datasketches-memory-java8-tests/src/test/java/org/apache/datasketches/memory/test/Utf8Test.java
* datasketches-memory-java8-tests/src/test/java/org/apache/datasketches/memory/test/IsValidUtf8TestUtil.java
and adapted from Java source code located at:
* https://github.com/protocolbuffers/protobuf/blob/master/java/core/src/main/java/com/google/protobuf/Utf8.java,
* https://github.com/protocolbuffers/protobuf/blob/master/java/core/src/test/java/com/google/protobuf/Utf8Test.java,
* https://github.com/protocolbuffers/protobuf/blob/master/java/core/src/test/java/com/google/protobuf/IsValidUtf8TestUtil.java,



=============================================================
BSD-2-Clause License
=============================================================
Expand Down Expand Up @@ -330,14 +263,13 @@ APPENDIX B: Additional licenses relevant to this product:

Code locations:
-------------------------------------------------------------
This product contains code for encoding, decoding and testing UTF8:
This product contains code for encoding, decoding and testing XxHash:
* datasketches-memory-java8/src/main/java/org/apache/datasketches/memory/internal/XxHash64.java
and adapted from C++ code located at:
* https://github.com/Cyan4973/xxHash/blob/dev/xxhash.c,
* https://github.com/Cyan4973/xxHash/blob/dev/xxhash.h



=============================================================
Public Domain (optional notice)
=============================================================
Expand Down
4 changes: 2 additions & 2 deletions NOTICE
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Apache DataSketches Memory
Copyright 2022 - The Apache Software Foundation
Copyright 2024 - The Apache Software Foundation

Copyright 2015-2018 Yahoo Inc.
Copyright 2019-2020 Verizon Media
Copyright 2021 Yahoo Inc.
Copyright 2021- Yahoo Inc.

This product includes software developed at
The Apache Software Foundation (http://www.apache.org/).
Expand Down
162 changes: 36 additions & 126 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,6 @@

[![Build Status](https://travis-ci.org/apache/datasketches-memory.svg?branch=master)](https://travis-ci.org/apache/datasketches-memory)
[![Maven Central](https://maven-badges.herokuapp.com/maven-central/org.apache.datasketches/datasketches-memory/badge.svg)](https://maven-badges.herokuapp.com/maven-central/org.apache.datasketches/datasketches-memory)
[![Language grade: Java](https://img.shields.io/lgtm/grade/java/g/apache/datasketches-memory.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/apache/datasketches-memory/context:java)
[![Total alerts](https://img.shields.io/lgtm/alerts/g/apache/datasketches-memory.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/apache/datasketches-memory/alerts/)
[![Coverage Status](https://coveralls.io/repos/github/apache/datasketches-memory/badge.svg?branch=master)](https://coveralls.io/github/apache/datasketches-memory?branch=master)

=================
Expand All @@ -29,110 +27,72 @@
This Memory component is general purpose, has no external runtime dependencies and can be used in any
application that needs to manage data structures inside or outside the Java heap.

The goal of this component of the DataSketches library is to provide a high performance access API for accessing four different types of memory resources. Each of the four resource types is accessed using different API methods in the Memory component.

* **Heap:** Contiguous bytes on the Java Heap constructed by, e.g., *WritableMemory.writableWrap(byte[])* or using the *WritableMemory.allocate(int)* method. For purposes of this document this includes on-heap ByteBuffers constructed using *ByteBuffer.allocate(int)*.

* **DirectByteBuffer:** Contiguous bytes of a ByteBuffer constructed by, e.g., *WritableMemory.writableWrap(ByteBuffer)* where the ByteBuffer was previously constructed using *ByteBuffer.allocateDirect(int)*; or, is a slice() thereof.

* **Direct:** Contiguous bytes off the Java Heap constructed by, e.g., *WritableMemory.allocateDirect(long)* method.

* **Memory-Mapped Files** Contiguous bytes of a file represented in off-heap memory and created using, e.g., the *WritableMemory.writableMap(File)* method.

Please visit the main [DataSketches website](https://datasketches.apache.org) for more information.

If you are interested in making contributions to this Memory component please see our
[Community](https://datasketches.apache.org/docs/Community/) page.

## Release 2.0.0+
Starting with release *datasketches-memory-2.0.0*, this Memory component supports Java 8 through Java 13. Providing access to the four contiguous byte resources (mentioned above) in Java 8 only requires reflection. However, **Java 9 introduced the Java Platform Module System (JPMS) where access to these internal classes requires starting up the JVM with special JPMS arguments.** The actual JVM arguments required will depend on how the user intends to use the Memory API, the Java version used to run the user's application and whether the user's application is a JPMS application or not.
The goal of this component of the DataSketches library is to provide a high performance access API for accessing four different types of memory resources:

Also see the [usage examples](docs/usage-examples.md) for more information.
* On-heap Memory via primitive arrays
* On-heap Memory via ByteBuffers
* Off-heap (Direct) Memory via direct allocation
* Off-Heap Memory-Mapped files

## USE AS A LIBRARY (using jars from Maven Central)
In this environment, the user is using the Jars from Maven Central as a library dependency and not attempting to build the Memory component from the source code or run the Memory component tests.
Each of the four resource types is accessed using different API methods in the Memory component.

* If you are running Java 8, no extra JVM arguments are required.
* If you are running Java 11-13 and only using the **Heap** related API, no extra JVM arguments are required.
Note: *primitive* := *{byte, short, int, long, float, double}*

Otherwise, if you are running Java 11-13 and ...
### Contiguous bytes on the Java Heap constructed by these examples:

* If your application **is not a JPMS module** use the following table. Choose the columns that describe your use of the Memory API. If any of the columns contain a *Yes*, then the JVM argument in the first column of the row containing a *Yes* will be required. If you are not sure the extent of the Memory API being used, there is no harm in specifying all 4 JVM arguments. Note: do not embed any spaces in the full argument.
* **Heap via primitive arrays:**
* *Memory.wrap(primitive[])* (read only)
* *WritableMemory.allocate(int)*
* *WritableMemory.writableWrap(primitive[])*

| JVM Arguments for non-JPMS Applications | Direct ByteBuffer | Direct | MemoryMapped Files |
| :----------------------------------------------------: | :---------------: | :----: | :----------------: |
| --add-exports java.base/jdk.internal.misc= ALL-UNNAMED | | Yes | |
| --add-exports java.base/jdk.internal.ref= ALL-UNNAMED | | Yes | Yes |
| --add-opens java.base/java.nio= ALL-UNNAMED | | Yes | Yes |
| --add-opens java.base/sun.nio.ch= ALL-UNNAMED | | | Yes |
* **Heap via ByteBuffer**
* *Memory.wrap(ByteBuffer.wrap(byte[]))* (read only)
* *WritableMemory.writableWrap(ByteBuffer.allocate(int))*

* If your application **is a JPMS module** use the following table. Choose the columns that describe your use of the Memory API. If any of the columns contain a *Yes*, then the JVM argument in the first column of the row containing a *Yes* will be required. If you are not sure the extent of the Memory API being used, there is no harm in specifying all 4 JVM arguments. Note: do not embed any spaces in the full argument.
### Contiguous bytes off the Java Heap constructed by these examples:

| JVM Arguments for JPMS Applications | Direct ByteBuffer | Direct | MemoryMapped Files |
| :-----------------------------------------------------------------------: | :---------------: | :----: | :----------------: |
| --add-exports java.base/jdk.internal.misc= org.apache.datasketches.memory | | Yes | |
| --add-exports java.base/jdk.internal.ref= org.apache.datasketches.memory | | Yes | Yes |
| --add-opens java.base/java.nio= org.apache.datasketches.memory | | Yes | Yes |
| --add-opens java.base/sun.nio.ch= org.apache.datasketches.memory | Yes | | Yes |
* **Off-Heap, Direct via direct allocation:**
* *WritableMemory.allocateDirect(long)*

* **Off-Heap, Direct via (Direct) ByteBuffer**
* *WritableMemory.writableWrap(ByteBuffer.allocateDirect(int))*

## DEVELOPER USAGE
In this environment the developer needs to build the Memory component from source and run the Memory Component tests. There are two use cases. The first is for a *System Developer* that needs to build and test their own Jar from source for a specific Java version. The second use case is for a *Memory Component Developer and Contributor*.
* **Off-Heap, Memory-Mapped Files**
* *Memory.map(File)* (read only)
* *WritableMemory.writableMap(File)*

* System Developer
* Compile, test and create a Jar for a specific Java version
* use the provided script for this purpose
## Release 4.0.0 (inclusive) to 5.0.0 (exclusive)
Starting with release *datasketches-memory-4.0.0*, this Memory component supports Java 17 when compiling from source and should work with later Java versions at runtime. This component is not designed as a Java Module, so the Jar file should be part of the application classpath.

* Memory Component Developer / Contributor
* Compile & test the library from source code using:
* Eclipse (version)
* IntelliJ (version)
* Maven (version)
* Command-line or scripts
* The developer must have installed in their development system at least JDK versions 8 and 11.
* Unless building with the provided script, the developer must have a valid [Maven toolchain configuration](docs/maven-toolchains.md).


### Build Instructions
## Build Instructions
__NOTES:__

1) This component accesses resource files for testing. As a result, the directory elements
of the full absolute path of the target installation directory must qualify as Java identifiers.
In other words, the directory elements must not have any space characters (or non-Java identifier
characters) in any of the path elements. This is required by the Oracle Java Specification in
order to ensure location-independent access to resources:
[See Oracle Location-Independent Access to Resources](https://docs.oracle.com/javase/8/docs/technotes/guides/lang/resources.html)
* This component accesses resource files for testing.
As a result, the directory elements of the full absolute path of the target installation directory must qualify as Java identifiers.
In other words, the directory elements must not have any space characters (or non-Java identifier characters) in any of the path elements. This is required by the Oracle Java Specification in order to ensure location-independent access to resources:
[See Oracle Location-Independent Access to Resources](https://docs.oracle.com/javase/8/docs/technotes/guides/lang/resources.html)
* The compile command line must contain the JVM flag *--add-modules=jdk.incubator.foreign*.
* This component is not designed as a Java Module, so the Jar file should be part of the application classpath.

#### Dependencies
There are no run-time dependencies. See the pom.xml file for test dependencies.

#### Maven build instructions
The Maven build requires the following JDKs to compile:

- JDK8/Hotspot
- JDK11/Hotspot

Before building, first ensure that your local environment has been configured according to the [Maven Toolchains Configuration](docs/maven-toolchains.md).

There are three types of tests: normal unit tests, tests run by the strict profile and continuous integration(CI) tests.
The CI tests target the Multi-Release (MR) JAR and run the entire test suite using a specific version of Java. Running the CI test command also runs the default unit tests.
The Maven build requires JDK-17 to compile:

To run normal unit tests:

mvn clean test

To run the strict profile tests (only supported in Java 8):

mvn clean test -P strict

To run javadoc on this multi-module project, use:
To run javadoc:

mvn clean javadoc:javadoc -DskipTests=true

To build the multi-release JAR, use:

mvn clean package

To run the eclipse plugin on this multi-module project, use:

mvn clean eclipse:eclipse -DskipTests=true
Expand All @@ -149,57 +109,7 @@ This will create the following Jars:
* datasketches-memory-X.Y.Z-test-sources.jar The test source files
* datasketches-memory-X.Y.Z-javadoc.jar The compressed Javadocs.

#### Building for a specific java version

A build script named **package-single-release-jar.sh** has been provided to package a JAR for a specific java version. This is necessary in cases where a developer is unable to install all the required versions of the JDK that are required as part of the Maven build.

The build script performs the following steps:

1. Sets up staging directories under **target/** for the package files
2. Uses git commands to gather information about the current Git commit and branch
3. Compiles java source tree
4. Packages a JAR containing compiled sources together with the Manifest, License and Notice files
5. Checks and tests the assembled JAR by using the API to access four different resource types

The build script is located in the **tools/scripts/** directory and requires the following arguments:

* JDK Home Directory - The first argument is the absolute path of JDK home directory e.g. $JAVA_HOME
* Git Version Tag - The second argument is the Git Version Tag for this deployment e.g. 1.0.0-SNAPSHOT, 1.0.0-RC1, 1.0.0 etc.
* Project Directory - The third argument is the absolute path of project.basedir e.g. /src/apache-datasketches-memory

For example, if the project base directory is `/src/datasketches-memory`;

To run the script for a release version:

./tools/scripts/package-single-release-jar.sh $JAVA_HOME 2.1.0 /src/datasketches-memory

To run the script for a snapshot version:

./tools/scripts/package-single-release-jar.sh $JAVA_HOME 2.2.0-SNAPSHOT /src/datasketches-memory

To run the script for an RC version:

./tools/scripts/package-single-release-jar.sh $JAVA_HOME 2.1.0-RC1 /src/datasketches-memory

Note that the script does **not** use the _Git Version Tag_ to adjust the working copy to a remote tag - it is expected that the user has a pristine copy of the desired branch/tag available **before** using the script.

---

### Further documentation for contributors

For more information on the project configuration, the following topics are discussed in more
detail:

* [Maven Configuration](docs/maven.md)
* [Maven Toolchains Configuration](docs/maven-toolchains.md)
* [Multi-Release Jar](docs/multi-release-jar.md)
* [Java Platform Module System](docs/module-system.md)
* [Usage examples](docs/usage-examples.md)

In order to build and contribute to this project, please read the relevant IDE documentation:

- [Eclipse IDE Setup](docs/eclipse.md)
- [IntelliJ IDE Setup](docs/intellij.md)
### Deployment to Nexus

For releasing to AppNexus, please use the `sign-deploy-jar.sh` script in the scripts directory.
For releasing to Nexus, please use the `sign-deploy-jar.sh` script in the scripts directory.
See the documentation within the script for usage instructions.
Loading