Preliminary support for ARM #25318

drewr · 2017-06-20T18:05:48Z

These commits produce a working Elasticsearch binary and get the full test suite to pass. It has not been tested in any production capacity. It depends on elastic/jna-build#1, which adds aarch64 support to our custom JNA jar.

Development system is currently an ARMv8, 64-core, Huawei Hi1616 Type 2a2 from our friends at Packet.

% bin/elasticsearch
[...]
[2017-06-20T14:20:35,064][INFO ][o.e.n.Node   ] version[6.0.0-alpha3-SNAPSHOT], pid[52835], build[64aebe7/2017-06-19T19:06:06.582Z], OS[Linux/4.9.0/aarch64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_131/25.131-b11]
[...]
% curl -s localhost:9200/_nodes/os | jq ".nodes[].os"
{
  "refresh_interval_in_millis": 1000,
  "name": "Linux",
  "arch": "aarch64",
  "version": "4.9.0",
  "available_processors": 64,
  "allocated_processors": 64
}

jasontedor

I left some comments.

jasontedor · 2017-06-20T18:12:39Z

core/licenses/jna-4.4.0.jar.sha1

I'm confused by this change? How is the SHA changing without the version? And I think we shouldn't change the version unless upstream has incremented the version and we are incrementing with them.

The SHA changes because there are more bits (the aarch64-related files). The version doesn't change because it's apparently tied to the upstream JNA version, which is still 4.4.0 AFAICT. If we can bump the version that would definitely be preferred.

Right, I understand that, but we can not re-release 4.4.0 so we can not change the SHA without changing the version as is done here. I would prefer that we not bump the version until upstream bumps, and they are still on 4.4.0. We can consider releasing a 4.4.0-1 or some other moniker to differentiate it from 4.4.0 if upstream does not bump soon.

I would actually prefer the extension to permanently differentiate (like an OS package), but this kind of change is so infrequent maybe it's not worth it. Maybe I should take it up on elastic/jna-build#1.

Right now we differentiate based on the group ID, I don't think we need more than that as this need release out of sync should be infrequent as you say indeed.

like an OS package

I was thinking along those lines too....

I've implemented this as part of elastic/jna-build#1 in elastic/jna-build@b6661ab. Please review that particular change over there. I'll bump the version and SHA here.

jasontedor · 2017-06-20T18:13:24Z

core/src/main/java/org/elasticsearch/bootstrap/SystemCallFilter.java

@@ -243,6 +243,8 @@ static SockFilter BPF_JUMP(int code, int k, int jt, int jf) {
        Map<String,Arch> m = new HashMap<>();
        m.put("amd64", new Arch(0xC000003E, 0x3FFFFFFF, 57, 58, 59, 322, 317));
        m.put("i386",  new Arch(0x40000003, 0xFFFFFFFF, 2, 190, 11, 358, 354));
+        // 64-bit only!
+        m.put("aarch64",  new Arch(0xC00000B7, 0xFFFFFFFF, 1079, 1071, 221, 281, 277));


These make sense. I haven't checked the system call numbers but I trust that they are right. I will double check before final review.

I used some code from @rmuir to double-check.

Yeah, I checked https://github.com/torvalds/linux/blob/9705596d08ac87c18aee32cc97f2783b7d14624e/include/uapi/asm-generic/unistd.h and they all look right to me.

Maybe change the comment to "arm is 64 bit only" so it doesn't get confused with the line above it supporting 32 bit intel.

jasontedor · 2017-06-20T18:14:04Z

client/rest-high-level/src/test/java/org/elasticsearch/client/SearchIT.java

@@ -269,7 +269,7 @@ public void testSearchWithMatrixStats() throws IOException {
        assertEquals(5, matrixStats.getFieldCount("num2"));
        assertEquals(29d, matrixStats.getMean("num2"), 0d);
        assertEquals(330d, matrixStats.getVariance("num2"), 0d);
-        assertEquals(-0.13568039346585542, matrixStats.getSkewness("num2"), 0d);
+        assertEquals(-0.13568039346585542, matrixStats.getSkewness("num2"), 1.0e-16);


What's happening here? If this is correct now for x64, and needs to be changed for ARM (why?) how is this not going to turn around and break on x64?

For whatever reason only getSkewness() has less precision on aarch64:

amd64: -0.13568039346585542 aarch64: -0.1356803934658554

It still passes on Intel, because the delta is still valid. It makes the assertion slightly more permissive, not less. I scratched my head over this one as well, curious why only this assertion needed a delta. I asked @jpountz about it. 😄

Yeah, it's puzzling!

And yes, I misread the assertion at first but the puzzle why a change is needed only here remains! 😄

The matrixStats PR is here - #18300 - with code from @nknize et al.

This is puzzling indeed. It could be a bug, but it could also be due to differences in the JVM/hardware that causes the documents to be stored in a different order in the index. Then at search time, aggs collect documents in a different order as well and we get different rounding errors?

vielmetti · 2017-06-20T20:21:08Z

The build check failed with this error message:

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':core:dependencyLicenses'.

BUILD FAILED

Total time: 2 mins 35.343 secs
> SHA has changed! Expected d85e9ceee4d97e738524a231687ff28446376c97 for jna-4.4.0.jar but got 6edc9b4514969d768039acf43f04210b15658cd7. 
  This usually indicates a corrupt dependency cache or artifacts changed upstream.
  Either wipe your cache, fix the upstream artifact, or delete /var/lib/jenkins/workspace/elastic+elasticsearch+pull-request/core/licenses/jna-4.4.0.jar.sha1 and run updateShas

drewr · 2017-06-20T20:23:51Z

This is because you're pulling the stock jna-4.4.0.jar @vielmetti. Once I publish the nascent 4.4.0-1 and update this PR you shouldn't have to do anything special. Stay tuned.

jasontedor

LGTM.

* master: (129 commits) Add doc note regarding explicit publish host Fix typo in name of test Add additional test for sequence-number recovery WrapperQueryBuilder should also rewrite the parsed query. Remove dead code and stale Javadoc Update defaults in documentation (#25483) [DOCS] Add docs-dir to Painless (#25482) Add concurrent deprecation logger test [DOCS] Update shared attributes for Elasticsearch (#25479) Use LRU set to reduce repeat deprecation messages Add NioTransport threads to thread name checks (#25477) Add shortcut for AbstractQueryBuilder.parseInnerQueryBuilder to QueryShardContext Prevent channel enqueue after selector close (#25478) Fix Java 9 compilation issue Remove unregistered `transport.netty.*` settings (#25476) Handle ping correctly in NioTransport (#25462) Tests: Remove platform specific assertion in NioSocketChannelTests Remove QueryParseContext from parsing QueryBuilders (#25448) Promote replica on the highest version node (#25277) test: added not null assertion ...

This commit adds preliminary support for 64-bit ARM architectures. Relates #25318

jasontedor · 2017-06-30T18:25:02Z

Thanks @drewr.

lefuturiste · 2019-07-31T20:52:49Z

Do you have any guides to install elasticsearch on a aarch64 ubuntu server or a docker image for aarch64 ?

drewr added 3 commits June 19, 2017 15:27

[aarch64] Update JNA jar sha1

3785fcd

[aarch64] Add seccomp syscall filter spec

de5794c

[aarch64] Add delta for float comparison

64aebe7

jasontedor reviewed Jun 20, 2017

View reviewed changes

jasontedor self-requested a review June 20, 2017 18:22

vielmetti mentioned this pull request Jun 20, 2017

WOA 9 editorial calendar (2017-W25) WorksOnArm/worksonarm-news#26

Closed

jasontedor mentioned this pull request Jun 20, 2017

Enable aarch64 support elastic/jna-build#1

Merged

[aarch64] jna-4.4.0-1

075c1b8

jasontedor approved these changes Jun 30, 2017

View reviewed changes

jasontedor merged commit 6deb18c into master Jun 30, 2017

jasontedor pushed a commit that referenced this pull request Jun 30, 2017

Preliminary support for ARM

5cd3115

This commit adds preliminary support for 64-bit ARM architectures. Relates #25318

jasontedor added :Delivery/Packaging RPM and deb packaging, tar and zip archives, shell and batch scripts v5.6.0 v6.0.0 review >non-issue labels Jun 30, 2017

jasontedor deleted the aarch64 branch June 30, 2017 18:24

clintongormley added v6.0.0-beta1 and removed v6.0.0 labels Jul 25, 2017

mark-vieira added the Team:Delivery Meta label for Delivery team label Nov 11, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Preliminary support for ARM #25318

Preliminary support for ARM #25318

drewr commented Jun 20, 2017

jasontedor left a comment

jasontedor Jun 20, 2017

drewr Jun 20, 2017

jasontedor Jun 20, 2017

drewr Jun 20, 2017

jasontedor Jun 20, 2017

nik9000 Jun 20, 2017

drewr Jun 20, 2017 •

edited

Loading

jasontedor Jun 20, 2017

drewr Jun 20, 2017

jasontedor Jun 20, 2017

nik9000 Jun 20, 2017

jasontedor Jun 20, 2017

drewr Jun 20, 2017

jasontedor Jun 20, 2017

jasontedor Jun 20, 2017

vielmetti Jun 20, 2017

jpountz Jul 4, 2017

vielmetti commented Jun 20, 2017

drewr commented Jun 20, 2017 •

edited

Loading

jasontedor left a comment

jasontedor commented Jun 30, 2017

lefuturiste commented Jul 31, 2019

Preliminary support for ARM #25318

Preliminary support for ARM #25318

Conversation

drewr commented Jun 20, 2017

jasontedor left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drewr Jun 20, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vielmetti commented Jun 20, 2017

drewr commented Jun 20, 2017 • edited Loading

jasontedor left a comment

Choose a reason for hiding this comment

jasontedor commented Jun 30, 2017

lefuturiste commented Jul 31, 2019

drewr Jun 20, 2017 •

edited

Loading

drewr commented Jun 20, 2017 •

edited

Loading