Test: arm64: Kconfig: Enable NODES_SPAN_OTHER_NODES config for NUMA

[adamliyi]

Note: This patch fixes incorrect memory size in OS, when SRAT memory map is overlapped. However, no memory overlap observed with current FW. We only calculate memory size from ACPI/SRAT, and check with memory size reported by OS.

Test steps:

1. In BIOS enable Slave 32bit

Memory Configuration ->  Enable Slave 32bit memory region       [Enabled]

2. Check kernel configure 'NODES_SPAN_OTHER_NODE' is enabled

3. dmesg | grep 'SRAT', got numa memory map. e.g:

[    0.000000] ACPI: SRAT: Node 2 PXM 2 [mem 0x88300000-0x883fffff]
[    0.000000] ACPI: SRAT: Node 2 PXM 2 [mem 0x90000000-0xbfffffff]
[    0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x80000000000-0x8007fffffff]
[    0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x800c0000000-0x83fffffffff]
[    0.000000] ACPI: SRAT: Node 3 PXM 3 [mem 0xc2000000-0xffffffff]
[    0.000000] ACPI: SRAT: Node 1 PXM 1 [mem 0x400000000000-0x4000bfffffff]
[    0.000000] ACPI: SRAT: Node 1 PXM 1 [mem 0x400100000000-0x403fffffffff]

4. Calculate memory size on each node based on above memory map.
   e.g, for Node 0, the memory size is calculated as:

1) 0x8007fffffff - 0x80000000000 = 0x7fffffff
2) 0x83fffffffff - 0x800c0000000 = 0x3F 3FFF FFFF

Total size = 0x7fff ffff + 0x3F 3FFF FFFF = 0x3F BFFF FFFE

5. cat /sys/devices/system/node/node0/meminfo
   Node 0 MemTotal: ...

6. Compare memory size between step 4 and step 5. They should be similar (not exactly the same). We can just print the memory size to log.

[adamliyi]

Confirmed 'CONFIG_NODES_SPAN_OTHER_NODES' is a nice-to-have patch. FW will not report overlapped memory region.

Tested on MJ with 1.05 FW.

1. In BIOS enable Slave 32bit memory region
   Memory Configuration -> Enable Slave 32bit memory region [Enabled]

2. disable CONFIG_NODES_SPAN_OTHER_NODES in kernel

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 1eacab57a7b7..11757e1f6ea0 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -933,7 +933,7 @@ config HOLES_IN_ZONE
# Even though a pfn is valid and between a node's start and end pfns,
# it may not reside on that node.
config NODES_SPAN_OTHER_NODES
-       def_bool y
+       def_bool n
        depends on ACPI_NUMA

3. rebuild 5.4.93 kernel

4. SRAT as bellow:

[    0.000000] ACPI: SRAT 0x00000000B0020000 000D18 (v03 Ampere Altra    00000000 AMP. 01000013)
[    0.000000] ACPI: SRAT: Node 2 PXM 2 [mem 0x88300000-0x883fffff]
[    0.000000] ACPI: SRAT: Node 2 PXM 2 [mem 0x90000000-0xbfffffff]
[    0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x80000000000-0x8007fffffff]
[    0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x800c0000000-0x83fffffffff]
[    0.000000] ACPI: SRAT: Node 3 PXM 3 [mem 0xc2000000-0xffffffff]
[    0.000000] ACPI: SRAT: Node 1 PXM 1 [mem 0x400000000000-0x4000bfffffff]
[    0.000000] ACPI: SRAT: Node 1 PXM 1 [mem 0x400100000000-0x403fffffffff]

5. numactl output as bellow:

numactl -H
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79
node 0 size: 260789 MB
node 0 free: 257240 MB
node 1 cpus: 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159
node 1 size: 260746 MB
node 1 free: 259222 MB
node 2 cpus:
node 2 size: 652 MB
node 2 free: 577 MB
node 3 cpus:
node 3 size: 926 MB
node 3 free: 854 MB
node distances:
node   0   1   2   3
  0:  10  20  14  24
  1:  20  10  24  14
  2:  14  24  10  24
  3:  24  14  24  10

5. Compare SRAM with node memory size reported by numactl, there isn't much difference:

CONFIG_NODES_SPAN_OTHER_NODES is not set
Numa Node 0: ACPI SRAT size 261119 MB, Kernel Reported 260789 MB, Delta 330 MB
Numa Node 1: ACPI SRAT size 261119 MB, Kernel Reported 260746 MB, Delta 373 MB
Numa Node 2: ACPI SRAT size 768 MB, Kernel Reported 652 MB, Delta 116 MB
Numa Node 3: ACPI SRAT size 991 MB, Kernel Reported 926 MB, Delta 65 MB

[adamliyi]

close. done.