Infinite Indexing Problem - indexing wrong mounts in Linux which will take infinite time

[Leo-Adlakha]

Description
I am a beginner with opengrok. I was following all the instructions given on https://github.com/oracle/opengrok/wiki/How-to-setup-OpenGrok to have my opengrok instance running. Earlier, when I was running my instance, it worked fine and as per expectations. But, suddenly when I try to re-index my sources it runs infinitely as it tries to scan other mounts than specified in the -s flag, which is supposed to take infinitely long time. I am not sure what is wrong here. When I try to stop the re-indexing by Ctrl+C in terminal, it doesn't even exit and when I try to kill my terminal instance, the command goes into a state.

Opengrok: 1.7.33
JDK: 11.0.8
OS: RHEL 7
Tomcat: 10.0.22

To Reproduce
Steps to reproduce the behavior: Followed the steps given in the documentation, nothing special. Ran intially smoothly, but doesn't index now.

Opengrok Logs

2022-07-15 17:13:45.723+0530 FINE t1 Executor.registerErrorHandler: Installing default uncaught exception handler
2022-07-15 17:13:45.884+0530 INFO t1 Indexer.parseOptions: Indexer options: [-c, /path/to/ctags, -s, /path/to/opengrok/src, -d, /path/to/opengrok/data, -H, -P, -S, -G, -W, /path/to/opengrok/etc/configuration.xml, -U, http://localhost:8080/source]
2022-07-15 17:13:45.927+0530 FINE t1 HostUtil.isReachable: URI http://localhost:8080/source is reachable via localhost/127.0.0.1
2022-07-15 17:13:46.194+0530 FINER t1 ReflectionHelper$2.run: Unable to load class org.osgi.framework.BundleReference using the current class loader.

///////////////////////////////////////////////////////////////////////
<Unwanted traversal of various files not mentioned anywhere like my .bashrc, and other paths not intentional>
///////////////////////////////////////////////////////////////////////

2022-07-15 17:13:46.939+0530 FINER t1 HistoryGuru.addRepositories: Discovered repositories: []
2022-07-15 17:13:46.939+0530 INFO t1 Statistics.logIt: Done scanning for repositories, found 0 repositories (took 17 ms)
2022-07-15 17:13:46.940+0530 INFO t1 Indexer.prepareIndexer: Generating history cache for all repositories ...
2022-07-15 17:13:46.940+0530 WARNING t1 HistoryGuru.createCacheReal: History cache is enabled however the list of repositories is empty. Either specify the repositories in configuration or let the indexer scan them.
2022-07-15 17:13:46.940+0530 INFO t1 Indexer.prepareIndexer: Done generating history cache
2022-07-15 17:13:46.943+0530 INFO t1 Indexer.doIndexerExecution: Starting indexing
2022-07-15 17:13:47.004+0530 INFO t1 Indexer.doIndexerExecution: Waiting for the executors to finish

The log stops here and now it runs infinitely. Obviously because its is trying to scan everythig in linux server.

strace.out

11480 open("/sys/<unwanted-path>", O_RDONLY <unfinished ...>
11474 <... gettid resumed>)             = 11474
11502 <... mmap resumed>)               = 0x7ff0cc000000
11496 <... gettid resumed>)             = 11496
11486 gettid( <unfinished ...>
11481 <... gettid resumed>)             = 11481
11480 <... open resumed>)               = 78</sys/<unwanted-path>>
11502 mprotect(0x7ff0cc000000, 135168, PROT_READ|PROT_WRITE <unfinished ...>
11496 gettid( <unfinished ...>
11486 <... gettid resumed>)             = 11486
11481 gettid( <unfinished ...>
11480 fstat(78</sys/<unwanted-path>>,  <unfinished ...>
11474 gettid( <unfinished ...>
11502 <... mprotect resumed>)           = 0
11496 <... gettid resumed>)             = 11496
11486 gettid( <unfinished ...>
11481 <... gettid resumed>)             = 11481
11480 <... fstat resumed>{st_mode=S_IFREG|0444, st_size=0, ...}) = 0
11474 <... gettid resumed>)             = 11474
11502 sched_getaffinity(11502, 32,  <unfinished ...>
11486 <... gettid resumed>)             = 11486
11481 gettid( <unfinished ...>
11480 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
11502 <... sched_getaffinity resumed>0x7ff0cc0008c0) = -1 EINVAL (Invalid argument)
11496 gettid( <unfinished ...>
11481 <... gettid resumed>)             = 11481
11474 gettid( <unfinished ...>
11502 sched_getaffinity(11502, 64,  <unfinished ...>
11496 <... gettid resumed>)             = 11496
11486 gettid( <unfinished ...>
11481 gettid( <unfinished ...>
11480 <... mmap resumed>)               = 0x7ff17b7d7000
11474 <... gettid resumed>)             = 11474
11502 <... sched_getaffinity resumed>[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55]) = 64
11496 gettid( <unfinished ...>
11486 <... gettid resumed>)             = 11486
11481 <... gettid resumed>)             = 11481
11480 read(78</sys/<unwanted-path>>,  <unfinished ...>
11502 sched_getaffinity(11502, 32,  <unfinished ...>
11496 <... gettid resumed>)             = 11496
11481 gettid( <unfinished ...>
11474 gettid( <unfinished ...>
11502 <... sched_getaffinity resumed>0x7ff0cc000910) = -1 EINVAL (Invalid argument)
11496 gettid( <unfinished ...>
11481 <... gettid resumed>)             = 11481
11480 <... read resumed>"13893746688\n", 4096) = 12
11474 <... gettid resumed>)             = 11474
11502 sched_getaffinity(11502, 64,  <unfinished ...>
11496 <... gettid resumed>)             = 11496
11481 gettid( <unfinished ...>
11480 close(78</sys/<unwanted-path>> <unfinished ...>
11502 <... sched_getaffinity resumed>[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55]) = 64
11481 <... gettid resumed>)             = 11481
11496 gettid( <unfinished ...>

Above shows I think that it tries to scan unintentional mounts.

Command Run

java \
    -Djava.util.logging.config.file=/path/to/opengrok/etc/logging.properties \
    -jar /path/to/opengrok/dist/lib/opengrok.jar \
    -c /path/to/bin/ctags \
    -s /path/to/opengrok/src -d /path/to/opengrok/data -H -P -S -G \
    -W /path/to/opengrok/etc/configuration.xml -U http://localhost:8080/source

Please help to resolve this issue if its a bug. Thanks!

[vladak]

The main question is how does the source root (/path/to/opengrok/src) correspond to the rest of the directory tree (any funky mounts ?) and what is inside the source root directory. By default, if there is a symlink directly underneath the source root, the indexer will follow it. Other symlinks will be ignored.

You can try running find -type d /path/to/opengrok/src/ with and without the -L/-follow option to observe the directory traversal. This will not be the same as the traversal performed by the indexer (because of the said default symlink restrictions) however might provide some clues.

Also, it might help to increase verbosity of the logging (possibly complemented with the --progress indexer option) to see how does the directory traversal reach the unwanted entries.