excludes doesn't appear to work with subdirectories/paths

[Kevin-McKee]

For example if I have a directory /mypath/folder with subdirectories

/mypath/folder
├── folderA
│   ├── subfolderA
│   ├── subfolderB
│   └── subfolderC
├── folderB
│   ├── subfolderA
│   ├── subfolderB
│   └── subfolderC
└── folderC
├── subfolderA
├── subfolderB
└── subfolderC

I would like to be able to start crawling at /mypath/folder and crawl everything except folderB/subfolderB for example.

With the way that I believe "excludes" works currently, I could put "folderB" which would exclude all of "folderB", or I could put "subfolderB" which would exclude folderA/subfolderB, folderB/subfolderB and folderC/subfolderB

I would like to be able to put "excludes": ["folderB/subfolderB"] or even a wildcard like "excludes": ["folderB/subfolder*"]

[dadoonet]

I feel like excludes and includes actually works only for now on folder name, not on the relative path from root folder.

Which means that folderB/subfolderB is never evaluated but subfolderB is.

I believe I need to rewrite this part. I started to POC around this today and will see how it goes but this could be seen as a breaking change though.

That said, I'm not sure a lot of users are using includes or excludes for dir names so it might be OK to break this behavior.
Stay tuned.

[Kevin-McKee]

OK thanks. It sure would be useful for me as I have some specific indexes that I’ve built off specific folders, but I’d like to build an “other” index which encapsulates everything else around those directories but excludes the ones I’ve made specific indexes out of. If you need some testing done I’d be happy to try it out for you. Thanks,

[dadoonet]

@a344254 I removed some personal information from the last answer you sent (email signature).

I created this branch https://github.com/dadoonet/fscrawler/tree/fix/553-exclude-dirs which supports the new feature.
I did some tests and it seems to be ok but I'd love if you can try it out.

Compile the project with

mvn package -DskipTests

And get the zip distribution file.

Documentation about this change is:

• https://fscrawler.readthedocs.io/en/fix-553-exclude-dirs/installation.html#upgrade-to-2-5
• https://fscrawler.readthedocs.io/en/fix-553-exclude-dirs/admin/fs/local-fs.html#includes-excludes

Let me know! :)

[Kevin-McKee]

I did some testing this morning and it appears to work. Thanks!
P.S. - the download link on http://fscrawler.readthedocs.io/en/fscrawler-2.5/installation.html doesn't work - it's pointing at https://repo1.maven.org/maven2/fr/pilato/elasticsearch/crawler/fscrawler/2.5/fscrawler-2.5.zip but gives a 404 Not Found error.

[dadoonet]

Thanks @a344254. I just forgot to close and release the sonatype repo :)
It's done now. It can take some time to be synch'ed to Maven central though. Let's see that in some hours.