-
|
I want to crawl all websites with different domains, limited regions (such as Thailand), and unlimited depth. In short, I want to crawl all websites in the specified country, but skip subdirectories of the same website category It is mainly used to collect basic information of all websites in specific countries |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
|
@hktalent you can use crawl scope option for this, for example: You can adopt the keyword, regex as you want. |
Beta Was this translation helpful? Give feedback.
-
|
It can't be fully satisfied. You know, there are many IP designed url in the webpage similar: http://33.3.79.1:9080/xxx.login |
Beta Was this translation helpful? Give feedback.
@hktalent you can use crawl scope option for this, for example:
You can adopt the keyword, regex as you want.