Improved performance of URL modification generation function #45
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Uma maneira de melhorar a performance dessa função seria mudar o loop interno que itera sobre alteration_words para criar as URLs para um loop externo. Dessa forma, ao invés de iterar sobre alteration_words a cada iteração do loop principal, você poderia iterar sobre alteration_words apenas uma vez e gerar todas as URLs de uma vez. Isso deve ser mais rápido do que fazer isso a cada iteração do loop principal. Outra sugestão seria mudar o loop principal para iterar sobre uma lista de tuplas, onde cada tupla é formada pelo subdomínio atual e os alteration_words a serem adicionados nele. Dessa forma, você poderia gerar todas as URLs com os subdomínios modificados de uma vez, em vez de ter que ler e escrever cada linha individualmente no arquivo de saída. Além disso, você poderia considerar usar os módulos concurrent.futures ou multiprocessing do Python para paralelizar essa operação e melhorar ainda mais a performance. Isso pode ser especialmente útil se você tiver um grande número de subdomínios para processar.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
To improve the performance of this function, one way would be to change the inner loop that iterates over alteration_words to create the URLs to an outer loop. This way, instead of iterating over alteration_words on each iteration of the main loop, you could iterate over alteration_words just once and generate all the URLs at once. This should be faster than doing it on each iteration of the main loop.
Another suggestion would be to change the main loop to iterate over a list of tuples, where each tuple is formed by the current subdomain and the alteration_words to be added to it. This way, you could generate all the URLs with the modified subdomains at once, instead of having to read and write each line individually to the output file.