Add preprocessing in summarize function to remove newlines in the middle of sentences

[diegospd]

Description

The summarize function depends on the text having at least 10 sentences as measured by clean_text_by_sentences. If the text is shorter than that then the summarization fails in an undocumented way. Moreover clean_text_by_sentences cannot handle properly a text with new lines
at the middle of a sentence. I suggest a preprocessing step to purge those.

I'm currently using this to workaround this bug

import re
text = re.sub(r'\n|\r|\t', ' ', text)
text = re.sub(r'\s+', ' ', text)

Versions

• Python 3.6.2 (default, Jul 17 2017, 16:44:45)
  [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)]
• NumPy 1.13.0
• SciPy 0.19.1
• gensim 2.3.0
• FAST_VERSION 0