Code

Add PDF parser for indexing#109

Code

Merged

tanaysoni merged 32 commits intomasterdeepset-ai/haystack:masterfrom pdfdeepset-ai/haystack:pdfCopy head branch name to clipboardJun 8, 2020

+424-65

Commits

Commits on Jun 2, 2020

• Add PDF parser for indexing

  tanaysoni

  committed
  f41accf

• Add param to remove headers in PDF

  tanaysoni

  committed
  adb5458

• Add xpdf conversion and header/footer cleaning

  tanaysoni

  committed
  f81f23b

• Add a module for file converters

  tanaysoni

  committed
  239f3d0

• Remove unused requirements

  tanaysoni

  committed
  ae08859

• Simplify pdftotext conversion pipeline

  tanaysoni

  committed
  fbe7115

• Add tests for PDF conversion

  tanaysoni

  committed
  0198747

• Add sample pdf file for tests

  tanaysoni

  committed
  b00512e

• Remove xpdf install

  tanaysoni

  committed
  5035a27

• Update tests

  tanaysoni

  committed
  86948d9

• Add PyMuPDF in requirements

  tanaysoni

  committed
  465058c

• Add langdetect in requirements

  tanaysoni

  committed
  6ce49d9

• Add xpdf install in CI

  tanaysoni

  committed
  8523de3

Commits on Jun 3, 2020

• Add a pytest fixture for xpdf

  tanaysoni

  committed
  5c57657

• Remove xpd install from CI

  tanaysoni

  committed
  e30281a

• Raise privileges for CI

  tanaysoni

  committed
  aac72a7

• Remove type-cast to Document schema

  tanaysoni

  committed
  145aada

• Add sudo for pdftotext fixture

  tanaysoni

  committed
  d8680d3

Commits on Jun 4, 2020

• Make file extension case insensitive

  tanaysoni

  committed
  7a347b1

• Update xpdf download instructions

  tanaysoni

  committed
  6dc120f

• Simplify glob pattern for reading input files

  tanaysoni

  committed
  d45e72b

Commits on Jun 5, 2020

• Add check if pdftotext is already installed

  tanaysoni

  committed
  b255372

• Add test for header/footer removal

  tanaysoni

  committed
  b3e9bef

• Renamve header removal params

  tanaysoni

  committed
  7d4c6cf

• Add check if string is None for header/footer removal

  tanaysoni

  committed
  83bd686

• fix header/footer heuristic

  tholor

  committed
  23584cf

• merge latest changes

  tholor

  committed
  b7ed431

• remove caplog from test

  tholor

  committed
  6624114

• Add docstrings and code formatting

  tanaysoni

  committed
  9ce2a5c

• Rename indexing utils

  tanaysoni

  committed
  4372f41

Commits on Jun 8, 2020

• Add PDF in README

  tanaysoni

  committed
  3f765ea

• Rename utils func in tests

  tanaysoni

  committed
  fd1a203