Non-break space character in PDF causing load problem with text Brat sentence view

[GiantEnemyCrab]

Describe the bug and To Reproduce
Wen PDF file contains non-breaking space (U+00A0), it loads the document fine in PDF view, but when switch to Brat-sentence text view, it has an issue

Expected behavior
Documents load with non-breaking space in Brat sentence text view if it works with PDF view

Screenshots

Please complete the following information:

• Version and build ID: version 35
• OS: Windows, Mac, Linux
• Browser: Edge, Chrome

[reckart]

That should actually have been fixed in recent versions. If you still have a project lying around from an older version, you can try this to fix your project:

Open the CAS Doctor in the project settings and then follow the steps below (leaving all the repair options that are enabled by default active as well)

1. enable TrimAnnotationsRepair & run repair
2. disable TrimAnnotationsRepair & enable RemoveZeroSizeTokensAndSentencesRepair & run repair
3. disable RemoveZeroSizeTokensAndSentencesRepair & enable RemoveDanglingRelationsRepair & run repair

Hopefully, no errors should be left now and the document should render.

[reckart]

Probably duplicate of #5035

[GiantEnemyCrab]

Thanks for the guidance on the CAS Doctor repair feature, but this PDF seems more tricky, and the issue couldn't be resolved via the CAS Doctor approach.

I wish I could share the exact document here but I can't due to the file containing sensitive data.
Hello world. \xa0Hello World., where \xa0 is the non-breaking space.

It still loads fine in PDF viewer mode, but not in text viewing mode of the PDF file. What I tried was Sentence (BRAT) view.

I use pymupdf and the work-around was to delete \xa0 non-break space character via pymupdf's "applyRedaction()" method to delete that character. Then it worked fine in INCEpTION whether PDF view or text view.

I am not sure of the exact condition of the error. It looks like there are other \xa0 characters that are not causing any issue, too.

[reckart]

Does CAS Doctor detect an issue or does it claim everything to be in order?

[GiantEnemyCrab]

For INITIAL andCURATION, nothing to report for that document, but for annotation I am seeing UnreachableAnnotationsCheck saying Type webanno.custom.LayerName ~~~~~ has X unreachable instances where X is an integer.

[reckart]

The unreachable annotations are normal. This happens when a user deletes annotations. The stick around for a while until the document is opened for annotation again at which point they are garbage-collected. I think this is only an info-level message.

[reckart]

I have used pdfbox to create a little toy PDF that contains a 0xA0 char - but I don't seem to be having problems with that one either in the PDF or in the Brat (sentence-based) view.

nbsp.pdf

[reckart]

Code I used with pdfbox v3:

import org.apache.pdfbox.pdmodel.PDDocument;
import org.apache.pdfbox.pdmodel.PDPage;
import org.apache.pdfbox.pdmodel.PDPageContentStream;
import org.apache.pdfbox.pdmodel.font.PDType1Font;
import org.apache.pdfbox.pdmodel.font.Standard14Fonts.FontName;

public class PDFWithNBSP
{
    public static void main(String[] args) throws Exception
    {
        try (var document = new PDDocument()) {
            var page = new PDPage();
            document.addPage(page);

            var font = new PDType1Font(FontName.HELVETICA_BOLD);

            try (var contentStream = new PDPageContentStream(document, page)) {
                contentStream.setFont(font, 12);
                contentStream.beginText();
                contentStream.setLeading(14.5f);
                contentStream.newLineAtOffset(100, 700);
                var text = "Hello\u00A0World";
                contentStream.showText(text);
                contentStream.endText();
            }

            // Save the document
            document.save("output.pdf");
            System.out.println("PDF created successfully: output.pdf");
        }
    }
}

[GiantEnemyCrab]

redacted.pdf

I uploaded file here that can reproduce the issue.
My discovery was that in fact, most of the case, non-breaking spaces happened to be ok. Sorry about misleading earlier, and I investigated more cases since then. But this one, I can't figure out what's going on.

The step to reproduce the error with this redacted.pdf:

• import the document with PDF file format
• Load the document in annotation view
• Switch to BRAT (sentence-oriented)
• It should result in error like the screen shot:

[reckart]

Thanks for helping to track this down. There was some duplicate code and the recent fix wrt. nbsp issues only fixed one copy, not the other. I have removed the second copy now, using only the one that works. Looks like that fixes the issue.

[GiantEnemyCrab]

Thank you for applying the fix so quickly!