WTPDF / PDF/UA-2 Examples by the LaTeX Project #72
Replies: 10 comments 32 replies
-
|
Awesome work! But I did note 2 files with errors and a few other issues:
Just for discussion: several files have private PTEX entries for XObjects, etc. such as PTEX.FileName and PTEX.InfoDict which can include author, filename, etc as per the pdfTEX documentation (https://texdoc.org/serve/pdftex-a.pdf/0). Since PDF/A files are intended for long-term preservation, this has the potential to cause issues for FOIA and similar requests since the presence of private data might slip past various redaction workflows. A modern equivalent is to use an XMP Metadata stream instead of 2nd class custom PDF keys which makes this more discoverable. |
Beta Was this translation helpful? Give feedback.
-
|
@petervwyatt Thanks for the report. I will add an appearance to PHY-exam.pdf. But while testing that I got two curious complains from arlington (I used the lastest verapdf version) from the attached pdf:
|
Beta Was this translation helpful? Give feedback.
-
|
@davidcarlisle one question: does LaTeX not warn for 2401.09965v1-tagged.pdf that it needs one more run or was this simply overlooked? |
Beta Was this translation helpful? Give feedback.
-
|
@FrankMittelbach yes it did warn but got lost in all the tagging debug logging, so user error, me :( If you see the build script in the sources it now grep's the log and re-runs as needed after an update this morning so that shouldn't happen in future. |
Beta Was this translation helpful? Give feedback.
-
|
@u-fischer I can't speak to veraPDF's Arlington implementation (I'm not 100% sure which version they used and whether they augmented the rules, or how they determine what each object is) but DA messages are most likely related to PDF Errata #323 as this is not well specified. |
Beta Was this translation helpful? Give feedback.
-
|
PHY-exam.pdf has been updated |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
Speaking as an accessibility professional ( am no expert in LaTeX), the dependence on VeraPDF is not wise. While it seems to be able to verify that tags exist in a nominal structure, the quality and usefulness of the actual tags and structure being generated sub-standard. The lack of ActualText in math equations means this fails to meet the PDF/UA-2 or WTPDF standards. Tables are very baseline and primitive, without any header cells or scoping. Image captions are not contained correctly and alt text seems to have some issue where it is not being picked up by screen readers. It is premature to claim any level of real compliance. All of these issues are ones that there is no automatic checker for and can only be picked up through human testing. |
Beta Was this translation helpful? Give feedback.
-
|
I'm in the process of gathering the information, and need time to injest ISO 32000-2:2020. I work at an academic institution, and we are approaching this from a policy/legal compliance standpoint, in our case WCAG 2.1 AA and Section 508. More importantly, we have to test for demonstrable accessibility which both the example files above and files we generated with TeXLive 2024 do not meet. Screen reader performance was especially poor, checking with Apple VO, NVDA, and Adobe's own reader. That is honestly where the rubber meets the road. Compliance to a standard that isn't implemented anywhere isn't a useful compliance, especially if it means not meeting real-world accessibility needs. Like I said, I am gathering more useful details to submit in one or more issues. |
Beta Was this translation helpful? Give feedback.
-
True to some extent, but one issue in this area is that without good examples (complex inputs meeting the agreed standards in terms of structure), viewers, etc., will not be developed that can read them. So simply saying 'target only what is readable now' doesn't really work: current reader implementations have significant gaps in coverage |
Beta Was this translation helpful? Give feedback.
-
|
Note that this collection is specifically a collection of examples for PDF/UA-2, that is the new PDF 2.0 based standard. It is explicitly here to allow implementors of PDF consuming software to have a collection of documents to test against. So while, to produce an accessible document for end users today, you do need to target PDF 1.x and PDF/UA-1 this collection is to allow us to test PDF/UA-2 generation and allow consuming applications to have a set of PDF/UA-2 documents to test. |
Beta Was this translation helpful? Give feedback.
-
|
"real-world accessibility" for eg math is currently not given anyway: in 1.7 not as the standard has no support for it, in 2.0 not as the implementations do not support it. Our files are meant to push development forward. I just was at the PDF week where we used the files to demonstrate and discuss the implementation problems. |
Beta Was this translation helpful? Give feedback.
-
|
@ErroneousBosch wrote:
This is relevant, since ensuring (as far as possible) that the PDF output is compliant
Therefore, for much that is important to compliance with WCAG provisions, it is impossible for PDF production software alone to ensure, or even provide support We probably need to look at this, to see if it helps to improve our WCAG compliance:
And it would be unwise for us to go anywhere near supporting such purely local "legal requirements”! |
Beta Was this translation helpful? Give feedback.
-
|
Am 14.05.24 um 01:54 schrieb ErroneousBosch:
More importantly, we have to test for
demonstrable accessibility which both the example files above and files
we generated with TeXLive 2024 do not meet.
The big question here is why are they not meeting it? Because there are
errors in them with respect to implementing UA-2 or because consuming
software up to now is not capable of properly handling PDF 2 structures yet?
So far industry hasn't bothered much with the improved structures
provided by PDF 2 (and necessary for higher quality accessibility)
because there were (nearly) no documents that used them --- so so why
bother if there is no use case?
Screen reader performance was especially poor, checking with Apple VO,
NVDA, and Adobe's own reader. That is honestly where the rubber meets
the road. Compliance to a standard that isn't implemented anywhere isn't
a useful compliance, especially if it means not meeting real-world
accessibility needs.
all true, but if your road is currently a gravel surface with huge holes
in it, the question is: do you want to continue running over it only
with noisy tanks because everything else that would be a comfortable car
will break down, or do you strife for improving the road?
right now accessibility of PDFs is so poor because consuming software is
based on 1.7 and UA-1 + a lot of heuristics (which differ from
implementation to implementation and therefore also do not give a good
user experience over all).
Now, by producing UA-2 docs you to not get magically better
accessibility, in fact you are likely to get even worse, because
consumer software handles the improved structures badly or not at all
and their heuristics fail with such documents.
But as it was pointed out, the goal of producing documents that comply
to the new (and better) standards, was to make showcases of where in the
current consumer software fails with PDF/UA-2 and this way drive good
implementations of the new standard in the consumer software. With the
ability of providing a corpus of complex documents that meet PDF/UA-2,
we are fairly confident that this could happen and in fact we already
see movements in this respect
Like I said, I am gathering more useful details to submit in one or more
issues.
please do, but also please keep in mind the purpose of the generated
documents, e.g.,
- things in which we go wrong should be improved on our end to make
the documents better
- but things that go wrong in consumer apps because they do not
understand the standard, should really (with some pressure)
communicated by the community to the vendors.
|
Beta Was this translation helpful? Give feedback.
-
|
I'm at the workshop right now! Can I request feedback for this PDF I made? The LaTeX source code uses amsthm, which has many votes to be made compatible: |
Beta Was this translation helpful? Give feedback.
-
|
@Jeffrey-Kuan Interesting results. Some of the alt text is better, and it is at least there. Looking at page 3:
The matrix representation's alt text is not usable, possibly cut off: My reader skipped the element entirely. Tagging structure is a mixed bag, since body text is separate from contained formulas. The real killer is that the reading order is completely useless. Inline formulas are sometimes read after their containing text, some text is not completely read, references are read inline as numbers, and footnotes behave very erratically. Overall still a hard fail. |
Beta Was this translation helpful? Give feedback.
-
|
@Jeffrey-Kuan There are a number of errors.
|
Beta Was this translation helpful? Give feedback.
-
|
I fail to get tagging to work. I have the following very minimal example: and run with This generates a file foo.pdf but neither Preview nor Adobe Acrobat manage to read it. I then tried to use the tagpdf example files from https://github.com/latex3/tagging-project/tree/main/project-examples/tagpdf. I run which fails with I have TeX Live 2024 for macOS. This is the version information when compiling: Any suggestions? |
Beta Was this translation helpful? Give feedback.
-
|
For what it's worth, your example works for me with Mac OS, TexLive 2024, updated about a week ago. But I ran lualatex not lualatex-dev. Preview and Adobe displayed it fine. I also ran the Adobe accessibility check, and the only that that failed was missing title. I also tested the "read out loud" feature and it worked. |
Beta Was this translation helpful? Give feedback.
-
|
Same with lualatex-dev, works too here. But the diff to my log shows that you have an old tagpdf-base in your system and that might be the cause of the issue: @kaimikael perhaps you should try to update your system again, it is certainly a somewhat singular behavior. |
Beta Was this translation helpful? Give feedback.
-
|
Updating pdfmanagement-testphase did the trick, I hadn’t realised that was a relevant package. |
Beta Was this translation helpful? Give feedback.
-
|
@kaimikael sorry I seem to have missed the notification. Apart from the pdfmanagement you should also update the tagpdf package. The current version is 2024-08-02, 0.99d. |
Beta Was this translation helpful? Give feedback.
-
|
@u-fischer I think we should add a task to update the documentation how to use the prototype, stating not only that one needs a current LaTeX (and preferanly use the -dev format), but also which packages have to be up-to-date, for people who use a small LaTeX distribution and do not auto-update all kind of packages. |
Beta Was this translation helpful? Give feedback.
-
|
@davidcarlisle doing more in-depth tests, we encountered the issue on 2401.09965v1-tagged: the inline math element and The file names also seem to indicate the mismatch: We suspect that these are other cases in this and other test PDFs like this, but we stopped at the first one. |
Beta Was this translation helpful? Give feedback.
-
|
Oh thank you. Fortunately (or perhaps unfortunately) this is not a systematic error but manual editing failure on my part. The arxiv examples were trialing obtaining mathml for each tex fragment by extracting it from the arxiv supplied xhtml version of the document. the tricky part (and why we never published the scripts used) is matching up the tex fragments as obtained by different systems The file fragment 18 is I'll fix the hash ahd re-generate.... |
Beta Was this translation helpful? Give feedback.
-
|
see 5dc2b94 |
Beta Was this translation helpful? Give feedback.
-
|
LaTeX is an incredible document format. My understanding is that PDF/UA and PDF/UA-2 are tagged extensively. Does that mean PDF/UA docs can be reverse engineered into LaTeX? Is this possible? |
Beta Was this translation helpful? Give feedback.
-
Pandoc is a good and converter for going to HTML and supports good features for translating math, esp LaTeX -> HTML. PDF to anything is a gamble. Just remember that you have to check anything that comes out. I did some experimentation with LaTeX -> ePub with varied success, and I need to revisit it now that there is some more support for alternative text in the tagging engine. I am hoping to build a workflow that will be useful sometime in the coming months |
Beta Was this translation helpful? Give feedback.
-
|
We have some traditional tools for PDF to HTML. My goal was looking into well tagged PDF/UA docs and formal derivations from the tags for a faster and more accurate conversion. |
Beta Was this translation helpful? Give feedback.
-
|
@ConProgramming you may wish to look at this industry specification: https://pdfa.org/resource/deriving-html-from-pdf/ |
Beta Was this translation helpful? Give feedback.
-
|
I also know of this early OSS implementation if you are after source code: https://github.com/Normex/PDF-Derivation but it has not been updated to the latest spec (based on commit dates) |
Beta Was this translation helpful? Give feedback.
-
|
Thank you Peter - I appreciate the both of thoss |
Beta Was this translation helpful? Give feedback.
-
WTPDF / PDF/UA-2 Examples by the LaTeX Project
The following files demonstrate various aspects of Well Tagged PDF documents conforming to PDF/UA-2.
They were all generated with LuaLaTeX (
lualatex-devin TeX Live 2024).The files are a mixture of small examples demonstrating specific features, older out of copyright documents that have been re-typeset as tagged pdf, and contemporary documents including recently published arXiv papers, course notes, and conference papers.
The files here are all PDF 2.0. PDF 1.7 versions of the same documents are available from PDF/UA-1 Examples by the LaTeX Project.
Access to the Files
The full collection of PDF files is available at Google Drive, where you may select one or more individual files to download, or, at the
top of the page is a Download all link which will generate a zip file and download the full collection.
Google drive directory of all example PDF files
The LaTeX sources are available from this repository also, where appropriate, we link to the original files used as source material.
Verification of PDF/UA-2 compliance
There are not yet many validators that correctly handle UA-2 (given that the standard was released in March 2024 not that
surprisingly). One online validator you can try on the smaller examples is
VeraPDF — PDF/A and PDF/UA Validation
Please note that some PDF viewers modify the PDF when opening it (to allow for annotations, for example). In some cases this is known to break the PDF/UA-2 standard. If that happens re-download and use a different viewer.
The Samples
Simple Examples with MathML Associated files
All three conform to: PDF/UA-2 PDF/A-4F WTPDF/Accessibility WTPDF/Reuse Arlington
Three small examples demonstrating the use of Associated Files to Tag mathematics. Each formula is associated with two associated files. A LaTeX fragment representing the original source, and a MathML document.
mathml-AF-ex1
mathml-AF-ex2
Sample-AF-Math-LaTeX
amsmathLaTeX package documentationConforms to: PDF/UA-2 PDF/A-4F WTPDF/Accessibility WTPDF/Reuse Arlington
The
amsmathpackage defines the main markup structures for mathematics in LaTeX.This manual has examples of many kinds of aligned equations and similar structures. This version has been enhanced to produce Well Tagged PDF.
amsldoc-tagged
tagpdfLaTeX package documentationConforms to: PDF/UA-2 PDF/A-4 WTPDF/Accessibility WTPDF/Reuse Arlington
The
tagpdfLaTeX package is a core part of the LaTeX support for tagged PDF.Its documentation already conforms to WTPDF and PDF/UA-2 and a snapshot is included here.
tagpdf
ArXiv publications
Tagged using MathML extracted from the arXiv-supplied html versions of the documents.
They were each submitted to arXiv under a CC Licence permitting re-use such as this experiment, The tagged documents are available under the same licence.
Conforms to: PDF/UA-2 PDF/A-4F WTPDF/Accessibility WTPDF/Reuse Arlington
2401.09965v1-tagged — Original Source
Conforms to: PDF/UA-2 PDF/A-4F WTPDF/Accessibility WTPDF/Reuse Arlington
2401.09436v1-tagged — Original Source
Conforms to: PDF/UA-2 PDF/A-4F WTPDF/Accessibility WTPDF/Reuse Arlington
2401.05361v1-tagged — Original Source
Niels Bohr: The Theory of Spectra and Atomic Constitution; Three Essays
Conforms to: PDF/UA-2 PDF/A-4F WTPDF/Accessibility WTPDF/Reuse Arlington
These essays by Niels Bohr are available as LaTeX source from The Project Gutenberg.
Additional TeX markup has been added to produce Tagged PDF. Also all math expressions were converted to MathML using LaTeXML.
47464-t-tagged — Original Source
William Shakespeare: MACBETH
Conforms to: PDF/UA-2 PDF/A-4 WTPDF/Accessibility WTPDF/Reuse Arlington
macbeth-tagged — Original Source
This document uses a provided LaTeX source of the play text. The LaTeX markup has been enhanced to produce Well Tagged PDF.
American Standard Version of the Bible (1901 text)
Conforms to: PDF/UA-2 PDF/A-4 WTPDF/Accessibility WTPDF/Reuse Arlington
The plain text source of the ASV Bible, 1901 as provided by Wikisource. This has been marked up as LaTeX to generate well tagged PDF. This example demonstrates a custom role map with structured tagging corresponding to the Testament/Book/Chapter/Verse structure shown in this work.
ASV Bible — Original Source
DEIMS 2024 Conference paper
Conforms to: PDF/UA-2 PDF/A-4 WTPDF/Accessibility WTPDF/Reuse Arlington
The paper Enhancing LATEX to Automatically Produce Tagged and Accessible PDF submitted to DEIMS 2024, Tokyo.
As well as describing the approach to PDF tagging used for these examples, the paper does itself form an example of tagging a
contemporary conference paper. This is the version as prepared for the TeX Users Group publication, TUGBoat.
tb139mitt-deims24
The presentation at the DEIMS conference including a demonstration is available as a video.
PDF Association sample poster
Conforms to: PDF/UA-2 PDF/A-4 WTPDF/Accessibility WTPDF/Reuse Arlington
An article describing the PDF Association work on accessibility produced for the PDF Association launch of Well Tagged PDF.
pdfa-art
Sample Chemistry/Math notes
This is a small contemporary document used as notes on mathematical aspects of Chemistry.
In this example, the math is associated with just LaTeX source Associated files, not MathML.
Conforms to: PDF/UA-2 PDF/A-4F WTPDF/Accessibility WTPDF/Reuse Arlington
525Da-23-group-theory
A small template exam paper.
Conforms to: PDF/UA-2 PDF/A-4F WTPDF/Accessibility WTPDF/Reuse Arlington
PHY-exam
Wilhelm Busch: Max and Moritz
Conforms to: PDF/UA-2 PDF/A-4 WTPDF/Accessibility WTPDF/Reuse Arlington
A LaTeX document that does not have math and the main language is not English. Showing tagging of images, verse structures and the use of more than one (marked up) language in a document.
pg17161-tagged — Original Source
Beta Was this translation helpful? Give feedback.
All reactions