attachments working again (issue with /EmbeddedFiles)
Hans, many thanks for having fixed the issues with attachments (in latest beta from 2020.02.07 18:36). I haven’t tested attachments with PDF/A-3a. But there is still an issue with /EmbeddedFiles. MWE: \setupinteraction[state=start] \starttext \startTEXpage[offset=1em] a\attachment[file=abdfaeasd.txt, name=hola, method=hidden] \contextversion \stopTEXpage \stoptext File name is changed: 2 0 obj << /AFRelationship /Unspecified /EF << /F 1 0 R >> /F (hola.txt) /Type /Filespec /UF <FEFF0068006F006C0061002E007400780074> >> endobj But embedded names have the name from the file, leaving it unchanged (abridged version): 18 0 obj << /EmbeddedFiles << /Names [ (abdfaeasd.txt) 2 0 R ] >> /Type /Names >> endobj Could you fix the name change for /EmbeddedFiles? Many thanks for your help, Pablo -- http://www.ousia.tk
On 2/7/20 9:19 PM, Rik Kabel wrote:
On 2/7/2020 14:46, Pablo Rodriguez wrote:
many thanks for having fixed the issues with attachments (in latest beta from 2020.02.07 18:36). I haven’t tested attachments with PDF/A-3a.
PDF/A-3a attachments still fail validation with the same issues.
Hi Rik, it has way less issues, at least using verapdf-1.5.18 and the following sample: \setuptagging[state=start] \setupbodyfont[30pt] \setupbackend [format=PDF/A-3a, intent=sRGB IEC61966-2.1, profile={sRGB.icc,default_gray.icc}, level=0] \setupcolors[pagecolormodel=auto] \setupinteraction[state=start] \starttext \startTEXpage[offset=1em] an attachment\attachment[file=xml-mkiv.pdf, type={application/pdf}] \stopTEXpage \stoptext With "method=hidden" (no attachment annotation) it has no issue. Both issues are related to annotations in general: - If the flag annotation is present (/F key), it should be set to print and disable everything else. - Annotations need an appearance dictionary (unless their /Rect is one and the same point, or /Popup or /Link annotations). Actual value of the annotation flag key is "/F null". I wonder whether this is a bug (for having wanted to avoid the presence of /F at all). Otherwise, I don’t think that setting all embedded file annotations to printable is a good default. Just in case it helps to the discussion, Pablo -- http://www.ousia.tk
On 2/7/20 9:19 PM, Rik Kabel wrote:
On 2/7/2020 14:46, Pablo Rodriguez wrote:
many thanks for having fixed the issues with attachments (in latest beta from 2020.02.07 18:36). I haven’t tested attachments with PDF/A-3a.
PDF/A-3a attachments still fail validation with the same issues.
Hi Rik,
it has way less issues, at least using verapdf-1.5.18 and the following sample:
\setuptagging[state=start]
\setupbodyfont[30pt]
\setupbackend [format=PDF/A-3a, intent=sRGB IEC61966-2.1, profile={sRGB.icc,default_gray.icc}, level=0]
\setupcolors[pagecolormodel=auto]
\setupinteraction[state=start] \starttext \startTEXpage[offset=1em] an attachment\attachment[file=xml-mkiv.pdf, type={application/pdf}] \stopTEXpage \stoptext
With "method=hidden" (no attachment annotation) it has no issue.
Both issues are related to annotations in general:
- If the flag annotation is present (/F key), it should be set to print and disable everything else. - Annotations need an appearance dictionary (unless their /Rect is one and the same point, or /Popup or /Link annotations).
Actual value of the annotation flag key is "/F null". I wonder whether this is a bug (for having wanted to avoid the presence of /F at all).
Otherwise, I don’t think that setting all embedded file annotations to printable is a good default.
Just in case it helps to the discussion, Peter is stepwise looking into all these issues but we decided to also see how that checking and standards evolve in cases where it's a confusing mess. And these 'appearance dicts' are an example of a mess. On the one hand there's predefined appearances and on the other hand enforced renderings which gives some chicken-egg issue. It smells a lot
On 2/7/2020 9:55 PM, Pablo Rodriguez wrote: like bugs became features (standards) or 'acrobat behaviour' made the standard or ... (like the zero rect thing, which, given t e plenty of flags there are and verbosity there is in pdfm is pretty weird and actually can make viewers bark. Irr the current approach we follow is kind of a compromis. Now, of course we can play safe and *always* have some fake appearance (we could basically choose whatever funny shape we like as over time people will interpret hard codes symbols for attachments differently) and drop support for standardized renderings (that could adapt over time). So ... no changes etc till Peter gives his blessing as he is testing all this in the framework we've set up for it. Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------
On 2/7/20 10:37 PM, Hans Hagen wrote:
Peter is stepwise looking into all these issues but we decided to also see how that checking and standards evolve in cases where it's a confusing mess. And these 'appearance dicts' are an example of a mess. On the one hand there's predefined appearances and on the other hand enforced renderings which gives some chicken-egg issue. It smells a lot like bugs became features (standards) or 'acrobat behaviour' made the standard or ... (like the zero rect thing, which, given t e plenty of flags there are and verbosity there is in pdfm is pretty weird and actually can make viewers bark. Irr the current approach we follow is kind of a compromise.
Just a comment on one of the issues. Maybe the simplest way to avoid the issues with the annotation flag is not including it at all. Once annotations have appearance dictionaries, Acrobat will print those ones with an annotation flag set to print. Pablo -- http://www.ousia.tk
Am 07.02.2020 um 22:37 schrieb Hans Hagen:
On 2/7/20 9:19 PM, Rik Kabel wrote:
On 2/7/2020 14:46, Pablo Rodriguez wrote:
many thanks for having fixed the issues with attachments (in latest beta from 2020.02.07 18:36). I haven’t tested attachments with PDF/A-3a.
PDF/A-3a attachments still fail validation with the same issues.
Hi Rik,
it has way less issues, at least using verapdf-1.5.18 and the following sample:
\setuptagging[state=start]
\setupbodyfont[30pt]
\setupbackend [format=PDF/A-3a, intent=sRGB IEC61966-2.1, profile={sRGB.icc,default_gray.icc}, level=0]
\setupcolors[pagecolormodel=auto]
\setupinteraction[state=start] \starttext \startTEXpage[offset=1em] an attachment\attachment[file=xml-mkiv.pdf, type={application/pdf}] \stopTEXpage \stoptext
With "method=hidden" (no attachment annotation) it has no issue.
Both issues are related to annotations in general:
- If the flag annotation is present (/F key), it should be set to print and disable everything else. - Annotations need an appearance dictionary (unless their /Rect is one and the same point, or /Popup or /Link annotations).
Actual value of the annotation flag key is "/F null". I wonder whether this is a bug (for having wanted to avoid the presence of /F at all).
Otherwise, I don’t think that setting all embedded file annotations to printable is a good default.
Just in case it helps to the discussion, Peter is stepwise looking into all these issues but we decided to also see how that checking and standards evolve in cases where it's a confusing mess. And these 'appearance dicts' are an example of a mess. On the one hand there's predefined appearances and on the other hand enforced renderings which gives some chicken-egg issue. It smells a lot
On 2/7/2020 9:55 PM, Pablo Rodriguez wrote: like bugs became features (standards) or 'acrobat behaviour' made the standard or ... (like the zero rect thing, which, given t e plenty of flags there are and verbosity there is in pdfm is pretty weird and actually can make viewers bark. Irr the current approach we follow is kind of a compromis.
Now, of course we can play safe and *always* have some fake appearance (we could basically choose whatever funny shape we like as over time people will interpret hard codes symbols for attachments differently) and drop support for standardized renderings (that could adapt over time).
So ... no changes etc till Peter gives his blessing as he is testing all this in the framework we've set up for it.
Hans
As Hans stated, we are working on it and two of the three validation problems are already solved (needs more testing though). The only remaining problem is this https://github.com/veraPDF/veraPDF-validation-profiles/wiki/PDFA-Parts-2-and... I attached the current code, so you and Rik are able to test it if you want (creating a new format is required). Best contact me off-list if you run into new problems. Peter
On 2/8/20 1:45 PM, Peter Rolf wrote:
[...]
I attached the current code, so you and Rik are able to test it if you want (creating a new format is required). Best contact me off-list if you run into new problems.
Hi Peter, the new version is fine for me. From what I see, /F is now set to 4 (print). Many thanks for your help, Pablo -- http://www.ousia.tk
On 2/8/2020 12:32, Pablo Rodriguez wrote:
On 2/8/20 1:45 PM, Peter Rolf wrote:
[...]
I attached the current code, so you and Rik are able to test it if you want (creating a new format is required). Best contact me off-list if you run into new problems. Hi Peter,
the new version is fine for me.
From what I see, /F is now set to 4 (print).
Many thanks for your help,
Pablo -- http://www.ousia.tk
Not quite ready yet. When the hidden method is used, it passes the online validator[1], but fails VeraPDF[2] for clause 6.8-1. When visible, it fails the online validator stating that the appearance dictionary doesn't contain an entry, and fails VeraPDF for clauses 6.8-1 and 6.3.3-1. Oddly, the bookmarks do not appear either way under MkIV, and they do not appear when the attachment is visible under LMTX. The bookmarks do appear with a hidden attachment under LMTX. The results are generally better than before this updated lpdf-wid.lua. [1] https://www.pdf-online.com/osa/validate.aspx [2] Version 1.14.8, which I believe is the current version -- Rik
On 2/8/20 10:13 PM, Rik Kabel wrote:
Not quite ready yet. When the hidden method is used, it passes the online validator[1], but fails VeraPDF[2] for clause 6.8-1. When visible, it fails the online validator stating that the appearance dictionary doesn't contain an entry, and fails VeraPDF for clauses 6.8-1 and 6.3.3-1.
Hi Rik, as for 6.8-1, did you set MIME type of the attached file (such as in "type={application/pdf}")?
Oddly, the bookmarks do not appear either way under MkIV, and they do not appear when the attachment is visible under LMTX. The bookmarks do appear with a hidden attachment under LMTX.
As far as I understand, only LMTX will receive regular updates now. (But Hans knows better 😅.) I get both visible attachments and bookmarks in LMTX (latest and previous latest [“latest” is the new beta]). Could you provide a minimal sample that shows the issue? Pablo -- http://www.ousia.tk
On 2/12/2020 14:20, Pablo Rodriguez wrote:
On 2/8/20 10:13 PM, Rik Kabel wrote:
Not quite ready yet. When the hidden method is used, it passes the online validator[1], but fails VeraPDF[2] for clause 6.8-1. When visible, it fails the online validator stating that the appearance dictionary doesn't contain an entry, and fails VeraPDF for clauses 6.8-1 and 6.3.3-1.
Hi Rik,
as for 6.8-1, did you set MIME type of the attached file (such as in "type={application/pdf}")?
Oddly, the bookmarks do not appear either way under MkIV, and they do not appear when the attachment is visible under LMTX. The bookmarks do appear with a hidden attachment under LMTX. As far as I understand, only LMTX will receive regular updates now. (But Hans knows better 😅.)
I get both visible attachments and bookmarks in LMTX (latest and previous latest [“latest” is the new beta]).
Could you provide a minimal sample that shows the issue?
Pablo -- http://www.ousia.tk
Hello Pablo (and everyone else), Here is my test file. You can use "context --mode=hidden file.tex" to hide the pushpin, or leave out the mode to expose it. %% Test file for pdf3a compliance. %% %% So far, this fails when the attachment is included. Other than the %% handling of attachments, ConTeXt does a good job with the various %% PDF/A variants. %% %% Testing done locally with the VeraPDF validator (most recently with %% version 1.14.8) and online with the 3-Heights validator found at %% https://www.pdf-online.com/osa/validate.aspx (current 2020-01-20). %% %% This is not a minimal example. \definemode [hidden][keep] \setuplanguage [en] \setuppapersize [A6,landscape] \setuppagenumbering [location=] \setuptagging [state=start] \setupinteraction [state=start] %setupbackend [format={pdf/a-3u}, \setupbackend [format={pdf/a-3a}, level=0, profile={default_gray.icc}, intent={ISO coated v2 300\letterpercent\space (ECI)}] \setupstructure [state=start, method=auto] \setupcolors [pagecolormodel=auto, start=yes, rgb=no, cmyk=no] \setupbodyfont [palatino,18pt] \starttext This document was produced using version \contextversion\ of \ConTeXt\ \doifelse{\texenginename}{luametatex}{LMTX}{\contextmark} with the version \texengineversion\ of \texenginename. \attachment [file=./\jobname.tex, method={\doifmodeelse{hidden}{hidden}{normal}}, type={text/plain;charset=UTF-8}] The source code for this document is attached as a plain text file. It should show up in the pdf bookmarks for the document\doifmodeelse {hidden}{}{ and through the icon at the start of this paragraph}. \stoptext I have tested with LMTX 2020.02.11 16:41. *Mime type* * You can see that the mime type is set in the ConTeXt file and it appears in the resulting pdf, although not knowing enough about the pdf file structure I do not know if it appears correctly. *Validation* * With the pin hidden, VeraPDF 1.14.8 complains about two occurrences of clause 6.8 test 1 failure. With the pin exposed, it complains about one occurrence of clause 6.8 test 1 failure and one occurrence of clause 6.3.3 test 1 failure (appearance dictionary). * With the pin hidden, pdf-online.com declares the valid. With the pin exposed, pdf-online.com fails it because "[t]he appearance dictionary doesn't contain an entry." They are aware of the difference between their report and that of VeraPDF. *Viewing* * Acrobat Reader DC shows the pushpin, changes the pointer, and displays flyover text when appropriate and shows the bookmark for both variants. * SumatraPDF shows the pushpin, changes the pointer, and displays flyover text when appropriate. It does not show the bookmark when the pushpin is visible, but does show it when it is hidden. * MuPDF does not show the pushpin, instead leaving empty space, when it should be visible. It does not show bookmarks or any other indication that there is an attachment for either variant. (MuPDF can show bookmarks for some other documents.) * Firefox 73 does not show the pushpin when it should be visible, but does change the pointer and show the attachment name as a flyover. Firefox 73 does show the bookmark for both variants. * Chrome 80 does not show the pushpin when it should be visible, but does show the attachment name as a flyover. Chrome 80 does not show the bookmark for either variant. (Chrome 80 does show bookmarks for some other PDF document.) * The three viewers listed here that do not show the pushpin do reserve the appropriate space – changing the symbol to the Paperclip demonstrates that even though the paperclip is not displayed. Clearly there are issues with the various viewers. How much ConTeXt can or should do to manage the differences is not clear to me. -- Rik Kabel
Following up on my viewer observations, I downloaded Evince for Windows version 2.32 and see something that might be meaningful. Evince has a side-panel with a drop-down for selecting Thumbnails, Index, Attachments, Layers, or Annotations. Those that are not present in the document are greyed out. For the document with the pushpin (which Evince does display) the available choices are Thumbnails and Annotations. The Annotations selection shows the attached file on a tree under Page 1. For the document with the hidden attachment, both the Attachments and the Annotations choices are available. When Annotations is selected, the message "/Document contains no annotations/" is shown. When Attachments is selected, the attachment is properly listed. Perhaps there is something in the difference between annotations and attachments that results in different treatments by the various viewers. -- Rik
Perhaps there is something in the difference between annotations and attachments that results in different treatments by the various viewers. it has always been that messy; there's also the annotation symbol
On 2/13/2020 12:45 AM, Rik Kabel wrote: treatment that can differ; it doesn't help that acrobat changed behaviour a couple of times i think this is why the whole attachment handling was as it is: some comprimise that kind of made them show up ok on the average; but validatin spoiled this and now we're stick with these incompatible viewer behaviour .. not much we can do about it i fear (For instance, i remember that at some point acrobat showed attachments twice in some lists. Often these features ended up in specs without viewer support yet so probably not well tested for interferences either.) Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------
On 2/13/2020 03:56, Hans Hagen wrote:
On 2/13/2020 12:45 AM, Rik Kabel wrote:
Perhaps there is something in the difference between annotations and attachments that results in different treatments by the various viewers. it has always been that messy; there's also the annotation symbol treatment that can differ; it doesn't help that acrobat changed behaviour a couple of times i think
this is why the whole attachment handling was as it is: some comprimise that kind of made them show up ok on the average; but validatin spoiled this and now we're stick with these incompatible viewer behaviour .. not much we can do about it i fear
(For instance, i remember that at some point acrobat showed attachments twice in some lists. Often these features ended up in specs without viewer support yet so probably not well tested for interferences either.)
Hans
The further I look into bookmark, annotation, and attachment handling by a number of viewers, the more confusing things appear. And Acrobat still shows attachments twice under some circumstances, with the helpful flyover for one entry giving the location as the associated page number, and for the other giving it as the attachments tab. What is more troublesome on the ConTeXt side is the validation issue with the appearance dictionary for visible attachments. The difference between validators when looking at the hidden attachment in my example hinges on the handling of the attachment mime type. I have filed a defect report with VeraPDF, which appears to be rejecting valid mime type that includes a charset clause. -- Rik
On 2/14/20 3:43 AM, Rik Kabel wrote:
[...] The further I look into bookmark, annotation, and attachment handling by a number of viewers, the more confusing things appear.
Hi Rik, this is why following the prescription (the PDF spec) is better in order to check the standard than gathering information in different viewers.
And Acrobat still shows attachments twice under some circumstances, with the helpful flyover for one entry giving the location as the associated page number, and for the other giving it as the attachments tab.
That practice is explictly discouraged by the PDF Association, citing the behaviour of some viewers (but not naming Acrobat). See https://www.pdfa.org/wp-content/uploads/2018/10/PDF20_AN002-AF.pdf#search=us....
What is more troublesome on the ConTeXt side is the validation issue with the appearance dictionary for visible attachments.
This is my (educated?) guess: appearance dictionaries are required to print annotation icons. This isn’t especially directed to attachments, but to all annotation types (excepting popups and links).
The difference between validators when looking at the hidden attachment in my example hinges on the handling of the attachment mime type. I have filed a defect report with VeraPDF, which appears to be rejecting valid mime type that includes a charset clause.
If this is a bug, they should fix it. They may have missed a valid MIME type. Pablo -- http://www.ousia.tk
participants (4)
-
Hans Hagen -
Pablo Rodriguez -
Peter Rolf -
Rik Kabel