Hi Karl, On Jun 11, 2016, at 8:57 AM, Karl Berry mailto:karl@freefriends.org> wrote: Hi Ross - I don't have the DavidCLM-Medium font, and had never heard of the culmus package; I see now it is a pretty big effort on sf. At any rate, I (or Thanh or whomever) need an example demonstrating the failure that can be run. Maybe you can make a self-contained set of files (the font(s) in needed formats, the tex source, etc.)? Or: it doesn't matter if there is a workaround for it, as long as it shows the failure. So if you have an example that uses fonts already in TL, so much the better. Certainly. The texlive-de.tex documentation uses a font that is affected this way. Attached is a cut-down example source, test-berafont.tex with PDFs: test-berafont-pdftex.pdf — PDF/A using pdfTeX and test-berafont-gs.pdf — plain PDF using dvips+GS You can see the difference in the attached PDFs: SCI:TL-DE16 ross$ grep -a CharSet test-berafont-gs.pdf /CharSet(/A/E/L/T/V/X/a/aacute/acute/e/g/i/l/n/o/one/r/s/six/t/two/u/v/z/zero)/FontFile3 12 0 R>> SCI:TL-DE16 ross$ mv test-berafont.pdf test-berafont-pdftex.pdf SCI:TL-DE16 ross$ grep -a CharSet test-berafont-pdftex.pdf /CharSet (/A/E/L/T/V/X/aacute/e/g/i/l/n/o/one/r/s/six/t/two/u/v/z/zero) where pdfTeX just puts /aacute GS includes /a/aacute/acute . To get a valid PDF/A directly using dvips+GS requires stuff you don’t yet have, and I’m in discussions with a GS developer to perfect. It should certainly be ready for release later this year.

Ross and all - back on your mail about /CharSet from two years ago. Date: Sat, 11 Jun 2016 00:15:05 +0000 From: Ross Moore [ https://mailman.ntg.nl/pipermail/ntg-pdftex/2016-June/004087.html ] As far as I can tell, the problem as reported relates to the seac operator. Heiko, Thanh, Ross, anyone, up for looking into the code to get the seac referents into the output /CharSet list? Not something I am familiar with, so it would take me a while. (More below.) For example, this pdftex -ini file shows the problem, unrelated to the pdf/x or latex: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \pdfoutput=1 \catcode`\{=1 \catcode`\}=2 \pdfcompresslevel=0 \pdfobjcompresslevel=0 \pdfglyphtounicode{aacute}{00E1}\pdfgentounicode=1 \font\b = fver8t \b % ecrm1000 \hsize=5pt \hfil\char225 % aacute \end %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% $ pdftex -ini foo.tex .. $ fgrep -a /CharSet try.pdf /CharSet (/aacute) Whereas the correct output should also include /a and /acute (as it does with gs, which has code to handle the seac pieces): /CharSet(/a/aacute/acute) Looking at the definition for /aacute (t1disasm gl_tree); glyph != NULL; glyph = (char *) avl_t_next(&t)) pdf_printf("/%s", glyph); And the code to handle seac is in writet1.c: case CS_SEAC: a1 = cc_get(3); a2 = cc_get(4); cc_clear(); mark_cs(standard_glyph_names[a1]); mark_cs(standard_glyph_names[a2]); break; "Just" have to get these pieces together, which doesn't seem like it should be too hard ... ? By the way, I checked a few other fonts. For EC (ecrm1000), Latin Modern (ec-lmr10), and txfonts (t1xr) (mentioned in tex.stackexchange.com/questions/81927), seac is not used. This is a reasonable choice for font implementors, as seac is deprecated all over the place, as it assumes AdobeStandardEncoding etc. dvips|ps2pdf does not output the /a and /acute for those fonts either; presumably Adobe programs don't either. This seems correct, since the /a and /acute character definitions are not in fact used in those cases. Hopefully that is ok with the new standards. I can't imagine a decent way to change it. --best, karl. P.S. Unrelated to the problem, but I noticed while looking into it ... although at one time writet1.c in pdftex and dvips were close, Pali's changes last year to support encodings for bitmap fonts made the two versions very different. I doubt they can reasonably be merged again. :(.

Perhaps we could just have a command-line switch that allows writing the /CharSet to be omitted? Not a command-line switch. It could be another \pdf... parameter, but I wonder if we can just make things simpler and never output the /CharSet, instead of bothering to make it conditional. I know that's ok for validation, the question is whether it would cause problems in practice. Since, as we know, its contents are likely to be incorrect, I surmise/hope the answer is no. So: Does anyone know of anything using the /CharSet value from pdftex's output? --karl

On 7/7/2018 2:31 AM, Reinhard Kotucha wrote:

On 2018-07-06 at 21:32:04 GMT, Karl Berry wrote:

...

Perhaps we could just have a command-line switch that allows writing the /CharSet to be omitted?

Not a command-line switch. It could be another \pdf... parameter, but I wonder if we can just make things simpler and never output the /CharSet, instead of bothering to make it conditional. I know that's ok for validation, the question is whether it would cause problems in practice. Since, as we know, its contents are likely to be incorrect, I surmise/hope the answer is no.

So: Does anyone know of anything using the /CharSet value from pdftex's output?

According to the PDF Reference[1], /CharSet is optional and meaningful only in Type 1 fonts. Thus no program can rely on it. If no program can rely on it and it causes trouble, it should better be removed without substitution (no command line option and no new primitive).

I suppose that luatex developers are listening here as well.

only by accident as most pdftex mails filter to the pdftex bin anyway, it's no big deal to add an option (omitcharset) for it as we also do it for cidsets (omitcidset) Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------

Hi guys, Can anyone tell me how this issue has been resolved, if at all ? On 11 Jul 2018, at 8:25 am, Reinhard Kotucha mailto:reinhard.kotucha@web.de> wrote: On 2018-07-07 at 11:05:49 +0200, Hans Hagen wrote:

On 7/7/2018 2:31 AM, Reinhard Kotucha wrote:

...

According to the PDF Reference[1], /CharSet is optional and meaningful only in Type 1 fonts. Thus no program can rely on it. If no program can rely on it and it causes trouble, it should better be removed without substitution (no command line option and no new primitive).

...

I suppose that luatex developers are listening here as well.

anyway, it's no big deal to add an option (omitcharset) for it as we also do it for cidsets (omitcidset)

I see no new options with man pdftex or pdftex --help . Sounds good. Please proceed. The reason I’m asking is that I have an example file that someone else has provided that produces a “missing CharSet” error when validated with Adobe’s Preflight for PDF/A-1b. It was built with: This is pdfTeX, Version 3.14159265-2.6-1.40.19 (MiKTeX 2.9.6930 64-bit) (preloaded format=pdflatex 2019.1.11) 23 JAN 2019 16:46 but there is no error when I build the analogous PDF with: This is pdfTeX, Version 3.14159265-2.6-1.40.19 (TeX Live 2018) (preloaded format=pdflatex 2018.12.24) 25 JAN 2019 17:16 The attached image shows that the CharSet is included on a Mac, using MacTeX, but not with MikTeX, even though the version of pdfTeX seems to be identical. [cid:761CF147-059F-4934-96BB-747FFD2CEA18@telstra.com.au] The only explanation that I can think of is that a primitive or option is being used in one case, but not the other. Can someone please shed some light on what may be causing this difference? Could it really be connected with the LaTeX format version? For me, tlmgr update --list doesn’t list anything needing an update that is obviously relevant. Regards, Reinhard -- ------------------------------------------------------------------ Reinhard Kotucha Phone: +49-511-3373112 Marschnerstr. 25 D-30167 Hannover mailto:reinhard.kotucha@web.de ------------------------------------------------------------------ _______________________________________________ ntg-pdftex mailing list ntg-pdftex@ntg.nlmailto:ntg-pdftex@ntg.nl https://mailman.ntg.nl/mailman/listinfo/ntg-pdftex All the best, Ross Dr Ross Moore Department of Mathematics and Statistics 12 Wally’s Walk, Level 7, Room 734 Macquarie University, NSW 2109, Australia T: +61 2 9850 8955 | F: +61 2 9850 8114 M:+61 407 288 255 | E: ross.moore@mq.edu.aumailto:ross.moore@mq.edu.au http://www.maths.mq.edu.au [cid:image001.png@01D030BE.D37A46F0] CRICOS Provider Number 00002J. Think before you print. Please consider the environment before printing this email. This message is intended for the addressee named and may contain confidential information. If you are not the intended recipient, please delete it and notify the sender. Views expressed in this message are those of the individual sender, and are not necessarily the views of Macquarie University. http://mq.edu.au/ http://mq.edu.au/

Hi Karl, On 26 Jan 2019, at 9:20 am, Karl Berry mailto:karl@freefriends.org> wrote: [ https://mailman.ntg.nl/pipermail/ntg-pdftex/2018-June/004251.htmlhttps://protect-au.mimecast.com/s/m-fMC0YKgRsJNmqKIDO6WH?domain=mailman.ntg.... ] Can anyone tell me how this issue has been resolved, if at all ? I removed the inclusion of /CharSet in the output in the pdftex source (back in July). We did not compile new TL binaries with that change. As things stand, it will be included in TL19. I imagine Christian has compiled new MiKTeX binaries since then. Hence the difference in behavior. has provided that produces a œôòümissing CharSetœôòý error when validated with Adobeœôòùs Preflight for PDF/A-1b. That "error" from preflight contradicts the previous information, which was that CharSet was optional, but had to be completely correct if it was included. It seems that the “optionality” applies to PDF/A-2 and PDF/A-3 but /CharSet is required in PDF/A-1b (presumably PDF/A-1a also). Since PDF/A-1b is what most people try to create when instructed that they need to submit their thesis as PDF/A , then it is wrong to always have the CharSet omitted. Inclusion/omission should be controllable by a primitive, allowing a user or package-writer to select the proper mode. Here’s how it is worded within the ISO 19005-2 spec. document: If the FontDescriptor dictionary of an embedded Type 1 font contains a CharSet string, then it shall list the character names of all glyphs present in the font program, regardless of whether a glyph in the font is referenced or used by the PDF or not. NOTE 2 The above requirement makes normative the statements in ISO 32000-1:2008, 9.8. For subsetted fonts, this is not 100% clear whether it means: 1. all glyphs in the unsubsetted font program (something which should be easy to achieve) or 2. all glyphs within the font program which is the subsetted font. There is no way to make it completely correct (precomposed accent agony, etc.). Therefore the idea was to omit it. It is surely just the latter (2.) that is harder to achieve, since current methods look at just the names of referenced glyphs, and not whether any of these reference any further glyphs. Yet the font subset program being built must include the subroutines for those extra referenced glyphs, so at some point the required tests are being performed. That information has to be passed back to the listing of glyphs used to build the CharSet. Thus it is really a timing problem, not anything that is inherently impossible to achieve. -k So can you please consider introducing a primitive, having a numerical value; e.g. 0 = omit the CharSet 1 = build CharSet as at present 2 = build CharSet from *all* glyph names in the unsubset font Then we can test whether option 2 gives valid documents in all PDF/A levels and flavours; if so it can become the default. If option 2 doesn’t work then the coding for option 1 will need to be enhanced. Then there could be an experimental option 3, that will become the default once perfected. All the best, Ross

Hi Karl, On 26 Jan 2019, at 10:01 am, Karl Berry mailto:karl@freefriends.org> wrote: If the FontDescriptor dictionary of an embedded Type 1 font contains a CharSet string, then I see nothing in that wording that implies CharSet is anything but entirely optional. That wording is for PDF/A-2, not for PDF/A-1. The PDF doc from MikTeX, which alerted me to this, does *not* show the CharSet error when validated for PDF/A-2 or PDF/A-3. It *does* show the error for PDF/A-1 validation. (I’ll copy you my response to the author, in a separate email.) There are many ways in which PDF/A-1 is stricter than later versions. See here: (page 3) https://www.pdfa.org/wp-content/until2016_uploads/2011/06/19005-1_FAQ.pdf PDF/A-1 files must include: • Embedded fonts • Device-independent color • XMP metadata PDF/A-1 files may not include: • Encryption • LZW Compression • Embedded files • External content references • PDF Transparency • Multi-media • JavaScript PDF/A-2 and PDF/A-3 relax many of those 'may not include’s, which are mostly things that TeX does support. The optionality of /CharSet is just another such relaxation. Anyway, right now the choices are a) omit /CharSet or b) output a possibly-incorrect CharSet. If there was a primitive that can control this, then that would potentially be enough, at least for the present. It would allow the CharSet to be omitted with PDF/A-2,3 but included with PDF/A-1. This distinction would need to be documented (in pdfx.pdf say ) so that authors can understand the issue and choose the appropriate package-loading option for their own circumstances. I’m happy to do this. If you want to have a third option c) <something else>, you (or someone) will need to send me a patch. I’ve looked at the coding in writefont.c for how gl_tree is set and used. But I’ve not yet looked at how the subsetted font is constructed. My thought is that the latter needs to adjust the gl_tree before it is used. As I said previously, this will be a timing issue; so I’m not confident that I could correctly write the necessary coding, using programming structures that I don’t fully understand. (I highly doubt that Thanh has time to look into this.) Sorry, but that's the reality. -k Cheers, Ross

Hi Hans, and others On 26 Jan 2019, at 8:09 pm, Hans Hagen mailto:j.hagen@xs4all.nl> wrote:

PDF/A-2 and PDF/A-3 relax many of those 'may not include’s, which are mostly things that TeX does support. The optionality of /CharSet is just another such relaxation.

just wondering: do you see any technical advantage in this CharSet bit array, other than it being an option to predict maybe font memory allocation demands or so (which then in turn is useless as the pdf format has many aspects that will bloat memory usage anyway) I can envisage a possible use for having this knowledge of which glyphs are available internally in a font subset. PDFs are now editable, at least in Acrobat Pro. So knowing what characters are available lets software easily determine whether a simple edit that changes or adds characters to a text block can simply be performed using the embedded font subset, or whether a font substitution is needed to do the specific edit. Of course it is preferable to not have to substitute, as this can change the metrics, hence potentially making a noticeable change to the visual appearance of that text block. If you have ever tried to edit a PDF made by someone else (with TeX or Word or …) then you should have experienced how things can move around significantly within the same paragraph.

...

Anyway, right now the choices are a) omit /CharSet or b) output a possibly-incorrect CharSet.

If there was a primitive that can control this, then that would potentially be enough, at least for the present. It would allow the CharSet to be omitted with PDF/A-2,3 but included with PDF/A-1.

in luatex it's an option At what level? Can it be done on a font-by-font basis? That would be ideal. If just a command-line option when calling lualatex then that is kind of workable. Essentially it would require a user to have done a preflight check and found that one of the fonts has a CharSet problem. Then rerun with the option set, to get a valid PDF/A-2 (or 3) document. It would be affecting all the Type-1 fonts, not just one of them. The ability (described above) to later edit the PDF would be lost pretty much entirely.

This distinction would need to be documented (in pdfx.pdf say ) so that authors can understand the issue and choose the appropriate package-loading option for their own circumstances. I’m happy to do this.

But I’ve not yet looked at how the subsetted font is constructed. My thought is that the latter needs to adjust the gl_tree before it is used. As I said previously, this will be a timing issue; so I’m not confident that I could correctly write the necessary coding, using programming structures that I don’t fully understand.

i don't know about pdftex but it is something delayed to the last when the 'combined' font resource is added as different tex fonts using the same resource can get different entries (and width arrays) but share the blobs My understanding of the code in writefont.c is that the Font Descriptor dictionary is constructed (and written) as a complete object, before the font subset itself is constructed. For the CharSet, the entries in gl_tree are used, based upon a list of the characters explicitly using that font. This does *not* include implicit glyphs, such as /grave (and perhaps /a ) with /agrave . It was such a circumstance that initiated this conversation roughly a year ago. I looked at solutions like writing the accent characters in white, outside the page boundaries, as an /Artifact say. But this begets a range of difficulties, and could potentially affect the pagination or typesetting, and can fail other accessibility checks. I want to develop reliable means to construct documents simultaneously for both Archivability and Accessibility.

...

(I highly doubt that Thanh has time to look into this.) Sorry, but that's the reality. -k it's probably not that complex; i also doubt if the quality of that vector should be perfect as probably only its prensence is checked, not its internal validity (which then would also demand checking fonts which afaik doesn't happen in detail); and i bet that viewers ignore its content anyway

From the veraPDF link that Reinhard provided, it seems that presence is checked with PDF/A-1, but not accuracy. But for PDF/A-2 and 3, there is an more detailed check for accuracy. Perhaps true for viewers; but PDFs are becoming about *more* than just the visual view. We want to be providing the structures required for accurate text extraction and editing. TeX was never designed with this in mind, but because of its programmability this is something that should be achievable. Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nlhttps://protect-au.mimecast.com/s/i9o5CgZ05JfPE8zwCEkFIz?domain=pragma-ade.n... | www.pragma-pod.nlhttps://protect-au.mimecast.com/s/EEOhCjZ12RflogxGHnU_YL?domain=pragma-pod.n... ----------------------------------------------------------------- Cheers. Ross

Hans Hagen wrote: On 1/27/2019 10:00 PM, Ross Moore wrote: PDFs are now editable, at least in Acrobat Pro. weren't they always, given fonts being available? They were always trivially editable; now they are very powerfully editable indeed. Before, an edit could not change the overall bounding box of that particular stretch of text, so one was very restricted in what one can do. Now the bounding box can be re-sized, even moved — Adobe Acrobat DC is a very powerful tool for editing PDFs. i never edit pdf documents (ok, i remember that once i had to strip stuff in order to get a logo, but not adding something) I frequently do, since many "forms" that are sent out are intended to be completed by hand; I simply complete them using AADC. imo editing a pdf makes no sense (and reflow even less) ... also, with respect to fonts, editing assumes all glyphs being present and with open type fonts one also enters a feature mess and gsub/gpos are not embedded All I can say is that since using AADC, editing PDFs has become a real pleasure. ** Phil. -- <Signature> Philip Taylor

David Kastrup wrote: "Taylor, P" mailto:P.Taylor@rhul.ac.uk writes: Hans Hagen wrote: On 1/27/2019 10:00 PM, Ross Moore wrote: PDFs are now editable, at least in Acrobat Pro. So are PostScript files using Tailor on a NeXtstation. But that's not an inherent property of the PDF format even though it is better parseable than PostScript. Can we stop the fanbuoyship for proprietary applications on this list? "We" ? You are welcome to discuss only non-proprietary software if you wish. Some of us live in the real world, where proprietary and non-proprietary tools and software happily co-exist, and where one is free to choose (and discuss) the best tool for the job, regardless of its proprietary/non-proprietary nature. They are nothing but a distraction to what TeXlive can and should provide. Since this is not the TeX Live list, I fail to see the relevance of your remark. -- [<Signature>] Philip Taylor

Hans Hagen wrote: hm, so where does the required information about kerning, ligature building, hyphenation, contextual substitution, anchoring etc come from? No idea, Hans; I simply use the tools that Adobe provide, and everything seems to work like magic. A very great step forward from the days of Acrobat 7. i presume we're anyway not talking of documents made by tex Probably not (I would imagine that most originated in Word) but if you like to send me a sample of something (relatively small) created using LuaTeX, I will see to what extent I can edit it using AADC. but forms use widgets and those are not really typeset You know that, and Adobe know that, but the people who send out forms do not. They send out bog-standard Word (or similar) documents with spaces for the respondent to write his (or her) answer, converted to Adobe PDF. Not wishing to hand-write on such things, I instead use AADC to insert and/or modify text and images. Even our national court service were happy to accept such a think as a part of a formal legal proceedings, with the "signature" (a jpeg from a scan) inserted using AADC. imo editing a pdf makes no sense (and reflow even less) ... also, with respect to fonts, editing assumes all glyphs being present and with open type fonts one also enters a feature mess and gsub/gpos are not embedded As I say, I have not looked into what happens internally; all I can say is "it works", in almost all cases where I have tried. All I can say is that since using AADC, editing PDFs has become a real pleasure. instead of fixing the source? (i'd probably opt for html then) I have no source; I have a PDF, originated by a third party, which I need to amend (e.g., to illustrate what can be achieved or what is desired). I have done this with wine lists, advertisements, posters, etc., sent them back to the originator, and he or she has then been able to incorporate the requested amendments in the original (Photoshop / InDesign / Illustrator / etc.) source. but ... given this thread: we're talking of archiving and editing a scientific archived article is imo "not done' Fine, that is a very different matter. and ... for texies, if then make a pdf, they do have a source, so they can fix the source and regenerate the pdf (which also keeps them in sync) Agreed. ** Phil.

On 1/28/2019 7:59 PM, Ross Moore wrote:

Hi Phil,

On 28/01/2019, at 22:44, "Taylor, P" mailto:P.Taylor@rhul.ac.uk> wrote:

...

Hans Hagen wrote:

...

...

All I can say is that since using AADC, editing PDFs has become a real pleasure. instead of fixing the source? (i'd probably opt for html then)

I /have /no source; I have a PDF, originated by a third party, which I need to amend (e.g., to illustrate what /can/ be achieved or what is desired).  I have done this with wine lists, advertisements, posters, etc., sent them back to the originator, and he or she has then been able to incorporate the requested amendments in the original (Photoshop / InDesign / Illustrator / etc.) source.

...

but ... given this thread: we're talking of archiving and editing a scientific archived article is imo "not done'

Why is it about *just* that? It is about adjusting pdfTeX to be able to produce PDFs that are useable in a variety of modern contexts. Editability is one of those aspects.

the "just" applies to the archiving: render a document, make sure it's ok, and put it in an archive where it's not touched (if it's a real archive)

As a lecturer and academic, I use (La)TeX for a wide variety of document types. Why should I need to use completely different software applications for class materials, lecture notes, student tests, examinations, performance reports, grant applications, conference reports, special demonstrations, and much more?

Well, this is indeed why editing pdf is not an issue is it? You use tex so you edit your source! This means that pdftex only has to create a valid pdf and not worry about it being edited.

The answer is that I do not have to with TeX, provided it allows to do all the special things in a reasonably consistent way, and all can exist compatibly within a PDF file. A feature that is obviously useful in one context could be useful also for other types of document.

Sure but editible means embedding whole fonts and such.

...

Fine, that is a very different matter.

No, it is not. Everything that Philip has said that he does with PDFs is very appropriate for this discussion.

He edits forms! That is widgets! And widgets are not typeset (well kind of pseudo typeset) ... just strings and choices wrapped in annotations. If one can edit and tweak them, fine, but here we're talkin gof text streams generated with a tex program (with all things fonts involved).

Essentially we need to keep TeX relevant to document production in the current age, and into the future.

It has been relevant for a few decades and will probably survide a few more. No one knows what happens from then on. If advanced html rendering had been around two decades ago, would tex be where it is now (not even pdf would be around i guess).

...

...

and ... for texies, if then make a pdf, they do have a source, so they can fix the source and regenerate the pdf (which also keeps them in sync)

Even when working collaboratively, you do not always have the full source. I frequently use APro to add hyperlinks and logos into otherwise sterile-looking PDFs produced with TeX by others, before making the

resulting PDF available, linked from public pages of a website. Hm, so archiving got a new meaning. I wonder how that works on the long run in academia: patching documents, maybe even presented results. Does

Hm, that is worrying ... that you have to mess with the pdf instead of reprocessing ... so tex is not that portable. But anyway, aghain we're not really talkling about editing content here: a logo is an image (no document fonts involved) and making a url active is applying an annotyation which again involves no document fonts. So, these don't relate to embedding fonts and charsets and ... which is what this thread is about. (btw, i'd rather ot bet tex's future on some commercial programs: just see what happens with media etc in pdf ... pretty much a waste of time to spend tex related dev time on that (in retrospect)) that also invalidate a reference to it? Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------

On 1/26/2019 12:01 AM, Karl Berry wrote:

If the FontDescriptor dictionary of an embedded Type 1 font contains a CharSet string, then

I see nothing in that wording that implies CharSet is anything but entirely optional.

that was my impression so the problem is that as soon a some validator complains we tend to 'fix' it (by inclusion again) but we really should be willing to doubt validators (interesting is that for validating the correctness of a pdf document in terms of xref and such, many pdf tools are pretty useless: when it comes to viewing, gs, mupdf, qpdf, xpdf, web stuff all have tolerance and/or recovery built in so they react differently ... only (old) acrobats can be trusted in that department)

Anyway, right now the choices are a) omit /CharSet or b) output a possibly-incorrect CharSet.

maybe we should just make a statement and always do (a) ... and as most folks use resonable modern viewers, just ignore a validators complain in that area

If you want to have a third option c) <something else>, you (or someone) will need to send me a patch. (I highly doubt that Thanh has time to look into this.) Sorry, but that's the reality. -k _______________________________________________ ntg-pdftex mailing list ntg-pdftex@ntg.nl https://mailman.ntg.nl/mailman/listinfo/ntg-pdftex

-- ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------

On 1/26/2019 12:27 PM, Ross Moore wrote:

...

the problem is that as soon a some validator complains we tend to 'fix' it (by inclusion again) but we really should be willing to doubt validators

I understand what you imply here, but the practicality is different. Students (postgrads mainly) produce theses using TeX which need to be submitted to their institution's library. The library system requires valid PDF/A. This validity is checked using Acrobat, since their institution has spent money for this.

Instead of complaining about the pdf they could better filter (unfinishedor invalid) crap from useful stuff and prevent further internet polution -) btw, i don't know how it's in other countries, but i wonder to what extent universities really care about long term ... it's not unusual to remove web pages (even email accounts) and such of employees that retire that they created (forcing them to host documents on another not future safe place) of students after a few years

Our own view on whether this is sensible or not has no bearing whatsoever.

Isn't it funny that a major of body of contributers of pdf's has no say in this? How competent are those making those decisions and demands. Just accepting a bit more sane standard (in the whole range of pdf standards) makes sense. I bet that at some point it will be okay anyway. (I also bet that these same institutions happily update to html 5 or whatever comes around.)

Hence we must *allow* for workflows that produce PDFs to any sensible published standard. We have to work out how to do this, else TeX will be considered a dinosaur. And we *are* smart enough to do this, provided we have the will to play this game properly.

well, it definitely makes pdf a dinosaur (or maybe it already is) (it's really funny that the standards permit other dinosauric, more complex and sort of useless, aspects of pdf) (anyway, i understand your pov; i never have to submit anything to any place; i normally add to context what users need to make the receiver happy, as long as i don't have to use it myself i don't bother too much) Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------

On 2019-01-26 at 10:04:30 -0700, Karl Berry wrote:

I gather the PDF/A-1 standard is not available online. -k

See https://github.com/veraPDF/veraPDF-validation-profiles/wiki It provides at least section 6 (Technical Requirements). Statements in sections "Requirement" are from the ISO specs. Regards, Reinhard -- ------------------------------------------------------------------ Reinhard Kotucha Phone: +49-511-3373112 Marschnerstr. 25 D-30167 Hannover mailto:reinhard.kotucha@web.de ------------------------------------------------------------------

On 2019-01-26 at 21:21:17 +0000, Ross Moore wrote:

On 27 Jan 2019, at 4:53 am, Reinhard Kotucha wrote:

...

https://github.com/veraPDF/veraPDF-validation-profiles/wiki

Thanks for this link, [...]

Mojca thankfully mentioned veraPDF some time ago. IMO veraPDF is quite promising because it seems to be developed like TeX (investigate thoroughly before writing the first line of code). Furthermore, an invaluable feature of veraPDF is that its source code is available. An online validator once complained about an erroneous /GlyphSet in a PDF file produced by LuaTeX. I extracted the font (CharisSIL) and disassembled the TTF with TTX/FontTools. Then I reverse-engineered the /GlyphSet and came to the conclusion that the /GlyphSet produced by LuaTeX is fully compliant with the PDF and PDF/A standards. The web site offered me to convert my document to PDF/A-1b but they simply removed /GlyphSet and *their* validator said that the PDF/A-1b file I created with *their* tool is correct. But without access to the source code I'll never be able to find out why the /GlyphSet was considered wrong. In short: I'm convinced that veraPDF is a very good thing. Keep in mind that Adobe Acrobat Pro isn't available on platforms used by the majority of TeX users. And most TeX users (not only at schools and universities) cannot afford proprietary software or are not willing to pay for it. Regards, Reinhard -- ------------------------------------------------------------------ Reinhard Kotucha Phone: +49-511-3373112 Marschnerstr. 25 D-30167 Hannover mailto:reinhard.kotucha@web.de ------------------------------------------------------------------

All I need is the name of the new primitive, and which values will correspond to include/exclude, \pdfincludecharset=0 (default) -> no /CharSet \pdfincludecharset=1 -> /CharSet included Just now committed to pdftex and TL repos. Hope it works out. -k

The only thing is that you might want to consider swapping the default, for backwards compatibility reasons - nothing to do with PDF/A. I wish you had answered my earlier question about that. Or maybe I missed it. Changing the default means messing with everything again, since I'd want to name it differently for consistency with many other parameters. So forget this name, I'll change it to \pdfomitcharset and commit again. Will let you know. -k

I disabled the /CharSet output in pdftexdir/writefont.c with a compile-time conditional ENABLE_PDF_CHARSET (r48169 in TeX Live), but left all the code in place just in case we need to reenable it. As discussed, there is no feasible way to guarantee its correctness. I surmise Akira will build a new pdftex for w32tex soon enough, and so we'll get some testing that way before the next release. Norbert, maybe you would like to insert that change for the next round of Debian updates too? BTW, I noticed that /CharSet was already output only for subsetted Type1 fonts, so nothing could have reliably depended on it always being present anyway. -k

Hi again, Just one more thing to add. On Jun 11, 2016, at 8:57 AM, Karl Berry mailto:karl@freefriends.org> wrote: This is the same problem as reported 3 years ago on SX: http://tex.stackexchange.com/questions/81927/charset-missing-or-incomplete-e... A couple of days ago, I requested it be reopened ... At any rate, I (or Thanh or whomever) need an example demonstrating the failure that can be run. Maybe you can make a self-contained set of files (the font(s) in needed formats, the tex source, etc.)? … and now we have the MWE, sent in me previous message. — When it gets through on the list, that is. It’s held up by a limit of 40kb !!!! I really do not think that anything related to generating PDF/A from LaTeX documents can anymore be described as being "too localized”. Maybe someone can add a comment saying something like: “we now understand the reason for this, and expect a fix to be available in TeXLive by late 2016 or 2017." We try to keep the type1 code in sync between dvips and pdftex, so it's sad that the results differ. Clearly something is awry :(. --thanks, karl. Cheers Ross Dr Ross Moore Mathematics Dept | Level 2, S2.638 AHH Macquarie University, NSW 2109, Australia T: +61 2 9850 8955 | F: +61 2 9850 8114tel:%2B61%202%209850%209695 M:+61 407 288 255tel:%2B61%20409%20125%20670 | E: ross.moore@mq.edu.aumailto:rick.minter@mq.edu.au http://www.maths.mq.edu.auhttp://mq.edu.au/ [cid:image001.png@01D030BE.D37A46F0]http://mq.edu.au/ CRICOS Provider Number 00002J. Think before you print. Please consider the environment before printing this email.http://mq.edu.au/ This message is intended for the addressee named and may contain confidential information. If you are not the intended recipient, please delete it and notify the sender. Views expressed in this message are those of the individual sender, and are not necessarily the views of Macquarie University.http://mq.edu.au/

On 6/11/2016 1:32 AM, Reinhard Kotucha wrote:

On 2016-06-10 at 03:58:05 +0000, Ross Moore wrote:

...

1. TeX cannot easily be persuaded to include /space within the font subset;

Dear Ross, if the font you are using contains a glyph for /space, it should be sufficient to type \char 32 somewhere in your document and the /space should appear within the font subset.

this is why long ago (so maybe forgotten) \pdfincludechars was introduced (because in widgets one also needs to have glyphs present for editing) (i remember using the 'put wanted glyph typeset white one meter in the margin outside mediabox' trick but that could confuse viewers) Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------