Hans Hagen
in cases where you use tex as processsing engine one can avoid most tex and especially nasty parsing by just keeping the tricky stuff at the lua end or in the case of lilypond at the lisp end: just avoid parsing at the tex end and spit out predictable tex code ... no user is going to see it
The whole point was that you don't need to stay locked into Scheme for doing work you'd rather do in LilyPond. An extension language should be more than a scripting language you use for preparing batched work offline. Maybe comparing URL:http://news.lilynet.net/?The-LilyPond-Report-23#feature_story_prelude_1_... with the referenced URL:http://nicolas.sceaux.free.fr/prelude/prelude.html makes this point clearer. The pipelining (rather than lexer switching) approach used by LuaTeX also means that error messages related to Lua/TeX interoperation are useless: they can't refer to the original source location since that has been lost during tokenizing and detokenizing. That's also very user unfriendly.
i'm not so sure if messing with the input parser helps out ... and any special treatment of for instance \directlua's argument would slow down and for sure break already present code;
So you don't call the command \directlua. Problem solved.
there are some helpers (like \luaescapestring) and one can do catcode juggling as well as detokenize but in the end, \directlua is just a predictable simple and efficient interface.
"simple" in the implementation but not simple for doing actual interfacing and using the language for_extending_ the abilities for working in TeX rather than _replacing_ them.
at some point taco and i discussed a tex primitive that would sort of map more directly onto a function for instance by (optionally) avoiding tokenization (if possible)
but it's all very low priority stuff compared to other pending (opening up) issues and more something for > 1.00
Well, let's just say that for LilyPond it has shown to be an important enough issue regarding usability to make private users keep throwing significant amounts of money at me because that kind of work is what empowers people with a user rather than a programmer background. The kind of interfaces LuaTeX offers nowadays are appealing to people for whom TeX is too restricted. But that's not an area with mass appeal. Lua has a lot of potential to appeal to people for whom TeX is too _complex_. Those who find themselves _lost_ with TeX, not those who know it so well that they can work with the added complexity of the LuaTeX interface, juggling with catcode regimes and whatever else you need to understand how to do even simple tasks reliably. For example, it is quite silly that LuaTeX (at one point of time?) modified Lua's print routine so that it would output 1e-7 as 0.0. Why would you even want to turn a number into a _printable_ representation? Why not let Lua instead _return_ or pipeline a dimendef token with the right value? If your value is a dimension, you don't want it disappearing in a comment or be combined with a preceding ^^ or have its decimal point be interpreted as an active character doing something else. You want it to be a dimension. You are making LuaTeX interesting for the best of the TeX programmers. It would have the potential to be interesting for the worst of the TeX programmers... -- David Kastrup