Steps to become a ConTeXt developer

Shiv Shankar Dayal

28 Aug 2024 28 Aug '24

3:16 a.m.

Hi, I am not sure if this is the best place to ask this question but here it goes. Suppose a C programmer wants to become a core ConTeXt developer then what steps one should follow. Assume that the programmer knows only the basics of LaTeX and never wrote macros etc using TeX or ConTeXt. -- Respect, Shiv Shankar Dayal

Show replies by date

Jethro Djan

28 Aug 28 Aug

4:43 a.m.

Hi, Not sure what you mean by a core ConTeXt developer. I think, as with anything, you get better at it by using it extensively. I would focus on a use case of interest and take advantage of ConTeXt’s unmatched extensibility to automate much of it as possible. Then you ask for help when you get stuck. If you want to contribute to ConTeXt, maybe start by having a look at the source and try to understand what’s going on (should be accessible to you since you are a C developer). If you identify a bug, or want to propose a feature, I am sure the community will support you however it can. Jethro On Wed, 28 Aug 2024 at 1:20 AM, Shiv Shankar Dayal < shivshankar.dayal@gmail.com> wrote:

...

Hi,

I am not sure if this is the best place to ask this question but here it goes.

Suppose a C programmer wants to become a core ConTeXt developer then what steps one should follow. Assume that the programmer knows only the basics of LaTeX and never wrote macros etc using TeX or ConTeXt.

-- Respect, Shiv Shankar Dayal

___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://mailman.ntg.nl/mailman3/lists/ntg-context.ntg.nl webpage : https://www.pragma-ade.nl / https://context.aanhet.net (mirror) archive : https://github.com/contextgarden/context wiki : https://wiki.contextgarden.net

___________________________________________________________________________________

Shiv Shankar Dayal

10:57 a.m.

On Wed, Aug 28, 2024 at 8:17 AM Jethro Djan wrote:

...

Hi,

Not sure what you mean by a core ConTeXt developer. I think, as with anything, you get better at it by using it extensively. I would focus on a use case of interest and take advantage of ConTeXt’s unmatched extensibility to automate much of it as possible. Then you ask for help when you get stuck.

If you want to contribute to ConTeXt, maybe start by having a look at the source and try to understand what’s going on (should be accessible to you since you are a C developer). If you identify a bug, or want to propose a feature, I am sure the community will support you however it can.

Jethro

I want to contribute to ConTeXt. I think that the steps for me should be: 1. Learn TeX 2. Learn LuaMetaTeX and then I can approach the C code. It won't make any sense if I just start reading the LuaMetaTeX code. -- Respect, Shiv Shankar Dayal

Henning Hraban Ramm

11:12 a.m.

Am 28.08.24 um 10:57 schrieb Shiv Shankar Dayal:

...

On Wed, Aug 28, 2024 at 8:17 AM Jethro Djan wrote:

...
Hi,

Not sure what you mean by a core ConTeXt developer. I think, as with anything, you get better at it by using it extensively. I would focus on a use case of interest and take advantage of ConTeXt’s unmatched extensibility to automate much of it as possible. Then you ask for help when you get stuck.

If you want to contribute to ConTeXt, maybe start by having a look at the source and try to understand what’s going on (should be accessible to you since you are a C developer). If you identify a bug, or want to propose a feature, I am sure the community will support you however it can.

Jethro

I want to contribute to ConTeXt. I think that the steps for me should be:

1. Learn TeX 2. Learn LuaMetaTeX

and then I can approach the C code. It won't make any sense if I just start reading the LuaMetaTeX code.

Sure, without a deep understanding of TeX it makes no sense to try anything with the C code. Also, most of ConTeXt is written in Lua. If you want to contribute, help with documentation – there are still too many commands and options undocumented in the wiki. If you don’t understand enough of ConTeXt to do that, anything deeper makes no sense anyway. Hraban

Hans Hagen

11:21 a.m.

On 8/28/2024 10:57 AM, Shiv Shankar Dayal wrote:

...

On Wed, Aug 28, 2024 at 8:17 AM Jethro Djan wrote:

...
Hi,

Not sure what you mean by a core ConTeXt developer. I think, as with anything, you get better at it by using it extensively. I would focus on a use case of interest and take advantage of ConTeXt’s unmatched extensibility to automate much of it as possible. Then you ask for help when you get stuck.

If you want to contribute to ConTeXt, maybe start by having a look at the source and try to understand what’s going on (should be accessible to you since you are a C developer). If you identify a bug, or want to propose a feature, I am sure the community will support you however it can.

Jethro

I want to contribute to ConTeXt. I think that the steps for me should be: Just curious: is there something that you miss.

...

1. Learn TeX 2. Learn LuaMetaTeX

Makes sense.

...

and then I can approach the C code. It won't make any sense if I just start reading the LuaMetaTeX code. Development of LuaMetaTeX (the engine) is mostly finished. We now entered the stage where we conduct experiment with the more difficult aspects (often unsolvable due to constraints). For that one needs to be quite a bit experienced in tex and the way context approaches things.

As an example, project like updating math took many man-years and intense testing, discussion and experimenting (so also quite some long term commitment). In a similar fashion (and related) we now are playing with the par and pagebuilder where user input and testing is required. Things like that get discussed at meetings and personal cummunication. The source code is under rather stict dev regime. We don't want to break compatibility, all has to fit into the user interface, performance might not be impacted significantly. We don't have an official dev team but everone knows what the de-facto team is (last week those that had time and were within reasonable traveling distance met at the context meeting) and it's also a kind of friends and fun thing where everyone brings in competence and inspiration. At such meetings we also set the objectives for the next year. That said, the usual way to come up with something that a user needs is to use a mix of lua, tex and metapost and wrap that in a module. That way the core is not crippled. Also, no all has to go into the engine but is done in Lua. Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------

Shiv Shankar Dayal

11:42 a.m.

...

Development of LuaMetaTeX (the engine) is mostly finished. We now entered the stage where we conduct experiment with the more difficult aspects (often unsolvable due to constraints). For that one needs to be quite a bit experienced in tex and the way context approaches things.

As an example, project like updating math took many man-years and intense testing, discussion and experimenting (so also quite some long term commitment). In a similar fashion (and related) we now are playing with the par and pagebuilder where user input and testing is required. Things like that get discussed at meetings and personal cummunication.

The source code is under rather stict dev regime. We don't want to break compatibility, all has to fit into the user interface, performance might not be impacted significantly. We don't have an official dev team but everone knows what the de-facto team is (last week those that had time and were within reasonable traveling distance met at the context meeting) and it's also a kind of friends and fun thing where everyone brings in competence and inspiration. At such meetings we also set the objectives for the next year.

That said, the usual way to come up with something that a user needs is to use a mix of lua, tex and metapost and wrap that in a module. That way the core is not crippled. Also, no all has to go into the engine but is done in Lua.

Hans

So I guess I need to learn TeX, Metapost and Lua. I know a bit of these but will go a bit deeper. Once I have learned these I will try to help by updating the Wiki documentation. -- Respect, Shiv Shankar Dayal

Mojca Miklavec

11:44 a.m.

Dear Shiv, On Wed, 28 Aug 2024 at 03:17, Shiv Shankar Dayal wrote:

...

Suppose a C programmer wants to become a core ConTeXt developer then what steps one should follow. Assume that the programmer knows only the basics of LaTeX and never wrote macros etc using TeX or ConTeXt.

I'm really sorry, please don't take it personally, but I really couldn't resist to say that the first thing that comes to my mind after reading your question is:

...

...
Hello, I'm Jia Tan and I would like to become a core XZ developer. Where can I start?

(Or perhaps: Hello, I'm John Doe and I would like to become the president of the USA. What steps should I follow to achieve that?) The question was asked in a super weird way and you cannot expect to get a satisfactory answer unless others understand exactly what you are asking. If you said "I really miss feature X in ConTeXt and would like to try to implement it. I'm familiar with Y, but don't know anything about Z yet. What pointers would you suggest to me to be able to achieve X?" Or if you asked "I would like to participate in Google Summer of Code. Because I really like ConTeXt, I would love to be able to dive deeper into ConTeXt development, so maybe you have some suggestions for a project that I could do?" Or "I love ConTeXt, but I have never written any macros. I would love to start a small business offering X to customers. What's the best way to learn?" ... then this would be somehow easier to answer. But your question is both unclear and a very strange way to start. You don't immediately aim for the core development. If anything, you start contributing patches etc, but you need to have some passion where you want to work on. You have been following this mailing list for a very very long time already, so you should be to some extent familiar with ConTeXt and what you would like to achieve. Be more specific. Mojca

Shiv Shankar Dayal

12:01 p.m.

...

You have been following this mailing list for a very very long time already, so you should be to some extent familiar with ConTeXt and what you would like to achieve. Be more specific.

Mojca

I have two specific interests. I want to make asymptote work with ConTeXt but I can live with including generated images and use PDFs. The second interest is in Aditya Mahajan's syntax highlighting module. It is very nice but it invokes VI making the entire thing slow. I understand the advantage of using VI is that we do not have to implement anything when a new language comes. VI will have syntax highlighting and it will be automatically done for us. -- Respect, Shiv Shankar Dayal

Mojca Miklavec

4:47 p.m.

On Wed, 28 Aug 2024 at 12:03, Shiv Shankar Dayal wrote:

...

I have two specific interests. I want to make asymptote work with ConTeXt but I can live with including generated images and use PDFs.

This is a much better specified question. It would ideally require some work both on the Asymptote side, as well as some work on the ConTeXt side. Asymptote is in fact written in C++, so it might be a lot closer to your particular expertise. I would say that what's most needed is someone with a great passion to get this complete, and some clear communication pathway between the two projects. Asymptote is very LaTeX-centric and should ideally be ConTeXt-ilized to simplify some operations. I would start by playing with a super simple minimum example written in plain asymptote. Figure out how to compile Asymptote on your computer and try to process the asy file into a ConTeXt file. This conversion then needs to be tweaked in order to produce two different results: - A standalone file that could be processed directly by ConTeXt. - A minimalistic file that could be included from another ConTeXt source to generate the final document. Check the file that you get right now, figure out what goes wrong, what could be improved ... If you have a simple file that no longer needs asymptote processing, you can check on the ConTeXt list what could still be improved, and then you can go and start fiddling with Asymptote source code (with help of Asy developers) until the files get progressively better. Iterate between the Asymptote and ConTeXt developers until the results are fully satisfactory. It has been a while since I touched any asymptote, but I suspect that what you want to achieve is something like \starttext \startasymptote some graphics \stopasymptote \placefigure[label]{really nice 3D model}{% \startasymptote another 3D graphics \stopasymptote} \stoptext where ConTeXt would process this file, prepare some input to asymptote, and then asymptote would have to be called just once to generate all the required graphics at once, and ConTeXt would eventually include them, properly scaled etc. Start with graphics, continue with simple text labels ... progress towards proper interactive 3D models ...

...

The second interest is in Aditya Mahajan's syntax highlighting module. It is very nice but it invokes VI making the entire thing slow. I understand the advantage of using VI is that we do not have to implement anything when a new language comes. VI will have syntax highlighting and it will be automatically done for us.

ConTeXt does support native syntax highlighting. You can try to create some new grammars to extend language coverage (Hans only made sure that TeX, lua, metapost etc. work correctly). Or you can try to start brainstorming whether some existing OpenSource grammars could be integrated to do syntax highlighting using Lua. Vi was a shortcut to get the syntax highlighting done in some way at all. In either case start a separate thread covering exactly that one topic. Mojca

Henning Hraban Ramm

5:06 p.m.

Am 28.08.24 um 16:47 schrieb Mojca Miklavec:

...

On Wed, 28 Aug 2024 at 12:03, Shiv Shankar Dayal wrote:

...
I have two specific interests. I want to make asymptote work with ConTeXt but I can live with including generated images and use PDFs.

There’s the built-in asymptote module: https://source.contextgarden.net/tex/context/modules/mkiv/m-asymptote.mkiv I don’t know if it still works and if it could/should be improved. There’s also Aditya’s filter module that alleviates calling external programs (I’m using it with LilyPond): https://github.com/adityam/filter But calling external programs is slow. MetaPost is fast, (also) because it is integrated as a library. It’s possible to address C libraries using Lua’s FFI (don’t ask me how, there is/was a manual). Also, Hans announced to include some "2.5D" functions (perspective projections) into MetaFun soon. That might obsolete some uses of Asymptote. But a proper 3D model for MetaPost seems to be missing (not my expertise).

...

...
The second interest is in Aditya Mahajan's syntax highlighting module. It is very nice but it invokes VI making the entire thing slow. I understand the advantage of using VI is that we do not have to implement anything when a new language comes. VI will have syntax highlighting and it will be automatically done for us.

ConTeXt does support native syntax highlighting. You can try to create some new grammars to extend language coverage (Hans only made sure that TeX, lua, metapost etc. work correctly). Or you can try to start brainstorming whether some existing OpenSource grammars could be integrated to do syntax highlighting using Lua.

Vi was a shortcut to get the syntax highlighting done in some way at all.

There’s also the scite module that uses SciTE’s lexers for syntax highlighting. The builtin highlighters are limited to TeX/ConTeXt, MetaPost, Lua and XML. Hraban

Hans Hagen

8:18 p.m.

On 8/28/2024 5:06 PM, Henning Hraban Ramm wrote:

...

Also, Hans announced to include some "2.5D" functions (perspective projections) into MetaFun soon. That might obsolete some uses of Asymptote. But a proper 3D model for MetaPost seems to be missing (not my expertise).

Indeed, Mikael and I have plans to look into it ... a pet project for metapost lovers.

...

There’s also the scite module that uses SciTE’s lexers for syntax highlighting. The builtin highlighters are limited to TeX/ConTeXt, MetaPost, Lua and XML. They are actually just lpeg lexers that we made for use in scite so no scite installation is needed.

Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------

Wolfgang Schuster

8 Sep 8 Sep

12:12 p.m.

Hans Hagen via ntg-context schrieb am 28.08.2024 um 20:18:

...

On 8/28/2024 5:06 PM, Henning Hraban Ramm wrote:

...
There’s also the scite module that uses SciTE’s lexers for syntax highlighting. The builtin highlighters are limited to TeX/ConTeXt, MetaPost, Lua and XML. They are actually just lpeg lexers that we made for use in scite so no scite installation is needed.

Which highlighter should extended when support for additional languages is needed, the built-in one or the scite version? Wolfgang

Henning Hraban Ramm

3 p.m.

New subject: syntax highlighting (was: Steps to become a ConTeXt developer)

Am 08.09.24 um 12:12 schrieb Wolfgang Schuster:

...

Hans Hagen via ntg-context schrieb am 28.08.2024 um 20:18:

...
On 8/28/2024 5:06 PM, Henning Hraban Ramm wrote:

...
There’s also the scite module that uses SciTE’s lexers for syntax highlighting. The builtin highlighters are limited to TeX/ConTeXt, MetaPost, Lua and XML. They are actually just lpeg lexers that we made for use in scite so no scite installation is needed.

Which highlighter should extended when support for additional languages is needed, the built-in one or the scite version?

The scite module doesn’t support escaping, and the TeX highlights of both are conflicting. It would be nice if that could be fixed. (I need the scite highlighters for e.g. bash and escaping in TeX highlighting, ATM I can’t have both.) Hraban

Hans Hagen

5:01 p.m.

...

Hans Hagen via ntg-context schrieb am 28.08.2024 um 20:18:

...
On 8/28/2024 5:06 PM, Henning Hraban Ramm wrote:

...
There’s also the scite module that uses SciTE’s lexers for syntax highlighting. The builtin highlighters are limited to TeX/ConTeXt, MetaPost, Lua and XML. They are actually just lpeg lexers that we made for use in scite so no scite installation is needed.

Which highlighter should extended when support for additional languages is needed, the built-in one or the scite version?

On 9/8/2024 12:12 PM, Wolfgang Schuster wrote: probably easiest is the built in but if one wants to nest them then the scite ones are aeasier (in a way the scite ones are like the built in ones but they provide data structures that accomodate scite and as a side effect i then added nexting) for most languages the simple ones are good enough (no need to have real parsers, just coloring keywords an dspecial symbols) Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------

Hans Hagen

28 Aug 28 Aug

8:15 p.m.

On 8/28/2024 12:01 PM, Shiv Shankar Dayal wrote:

...

...
You have been following this mailing list for a very very long time already, so you should be to some extent familiar with ConTeXt and what you would like to achieve. Be more specific.

Mojca

I have two specific interests. I want to make asymptote work with ConTeXt but I can live with including generated images and use PDFs.

There is zero change that that will be integrated as it introduces a (long term) dependency. It's pretty much geared at latex anyway and we would need some backend that fits in. We already have metapost (and will extend that over time). It is however possible to load libraries at runtime as we can do with some optional libraries (these linbraries themselves are not included either but loaded on demand). One can add extra ones locally in which case we don't have to worry about the style of their interface. Of course that also means maintaining them as one needs to keep an eye on (changing) api's. Keep in mind that if you do that you need to maintain it for decades to come. We also have a foreign interface (no ffi but kind of) but afaik no one ever showed interest in that. With respect to external pdf files ... there are several ways to optimize that and only process files when they change as we do with other inclusions.

...

The second interest is in Aditya Mahajan's syntax highlighting module. It is very nice but it invokes VI making the entire thing slow. I understand the advantage of using VI is that we do not have to implement anything when a new language comes. VI will have syntax highlighting and it will be automatically done for us.

Again, when one caches these snippets there is little overhead involved. One could look into some socket communication but language servers tend to be huge and evolving. But of course one can play with that in a local variant, after all the engine code is included in the context distribution. Syntax highlighting is kind of personal anyway, we have soem 'reference'highlighting (which reflects how contetx is codes) but that is also personal. No solution suits all. Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------

Age (days ago)

Last active (days ago)

List overview

Download

14 comments

7 participants

participants (7)

Hans Hagen
Hans Hagen
Henning Hraban Ramm
Jethro Djan
Mojca Miklavec
Shiv Shankar Dayal
Wolfgang Schuster

Steps to become a ConTeXt developer

tags

participants (7)