L A TEX3 News

(1)

L ^A TEX3 News, Issues 1–12

L ^A TEX3 News

Issue 1, February 2009

Welcome to L

^A

TEX3

Momentum is again starting to build behind The LÂTEX Project. For the last few releases of TEX Live, the experimental programming foundation for LÂTEX3 has been available under the nameexpl3. Despite large warnings that the code would probably change in the future, we wanted to show that there was progress being made, no matter how slowly. Since then, some people have looked at the code, provided feedback, and — most importantly — actually tried using it. Although it is yet early days, we believe that the ideas behind the code are sound and there are only ‘cosmetic improvements’ that need to be made before expl3is ready for the LÂTEX package author masses.

What currently exists

The current L^ATEX3 code consists of two main branches:

theexpl3modules that define the underlying programming environment, and the ‘xpackages’, which are a suite of packages that are written with theexpl3 programming interface and provide some higher-level functionality for what will one day become L^ATEX3 proper.

Both expl3and parts of thexpackagesare designed to be used on topof L^ATEX 2ε, so new packages can take advantage of the new features while still allowing to be used alongside many of the vast number of L^ATEX 2ε packages on ctan.

What’s happening now

In preparation for a minor overhaul of theexpl3 code, we are writing a comprehensive test suite for each module. These tests allow us to make implementation changes and then test if the code still works as before.

They are also highlighting any minor shortcomings or omissions in the code. As the tests are being written, our assumptions about what should be called what and the underlying naming conventions for the functions and datatypes are being questioned, challenged, and noted for further rumination.

At the time of writing, we are approximately half-way through writing the test suite. Once this task is complete, which we plan for the first half of 2009, we will be ready to make changes without worrying about breaking anything.

What’s happening soon

So what do we want to change? The current expl3 codebase has portions that date to the pre-L^ATEX 2ε days, while other modules have been more recently con- ceived. It is quite apparent when reading through the sources that some unification and tidying up would improve the simplicity and consistency of the code. In many cases, such changes will mean nothing more than a tweak or a rename.

Beyond these minor changes, we are also re-thinking the exact notation behind the way functions are defined. There are currently a handful of different types of arguments that functions may be passed (from an untouched single token to a complete expansion of a token list) and we’re not entirely happy with how the original choices have evolved now that the system has grown somewhat. We have received good feedback from several people on ways that we could improve the argument syntax, and as part of the upcoming changes to theexpl3packages we hope to address the problems that we currently perceive in the present syntax.

What’s happening later

After the changes discussed above are finished, we will begin freezing the core interface of the expl3mod- ules, and we hope that more package authors will be interested in using the new ideas to write their own code. While the core functions will then remain un- changed, more features and new modules will be added as L^ATEX3 starts to grow.

Some new and/or experimental packages will be changing to use theexpl3programming interface, including breqn,mathtools,empheq, fontspec, andunicode-math.

(Which is one reason for the lack of progress in these latter two in recent times.) There will also be a version of thesiunitxpackage written in expl3, in parallel to the current LÂTEX 2ε version. These developments will provide improvements to everyday LÂTEX users who haven’t even heard of The LÂTEX Project.

Looking towards the long term, L^ATEX3 as a document preparation system needs to be written almost from scratch. A high-level user syntax needs to be designed and scores of packages will be used as inspiration for the ‘out-of-the-box’ default document templates.

L^ATEX 2ε has stood up to the test of time — some fifteen years and still going strong — and it is now time to write a successor that will survive another score.

(4)

L TEX3 News

Issue 2, June 2009

TEX Live and the expl3 code

TEX Live 2009 is almost upon us, and the L^ATEX3 team have been readying a new release of the experimental L^ATEX3 code for this. Very dramatic changes have occurred since the last public release of the code in TEX Live 2008; no backwards compatibility has been maintained (as warned in the beginning of the documentation) but we believe the changes made are all much for the better. Almost every single part of expl3 has been scrutinized, resulting in a far more coherent code base.

Theexpl3code is now considered to be much more stable than it was before; a comprehensive test suite has been written that helps to ensure that we don’t make any mistakes as we change things in the future. In the process of writing the test suite, many minor bugs were fixed; we recommend such test suites for all similar de- velopmental projects! Some small underlying changes are still expected in theexpl3code, but major, disrup- tive, changes aren’t planned.

Planned updates

Until now, the last update toctanof theexpl3 bun- dle was for TEX Live 2008. Now that work on the code is happening on a semi-steady basis, we plan to keep updates rolling out toctanmore frequently. This will allow anyone who wishes to experiment with the new code to use the TEX Live or MiKTEX updaters to install a recent version without having to ‘check out’ the svn repository and install the packages manually.

New members

We didn’t say anything about it in the last status update, but Joseph Wright and Will Robertson are now members of the L^ATEX Team. They have been working fairly exclusively on the expl3code.

It’s worth repeating that L^ATEX 2ε is essentially frozen in order to prevent any backwards compatibility problems. As desirable as it is to benefit from the new features offered by new engines X E TEX and LuaTEX, we cannot risk the stability of production servers running older versions of L^ATEX 2εwhich will inevitably end up processing documents written into the future.

L^ATEX3 will not be inheriting the same restraints, so stay tuned.

Some specifics

Morten Høgholm will be presenting the recent changes in much more detail at tug2009. Here are some quick specifics for those interested. New code written and broad changes made to theexpl3modules:

More logical function names Many function names that were hold-outs from the TEX naming system have been changed to fit into the more logical scheme of expl3; e.g.,\def:Npnand\let:NNare now\cs_set:Npnand\cs_set_eq:NN.

Defining functions and conditionals Much thought was put into new ways to define functions and conditionals with a minimum of code. See\cs_set:Nn and\prg_set_conditional:Nnn.

Smart comparisons Comparisons can be made much more easily now, with familiar notation such as

\intexpr_compare_p:n{ #1+3 != \l_tmpa_int }.

Data from variables A new function argument spec- ifierVhas been added for extracting information from variables of different types, without needing to know the underlying variable structure. Some other tidy-ups on the argument specifiers offered, partially as a result of the addition of this new one.

l3msg New module to deal with communication between L^ATEX3 code and the user (info messages, warnings, and errors), including message filtering partially inspired by thesilencepackage.

The next six months

Having overhauled theexpl3code, we now plan to perform an analogous process with the foundations of the xpackages. These are the higher-level packages that will provide the basic needs such as control of the page layout and rich document-level interaction with the user. As the groundwork for this layer of the document processing matures, we will be able to start building more packages for a L^ATEX3 kernel; these packages will also be usable on top of L^ATEX 2εand serve as broadly customisable templates for future document design.

As gaps in the functionality offered by expl3are found (in some cases, we know that they exist already), the programming layer will be extended to support our needs. In other cases, wrappers around TEX functions that can be more usefully handled at a higher level will be written.

(5)

In terms of what we’re planning to work on next, three xpackages will take the focus of our attention.

xbase ‘xbase’ is actually two packages: xparseandtem- plate. These contain code for, respectively, defining new document commands (such that a user would use; e.g.,\section,\makebox, . . . ) and for handling keyval lists for user input and document specification. xparsewas presented attug1999¹ and Lars Hellstr¨om wrote some notes ontemplate in 2000². Functionality coverage for these packages is good but concepts need a good “airing”.

There are various approaches taken for keyval input, some more recent than the templatecode, so there are some alternatives to evaluate.

galley2 Sophisticated handling for constructing para- graphs and other document elements. Morten spoke on this at tug2008³. Design needs to be revisited after some stress testing.

xor This is the L^ATEX3 output routine for splitting the galley into page and sub-page sized chunks. Ideas and code need work to move to “production ready”

status. Early developments with this package were published by Frank in 2000⁴.

Expect to hear again from us at Christmas. If you’d like to discuss any of these ideas, please join us on the latex-l mailing list⁵.

1http://www.latex-project.org/papers/tug99.pdf

2http://www.latex-project.org/papers/template-notes.pdf

3http://river-valley.tv/the-galley-module/

4http://www.latex-project.org/papers/xo-pfloat.pdf

5http://www.latex-project.org/code.html

–5

(6)

L TEX3 News

Issue 3, January 2010

Happy New Year

Welcome to the holiday season edition of ‘news of our activities’ for the L^ATEX3 team.

Recent developments

The last six months has seen two significant releases in the L^ATEX3 code. In thectanrepository for thexpackages,¹you’ll find two items of interest:

A revised version of xparse; and

The new packagextemplate, a re-implementation of template with a new syntax.

Special thanks to Joseph Wright who handled the implementations above almost single-handedly (with lots of input and feedback from other members of the team and members of the latex-lmailing list).

These two packages are designed for the L^ATEX package author who wishes to define document commands and designer interfaces in a high-level manner.

xparse This package allows complex document commands to be constructed with all sorts of optional arguments and flags. Think of how \newcommandallows you to create a command with a single optional argument and xparseis a generalisation of that idea.

xtemplate This package requires more explanation.

Xtemplate is designed to separate the logical information in a document from its visual representation.

‘Templates’ are constructed to fulfil individual typesetting requirements for each set of arguments; to change the look of a certain part of a document, instantiations of templates can be swapped out for another without (a) having to change the markup of the source document, or (b) having to edit some internal L^ATEX macro.

L^ATEX 2εpackages, such asgeometry ortitlesec, already provide parameterized interfaces to specific document elements. For example, one may usetitlesecto change the layout of a \section: one modifies its layout parameters via \titleformatand\titlespacing. In a way, such packages define a template for a specific document element and some manipulation commands to instantiate it. However, the moment the intended lay-

1http://mirror.ctan.org/tex-archive/macros/latex/

contrib/xpackages/

out is not achievable with one package you are on your own: either you have to resort to low-level programming or find some other high-level package which, of course, comes with its own set of conventions and manipulation commands.

Thextemplatepackage can be thought of a generaliza- tion of such ideas. It provides a uniform interface for defining and managing templates for any kind of document element and most importantly provides a uniform interface for instantiating the layout.

Thus the designer activity of defining or modifying a document class is limited to selecting the document elements that should be provided by the class (e.g.,

\chapter,\section \footnote, lists, . . . ), selecting appropriate “named” templates for each of them, and instantiating these templates by specifying values for their layout parameters. If a desired layout can’t be achieved with a given template a different template for the same document element can be selected.

Programming is only necessary if no suitable template for the intended layout is available. It is then that a L^ATEX programmer has to build a new template that supports the layout requirements. Once this task is complete, the template may be added to the selection of templates that designers and users may choose from to define or adjust document layouts seamlessly.

This is a slight gloss over the complexities of the package itself, which you can read about in the documentation. We’ve tried to documentxtemplateclearly but we’d love feedback on whether the ideas make sense to you.

As an addendum to the introduction of xtemplate, the older templatepackage will be retired in the near future. To our knowledge there is only a single package onctanthat usestemplate, namely xfrac, and members of the L^ATEX team are in the process of switching this package over toxtemplate. If you have any private code that usestemplate, please let us know!

Upcoming plans

Having announced the updatedxparseand the new xtemplate, the next stage of development will revolve around using these two systems in the other components of thexpackages, feeding back our experience in practise with the original ideas behind the designs and evaluating if the packages are meeting our expectations.

(7)

Packages to tackle

xhead The first work will be to create a new xpackage (probably calledxhead), for typesetting section headings and other document divisions. Section headings are one of the more complex areas to work with, so the work should stressxtemplate enough to know if its current design is sufficient for most needs. Nothing has been released yet, but we’ll announce further developments on the latex-lmailing list² in the mean time.

galley We also need to givegalley the same treatment as xparseandxtemplatehave already had. That is, we have an older implementation (in fact two) that needs some work before we’re ready to release it to ctan. Thegalley package is used to place material into the vertical list while typesetting but before page breaks occur. Since it works at such a low level, it is important to solidify this package before writing higher level design templates.

An issue we have to face is that to achieve best results, galley cannot be used in concert with L^ATEX 2ε code.

This could limit its usefulness, and we may decide that it’s better to scale back the features we’re attempting, to allow better interoperability for existing packages and documents. More work remains before we can decide between these options.

2For details, seehttp://www.latex-project.org/code.html

–7

(8)

L TEX3 News

Issue 4, July 2010

Now that we’re back from the TEX Users Group conference in San Francisco, it’s time to discuss what’s been going on over the last six months. Due to some extra travel plans after the conference, this issue is slightly late in coming out.

expl3 in practice

Joseph Wright and Will Robertson have both released significant new versions of their packages, resp.,siu- nitxandfontspec. These have been re-written in the L^ATEX3 programming languageexpl3, which we have discussed here previously. Usingexpl3for production code has been very successful, both in demonstrating that the concepts are sound and highlighting areas that still need some attention.

In the case of fontspec,expl3programming is being used to target L^ATEX running on either X E TEX and LuaTEX. In the latter case, the package is a mixture of Lua code andexpl3code; Will presented theunicode- math package at TUG 2010, which is developed in the same style.

New xpackages

Frank Mittelbach has started to work on a new experimental L^ATEX3 packagexheadthat provides templates for one of the most complex areas of document design: section headings and document divisions. This is the beginning of an ambitious idea to map out the requirements for typesetting most documents currently processed with L^ATEX.

One of the challenges here is providing a “natural” design language for describing the two-dimensional spatial relationships of objects participating in the design, e.g., the placement of a heading number in relation to the heading title, a possible sub-title, etc. In answer to this challenge Frank developed the xcoffinpackage, which he presented at TUG 2010. It is designed as a high- level interface for placing and aligning boxes on a page, allowing a ‘designer’s approach’ for indicating the po- sitional relationship between boxes. (A ‘coffin’ is a box with handles.) As an example, it is possible to repre- sent ideas such as ‘align the lower-left corner of box A with the upper-right corner of box B after rotating it ninety degrees’, without having to calculate the inter- mediate positions.

We expect a future version of xcoffin(after some further work on its interface layer and its internal imple-

mentation) to play a major role in all packages providing layout templates for higher-level document objects, such as table of contents designs, floats, etc.

Finally, Joseph Wright has begun work with the current

‘galley’ packages, producing the new, minimal,xgalley based onxfm-galleyas a testbed for what we need and what will work.

Developments with expl3

Meanwhile, Joseph’salso been writing a new floating- point calculation module, calledl3fp, forexpl3. This module allows manipulation and calculation of numbers with a much larger range than TEX allows naturally.

Thel3fpmodule has already been utilised in thexcoffin code for calculatations such as coordinate rotations and intersection points of vectors.

The modulesl3ioand l3filehave been revised, rethink- ing the way that read and write streams are dealt with. TEX has a hard limit of sixteen input and output streams open at any one time, and the new implementation forexpl3provides more flexibility in how they are allocated; there’s now much less chance of running into a ‘No room for a new \read’ (or \write) error.

Sometimes we discuss ideas forexpl3 thatdon’t end up making it into the final code. One example of this is the concept of having ‘local registers’ for integers, boxes, and so on, that do not survive outside of the group they are defined in (in contrast to Plain TEX and L^ATEX, where allocators such as\newcountand

\newboxare always global). Despite the scope for some small benefit, we decided that the extra complexity that the additional functions required, in both syntax and documentation, was not justified.

TUG 2010 reflections

Our interpretation of the broad themes discussed at the conference are that TEX-based systems are still thriving and there are some big problems to solve with robust solutions to transform LÂTEX source, including mathematics, into a form such as HTML. While there are big pushes for standardising various aspects of the LÂTEX syntax, we also believe that it is LÂTEX’s very flexibility—its inherently non-standardised markup—

that has allowed it to survive for so many years. There is a delicate trade-off here between moving forward into more standards-based territory while also retaining the extensibility of the third-party package system.

(9)

L ^A TEX3 News

Issue 5, January 2011

Happy new year

Seasons greetings for 2011! As the previous news issue was released late, this season’s issue will be shorter than usual.

The LPPL is now OSI-approved

We are happy to report that earlier this year the L^ATEX Project Public License (LPPL) has been approved by the OSI as an open source licence.¹ Frank Mittelbach will be publishing further details on this news in a ret- rospective on the LPPL in an upcoming TUGboat article.

Reflections on 2010

We are pleased to see the continued development and discussion in the TEX world. The L^ATEX ecosystem continues to see new developments and a selection of notable news from the second half of last year include:

June The TUG 2010 conference was held very successfully in San Francisco; videos, slides, and papers from L^ATEX3 Project members are available from our website.²

Aug. The TEX Stack Exchange³ question & answer website was created and has since grown quickly. At time of writing, some 2800 people have asked 2600 questions with 5600 answers total, and 2200 users are currently visiting daily.

Sept. TEX Live 2010 was released: each year the ship- ping date is earlier; the production process is be- coming more streamlined and we congratulate all involved for their hard work. One of the most notable new components of TEX Live 2010 includes the ‘restricted shell escape’ feature to allow, among other things, automatic EPS figure conversion for pdfL^ATEX documents.

Oct. TLContrib⁴was opened by Taco Hoekwater as a way to update a TEX Live installation with material that is not distributable through tlmgritself.

Such material includes executables (e.g., new versions of LuaTEX), non-free code, or test versions of packages.

1http://www.opensource.org/licenses/lppl

2http://www.latex-project.org/papers/

3http://tex.stackexchange.com

4http://tlcontrib.metatex.org/

Nov. Philipp Lehman released the first stable version of biblatex. One of the most ambitious L^ATEX packages in recent memory,biblatexis a highly flexible package for managing citation cross-referencing and bibliography typesetting. In ‘beta’ status for some years now, reaching this point is a great mile- stone.

Dec. LuaTEX 0.65. We are happy to see LuaTEX development steadily continuing. L^ATEX users may use LuaTEX with thelualatexprogram. Like xelatex, this allows L^ATEX documents to use mul- tilingual OpenType fonts and Unicode text input.

Current progress

Theexpl3programming modules continue to see revi- sion and expansion; we have added a LuaTEX module, butexpl3 continues to support all three of pdfLÂTEX, X E LÂTEX, and LuaLÂTEX equally.

Thel3fpmodule for performing floating-point arith- metic has been extended and improved. Floating point maths is important for some of the calculations required for complex box typesetting performed in the new ‘coffins’ code. Thel3coffinmodule has been added based on the originalxcoffinspackage introduced at TUG 2010 as reported in the last news issue; this code is now available from CTAN for testing and feedback.

We have consolidated thel3intandl3intexprmod- ules (which were separate for historical purposes);

all integer/count-related functions are now contained within the ‘int’ code and have prefix\int_. Backwards compatibility is provided for, but eventually we will drop support for the older\intexpr_function names.

Plans for 2011

In the following year, we plan to use the current L^ATEX3 infrastructure to continue work in building high-level code for designing L^ATEX documents using thextemplate package. Our first priority is to look at section headings and document divisions, as we see this area as one of the most difficult, design-wise, of the areas to address.

From there we will broaden our scope to more document elements.

We will also do some low-level work on the ‘galley’, which is the code that L^ATEX3 uses to build material for constructing pages, and we will continue to extend expl3into a more complete system from which we can, one day, create a pure L^ATEX3 format.

(10)

L TEX3 News

Issue 6, June 2011

A key aim of releasing ‘stable’ L^ATEX3 material to CTAN is to allow users to benefit from new ideas now, and also to raise the profile of usable L^ATEX3 ideas.

This is clearly being successful, with xparsebeing of particular utility to end users. This increase in interest has been particularly notable on the new TeX.SX Q&A site.

The L

^A

TEX3 Team expands

Raising interest in L^ATEX3 developments has inevitably led to feedback on cases where the code base has required attention. It has also attracted new programmers to using L^ATEX3 ideas, some more than others!

Bruno Le Floch has over the past few months made many useful contributions to L^ATEX3, and we are very pleased that he has recently joined The LATEX Project.

Bruno has taken a particular interest in improving the performance and reliability of theexpl3language. This has already resulted in new implementations for the prop andseqdata types. At the same time, he has identified and fixed several edge-case issues in core expl3 macros.

The ‘Big Bang’

In parallel to Bruno’s improvements, Joseph Wright ini- tiated a series of ‘Big Bang’ improvements to L^ATEX3.

The aim of the Big Bang was to address a number of long-standing issues with the L^ATEX3 code base. Devel- opment has taken place over many years, with the status of some of the resulting code being less than clear, even to members of The L^ATEX Project! At the same time, different conventions had been applied to different parts of the code, which made reading some of the code rather ‘interesting’. A key part of the Big Bang has been to address these issues, cleaning up the existing code and ensuring that the status of each part is clear.

The arrangement of L^ATEX3 code is now the same in the development repository and on CTAN, and splits the code into three parts.

l3kernel The core of LATEX3, code which is expected to be used in a L^ATEX3 kernel in more or less the current form. Currently, this part is made up of the L^ATEX3 programming layer,expl3.

l3packages L^ATEX 2εpackages making use of L^ATEX3 concepts and with stable interfaces. Thexparse

andxtemplatepackages are the core of this area.

While many of theideas explored here may eventually appear in a L^ATEX3 kernel, the interfaces here are tied to L^ATEX 2ε.

l3experimental L^ATEX 2ε packages which explore more experimental L^ATEX3 ideas, and which may see interface changes as development continues. Over time, we expect code to move from this area to ei- therl3kernelorl3packages, as appropriate.

In addition to these release areas, the development code also features al3trialsection for exploring code ideas.

Code inl3trialmay be used to improve or replace other parts of L^ATEX3, or may simply be dropped!

As well as these improvements to thecode used in L^ATEX3, much of the documentation forexpl3 has been made more precise as part of the Big Bang. This means thatsource3.pdf is now rather longer than it was previously, but also should mean that many of the inaccu- racies in earlier versions have been removed. Of course, we are very pleased to receive suggestions for further improvement.

L

^A

TEX3 on GitHub

The core development repository for L^ATEX3 is held in an SVN repository, which ispublicly viewablevia the Project website. However, this interface misses out on some of the ‘bells and whistles’ of newer code-hosting sites such as GitHubandBitBucket. We have therefore established a mirror of the master repository on GitHub¹. This is kept in synchronisation with the main SVN repository by Will Robertson (or at least by his laptop!).

The GitHub mirror offers several useful features for people who wish to follow the L^ATEX3 code changes.

GitHub offers facilities such as highlighted differences and notification of changes. It also makes it possible for non-Team members to submit patches for L^ATEX3 as

‘pull requests’ on GitHub.

As well as offering a convenient interface to the LÂTEX3 code, the GitHub site also includes an issue database². Given the very active nature of LÂTEX3 development, and the transitory nature of many of the issues, this provides a better approach to tracking issues than the main LÂTEX bug database³. Developers and users are

1http://github.com/latex3/svn-mirror

2http://github.com/latex3/svn-mirror/issues

3http://www.latex-project.org/bugs.html

(11)

therefore asked to report any issues with L^ATEX3 code via the GitHub database, rather than on the main Project homepage. Discussion on the LaTeX-L mailing listis also encouraged.

Next steps

The ‘Big Bang’ involves making a number of changes to expl3function names, and is likely to break at least some third-party code. As a result, the updates will not appear on the TEX Live 2011 DVD release, but will instead be added to TEX Live once regular updates restart (probably August).

Bruno is working on a significant overhaul of the l3fp floating-point unit for L^ATEX3. He has developed an approach which allows expandable parsing of floating- point expressions, which will eventually allow syntax such as

\fp_parse:n { 3 * 4 ( ln(5) + 1 ) } This will result in some changes in the interface for floating-point numbers, but we feel that the long-term benefit is worth a small amount of recoding in other areas.

Joseph has completed documentation of the xgalley module, and this is currently being discussed. Joseph is hoping to move on to implement other more visible ideas based on thextemplateconcept over the next few months.

–11

(12)

L TEX3 News

Issue 7, February 2012

After the ‘Big Bang’

The last L^ATEX3 News gave details of the ‘Big Bang’, in which the team have revised the layout and coverage of the L^ATEX3 codebase. This process has made the status of different modules clearer, so that both the team themselves and everyone else know what is going on.

The ‘Big Bang’ changes were not shipped toctanun- til after the TEX Live 2011 freeze, as we did not want to end up with advd containing badly broken code.

The update went toctansoon after TEX Live 2011 shipped, and has now propagated around the world.

The new package naming (l3kernel,l3packages and l3experimental) has caused some surprises for a small number of users, but there have not been any major issues with the changes at the code level.

The ‘Big Bang’ has attracted attention from programmers outside of the L^ATEX3 team, with useful feedback arriving on theLaTeX-Llist and TeX.sx, in particular.

One area that this has highlighted is the need to document carefully when changes to the ‘stable’ parts of the L^ATEX3 codebase occur. All changes tol3kernelnow come with an explicit date for the change in the documentation, which means that programmers can check exactly when the features they want were introduced.

Another key part of supporting LÂTEX3 use beyond the team is making it easy to check on the version of LÂTEX3 installed. To support that, the file date of the main expl3package is now set each time there is a release of the LÂTEX3 material toctan. This means that the LÂTEX 2ε \@ifpackagelatertest can be used reli- ably to detect if the installed version of LÂTEX3 is going to supply the functions that a programmer is using.

Deforming boxes

Additions to both the L^ATEX3 stable material and more experimental modules continue. Joseph Wright has been working on adding ‘native’ drivers for L^ATEX3 to support box transformations. These allow box rotation, clipping and scaling with the driversdvips,xdvipdfmx and direct pdfoutput.

The development of clipping support for thexdvipdfmx driver has also allowed us to suggest improvements to the L^ATEX 2ε graphics drivers, enabling clipping with the X E TEX engine.

A TEX-based regex engine

Bruno Le Floch has been improving the efficiency and robustness of a number of L^ATEX3 functions. Most no- tably, he has created a purely TEX-based regular expression (regex) system for L^ATEX3. This is currently experimental, but is already proving useful and will hopefully stabilise over the coming months.

Bruno’s regex system works with all of the supported engines (pdfTEX, X E TEX and LuaTEX). He has imple- mented the core ideas of standard regex systems, along with some TEX-specifics to allow matching and replac- ing the content of token lists by category code.

xparse improves

Thexparsemodule has been overhauled, making the internal code more efficient and adding additional argument types. This has also allowed us to deal with a number of internal bugs, meaning that argument grab- bing is now more reliable.

The argument grabbers themselves have been reworked so that in the event of an error, the user will normally get a meaningful message from TEX rather than one pointing toxparseinternal function names. This should help in tracking down erroneous input in real documents.

The galley

As detailed in the last issue, work on the galley module has been continuing. Discussion of Joseph’s reimplementation of the galley concepts highlighted some important areas to work on, with the nature of the template concept being particularly significant.

More work is still needed to finalise the galley concepts, but it is clear that some of this will require feedback from other areas. Joseph therefore hopes to finish work on the current round of galley improvements by the end of February, and to return to them once some other areas have been addressed.

Relationships between document items

Thetug2011 meeting took place in October in India.

Frank Mittelbach spoke there about ideas for describing the design relationship between document elements.

These ideas allow a document designer to specify the design of a document element based on its context within a document, and progress in this area will likely lead to an extension in thextemplatesystem.

(13)

L ^A TEX3 News

Issue 8, July 2012

Extended floating point support

Bruno Le Floch has been re-writing the floating point module to function in an ‘expandable’ manner. This allows floating point calculations to be computed far more flexibly and efficiently than before. The expandable nature of the new code allows its use inside operations such as writing to external files, for which previously any such calculations would have to be pre- calculated before any of the writing operations began.

Bruno’s work on the floating point module has been officially released into the main svnrepository for l3kernel; TEX Live 2012 will contain the ‘old’ code for stability while this year is spent testing the new code in production environments and ironing out any wrinkles.

Here’s a neat example as suggested in the documentation, which produces ‘6.278 400 000 000 000×10²’:

\usepackage{xparse, siunitx}

\ExplSyntaxOn

\NewDocumentCommand { \calcnum } { m } { \num { \fp_to_scientific:n {#1} } }

\ExplSyntaxOff

\calcnum {

round ( 200 pi * sin ( 2.3 ^ 5 ) , 2 ) }

This feature is invaluable for simple (and not-so-simple) calculations in document and package authoring, and has been lacking a robust solution for many years.

While LuaLÂTEX can perform similar tasks within its Lua environment, the floating point support is written using theexpl3programming language only, and is thus available in pdfLÂTEX and X E LÂTEX as well.

Regular expressions in TEX

As if expandable floating point support wasn’t enough, Bruno has also written a complete regular expression engine inexpl3 as well. Many reading this will be familiar with the quote attributed to Jamie Zawinski:

Some people, when confronted with a problem, think “I know, I’ll use regular expressions.”

Now they have two problems.

And as humorous as the saying is, it’s still fair to say that regular expressions are a great tool for manipulat- ing streams of text. We desperately hope that people will not start using the regex code to do things like parse xmldocuments; however, for general search–

replace duties, there’s never been anything like l3regex

for the L^ATEX world. As a trivial example, there are 0 instances of the word ‘We’ or ‘we’ in this document (including those two). This value is counted automatically in two lines of code.

And again, it is available for pdfLÂTEX and X E LÂTEX users as well as LuaLÂTEX ones; it also bears noting that this provides an easy solution for applying regular expression processing to LÂTEX documents and text data even on the Windows operating system that does not have native support for such things.

Separating internal and external code

L^ATEX packages are written by a wide range of package authors and consist of code and commands for various purposes. Some of these commands will be intended for use by document authors (such as \pboxfrom thepbox package); others are intended for use by other package writers (such as\@ifmtargfrom theifmtargpackage).

However, a large portion of them are internal, i.e., are intended to be used only within the package or within the L^ATEX kernel and should not be used elsewhere.

For example,\@floatis the L^ATEX kernel interface for floats to be used in class files, but the actual work is done by a command called\@xfloatwhich should not be used directly. Unfortunately the L^ATEX 2εlan- guage makes no clear distinction between the two, so it is tempting for programmers to directly call the latter to save some “unnecessary” parsing activity.

The downside of this is that the “internal” commands suddenly act as interfaces and a reimplementation or fix in any of them would then break add-on packages as they rely on internal behavior. Over the course of the last twenty years probably 80% of such “internal” commands have found their way into one or another package. The consequences of this is that nowadays it is next to impossible to change anything in the L^ATEX 2ε kernel (even if it is clearly just an internal command) without breaking something.

In L^ATEX3 we hope to improve this situation drastically by clearly separating public interfaces (that extension packages can use and rely on) and private functions and variables (that should not appear outside of their module). There is (nearly) no way to enforce this without severe computing overhead, so we implement it only through a naming convention, and some support mech- anisms. However, we think that this naming convention is easy to understand and to follow, so that we are

(14)

desired results.

Naming convention for internals

We’ve been throwing around some ideas for this for a number of years but nothing quite fit; the issue comes down to the fact that TEX does not have a ‘name- spacing’ mechanism so any internal command needs to have a specific prefix to avoid clashing with other packages’ commands. The prefix we have finally decided on forexpl3code is a double underscore, such that functions like \seq_count:Nare intended for external use and \__seq_item:nis an internal command that should not be used or relied upon by others.

All this is well and good, but it can be inconvenient to type long prefixes such as\__seq_before all command names, especially in a package for which nearly all package functions are internal.

We therefore also extendedDocStrip slightly by adding a ‘shorthand’ for internal package prefixes. Commands and variables in .dtxcode may now contain@@which is expanded to the function prefix when the .styfile is extracted. As an example, writing

%@@=seq

\cs_new:Npn \@@_item:n ...

is equivalent to

\cs_new:Npn \__seq_item:n ...

There are clear advantages to this syntax. Function names are shorter and therefore easier to type, and code can still be prototyped using the@@syntax (e.g., pasting code between a.dtx file and a regular.tex document). Most importantly, it is explicitly clear from the code source which commands are intended to be used externally and which should be avoided.

We hope that this syntax will prove popular; in our initial experiments we think it works very well. In fact we found a good number of smaller errors when being forced to think about what is internal and what is an external function.

Continual revolution—the ‘small bang’

In addition to the major additions introduced above, Frank Mittelbach has been examining expl3with a fresh eye to resolve any outstanding issues in the consistency or logic of the names of functions.

We are very mindful of the fact that for people to findexpl3 a useful tool, it must have a stable interface. This said, there are still some musty corners that we can show where people simply haven’t been using certain functions. In select cases, we’re re-assessing whether all of the (sometimes esoteric) odds and ends that have been added to expl3really belong; in other

choices weren’t correct the first time around.

To address this tarnish, we’re carefully making some minor changes to parts of theexpl3interface and we’d like to allay any fears thatexpl3 isn’t stable. The expl3language now offers a wide range of functions plus their variants, and we’re talking about changing but a very small percentage of these, and not common ones at that. We don’t want it to become a mess, so we think it’s better to tidy things up as we go. Follow the LaTeX-Lmailing list for such details as they arise.

(15)

dropped/replaced:

The L^ATEX3 codebases ranges between these two ex- tremes; the packages in l3packagesare largely the former while the modules composingexpl3 are largely the latter type. In both cases, the ‘external’ commands (whether for document author or package writer) are usually defined in terms of other internal package commands that should not be used by anyone else, but often when reading the internal package code it’s not always clear which is which.

For L^ATEX3 we are experimenting with an extension to theDocStrip mechanism to provide a clear distinction between internal and external package commands.

–15

(16)

L TEX3 News

Issue 9, March 2014

Contents Hiatus?

Well, it’s been a busy couple of years. Work has slowed on the L^ATEX3 codebase as all active members of the team have been — shall we say — busily occupied with more pressing concerns in their day-to-day activities.

Nonetheless, Joseph and Bruno have continued to fine-tune the L^ATEX3 kernel and add-on packages.

Browsing through the commit history shows bug fixes and improvements to documentation, test files, and internal code across the entire breadth of the codebase.

Members of the team have presented at two TUG conferences since the last L^ATEX3 news. (Has it really been so long?) In July 2012, Frank and Will travelled to Boston; Frank discussed the challenges faced in the past and continuing to the present day due to the limits of the various TEX engines; and, Frank and Will to- gether covered a brief history and recent developments of theexpl3 code.

In 2013, Joseph and Frank wrote a talk on complex layouts, and the “layers” ideas discussed in L^ATEX3;

Frank went to Tokyo in October to present the work.

Slides of and recordings from these talks are available on the L^ATEX3 website.

These conferences are good opportunities to intro- duce theexpl3language to a wider group of people; in many cases, explaining the rationale behind why expl3 looks a little strange at first helps to convince the audi- ence that it’s not so weird after all. In our experience, anyone that’s been exposed to some of the more awk- ward expansion aspects of TEX programming appreci- ates how expl3makes life much easier for us.

expl3 in the community

While things have been slightly quieter for the team, more and more people are adopting expl3for their own use. A search on the TEX Stack Exchange website for either “expl3” or “latex3” at time of writing yield around one thousand results each.

In order to help standardise the prefixes used in expl3 modules, we have developed a registration procedure for package authors (which amounts to little more than notifying us that their package uses a specific prefix, which will often be the name of the package itself).

Please contact us via the latex-lmailing list to reg- ister your module prefixes and package names; we ask that you avoid using package names that begin with

l3...sinceexpl3packages use this internally. Some authors have started using the package prefixlt3...as a way of indicating their package builds onexpl3 in some way but is not maintained by the L^ATEX3 team.

In the prefix database at present, some thirty package prefixes are registered by fifteen separate individ- uals (unrelated to The L^ATEX Project — the number of course grows if you include packages by members of the team). These packages cover a broad range of functionality:

acro Interface for creating (classes of) acronyms hobby Hobby’s algorithm in PGF/TiKZ for drawing

optimally smooth curves.

chemmacros Typesetting in the field of chemistry.

classics Traditional-style citations for the classics.

conteq Continued (in)equalities in mathematics.

ctex A collection of macro packages and document classes for Chinese typesetting.

endiagram Draw potential energy curve diagrams.

enotez Support for end-notes.

exsheets Question sheets and exams with metadata.

lt3graph A graph data structure.

newlfm The venerable class for memos and letters.

fnpct Interaction between footnotes and punctuation.

GS1 Barcodes and so forth.

hobete Beamer theme for the Univ. of Hohenheim.

kantlipsum Generate sentences in Kant’s style.

lualatex-math Extended support for mathematics in LuaL^ATEX.

media9 Multimedia inclusion for Adobe Reader.

pkgloader Managing the options and loading order of other packages.

substances Lists of chemicals, etc., in a document.

withargs Ephemeral macro use.

xecjk Support for CJK documents in X E L^ATEX.

xpatch,regexpatch Patch command definitions.

xpeek Commands that peek ahead in the input stream.

xpinjin Automatically add pinyin to Chinese characters zhnumber Typeset Chinese representations of numbers

(17)

zxjatype Standards-conforming typesetting of Japanese for X E L^ATEX.

Some of these packages are marked by their authors as experimental, but it is clear that these packages have been developed to solve specific needs for typesetting and document production.

Theexpl3language has well and truly gained traction after many years of waiting patiently.

A logo for the L

^A

TEX3 Programming Language

To show that expl3is ready for general use Paulo Cereda drew up a nice logo for us, showing a

hummingbird (agile and fast — but needs huge amounts of energy) picking at “l3”. Big thanks to Paulo!

Recent activity

L^ATEX3 work has only slowed, not ground to a halt.

While changes have tended to be minor in recent times, there are a number of improvements worth discussing explicitly.

1. Bruno has extended the floating point code to cover additional functions such as inverse trigono- metric functions. These additions round out the functionality well and make it viable for use in most cases needing floating point mathematics.

2. Joseph’s refinement of the experimental galley code now allows separation of paragraph shapes from margins/cutouts. This still needs some testing!

3. For some time nowexpl3has provided “native”

drivers although they have not been selected by default in most cases. These have been revised to improve robustness, which makes them probably ready to enable by default. The improvements made to the drivers have also fed back to more

“general” L^ATEX code.

Work in progress

We’re still actively discussing a variety of areas to tackle next. We are aware of various “odds and ends”

in expl3that still need sorting out. In particular, some experimental functions have been working quite well and it’s time to assess moving them into the “stable”

modules, in particular thel3str module for dealing with

catcode-twelve token lists more commonly known in expl3as strings.

Areas of active discussion including issues around uppercasing and lowercasing (and the esoteric ways that this can be achieved in TEX) and space skipping (or not) in commands and environments with optional arguments. These two issues are discussed next.

Uppercasing and lowercasing

The commands\tl_to_lowercase:nand

\tl_to_uppercase:nhave long been overdue for a good hard look. From a traditional TEX viewpoint, these commands are simply the primitive\lowercase and\uppercase, and in practice it’s well known that there are various limitations and peculiarities associ- ated with them. We know these commands are good, to one extent or another, for three things:

1. Uppercasing text for typesetting purposes such as all-uppercase titles.

2. Lowercasing text for normalisation in sorting and other applications such as filename comparisons.

3. Achieving special effects, in concert with manipu- lating\uccodeand the like, such as defining commands that contain characters with different cat- codes than usual.

We are working on providing a set of commands to achieve all three of these functions in a more direct and easy-to-use fashion, including support for Unicode in LuaL^ATEX and X E L^ATEX.

–17

(18)

We have also re-considered the behaviour of space- skipping inxparse. Consider the following examples:

\begin{dmath} \begin{dmath}[label=foo]

[x y z] = [1 2 3] x^2 + y^2 = z^2

\end{dmath} \end{dmath}

In the first case, we are typesetting some mathematics that contains square brackets. In the second, we are assigning a label to the equation using an optional argument, which also uses brackets. The fact that both work correctly is due to behaviour that is specifically programmed into the workings of the dmathenviron- ment of breqn: spaces before an optional argument are explicitly forbidden. At present, this is also how commands and environments defined using xparsebehave.

But consider a pgfplotsenvironment:

\begin{pgfplot}

[

% plot options ]

\begin{axis}

[

% axis options ]

...

\end{axis}

\end{pgfplot}

This would seem like quite a natural way to write such environments, but with the current state of xparsethis syntax would be incorrect. One would have to write either of these instead:

\begin{pgfplot}%

[

% plot options ]

\begin{pgfplot}[

% plot options ]

Is this an acceptable compromise? We’re not entirely sure here — we’re in a corner because the humble[has ended up being part of both the syntax and semantics of a L^ATEX document.

Despite the current design covering most regular use- cases, we have considered adding a further option to xparseto define the space-skipping behaviour as desired by a package author. But at this very moment we’ve rejected adding this additional complexity, because environments that change their parsing behaviour based on their intended use make a L^ATEX-based language more difficult to predict; one could imagine such behaviour causing difficulties down the road for automatic syntax checkers and so forth. However, we don’t make such decisions in a vacuum and we’re always happy to continue to discuss such matters.

There is one (understandable) misconception that shows up once in a while with people claiming that

expl3= L^ATEX3.

However, the correct relation would be a subset, expl3⊂L^ATEX3,

withexpl3forming the Core Language Layer on which there will eventually be several other layers on top that provide

higher-level concepts for typesetting (Typesetting Foundation Layer),

a designer interface for specifying document structures and layouts (Designer Layer),

and finally a Document Representation Layer that implements document level syntax.

Of those four layers, the lowest one —expl3— is available for use and withxparsewe have an instance of the Document Representation Layer modeled largely after L^ATEX 2ε syntax (there could be others). Both can be successfully used within the current L^ATEX 2ε framework and as mentioned above this is increasingly happening.

The middle layers, however, where the rubber meets the road, are still at the level of prototypes and ideas (templates,ldb, galley,xorand all the good stuff) that need to be revised and further developed to arrive at a L^ATEX3 environment that can stand on its own and that is to where we want to return in 2014.

An overview on this can be found in the answer to

“What can *I* do to help The L^ATEX Project?” on Stack Exchange,¹ which is reproduced below in slightly abridged form. This is of course not the first time that we have discussed such matters, and you can find similar material in other publications such as those at http://latex-project.org; e.g., the architecture talk given at the TUG 2011 conference.

1http://tex.stackexchange.com/questions/45838

(19)

What can you do for The L

^A

TEX Project?

By Frank Mittelbach

My vision of L^ATEX3 is really a system with multiple layers that provide interfaces for different kinds of roles.

These layers are

the underlying engine (some TEX variant)

the programming layer (the core language, i.e., expl3)

the typesetting foundation layer (providing higher- level concepts for typesetting)

the typesetting element layer (templates for all types of document elements)

the designer interface foundation layer

the class designer layer (where instances of document elements with specific settings are defined)

document representation layer (that provides the input syntax, i.e., how the author uses elements) If you look at it from the perspective of user roles then there are at least three or four roles that you can clearly distinguish:

The Programmer (template and functionality provider)

The Document Type Designer (defines which elements are available; abstract syntax and semantics)

The Designer (typography and layout)

The Author (content)

As a consequence The L^ATEX Project needs different kinds of help depending on what layer or role we are looking at.

The “Author” is using, say, list structures by specifying something like \begin{itemize} \itemin his documents. Or perhaps by writing<ul> ... </ul>or whatever the UI representation offers to him.

The “Document Type Designer” defines what kind of abstract document elements are available, and what attributes or arguments those elements provide at the author level. E.g., he may specify that a certain class of documents provides the display lists “enumerate”,

“itemize” and “description”.

The “Programmer” on the other hand implements templates (that offer customizations) for such document elements, e.g., for display lists. What kind of cus- tomization possibilities should be provided by the “Pro- grammer” is the domain of the “Document Designer”;

he drives what kind of flexibility he needs for the design. In most cases the “Document Designer” should be able to simply select templates (already written) from a template library and only focus on the design, i.e.,

instantiating the templates with values so that the desired layout for “itemize” lists, etc., is created.

In real life a single person may end up playing more than one role, but it is important to recognise that all of them come with different requirements with respect to interfaces and functionality.

Programming Layer

The programming layer consists of a core language layer (calledexpl3 (EXP erimental L aTeX 3) for historical reasons and now we are stuck with it:-)) and two more components: the “Typesetting Founda- tion Layer” that we are currently working on and the

“Typesetting Element Layer” that is going to provide customizable objects for the design layer. Whileexpl3 is in many parts already fairly complete and usable the other two are under construction.

Help is needed for the programming layer in

helping by extending and completing the regression test suite forexpl3

helping with providing good or better documentation, including tutorials

possibly helping in coding additional core functionality — but that requires, in contrast to the first two points, a good amount of commitment and experience with the core language as otherwise the danger is too high that the final results will end up being inconsistent

Once we are a bit further along with the “Typeset- ting Foundation Layer” we would need help in providing higher-level functionality, perhaps rewriting existing packages/code for elements making use of extended possibilities. Two steps down the road (once the

“Designer Layer” is closer to being finalized) we would need help with developing templates for all kinds of elements.

In summary for this part, we need help from people interested in programming in TEX andexpl3and/or interested in providing documentation (but for this a thorough understanding of the programming concepts is necessary too).

Design Layer

The intention of the design layer is to provide interfaces that allow specifying layout and typography styles in a declarative way. On the implementation side there are a number of prototypes (most notablyxtemplate and the recent reimplementation of ldb). These need to be unified into a common model which requires some more experimentation and probably also some further thoughts.

But the real importance of this layer is not the implementation of its interfaces but the conceptual view

–19

(20)

ods) to describe design and layout. I.e., enabling a designer to think not in programs but in visual representations and relationships.

So here is the big area where people who do not feel they can or want to program TEX’s bowels can help.

What would be extremely helpful (and in fact not just for L^ATEX3) would be

collecting and classifying ahuge set of layouts and designs

– designs for individual document elements (such as headings, TOCs, etc)

– document designs that include relationships between document elements

thinking about good, declarative ways to specify such designs

– what needs to be specified

– to what extent and with what flexibility I believe that this is a huge task (but rewarding in itself) and already the first part of collecting existing design specifications will be a major undertaking and will need coordination and probably a lot of work. But it will be a huge asset towards testing any implementations and interfaces for this layer later on.

Document Interface Layer

If we get the separation done correctly, then this layer should effectively offer nothing more than a front end for parsing the document syntax and transforming it into an internal standardised form. This means that on this layer one should not see any (or not much) coding or computation.

It is envisioned that alternative document syntax models can be provided. At the moment we have a draft solution inxparse. This package offers a document syntax in the style of L^ATEX 2ε, that is with*- forms, optional arguments in brackets, etc., but with a few more bells and whistles such as a more generalized concept of default values, support for additional delim- iters for arguments, verbatim-style arguments, and so on. It is fairly conventional though. In addition when it was written the clear separation of layers wasn’t well- defined and so the package also contains components for conditional programming that I no longer think should be there.

Bottom line on what is needed for this layer is to

think about good syntax for providing document content from “the author” perspective

think about good syntax for providing document content from an “application to typesetting” perspective, i.e., the syntax and structure for auto- mated typesetting where the content is prepared by a system/application rather than by a human

automation works much better with structures that do not have a lot of alternative possibilities and shortcuts, etc.) and even when just looking at the human author a lot of open questions need answering. And these answers may or may not be to painfully stick with existing L^ATEX 2εconventions in all cases (or perhaps with any?).

None of this requires coding orexpl3experience.

What it requires is familiarity with existing input concepts, a feel for where the pain points currently are and the willingness to think and discuss what alternatives and extensions could look like.

In Summary

Basically help is possible on any level and it doesn’t need to involve programming. Thoughts are sprinkled throughout this article, but here are a few more high- lights:

help with developing/improving the core programming layer by

– joining the effort to improve the test suite – help improving the existing (or not existing)

documentation

– joining the effort to produce core or auxiliary code modules

help on the design layer by

– collecting and classifying design tasks

– thinking and suggesting ways to describe layout requirements in a declarative manner

help on shaping the document interface layer These concepts, as well as their implementation, are under discussion on the listlatex-l.² The list has only a fairly low level of traffic right now as actual implementation and development tasks are typically discussed directly among the few active implementors.

But this might change if more people join.

And something else . . .

The people on the L^ATEX3 team are also committed to keeping L^ATEX 2ε stable and even while there isn’t that much to do these days there remains the need to resolve bug reports (if they concern the 2e core), provide new distributions once in a while, etc. All this is work that takes effort or remains undone or incomplete.

Thus here too, it helps the L^ATEX3 efforts if we get help to free up resources.

2Instructions for joining and browsing archives at:

http://latex-project.org/code.html

L A TEX3 News

L A TEX3 News, Issues 1–12

Contents

L A TEX3 News

Welcome to L

TEX3

What currently exists

What’s happening now

What’s happening soon

What’s happening later

L TEX3 News

TEX Live and the expl3 code

Planned updates

New members

Some specifics

The next six months

L TEX3 News

Happy New Year

Recent developments

Upcoming plans

L TEX3 News

expl3 in practice

New xpackages

Developments with expl3

TUG 2010 reflections

L A TEX3 News

Happy new year

The LPPL is now OSI-approved

Reflections on 2010

Current progress

Plans for 2011

L TEX3 News

The L

TEX3 Team expands

The ‘Big Bang’

L

TEX3 on GitHub

Next steps

L TEX3 News

After the ‘Big Bang’

Deforming boxes

A TEX-based regex engine

xparse improves

The galley

Relationships between document items

L A TEX3 News

Extended floating point support

Regular expressions in TEX

Separating internal and external code

Continual revolution—the ‘small bang’

L TEX3 News

Contents Hiatus?

expl3 in the community

A logo for the L

TEX3 Programming Language

Recent activity

Work in progress

What can you do for The L

TEX Project?

L ^A TEX3 News, Issues 1–12

L ^A TEX3 News

L ^A TEX3 News

L ^A TEX3 News