Mojca Miklavec wrote:

Any ideas if getting rid of dependency on environmental variables is doable?

I agree it would be great if mtxrun could call/do setuptex on itself if needed.

Oliver Buerschaper wrote:

...

...

Any ideas if getting rid of dependency on environmental variables is doable? I agree it would be great if mtxrun could call/do setuptex on itself if needed.

At the moment for me it would be enough to know where exactly LuaTeX looks for configuration information (path & file name) and which files take precedence over others in the case of conflicts ... my current understanding is:

1. Environment variables 2. setuptex.tmf

no, this one is only used when you run mtxrun --tree=.... i use this methos in web driven workflows where we can have multiple trees alongside (linux servers) and i never bother to setup my shell for that; i just call mtxrun then

3. texmf-context/web2c/context.cnf

we ship context.cnfas an example of what is needed (it's the one on my machine)

4. texmf/web2c/texmf.cnf

that one indeed unless it's in a usual place

5. ...

$TEXMF $TEXMFCACHE (since there is no agreement over where it lives) so, only two variables

As a second step, one would have to know what each of the settings in the above places is needed for. In particular, I would like to know which settings LuaTeX actually uses and which ones it ignores. But this can wait until the first question is figured out completely ;-)

$TEXMF is needed in order to locate the configuration file (currently still a CNF file but we may as well get rid of it since we use less and less info from it; if we would organize the minimals differently i.e. more flat that tds we can do without a configuration, especially fonts are an issue (why no tex/fonts/data/vendor/collection) $TEXMFCACHE is needed because distributers want control over the place where those files go; there is some default setup but ... Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

Hi Oliver, Hans and I are extending context in this area right now, so any current answer will likely be outdated before much longer. So, a little bit of extra patience is required still. All the best, Taco

Oliver Buerschaper wrote:

On the other hand, if you want ConTeXt distributions to always remain almost exact replicas of Hans' machine (like the current minimals are), fine. But don't expect them to be widely usable. The average user doesn't have the knowhow required to happily juggle with dozens of configurations sitting side by side.

In fairness the average user shouldn't be using any luatex/Mk IV stuff yet - it is still in alpha/beta. I have whole production environments built around ConTeXt minimals and they are extremely usable across multiple architectures, mainly because they are still based on Mk II. Mk IV is not (meant to be) production level yet. Just my humble opinion... :-) Duncan

Oliver Buerschaper wrote:

However, think of someone who has never heard of ConTeXt before but has been persuaded by a friend or colleague to give it a try. Perhaps they've been given a template document for a thesis to start from. If this person has a Mac they'll probably want to download some disk image, have ConTeXt deployed on their machine by the Apple installer program, fire up TeXShop and start writing their thesis with ConTeXt right away. This is what I'd consider the average user. With the current minimals this is impossible ...

hm, i really had it up and running on the mac pretty fast in texshop but since my mac is permantly broken i cannot test it right now ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

Oliver Buerschaper wrote:

...

...

However, think of someone who has never heard of ConTeXt before but has been persuaded by a friend or colleague to give it a try. Perhaps they've been given a template document for a thesis to start from. If this person has a Mac they'll probably want to download some disk image, have ConTeXt deployed on their machine by the Apple installer program, fire up TeXShop and start writing their thesis with ConTeXt right away. This is what I'd consider the average user. With the current minimals this is impossible ...

hm, i really had it up and running on the mac pretty fast in texshop but since my mac is permantly broken i cannot test it right now

Well, you're an IT pro ... that makes all the difference ;-)

i'm a complete mac-dummy so for me it was a matter of: - unzip a minimal - make sure that texshop intializes that tree btw, there will me a cross platform 'texshop' variant (tug funded project) and at some point we need to get context support in there Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

Taco Hoekwater wrote:

I wrote that message yesterday to let you know that we are actively working to make the situation better. It quite a cold shower to get flamed for doing exactly that.

also, keep in mind that taco and i are working nearly full time on it and asking for documentation of changing code would force us to skip sleeping Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

Oliver Buerschaper wrote:

Still I remember that from one MKII release to the next bugs kept creeping into the code that suddenly broke everything. Wrecked font handling in combination with XeTeX around the beginning of last year comes to my mind as just one of the topics people were having trouble

xetex is a moveing target too, and i think that in that respect latex/xetex users also have to update in sync with binaries; by now the xetex interface becomes more stable but it's still version < 1

with. From personal experience in the middle of a project at that time I also remember a few other problems with MKII ... besides I was having the impression that for each new MKII release that didn't solve a specific bug you'd have to adapt the original workaround.

depends, many parts are pretty stable but if you let a project depend on new features ...

understanding is that many Perl scripts have been superseded by Ruby scripts over time, still they're shipped with current MKII minimals.

i'm not ure what you mean here; we only keep perl scripts around for stuff that is not yet redone in ruby or because soem workflows may depend on it; if a user uses "texmfstart somescript" he or she will now notice if a perl or ruby script is used; therefore we could make a transition

Unfortunately there's no documentation about whether they could actually be deleted or whether there are still some (rare) tasks they're needed

depends on what kind of workflows one has; also some scripts are shipped that probably are used as pragma but since we use the same zips for updating i keep 'm there

for. And then they seem to access dozens of environment variables and configuration files so you have no clue about whether those are needed or not. You just have to know. Which I don't.

it all depends on your local system ... in the minimals 'setuptex' just plays safe ... it sets up all variables that can interfere with an existing installation

When I was trying to put together a Mac installer for MKII a couple of months ago I kept hearing the same definitely maybes about this configuration data issue as the ones I'm facing now -- only at that time the game wasn't called LuaTeX. Not having had the time to reverse engineer everything I gave up then.

hm, platform issues .. each platform has its own demands too and we can only adapt the minimals have been around for ages and not much changed for a long time; a few month ago we started thinking of and working on a garden based one and in the process of making engine specific variants (even more minimals) quite some experimenting is needed; these are not things that can be done in a few hours or weeks or months, especially not if you consider that tex distributions (and their contents, like fonts) change every now and then too

Recently I had the chance to work on the Mac installer again and after some progress I got stuck at the very same vital point called configuration information and how it's accessed.

i wonder what configuration info you refer too; in a minimals setup there is only 'setuptex' and optionally one may want to adapt the cont-sys.tex file

...

I wrote that message yesterday to let you know that we are actively working to make the situation better. It quite a cold shower to get flamed for doing exactly that.

Taco

Admittedly, I phrased my comment rather provocatively ... if you feel offended personally I apologize. I appreciate very much what both you and Hans do. Otherwise I wouldn't use ConTeXt at all and even more certainly wouldn't be concerned with a Mac package for it.

Still I find the current situation rather frustrating ... in fairness I think this is also quite understandable from the above. Anyway, for the moment I'll stay put and wait for the configuration mechanism to stabilize. Once that has happened I'd be delighted to get to know the details about how it works ... then the Mac installer could also enter the stage finally.

as taco mentioned ... mkiv is for the brave ... we will cook up something that makes it usable for tex live but even then ... it will take a while Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

As far as workflows are concerned my opinion is that keeping things *simple* should be the primary goal for any ConTeXt minimal distribution. I think you may have misunderstood the purpose of the minimals. "Minimal" here generally refers to effort - both on Hans' side and on

Oliver Buerschaper wrote: the user's side - rather than any notion of these distributions being stripped down. There has been some discussion on the main list of the stripped down distribution type that I think you're looking for, but it's important to know that the minimals are *not* it. The minimals are designed as an "unzip and go" *complete* package - "everything with minimal effort", not "the minimum you need to get started".

Now this is my one and only motivation for getting rid of the shell script "setuptex" and the need for all those environment variables. There's really nothing more to all my questions about configuration files ...

This is also illuminating. The concept of the setuptex script (as I understand it) is precisely to allow the minimals to operate completely independently of any other installed TeX systems. The point is that you only run the setuptex script in the shell in which you want to run ConTeXt. These environment variables aren't meant to be set globally (again, AFAIK), they are meant to be set for the session. To me this is what makes the minimals highly portable and easy to integrate into cross-platform scripts and applications. Your issues appear to me to be rather specific to your own environment, which perhaps does not allow you to run in this way.

One should agree on the simplest ConTeXt workflow possible and then design the minimals primarily with that idea in mind rather than trying to support each and every workflow under the sun right *out of the box*. The most common scenario will most probably be the average desktop user wanting to typeset a document interactively.

The simplest workflow possible is quite a different proposition, and should not be being addressed by the minimals. And the "average desktop user" is extremely difficult to define for ConTeXt - I suspect they don't use a Mac, for a start. :-) Duncan

I think you may have misunderstood the purpose of the minimals. "Minimal" here generally refers to effort - both on Hans' side and on the user's side - rather than any notion of these distributions being stripped down. There has been some discussion on the main list of the stripped down distribution type that I think you're looking for, but it's important to know that the minimals are *not* it. The minimals are designed as an "unzip and go" *complete* package - "everything with minimal effort", not "the minimum you need to get started".

Indeed, I followed the thread about the ultraminimals on the other list. However, I wasn't talking about stripping down Hans' standalone distribution to the bare minimum ... this would certainly need "maximal" rather than "minimal" effort on Hans' and Taco's side. What I was striving for is merely stripping down the complexity of the configuration mechanism for now. Most probably this would reduce a package developer's effort from "maximal" to "intermediate". Modularizing the whole distribution into small, cleanly separated components is quite another project. Without doubt ConTeXt would benefit a great deal from that but this is *not* the issue I was addressing in my post.

This is also illuminating. The concept of the setuptex script (as I understand it) is precisely to allow the minimals to operate completely independently of any other installed TeX systems. The point is that you only run the setuptex script in the shell in which you want to run ConTeXt. These environment variables aren't meant to be set globally (again, AFAIK), they are meant to be set for the session.

There I think you've exactly pinned down the problem. Each application wanting to run ConTeXt must call it from a wrapper script which sets these environment variables for the local shell. Keep in mind though that (at least on the Mac) this wrapper script is only needed for the minimals ... and certainly not for the ConTeXt from TeX Live, for example. If you use TeXShop as your frontend you'd then have to have different ConTeXt typesetting commands for each and every TeX distribution on your machine. Not exactly user-friendly ... Oliver

Oliver Buerschaper wrote:

Honestly, to my ears this sounds like a contradiction. If you consider MKIV for the brave then why include it in TeX Live at all? Once I read that the ConTeXt shipping with TeX Live should be regarded as the official stable release ...

We have not added mkiv to texlive. If it is included now, then that is thanks to Norbert's packaging. Best wishes, Taco

On Fri, May 23, 2008 at 7:45 PM, Norbert Preining wrote:

But AFAIS, the whole context could in fact be omitted. As far as I see *everyone* using context also uses his home-grown installation etc ;-)

- ConTeXt on MikTeX is completely/absolutely broken - TeX Live is outdated - ConTeX/XeTeX on gwTeX is broken - LuaTeX doesn't work at all, anywhere (except for the Debian) What else (apart from W32TeX) remains out there? Let's see how the new packaging in TeX Live is going to work ... :) :) :) Mojca

Norbert Preining wrote:

On Mi, 21 Mai 2008, Taco Hoekwater wrote:

...

...

that the ConTeXt shipping with TeX Live should be regarded as the official stable release ... We have not added mkiv to texlive. If it is included now, then that is thanks to Norbert's packaging.

No I didn't add anything to the TeX Live repository, that I leave to Taco/Hans/whoever. I would say it is up to them to ship what they consider fine.

I am trying to package the latest context for Debian.

But AFAIS, the whole context could in fact be omitted. As far as I see *everyone* using context also uses his home-grown installation etc ;-)

not true -) the so called minimals are based on my personal subset that i use in project; one can consider them a kind if reference subset and they can be used of course; however, it leave out all the other tex stuff: no other macro packages, less binaries, less fonts, etc Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

Oliver Buerschaper wrote:

Well, my feeling is some buggy parts weren't all that new at that time ...

some stuff is 'experimental' and therefore 'buggy' by definition ... if only because there is no one and perfect solution in traditional tex ... which is why we started the luatex project in the transition to mkiv it means that parts of context have to be split in kii and mkiv code (a rather tricky thing btw) so if you have a workflow wher eyou cannot update regularly, you should stick to the museum versions that taco mentioned (which also means, stick to pdftex for a while) we expect to need another two years to get luatex and mkiv in a state that most interfaces are stable

What I mean is that there is stuff lurking around in the minimals which is obsolete apart from rare (and presumably advanced) usage scenarios.

probably, so best is to ignore anything that does not ring a bell; i made the minimal because i needed them in projects and then found out that users could benefit from them (after all, there are the regular distributions (norbert, akira and gerben took care of that) for those who want things out of a box)

However, this stuff makes trying to figure out the interdepencies of configuration files and variables quite a daunting task to say the least. For a suggestion on how to remedy this situation see my lines below.

well, in the minimals there's only TEXMF (and TEXMFCNF) and TEXMFCACHE and all the rest is to prevent interference; all these are discussed in the TDS/KPSE documentation

...

...

Unfortunately there's no documentation about whether they could actually be deleted or whether there are still some (rare) tasks they're needed

depends on what kind of workflows one has; also some scripts are shipped that probably are used as pragma but since we use the same zips for updating i keep 'm there

Don't get me wrong but this is exactly the vague kind of answers I was referring to in my initial posting.

well, users only need texmfstart and texexec (unless they want to do tricky things like generating font metrics or manipulating graphics; this is described in docs on the web but not for average users)

As far as workflows are concerned my opinion is that keeping things *simple* should be the primary goal for any ConTeXt minimal distribution. One should agree on the simplest ConTeXt workflow possible

as duncan said .. minimal is not the same as small, it's just a subset

and then design the minimals primarily with that idea in mind rather than trying to support each and every workflow under the sun right *out of the box*. The most common scenario will most probably be the average desktop user wanting to typeset a document interactively.

that would mean that i'd have to cook up additional stuff for myself and there is no time for that (unless someone pays me for it); if i cannot use the minimals myself, i cannot spend too much time on it

That said, one could collect the legacy stuff you mentioned into a separate package for a start and clearly mark it as such. I think this should be feasible on all platforms ConTeXt runs on. The benefits of this approach abound: less scripts, less interdependencies to worry about, less environment variables, sleeker minimal distributions.

it would not make a difference, the only legacy stuff is some ruby scripts and some styles; it has nothing to do with environment variables

Note that neither power nor flexibility would suffer: experienced TeX users will be able to tailor their minimals installation to meet the needs of an enterprise workflow anyway. If you need legacy stuff for old projects you extend the minimals with the appropriate component. If you need multiple ConTeXt trees within a web based workflow you put in those setuptex.tmf files and anything else that may be required.

well, as said, having yet another subset would make my live misserable bit anyone is free to make yet another minimal-minimal; maybe someone only wants latin modern, etc etc

I hope this will clarify my point of view.

sure, but i just have no time for more variants

...

...

for. And then they seem to access dozens of environment variables and configuration files so you have no clue about whether those are needed or not. You just have to know. Which I don't.

it all depends on your local system ... in the minimals 'setuptex' just plays safe ... it sets up all variables that can interfere with an existing installation

If course, it depends on my local system. However, as a rule of thumb every installation of TeX (if there are several ones) should be responsible for not leaving behind any mess when it's not in use. Just my humble opinion how playing safe could be understood. Also, this is precisely what happens on a Mac when you have several versions of TeX Live, Fink-teTeX, MacPorts-teTeX etc. happily sitting side by side with zero effort on the user part ...

you're over optimistic ... i just logged in on one of our servers ... controller-1:~ # set | grep -i TEX TEXINPUTS=:/root/.TeX:/usr/share/doc/.TeX:/usr/doc/.TeX tex | latex | pdflatex) e='!*.+(tex|TEX|texi|latex)' see what i mean? and i really didn't install anything tex on that machine (older machines have even more)

Now this is my one and only motivation for getting rid of the shell script "setuptex" and the need for all those environment variables. There's really nothing more to all my questions about configuration files ...

what is the problem with running just one script to set up the system? after all, if you don't call setuptex, you still need to set up the path (and i never set up the path because i can have multiple tex trees)

...

...

Recently I had the chance to work on the Mac installer again and after some progress I got stuck at the very same vital point called configuration information and how it's accessed.

i wonder what configuration info you refer too; in a minimals setup there is only 'setuptex' and optionally one may want to adapt the cont-sys.tex file

I was referring to

setuptex.sh setuptex.tmf

all the same (there's also setuptex.bat) and the tmf file can be used when you run from an editor (in which case the --tree=... flag will look for that file)

context.cnf

an example fo what i use here; handy for users who want to know the mem values

texmf.cnf

always needed

environment variables

which ones? i never set any, since setuptex does it for me

and how the information contained therein is accessed (if at all) by the different engines and scripts.

ah ... that's another story ... for that you need to consult the tds documentation and the kpse/web2c documentation

...

...

Still I find the current situation rather frustrating ... in fairness I think this is also quite understandable from the above. Anyway, for the moment I'll stay put and wait for the configuration mechanism to stabilize. Once that has happened I'd be delighted to get to know the details about how it works ... then the Mac installer could also enter the stage finally.

as taco mentioned ... mkiv is for the brave ... we will cook up something that makes it usable for tex live but even then ... it will take a while

Honestly, to my ears this sounds like a contradiction. If you consider MKIV for the brave then why include it in TeX Live at all? Once I read that the ConTeXt shipping with TeX Live should be regarded as the official stable release ...

because there are brave tex live users and because luatex itself will be on tex live (used as scripting engine so be prepared for another kick out ruby/perl/bash round); we also need to 'proof that it's possible' -) Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

Taco Hoekwater wrote:

Please note this implementation logic is not set in stone yet. We preferred to start as simple as possible, but extensions may be needed. (currently we have a "works for me" system). Keep in mind that if your installation is really outlandish, you can just set TEXMFCNF in the environment explicitly.

an additional note: it is important to keep in mind that when you don't set TEXMFCNF and TEXMF, ther eis a potential clash with existing tex installations; the main reason for having setuptex in the minimals is that it permits many parallel installations (users like taco and me often have many 'independent' istallations (for projects) so, if the automatisms fail (i.e. you don't set TEXMF and TEXMFCNF yourself and depend on luatools cum suis to figure out where the configureations are) there is probably some interference with environment variables; i.e. when you for instance install tetex (or when the system came with it) you can be pretty sure that some variables are set and cf tds/kpse environment variables take precedence); in that case there is no way around overriding them at the shell level (as setuptex does) this also means that when you install the minimals allongside an existing tex installation, you probably still need to use setuptex

All such files will be read before any of the texmf.cnfs, and so they can be used to overrule texmf.cnf values, TEXMFCACHE in particular. The lua script has to return a table. Mine (I have only one) looks like this:

config = {} config.TEXMFCACHE = "/tmp" return config

there is an example contextcnf.lua file in the distribution that will have the variables that we support in addition to the normal ones in texmf.cnf ( Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

Taco Hoekwater wrote:

Taco Hoekwater wrote:

...

So, assuming /opt/tex/texmf-linux/bin, it attempts the following directories, in this order:

[/opt/tex]/texmf/web2c % two parents up [/opt/tex/texmf-linux]/texmf/web2c % one parent up [/opt/tex]/texmf-local/web2c [/opt/tex/texmf-linux]/texmf-local/web2c

I forgot to mention that in this case it will also set the texmf.cnf implicit variable $SELFAUTOPARENT to the single value that is the found TDS root (e.g. /opt/tex).

For Hans: it makes sense to do this if TEXMFCNF *is* set as well.

i think that this happens already,since i donnot test for the non-set case ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

Hi Taco,

This new release is available now.

I've just had the chance to test this new release and got mixed results ... the 2008.05.21 minimal distribution (LuaTeX only) now sits in /usr/local/context/2008-05-21 and there's a symbolic link /usr/texbin -> /usr/local/context/2008-05-21/texmf-osx-intel/bin to the binaries. This link is given by the distribution switching mechanism on the Mac (see my earlier post in this thread) and is appended to $PATH, too. Furthermore, *no* TEXMFCNF environment variable is set. Now there's a problem of finding the texmf.cnf file: --- $>/usr/texbin/luatools --generate --verbose LuaTools | version 1.2.0 - 2006+ - PRAGMA ADE / CONTEXT LuaTools | variable SELFAUTOLOC set to /usr/texbin LuaTools | variable SELFAUTODIR set to /usr/texbin/.. LuaTools | variable SELFAUTOPARENT set to /usr/texbin/../.. LuaTools | variable TEXMFCNF set to {$SELFAUTODIR,$SELFAUTOPARENT}{,{/ share,}/texmf{-local,.local,}/web2c} LuaTools | no cnf files found (TEXMFCNF may not be set/known) LuaTools | LuaTools | runtime: 0.004 seconds --- The problem persists when I call the binary directly: --- $> /usr/local/context/2008-05-21/texmf-osx-intel/bin/luatools -- generate --verbose LuaTools | version 1.2.0 - 2006+ - PRAGMA ADE / CONTEXT LuaTools | variable SELFAUTOLOC set to /usr/texbin LuaTools | variable SELFAUTODIR set to /usr/texbin/.. LuaTools | variable SELFAUTOPARENT set to /usr/texbin/../.. LuaTools | variable TEXMFCNF set to {$SELFAUTODIR,$SELFAUTOPARENT}{,{/ share,}/texmf{-local,.local,}/web2c} LuaTools | no cnf files found (TEXMFCNF may not be set/known) LuaTools | LuaTools | runtime: 0.007 seconds --- Obviously, the problem is the symbolic link because LuaTeX tries and finds argv[0] in $PATH but doesn't resolve /usr/texbin to its actual location on disk. It's clear that in this situation there can't be any difference between the distribution-independent syntax of calling luatex and the direct invocation. Do you think LuaTeX's path discovery mechanism could be adapted to handle symbolic links? All the best, Oliver

Hans Hagen wrote:

Oliver Buerschaper wrote:

...

Obviously, the problem is the symbolic link because LuaTeX tries and finds argv[0] in $PATH but doesn't resolve /usr/texbin to its actual location on disk. It's clear that in this situation there can't be any difference between the distribution-independent syntax of calling luatex and the direct invocation.

Do you think LuaTeX's path discovery mechanism could be adapted to handle symbolic links?

there is lfs.symlinkattributes but i'm somewhat reluctant to usinmg platform specific things; if i have time i'll see if this function returns something useful

No need, here is function that works and is portable to all platforms: function input.getownpath() local olddir = lfs.currentdir() if not input.ownpath then for p in string.gmatch(os.getenv("PATH"),"[^"..io.pathseparator.."]+") do lfs.chdir(p) p = lfs.currentdir() local b = file.join(p,input.ownbin) if lfs.isfile(b..".exe") or lfs.isfile(b) then input.ownpath = p break end end if not input.ownpath then input.ownpath = '.' end end lfs.chdir(olddir) return input.ownpath end Oliver, I have emailed you a patched version of luatools.lua. Can you try to make sure that it works for you as well? (It should, but ...) Best wishes, Taco

Taco Hoekwater wrote:

The chosen method has a few small flaws in bordercases that will be corrected eventually (in a new luatex release) but it seems to work well enough for now

Before Oliver tells me I am being vague again: ;-) What I mean is this: in the full path "/usr/texbin/luatex", "texbin" can be a symlink to a different directory and that will be followed, but if "luatex" itself is a symlink to a file in a different directory, that will not be noticed. Solving that in pure lua is pretty hard, so I will add a predefined "full executable path" variable in the next luatex beta. Best wishes, Taco

Oliver Buerschaper wrote:

Another question: what part of the LuaTeX chain is responsible for locating the configuration files? Is it the scripts mtxrun.lua and luatools.lua or the binary texlua itself? You mentioned something about luatools only but now that Hans uploaded the entire beta ...

this is done in the lua code and taco and i spent quit esome time experimenting with this; our filosophy is to hard code as less as possible and let lud du the work unless this is not possible; here we have a border case with these symlinks; an option would to add functions that provide info about the symlink, but then, we don;t wan tto add too many functions to the core; hoever in this case it may be that we need to let the binary provide info about the path where the binary resides (also because it may be a bit playform depedent) (btw, the lua code is in luat-inp but merged into mtxrun and luatools so that these scripts can operate independent of libraries, because otherwise we can have another chicken or egg problem) at some point part of the interface in luat-inp will be public i.e. users can use it themselves, but first i need to clean up that code Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

Who's "lud"?

Lua :-)

...

What I mean is this: in the full path "/usr/texbin/luatex", "texbin" can be a symlink to a different directory and that will be followed, but if "luatex" itself is a symlink to a file in a different directory, that will not be noticed.

I think this should be just fine for the moment ...

Extended report ... in short, format compilation fails because of file location problems. Apparently generating the LuaTeX cache succeeds though. See below. --- bash# luatools --generate --verbose LuaTools | version 1.2.0 - 2006+ - PRAGMA ADE / CONTEXT LuaTools | following symlink /usr/texbin to /usr/local/context/ 2008-05-24-beta/texmf-osx-intel/bin LuaTools | variable SELFAUTOLOC set to /usr/local/context/2008-05-24- beta/texmf-osx-intel/bin LuaTools | variable SELFAUTODIR set to /usr/local/context/2008-05-24- beta/texmf-osx-intel/bin/.. LuaTools | variable SELFAUTOPARENT set to /usr/local/context/ 2008-05-24-beta/texmf-osx-intel/bin/../.. LuaTools | variable TEXMFCNF set to {$SELFAUTODIR,$SELFAUTOPARENT}{,{/ share,}/texmf{-local,.local,}/web2c} LuaTools | loading configuration file /usr/local/context/2008-05-24- beta/texmf/web2c/texmfcnf.lua LuaTools | loading /usr/local/context/2008-05-24-beta/texmf/web2c/ texmf.cnf LuaTools | preparing configuration for /usr/local/context/2008-05-24- beta/texmf/web2c LuaTools | saving configuration in /usr/local/context/2008-05-24-beta/ texmf-cache/luatex-cache/context/e09ab1c5fd12ff1663572f339200d7b9/ trees/01087b0ca69d21f3ef3ffa1e45445465.tma LuaTools | compiling configuration to /usr/local/context/2008-05-24- beta/texmf-cache/luatex-cache/context/e09ab1c5fd12ff1663572f339200d7b9/ trees/01087b0ca69d21f3ef3ffa1e45445465.tmc LuaTools | locating list of /usr/local/context/2008-05-24-beta/texmf- project LuaTools | locating list of /usr/local/context/2008-05-24-beta/texmf- fonts LuaTools | locating list of /usr/local/context/2008-05-24-beta/texmf- local LuaTools | locating list of /usr/local/context/2008-05-24-beta/texmf- context LuaTools | locating list of /usr/local/context/2008-05-24-beta/texmf- extra LuaTools | locating list of /usr/local/context/2008-05-24-beta/texmf LuaTools | scanning path /usr/local/context/2008-05-24-beta/texmf-local LuaTools | 0 files found on 0 directories with 0 uppercase remappings LuaTools | scanning path /usr/local/context/2008-05-24-beta/texmf- context LuaTools | 1349 files found on 125 directories with 0 uppercase remappings LuaTools | scanning path /usr/local/context/2008-05-24-beta/texmf LuaTools | 1550 files found on 82 directories with 60 uppercase remappings LuaTools | preparing files for /usr/local/context/2008-05-24-beta/texmf LuaTools | saving files in /usr/local/context/2008-05-24-beta/texmf- cache/luatex-cache/context/e09ab1c5fd12ff1663572f339200d7b9/trees/ 058e57e5173e90883a3ba926c33fe22d.tma LuaTools | compiling files to /usr/local/context/2008-05-24-beta/texmf- cache/luatex-cache/context/e09ab1c5fd12ff1663572f339200d7b9/trees/ 058e57e5173e90883a3ba926c33fe22d.tmc LuaTools | preparing files for /usr/local/context/2008-05-24-beta/ texmf-context LuaTools | saving files in /usr/local/context/2008-05-24-beta/texmf- cache/luatex-cache/context/e09ab1c5fd12ff1663572f339200d7b9/trees/ 3c541a92ee2a3a79f77458e195b0a0f8.tma LuaTools | compiling files to /usr/local/context/2008-05-24-beta/texmf- cache/luatex-cache/context/e09ab1c5fd12ff1663572f339200d7b9/trees/ 3c541a92ee2a3a79f77458e195b0a0f8.tmc LuaTools | preparing files for /usr/local/context/2008-05-24-beta/ texmf-local LuaTools | saving files in /usr/local/context/2008-05-24-beta/texmf- cache/luatex-cache/context/e09ab1c5fd12ff1663572f339200d7b9/trees/ 678bda37a3a32e3b1917a023bf896e91.tma LuaTools | compiling files to /usr/local/context/2008-05-24-beta/texmf- cache/luatex-cache/context/e09ab1c5fd12ff1663572f339200d7b9/trees/ 678bda37a3a32e3b1917a023bf896e91.tmc LuaTools | LuaTools | runtime: 0.255 seconds --- So far, so good. Now creating the format fails although the file texmf- context/tex/context/base/context.tex is present and perfectly readable :-( --- bash# luatools --ini --compile --verbose cont-en LuaTools | version 1.2.0 - 2006+ - PRAGMA ADE / CONTEXT LuaTools | following symlink /usr/texbin to /usr/local/context/ 2008-05-24-beta/texmf-osx-intel/bin LuaTools | variable SELFAUTOLOC set to /usr/local/context/2008-05-24- beta/texmf-osx-intel/bin LuaTools | variable SELFAUTODIR set to /usr/local/context/2008-05-24- beta/texmf-osx-intel/bin/.. LuaTools | variable SELFAUTOPARENT set to /usr/local/context/ 2008-05-24-beta/texmf-osx-intel/bin/../.. LuaTools | variable TEXMFCNF set to {$SELFAUTODIR,$SELFAUTOPARENT}{,{/ share,}/texmf{-local,.local,}/web2c} LuaTools | loading configuration file /usr/local/context/2008-05-24- beta/texmf/web2c/texmfcnf.lua LuaTools | loading configuration for /usr/local/context/2008-05-24- beta/texmf/web2c from /usr/local/context/2008-05-24-beta/texmf-cache/ luatex-cache/context/e09ab1c5fd12ff1663572f339200d7b9/trees/ 01087b0ca69d21f3ef3ffa1e45445465 LuaTools | locating list of /usr/local/context/2008-05-24-beta/texmf- project LuaTools | locating list of /usr/local/context/2008-05-24-beta/texmf- fonts LuaTools | locating list of /usr/local/context/2008-05-24-beta/texmf- local LuaTools | locating list of /usr/local/context/2008-05-24-beta/texmf- context LuaTools | locating list of /usr/local/context/2008-05-24-beta/texmf- extra LuaTools | locating list of /usr/local/context/2008-05-24-beta/texmf LuaTools | loading files for /usr/local/context/2008-05-24-beta/texmf- local from /usr/local/context/2008-05-24-beta/texmf-cache/luatex-cache/ context/e09ab1c5fd12ff1663572f339200d7b9/trees/ 678bda37a3a32e3b1917a023bf896e91 LuaTools | loading files for /usr/local/context/2008-05-24-beta/texmf- context from /usr/local/context/2008-05-24-beta/texmf-cache/luatex- cache/context/e09ab1c5fd12ff1663572f339200d7b9/trees/ 3c541a92ee2a3a79f77458e195b0a0f8 LuaTools | loading files for /usr/local/context/2008-05-24-beta/texmf from /usr/local/context/2008-05-24-beta/texmf-cache/luatex-cache/ context/e09ab1c5fd12ff1663572f339200d7b9/trees/ 058e57e5173e90883a3ba926c33fe22d LuaTools | creating initialization file cont-en LuaTools | using library path : /usr/local/context/2008-05-24-beta/ texmf-context/tex/context/base LuaTools | using lua libraries: l-string.lua l-lpeg.lua l-table.lua l- boolean.lua l-number.lua l-set.lua l-unicode.lua l-md5.lua l-os.lua l- io.lua l-file.lua l-url.lua l-dir.lua l-utils.lua l-tex.lua luat- env.lua luat-lib.lua luat-inp.lua luat-tmp.lua luat-zip.lua luat-tex.lua LuaTools | using compiled initialization file cont-en.luc LuaTools | using lua initialization file cont-en.luc LuaTools | running command: luatex --ini --lua="cont-en.luc" "/usr/ local/context/2008-05-24-beta/texmf-context/tex/context/base/cont- en.tex" \\dump (/usr/local/context/2008-05-24-beta/texmf-context/tex/context/base/ cont-en.tex ! I can't find file `context.tex'. l.16 \input context.tex Please type another input file name: ^C bash# ! Emergency stop. l.16 \input context.tex mkiv lua stats : used config path - /usr/local/context/2008-05-24- beta/texmf/web2c/texmf.cnf mkiv lua stats : used cache path - /Users/oliver mkiv lua stats : input load time - 0.005 seconds mkiv lua stats : current memory usage - 612076 bytes mkiv lua stats : node memory usage - 1 dir --- Strangely LuaTeX seems to use yet another cache path during format generation. I'm puzzled. Oliver

Oliver Buerschaper wrote:

...

...

What I mean is this: in the full path "/usr/texbin/luatex", "texbin" can be a symlink to a different directory and that will be followed, but if "luatex" itself is a symlink to a file in a different directory, that will not be noticed. I think this should be just fine for the moment ...

Extended report ... in short, format compilation fails because of file location problems. Apparently generating the LuaTeX cache succeeds though. See below.

taco and i are clueless ... we need more info -- ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

There is a new beta release now. It would be nice if you could delete all luatex-cache's on your disk and then try again.

Will do. Oliver

Oliver Buerschaper wrote:

...

...

Extended report ... in short, format compilation fails because of file location problems. Apparently generating the LuaTeX cache succeeds though. See below. taco and i are clueless ... we need more info

Will try and have a look at luat-tmp later the day and see whether I'll understand enough in order to implement your suggestion ... my

Just ignore this suggestion, it is superseded by the new beta Best wishes, Taco

Hi Oliver, Oliver Buerschaper wrote:

/var/folders/d5/d5kShiN8Gla+FttIH2Kn4E+++TM/-Tmp-

How on earth did you manage to get this path? Is that $TMPDIR ? If so, it is probably different for each user, and in that case the distinction between root/user could be important. Best wishes, Taco

Oliver Buerschaper wrote:

Actually I've just realized that $TMPDIR isn't set for root. Most probably this is why LuaTeX chose my home folder (sudo -s made me root!) during database and format generation as its fall-back cache location because there was no $TMPDIR. Still, LuaTeX should've used the cache path specified in texmfcnf.lua in the first place.

Just checking: does /usr/local/context/2008-05-26-beta/texmf-cache exist and is it writeable for everybody?

Oliver Buerschaper wrote:

...

...

Actually I've just realized that $TMPDIR isn't set for root. Most probably this is why LuaTeX chose my home folder (sudo -s made me root!) during database and format generation as its fall-back cache location because there was no $TMPDIR. Still, LuaTeX should've used the cache path specified in texmfcnf.lua in the first place.

Just checking: does

/usr/local/context/2008-05-26-beta/texmf-cache

exist and is it writeable for everybody?

It exists but used to have permissions 775 which were the ones rsync set. Changed them to 777 and retried database generation as normal user first.

The directory above (set in texmfcnf.lua) is still ignored. Since $TMPDIR exists for a normal user LuaTeX chooses this one for its cache location.

After wiping all luatex-cache directories from my machine I made another fresh attempt at database generation, this time as root. Again the cache settings in texmfcnf.lua are ignored and since there's no $TMPDIR for root LuaTeX resorts to $HOME.

but is texmfcnf.lua loaded? (--verbose) ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

Oliver Buerschaper wrote:

...

but is texmfcnf.lua loaded? (--verbose)

Sure. Both for root and the normal user.

ok, another attempt .. new beta -- ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

On Wed, May 28, 2008 at 6:39 PM, Oliver Buerschaper wrote:

P.S. Can you put that "context" stub back into the minimals?

It's only missing in the scripts/context/stubs/unix/ folder, in scripts/context/stubs/mswin/ it is present :) Mojca

well, i copy a non existing unix stub into the zip

is this one ok?

#!/bin/sh mtxrun --script context "$@"

Perfect. It's exactly the one that used to be present in the beta 2008.05.24 20:06 (both in the binaries as well as the unix stubs directories). I was wondering why you removed it at all ;-) Oliver

Oliver Buerschaper wrote:

The directory above (set in texmfcnf.lua) is still ignored. Since $TMPDIR exists for a normal user LuaTeX chooses this one for its cache location.

Suddenly mine is ignored as well. Silly me, I should have tested that much earlier, sorry about that. Will get back to you.

Arthur Reutenauer wrote:

...

taco and i are clueless ... we need more info

I think there is a problem with input.aux.expanded_path ... I have more or less the same system as Oliver, and this week-end I had the same problem as him, but with today's release I have a different one:

(I changed only the PATH)

i'll look into it ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

Arthur Reutenauer wrote:

...

So, assuming /opt/tex/texmf-linux/bin, it attempts the following directories, in this order:

[/opt/tex/texmf]/web2c % two parents up [/opt/tex/texmf-linux]/texmf/web2c % one parent up [/opt/tex]/texmf-local/web2c [/opt/tex/texmf-linux]/texmf-local/web2c

Thats's great! It works on my machine without the need to set any environment variable (I even tried two different arrangments of the directories).

well, you need to set the path i assume -) ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

Arthur Reutenauer wrote:

...

well, you need to set the path i assume -)

Yes, sure :-) But it's much easier to control than all the variables set by setuptex. Now I can switch between the minimals and TeX Live by simply changing my PATH. Great!

By the way, the rsync minimals have been updated, so it also works with them now.

Be warned that this only affects mkiv for now. Last time I tried, texexec could not run without 4 or 5 variables (but admittedly that was quite a while ago, when it was still in perl). Best wishes, Taco

Hi, Hans Hagen wrote:

Taco Hoekwater wrote:

...

Arthur Reutenauer wrote:

...

...

well, you need to set the path i assume -) Yes, sure :-) But it's much easier to control than all the variables set by setuptex. Now I can switch between the minimals and TeX Live by simply changing my PATH. Great!

By the way, the rsync minimals have been updated, so it also works with them now.

Be warned that this only affects mkiv for now. Last time I tried, texexec could not run without 4 or 5 variables (but admittedly that was quite a while ago, when it was still in perl).

it depends a bit on the logic; the previous code in luatools/mtxrun that tried to figure out the current location of the cnf file was derived from texmfstart which itself was based on older distributions; there have been changes in tds, names of trees, etc etc so texmfstart may as well be wrong there; 9it's one of those things i cannot test)

I just tried texmfstart without any variables (but with adjusted PATH, of course), and it seems to work just fine here, much better than expected. I had to regenerate all my formats because the texmf-linux subtree is no longer part of the TEXMF searches, but after texexec --make all now seems to be fine (with the new formats in texmf-project). To other people: you may want to try this also. Perhaps setuptex has been obsolete for years without any of use realising it ;-) Best wishes, Taco

Arthur Reutenauer wrote:

...

well, you need to set the path i assume -)

Yes, sure :-) But it's much easier to control than all the variables set by setuptex. Now I can switch between the minimals and TeX Live by simply changing my PATH. Great!

hm, but that not much more keying than calling setuptex -) anyhow. i'm glad that it works (btw, it would be interesting to know to what extend linux/osx set up possible conflicting variables; when i made setuptex i had to take care of everything that a sysadm had set up (for instance in a project outside our company where the users has to deal with very old tex's hanging around and not being updated or lagging two years behind, and setuptex is then the only way to keep interference away) Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

(btw, it would be interesting to know to what extend linux/osx set up possible conflicting variables; when i made setuptex i had to take care of everything that a sysadm had set up (for instance in a project outside our company where the users has to deal with very old tex's hanging around and not being updated or lagging two years behind, and setuptex is then the only way to keep interference away)

On my machine (Mac OS X 10.5.2) with only the MacTeX distribution (TeX Live 2007 for Mac) installed no environment variable whatsoever is set that's related to TeX in any way ... I can't comment on older distributions though. Oliver

On Wed, May 21, 2008 at 7:27 PM, Arthur Reutenauer wrote:

...

So, assuming /opt/tex/texmf-linux/bin, it attempts the following directories, in this order:

[/opt/tex/texmf]/web2c % two parents up [/opt/tex/texmf-linux]/texmf/web2c % one parent up [/opt/tex]/texmf-local/web2c [/opt/tex/texmf-linux]/texmf-local/web2c

Mac (gwtex, mactex) also uses texmf.local. (Well, also many others, but that one is most evident.)

Thats's great! It works on my machine without the need to set any environment variable (I even tried two different arrangments of the directories).

I have just tested it and it didn't work as expected. MK II:

texexec --make --xtx --all TeXExec | using search method 'kpsewhich' TeXExec | updating file database mktexlsr: Updating /Users/mojca/context/tex/texmf/ls-R... mktexlsr: Updating /Users/mojca/context/tex/texmf-context/ls-R... mktexlsr: Updating /Users/mojca/context/tex/texmf-local/ls-R... mktexlsr: Done. TeXExec | using tex engine xetex TeXExec | using tex format path /Users/mojca/context/tex/texmf-context/web2c/xetex TeXExec | generating tex format cont-en This is XeTeX, Version 3.1415926-2.2-0.998.5-dev (Web2C 7.5.6) (INITEX) \write18 enabled. entering extended mode ... Beginning to dump on file metafun.mem (mem=metafun 2008.5.23) at most 3068 strings of total length 40865 49365 memory locations dumped; current usage is 12822&36400 1788 symbolic tokens Transcript written on metafun.log. TeXExec | TeXExec | tex engine path: /Users/mojca/context/tex/texmf-context/web2c/xetex TeXExec | mps engine path: /Users/mojca/context/tex/texmf-context/web2c TeXExec | TeXExec | tex: 23/05/2008 02:05:17 > /Users/mojca/context/tex/texmf-context/web2c/xetex/cont-en.fmt (4503548) TeXExec | tex: 23/05/2008 02:05:23 > /Users/mojca/context/tex/texmf-context/web2c/xetex/cont-nl.fmt (4544450) TeXExec | tex: 23/05/2008 02:05:25 > /Users/mojca/context/tex/texmf-context/web2c/xetex/mptopdf.fmt (1786949) TeXExec | mps: 23/05/2008 02:05:25 > /Users/mojca/context/tex/texmf-context/web2c/metafun.mem (491080) TeXExec | TeXExec | runtime: 15.745237

texexec --xtx a.tex TeXExec | processing document 'a.tex' TeXExec | no ctx file found TeXExec | tex processing method: context TeXExec | TeX run 1 TeXExec | writing option file a.top TeXExec | using randomseed 1213 TeXExec | tex engine: xetex TeXExec | tex format: cont-en This is XeTeX, Version 3.1415926-2.2-0.998.5-dev (Web2C 7.5.6) \write18 enabled. kpathsea: Running mktexfmt cont-en.fmt /usr/texbin/mktexfmt: line 336: /Users/mojca/context/tex/texmf/texconfig/tcfmgr: No such file or

(formats have always flied to texmf-osx-intel so far) But ... directory fmtutil: config file `fmtutil.cnf' not found. I can't find the format file `cont-en.fmt'! MK IV:

context a.tex

MtxRun | error unable to locate texmfcnf.lua MtxRun | loading configuration for /Users/mojca/context/tex/texmf/web2c from /Users/mojca/luatex-cache/context/bce2a35a5270714da084d11faf744f76/trees/71a1a190289e8fc90263c427daa9d235 MtxRun | locating list of /Users/mojca/context/tex/texmf-project MtxRun | locating list of /Users/mojca/context/tex/texmf-fonts MtxRun | locating list of /Users/mojca/context/tex/texmf-local MtxRun | locating list of /Users/mojca/context/tex/texmf-context MtxRun | locating list of /Users/mojca/context/tex/texmf-extra MtxRun | locating list of /Users/mojca/context/tex/texmf MtxRun | loading files for /Users/mojca/context/tex/texmf-local from /Users/mojca/luatex-cache/context/bce2a35a5270714da084d11faf744f76/trees/9cef58cec78e4ef962f7764cc0d0af77 MtxRun | loading files for /Users/mojca/context/tex/texmf-context from /Users/mojca/luatex-cache/context/bce2a35a5270714da084d11faf744f76/trees/05c539c10b6a70b446df0913d1c04b99 MtxRun | loading files for /Users/mojca/context/tex/texmf from /Users/mojca/luatex-cache/context/bce2a35a5270714da084d11faf744f76/trees/9cd0d756a20671533c8167942232650c MtxRun | using script: /Users/mojca/context/tex/texmf-context/scripts/context/lua/mtx-context.lua MtxRun | loading configuration for /Users/mojca/context/tex/texmf/web2c from /Users/mojca/luatex-cache/context/bce2a35a5270714da084d11faf744f76/trees/71a1a190289e8fc90263c427daa9d235 MtxRun | error no format found with name cont-en MtxRun | total runtime: 0.011 I definitely need to add texmfcnf.lua, but just a quick overview of non-working solutions :) Mojca

On Fri, May 23, 2008 at 9:13 AM, Taco Hoekwater wrote:

Mojca Miklavec wrote:

...

On Wed, May 21, 2008 at 7:27 PM, Arthur Reutenauer wrote:

...

...

So, assuming /opt/tex/texmf-linux/bin, it attempts the following directories, in this order:

[/opt/tex/texmf]/web2c % two parents up [/opt/tex/texmf-linux]/texmf/web2c % one parent up [/opt/tex]/texmf-local/web2c [/opt/tex/texmf-linux]/texmf-local/web2c

Mac (gwtex, mactex) also uses texmf.local. (Well, also many others, but that one is most evident.)

Try this: if you run

$ kpsewhich texmf.cnf

when there are *no* variables set, what file does it return? Extending the lua function is easy, we just need to know how.

Actually, I just tried that on my machine and it seems that it first tries three things we don't do:

/opt/tex/texmf-linux/bin/texmf.cnf and /opt/tex/texmf-linux/texmf.cnf and /opt/tex/texmf.cnf

the third of those should capture your file, right?

Yes, thanks :) I only need to test this a bit more.

...

I have just tested it and it didn't work as expected.

MK II:

...

texexec --make --xtx --all

(formats have always flied to texmf-osx-intel so far)

This depends on your actual texmf.cnf settings. Likely they are not quite te same as the ENV variables used to be.

Esp, check TEXFORMATS.

TEXMF = {!!$TEXMFPROJECT,!!$TEXMFFONTS,!!$TEXMFLOCAL,!!$TEXMFCONTEXT,!!$TEXMFEXTRA,!!$TEXMFMAIN} TEXFORMATS = .;$TEXMF/web2c{/$engine,}

...

I can't find the format file `cont-en.fmt'!

Same issue here. To get back to what you used to have, you probably want these two in texmf.cnf:

TEXMFOS = $SELFAUTODIR TEXFORMATS = .;$TEXMFOS/web2c/{$engine,}

Great, thanks :) This works wonderful :) mkii is now (almost) setuptex-free? I need to check the rest, but I'm already fixing this one. Mojca

Mojca Miklavec wrote:

Which variables on this list are still needed? only mpmems?

Perhaps pdftex still needs TEXPOOL?

Mojca Miklavec wrote:

Which variables on this list are still needed? only mpmems?

MFBASES = .;$TEXFORMATS MPMEMS = .;$TEXFORMATS

TEXPOOL = .;$TEXFORMATS MFPOOL = .;$TEXFORMATS MPPOOL = .;$TEXFORMATS

...

...

from texmf.cnf, and $TMP doesn't exist on mac, Which is odd, to say the least. Where *do* temporary files go on OSX?

i think that only mf still has a pool file (and probably also tex and etex and omega and aleph)

To /tmp, but there is no $TMP or $TEMP defined anywhere.

weird, maybe they use some apple name, like $WASTEBIN or $THRASH Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

...

To /tmp, but there is no $TMP or $TEMP defined anywhere.

weird, maybe they use some apple name, like $WASTEBIN or $THRASH

We already had that discussion: very few systems actually set TMP or TEMP (the first public releases of Mark IV used to assume it was set, and subsequently created luatex-cache in the current directory since it wasn't). Arthur

few? afaik all windows and linux systems -)

No :-) I have to insist like one year ago, *no* Linux machine I have an account on does set TMP or TEMP (using various distributions). Maybe that's a shell problem, I don't know, but on any case you can't rely on it.

anyhow, taco and i decided now to default as follows: if CACHEPATH is not set:

cachepath = os.env["TEXMFCACHE"] or os.env["TMP"] or os.env["TEMP"] or os.env["TMPDIR"] or os.get["HOME"] or os.env["HOMEPATH"] or nil

Sounds reasonable, but why not test for the existence of the /tmp directory also? Arthur

weird because what then is the purpose of that variable

But if there is no such variable, what's the point of looking for its purpose? :-) I suppose that on Unix, functions like mkstemp have to be preferred for truly temporary directories. Arthur

2008/5/23 Hans Hagen :

sure, but it depends on tmp being cleaned up too; afaik /tmp and c:/temp are only cleaned up periodically e.g. when one runs out of space so it's a kind of indication of 'may be cleaned up'

Many Unixes clear /tmp at bootup. Even SUSE can do this. :-) http://en.wikipedia.org/wiki/Temporary_directory "In Unix and Linux, the global temporary directories are /tmp and /var/tmp. Typically, /var/tmp is for more permanent files, and /tmp is for more temporary files. See Filesystem Hierarchy Standard. In addition, a user can set his TMPDIR environment variable to point to a preferred directory (where the creation and modification of files is allowed)." Best Martin

Arthur Reutenauer wrote:

Sounds reasonable, but why not test for the existence of the /tmp directory also?

ok, to please you, i now stepwise test for existence; will b ein next beta upload Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

...

Which is odd, to say the least. Where *do* temporary files go on OSX?

To /tmp, but there is no $TMP or $TEMP defined anywhere.

In Mac OS X 10.5.2 there's a $TMPDIR environment variable if that helps ... Oliver

when discussing the cache Arthur had a preference for HOME, so home is tested before TMPDIR

On a Mac the (persistent) user cache should probably go to a subdirectoy of $HOME/Library/Caches/ because this is where most applications store their cache data. Will have to check back with the Apple file system specification though ... Oliver

when discussing the cache Arthur had a preference for HOME, so home is tested before TMPDIR

Actually, I insisted that HOME came before system-wide locations like TEXMFVAR, so that users could override it, and I think that's the most important. Arthur

So, assuming /opt/tex/texmf-linux/bin, it attempts the following directories, in this order:

[/opt/tex/texmf]/web2c % two parents up [/opt/tex/texmf-linux]/texmf/web2c % one parent up [/opt/tex]/texmf-local/web2c [/opt/tex/texmf-linux]/texmf-local/web2c

Shouldn't local have some precedence over texmf? (Maybe I'm wrong.) Just a side note: Here I have the cnf file under /usr/local/gwTeX/texmf.cnf The binaries are under /usr/local/gwTeX/bin/i386-apple-darwin8.8.1 but I didn't test it on gwTeX yet. The main idea - cnf file will be overwritten by every update if it's only in texmf/web2c. The user needs to be able to put it either to texmf-local or to the top (at the same level where setuptex is), and that one needs to have precedence over the one in texmf. Unrelated - how exactly do I set TEXMFCACHE to be in texmf-cache alongside other texmf trees? I have tried (I know that this cannot work): TEXMFCACHE = '$TEXROOT/texmf-cache', but TEXROOT is probably unknown anyway Maybe it's still respecting the order in TEXMFCACHE = $TMP;$TEMP;$TMPDIR;$TEMPDIR;$HOME;$TEXMFVAR;$VARTEXMF;. from texmf.cnf, and $TMP doesn't exist on mac, so home is the first existing folder (where I hate to keep the cache - it should at least have a preceeding dot then). The only problem with mkii: basically works, but texexec --make puts formats under texmf-context and the kpathsea doesn't find them for some reason. If format is put under texmf-engine/web2c, everything works out of the box afterwards (with only path variable being set). Mojca

Mojca Miklavec wrote:

...

Unrelated - how exactly do I set TEXMFCACHE to be in texmf-cache alongside other texmf trees? I have tried (I know that this cannot work): TEXMFCACHE = '$TEXROOT/texmf-cache', but TEXROOT is probably unknown anyway

I forgot to finish the sentence:

with this setting I get

luatools --generate

LuaTools | loading configuration file /Users/mojca/context/tex/texmf/web2c/texmfcnf.lua LuaTools | loading /Users/mojca/context/tex/texmf/web2c/texmf.cnfShould I create the cache path /texmf-cache? [yes|no] [no]

every time, independent on whether I answer with yes or no. And everytime cache is generated under $HOME/luatex-cache. I guess that $TOXROOT is simply empty and /texmf-cache is not writable by me.

indeed, TEXROOT is an internal setuptex variable; actually, the only thing that setuptex sets up is TEXMF, TEXMFCNF, TEXFORMATS and TEXMFCACHE; the rest is nilling interfering variables and a few intermedia ones that save typing (TEXOS, TEXROOT) in the path bases variant, you cannot use TEXOS and TEXROOT since they don't exist; you can try TEXMFCACHE=/tmp ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

Mojca Miklavec wrote:

Here I have the cnf file under /usr/local/gwTeX/texmf.cnf

huh, in the root of the tree? is that also true for upcoming versions? i'm not sure if we should support all variants of trees and maybe gwtex assumes an env var to be set? Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

Mojca Miklavec wrote:

On Fri, May 23, 2008 at 9:31 AM, Hans Hagen wrote:

...

Mojca Miklavec wrote:

...

Here I have the cnf file under /usr/local/gwTeX/texmf.cnf huh, in the root of the tree? is that also true for upcoming versions? i'm not sure if we should support all variants of trees and maybe gwtex assumes an env var to be set?

In gwTeX there is only one variable, namely PATH is set to /usr/texbin, and that is a symlink to the proper version of binaries (/usr/local/gwTeX/bin/i386-apple-darwin8.8.1 in my case). Nothing else is needed, even though Oliver might explain more details if needed.

From what Taco explained, that configuration should already be OK.

gwtex can have a patched default for TEXMFCNF (i.e the period instead of the -) ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

Hi Mojca, (and Hans) Mojca Miklavec wrote:

Shouldn't local have some precedence over texmf? (Maybe I'm wrong.)

Yes (see below).

Unrelated - how exactly do I set TEXMFCACHE to be in texmf-cache alongside other texmf trees? I have tried (I know that this cannot work): TEXMFCACHE = '$TEXROOT/texmf-cache', but TEXROOT is probably unknown anyway

It is not (because texmfcnf.lua is read before everything else). There is SELFAUTOPARENT; I guess that is the only variable already defined at that point. TEXMFCACHE = '$SELFAUTOPARENT/texmf-cache', should work.

Maybe it's still respecting the order in TEXMFCACHE = $TMP;$TEMP;$TMPDIR;$TEMPDIR;$HOME;$TEXMFVAR;$VARTEXMF;.

Not that ordering *exactly* because texmf.cnf is not read yet, but a compiled-in version of that.

from texmf.cnf, and $TMP doesn't exist on mac,

Which is odd, to say the least. Where *do* temporary files go on OSX?

The only problem with mkii: basically works, but texexec --make puts formats under texmf-context and the kpathsea doesn't find them for some reason. If format is put under texmf-engine/web2c, everything works out of the box afterwards (with only path variable being set).

See above. Note: You may even have to run mktexlsr after format generation, because you are now totally at the mercy of the kpathsea library. ------------------------------------------------------------------- Mostly for Hans and mkiv, here is some more information about these AUTO-things: It turns out here are actually three variables, each a well-defined value, single value. Starting from the actual binary location, they are the relative paths: SELFAUTOLOC=. SELFAUTODIR=.. SELFAUTOPARENT=../.. And kpathsea uses these variables in a compile-time default of TEXMFCNF. My 'kpsewhich' actually tries for texmf.cnf in the following chain: /opt/tex/texmf-linux/bin /opt/tex/texmf-linux /opt/tex /opt/tex/texmf-linux/bin/share/texmf-local/web2c /opt/tex/texmf-linux /share/texmf-local/web2c /opt/tex /share/texmf-local/web2c /opt/tex/texmf-linux/bin/texmf-local/web2c /opt/tex/texmf-linux /texmf-local/web2c /opt/tex /texmf-local/web2c /opt/tex/texmf-linux/bin/share/texmf/web2c /opt/tex/texmf-linux /share/texmf/web2c /opt/tex /share/texmf/web2c /opt/tex/texmf-linux/bin/texmf/web2c /opt/tex/texmf-linux /texmf/web2c /opt/tex /texmf/web2c . /home/fmorel/Build/source/inst/texmf/web2c /home/fmorel/Build/source/inst/texmf/web2c And that is because my build/texk/kpathsea/texmf.cnf has TEXMFCNF = {$SELFAUTOLOC,$SELFAUTODIR,$SELFAUTOPARENT}{,{/share,}/texmf{-local,}/web2c} The '.' is also useful methinks, but the two '/home/fmorel' ones are a bit of a fluke: they are added based on the user that ran autoconf, which is almost certainly irrelevant. At first glance, kpsewich seems it indeed finds only the first texmf.cnf, but that may be because I only have one :-) Best wishes, Taco

On Mo, 19 Mai 2008, Mojca Miklavec wrote:

Any ideas if getting rid of dependency on environmental variables is doable?

And please also of that *stupid* user-specific luatex-cache. There must be a way to run-time merge a system-wide luatex-cache with a luatex-cache only for the user files. like mktexlsr. Please do NOT reinvent the wheel formed like a rectangle... see my other email. Best wishes Norbert ------------------------------------------------------------------------------- Dr. Norbert Preining Vienna University of Technology Debian Developer Debian TeX Group gpg DSA: 0x09C5B094 fp: 14DF 2E6C 0307 BE6D AD76 A9C0 D2BF 4AA3 09C5 B094 ------------------------------------------------------------------------------- The story goes that I first had the idea for THHGTTG while lying drunk in a field in Innsbruck (or `Spain' as the BBC TV publicity department authorititively has it, probably because it's easier to spell). --- Foreward by DNA. --- Douglas Adams, The Hitchhikers Guide to the Galaxy

Mojca Miklavec wrote:

On Mon, May 19, 2008 at 9:03 AM, Taco Hoekwater wrote:

...

Norbert Preining wrote:

...

Well not surprising ... there is NO context script anywhere in the distribution!

Am I missing something? The script is:

scripts/context/lua/mtx-context.lua

If mtxrun cannot find it, something went wrong running

luatools --generate

It seems to me that this is actually a yet unanswered FAQ: - the thread about AucTeX on ntg-context

dunno, i never used emacs

- Oliver asked me about how to run LuaTeX without having to set any environmental variables except for PATH adjustment; that would make the mac installer much easier to cope with - in most TeX distributions, one simply puts texmf.cnf somewhere above the binary (selfautoparent trickery), and then everything works out of the box

i've come to hate this autofoo trickery since there is no resl standard and whatever one cooks up eventually something else happens

- similar question raised by Vyatcheslav for Windows

it would ne no problem if we could be sure that the binary lives relative to the tree (same for cnf file) but there is no clear policy for this (for instance on linux a cnf file can live somewhere under etc too)

- I often have problems in my editor; if I launch it from terminal with proper environmental variables set, then it works OK, but if I want to launch it from menu, then it sometimes works and sometimes not (some weird interference with the existing distribution, in most cases it just doesn't find the cont-en.fmt and stops there already)

I just wanted to raise the question as it appears in just every installation of TeX where one wants to use ConTeXt with LuaTeX.

As long as one runs source setuptex in minimals, or source ~/.luatex in Thomas' instructions of hard-coded paths then LuaTeX runs OK, but if one would like to run LuaTeX without explicitly setting environmental variables (problematic within TeX editors), one soon runs into problems.

mtxrun --tree=<pathtotree> should work given that there is a setuptex.tmf file there

Minimals contain texmf.cnf hidden somewhere in the tree. But even if I put in in the top folder, it doesn't help much. Hans also uses setuptex.tmf for reading settings with LuaTeX in some way, but again, it doesn't happen automatically.

indeed, since there is no robust way to locate it

Jonathan Kew also told me that it should be enough to have a proper version of texmf.cnf in the top folder, and then setting PATH to include binaries should be enough. LuaTeX now works on standalone ConTeXt installation for more than a year, but it would be nice to make it work on Debian, TeX Live, MikTeX (that will soon provide it) .... as well.

Any ideas if getting rid of dependency on environmental variables is doable?

i keep thinking about it but i'm not in the mood to implement a messy fuzzy logic solution thet eventually will bite us for instance, can you cook up a list of locations for each platform that is guaranteed correct? (no problem for the minimals probably)

PS: ping hans@miktex

hm Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

Norbert Preining wrote:

Arggg... does that mean that anyone wanting to use simply context as a program as to set up all the luatex/context stuff, with independent database etc etc?

indeed, mkiv will not use kpse; at some point i want to play with texmf.zip texmf-local.zip technically it's already possible but i need some more time to play with it

Again, you should definitely implement some level of global setup possibility!!!

you mean shared for users?

Either you want context in distributions, or we forget about that and ask people to install context by themselves! It is impossible to setup a context installation for MkIV in a decent way on a multi-user system. It is simply *impossible* ATM.

well, mkiv is still somewhat experimental ... if a day had more hours ...

I already suggested several times that there is a *system*wide* luatex-cache which can be updated/regenerated everytime a distribution package installs new files, very similar to mktexlsr.

i know, and its noted somewhere ...

Now that is *really* broken on context.

should kind of work for pdftex an xetex (in the past texexec even called fmtutil on a unix system but when thomas refused to support $engine i kicked that out)

Again, if you want to be a system for strictly *single*user* systems, please let me know and I will remove context from Debian ...

keep in mind that i hav eno experience with multi user systems (ok, long ago on vms) ... although on my linux boxes i have a user hagen, i'm always logged in as root -)

AHHh, enough of my frustrated flames. Please don't take it personally. But I have written again and again about that, and it seems nobody seems to care nor to listen.

come on, we do care, but it's a moving train ... Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

What we actually need, and that is the same for TeX Live and for all distributions (at least inmy imagination), is:

- one (or several) system wide luatex cache thingies - one per user (if the user wants to) - a way to update the cache for specific trees only (luatools --system-only ...) - and the adaption of the lua functions that the different caches are merged at loading time, the later (user) taking precedence over the former.

That would already be some big step forward.

...

come on, we do care, but it's a moving train ...

;-) Let me sometimes get rid of some frustration .... ;-))) It does not happen often anyway.

I really wonder how (if at all) LuaTeX is going to work on MikTeX ... :) ConTeXt is broken there already, and ConTeXt is the only tool that really benefits from LuaTeX (apart from some of your tools on TL). The new search capabilities are likely to break the MikTeX's auto-install functionality as well if nobody will push Hans to fix it (or implement it himself). Mojca

Mojca Miklavec wrote:

...

What we actually need, and that is the same for TeX Live and for all distributions (at least inmy imagination), is:

- one (or several) system wide luatex cache thingies - one per user (if the user wants to) - a way to update the cache for specific trees only (luatools --system-only ...) - and the adaption of the lua functions that the different caches are merged at loading time, the later (user) taking precedence over the former.

That would already be some big step forward.

...

come on, we do care, but it's a moving train ... ;-) Let me sometimes get rid of some frustration .... ;-))) It does not happen often anyway.

I really wonder how (if at all) LuaTeX is going to work on MikTeX ... :) ConTeXt is broken there already, and ConTeXt is the only tool that really benefits from LuaTeX (apart from some of your tools on TL). The new search capabilities are likely to break the MikTeX's auto-install functionality as well if nobody will push Hans to fix it (or implement it himself).

mixtex is another issue mostly because it uses a diferent approach; we had it working so at some point it will work again; probably not soon for luatex/mkiv but that's experimental anyway i simply have no capacity to run everything here (e.g. in order to set up a proper windows vm for testing i need additional windows licences which i don't have Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

On Mo, 19 Mai 2008, Hans Hagen wrote:

for that to happen we need to define what system is and what is user in a pretty consistent and cross platform way (and not in the tetex n*m permutations way)

system: TEXMFDIST TEXMFMAIN TEXMFLOCAL TEXMFSYSCONFIG TEXMFSYSVAR user: TEXMFHOME TEXMFVAR TEXMFCONFIG THat is how TL2008 will work on *ALL* supported platforms, including windows. TEXMFHOME = ~/texmf where ~ -> $HOME on unix ~ -> %USERPROFILE% on win32 etc Yes, till TL2007 we didn't have the distinction of SYS <-> non-SYS on windows. But TL2008 will have exactely the same approach on all platforms, including the same scripts (updmap-sys vs updmap, ...) Best wishes Norbert ------------------------------------------------------------------------------- Dr. Norbert Preining Vienna University of Technology Debian Developer Debian TeX Group gpg DSA: 0x09C5B094 fp: 14DF 2E6C 0307 BE6D AD76 A9C0 D2BF 4AA3 09C5 B094 ------------------------------------------------------------------------------- MAVIS ENDERBY (n.) The almost-completely-forgotten girlfriend from your distant past for whom your wife has a completely irrational jealousy and hatred. --- Douglas Adams, The Meaning of Liff

On Mo, 19 Mai 2008, Hans Hagen wrote:

...

system: TEXMFDIST TEXMFMAIN TEXMFLOCAL TEXMFSYSCONFIG TEXMFSYSVAR

i suppose that this is the order in $TEXMF in tex live then

No no no, that was the order in which the stuff came to my mind!!!

just wondering ... how is a system wide update taking place (since local comes last) ... is the policy now "wipe out and install anew), if so, why still local?

update in TeX Live will happen simply by removing a sub-package and re-adding it, so the files will go normally to TEXMFDIST. But that depends on what is in the package. I don't understand with "wipe out and install anew"!? Yes, up to TL2007 every installation was practically a new one. With 2008 we will have the ability to install updated package and also other packages over the net into an already present installation. For you and Taco it might be interesting that if you want you can even set up some local repository and tell people to do: tlmgr -location www.pragme-ade.nl/context-tl update which will update the context package as shipped with TL to the one shipped from your server. Best wishes Norbert ------------------------------------------------------------------------------- Dr. Norbert Preining Vienna University of Technology Debian Developer Debian TeX Group gpg DSA: 0x09C5B094 fp: 14DF 2E6C 0307 BE6D AD76 A9C0 D2BF 4AA3 09C5 B094 ------------------------------------------------------------------------------- `That young girl is one of the least benightedly unintelligent organic life forms it has been my profound lack of pleasure not to be able to avoid meeting.' --- Marvin's first ever compliment about anybody. --- Douglas Adams, The Hitchhikers Guide to the Galaxy