t-filter: no table created when Unicode characters are included

Paul Menzel

29 Apr 2011 29 Apr '11

11:09 p.m.

Dear ConTeXt folks, the filter module installed using $ ./first-setup.sh --extras="t-filter" does not work with the following example, which is also attached. \usemodule[filter] \defineexternalfilter[rest][filtercommand={pandoc -f rst -t context -o \externalfilteroutputfile\space \externalfilterinputfile}] \starttext There should be a table here. \startrest ========= ========= test table ========= ========= α b c d ======== ======== \stoprest \stoptext The output file `minimal-temp-rest.tex` is just empty. Replacing »α« by »a« solves the problem. Running `filtercommand` in a terminal produces the *correct* output though and no empty file. Is that a known problem? I also include a patch adding this example into `tests/` in [1]. Thanks, Paul [1] https://github.com/adityam/filter

Attachments:

minimal.tex (text/x-tex — 326 bytes)
signature.asc (application/pgp-signature — 198 bytes)

Show replies by date

Paul Menzel

29 Apr 29 Apr

11:17 p.m.

New subject: [t-filter] [PATCH] tests: add test file for reST (reStructuredText)

Date: Fri, 29 Apr 2011 23:03:04 +0200 • This file includes Unicode characters. Signed-off-by: Paul Menzel --- tests/rest.tex | 13 +++++++++++++ 1 files changed, 13 insertions(+), 0 deletions(-) create mode 100644 tests/rest.tex diff --git a/tests/rest.tex b/tests/rest.tex new file mode 100644 index 0000000..8ec2f52 --- /dev/null +++ b/tests/rest.tex @@ -0,0 +1,13 @@ +\usemodule[filter] +\defineexternalfilter[rest][filtercommand={pandoc -f rst -t context -o \externalfilteroutputfile\space \externalfilterinputfile}] +\starttext +There should be a table here. +\startrest +========= ========= + test table +========= ========= + α b + c d +======== ======== +\stoprest +\stoptext -- 1.7.4.4

Aditya Mahajan

30 Apr 30 Apr

12:13 a.m.

New subject: t-filter: no table created when Unicode characters are included

On Fri, 29 Apr 2011, Paul Menzel wrote:

...

Dear ConTeXt folks,

the filter module installed using

$ ./first-setup.sh --extras="t-filter"

does not work with the following example, which is also attached.

\usemodule[filter] \defineexternalfilter[rest][filtercommand={pandoc -f rst -t context -o \externalfilteroutputfile\space \externalfilterinputfile}] \starttext There should be a table here. \startrest ========= ========= test table ========= ========= α b c d ======== ======== \stoprest \stoptext

The output file `minimal-temp-rest.tex` is just empty.

Replacing »α« by »a« solves the problem.

Weird.

...

Running `filtercommand` in a terminal produces the *correct* output though and no empty file.

Is that a known problem?

No. The content of the file should not affect whether the filter is run or not. Can you add \traceexternalfilters on the top and check the output (look for lines starting with t-filter).

...

I also include a patch adding this example into `tests/` in [1].

Thanks. I added it to the test suite. The file works correctly on my machine. Which version of pandoc are you using? Does the error also occur if you use the dev branch from github? $git checkout dev $context tests/rest Aditya

Paul Menzel

9:37 a.m.

New subject: t-filter: no table created when Unicode characters are included

Am Freitag, den 29.04.2011, 18:13 -0400 schrieb Aditya Mahajan:

...

On Fri, 29 Apr 2011, Paul Menzel wrote:

...

...
the filter module installed using

$ ./first-setup.sh --extras="t-filter"

does not work with the following example, which is also attached.

\usemodule[filter] \defineexternalfilter[rest][filtercommand={pandoc -f rst -t context -o \externalfilteroutputfile\space \externalfilterinputfile}] \starttext There should be a table here. \startrest ========= ========= test table ========= ========= α b c d ======== ======== \stoprest \stoptext

The output file `minimal-temp-rest.tex` is just empty.

Replacing »α« by »a« solves the problem.

Weird.

...
Running `filtercommand` in a terminal produces the *correct* output though and no empty file.

Is that a known problem?

No. The content of the file should not affect whether the filter is run or not. Can you add \traceexternalfilters on the top and check the output (look for lines starting with t-filter).

You already added it in your commit of the test file. […] resolvers > modules > loaded: 'filter' (t-filter.tex loading > ConTeXt User Module / Filter resolvers > modules > loaded: 'module-catcodes' (t-module-catcodes.tex loading > ConTeXt User Module / Module Catcodes )) (/opt/context/tex/texmf-context/tex/context/base/type-siz.mkiv) (/opt/context/tex/texmf-context/tex/context/base/type-otf.mkiv){/opt/context/tex/texmf-context/fonts/map/pdftex/context/mkiv-tx.map}{/opt/context/tex/texmf/fonts/map/dvips/lm/lm-math.map}{/opt/context/tex/texmf/fonts/map/dvips/lm/lm-rm.map}{/opt/context/tex/texmf-context/fonts/map/pdftex/context/mkiv-base.map} fonts > virtual math > the mapping is incomplete for 'txmath' at 7pt system > begin file tests/rest at line 11 t-filter > current filter : rest t-filter > base file : rest-temp-rest t-filter > input file : rest-temp-rest.tmp t-filter > output file : rest-temp-rest.tex t-filter > command : pandoc -f rst -t context -o rest-temp-rest.tex rest-temp-rest.tmp (rest-temp-rest.tex) backend > xmp > using file '/opt/context/tex/texmf-context/tex/context/base/lpdf-pdx.xml' pages > flushing realpage 1, userpage 1, subpage 1 system > end file tests/rest at line 21 ) […] $ more rest-temp-rest.t* :::::::::::::: rest-temp-rest.tex :::::::::::::: :::::::::::::: rest-temp-rest.tmp :::::::::::::: ========= ========= test table ========= ========= α b c d ======== ======== Please find `rest.log` attached.

...

...
I also include a patch adding this example into `tests/` in [1].

Thanks. I added it to the test suite.

You can save the mbox file and use `git am path/to/mbox` next time.

...

The file works correctly on my machine.

That is weired.

...

Which version of pandoc are you using?

$ pandoc --version pandoc 1.5.1.1 Compiled with syntax highlighting support for: Ada, Alert, Asp, Awk, Bash, Bibtex, C, Cmake, Coldfusion, Commonlisp, Cpp, Css, D, Djangotemplate, Doxygen, Dtd, Eiffel, Erlang, Fortran, Haskell, Html, Java, Javadoc, Javascript, Json, Latex, Lex, LiterateHaskell, Lua, Makefile, Matlab, Mediawiki, Modula3, Nasm, Objectivec, Ocaml, Octave, Pascal, Perl, Php, Postscript, Prolog, Python, Relaxngcompact, Rhtml, Ruby, Scala, Scheme, Sgml, Sql, SqlMysql, SqlPostgresql, Tcl, Texinfo, Xml, Xslt, Yacc Copyright (C) 2006-2010 John MacFarlane Web: http://johnmacfarlane.net/pandoc This is free software; see the source for copying conditions. There is no warranty, not even for merchantability or fitness for a particular purpose. As already written in my last mail. Executing the command directly works fine. $ pandoc -f rst -t context -o rest-temp-rest.tex rest-temp-rest.tmp $ more rest-temp-rest.t*:::::::::::::: rest-temp-rest.tex :::::::::::::: \placetable[here]{none} \starttable[|l|l|] \HL \NC test \NC table \NC\AR \HL \NC α \NC b \NC\AR \NC c \NC d \NC\AR \HL \stoptable :::::::::::::: rest-temp-rest.tmp :::::::::::::: ========= ========= test table ========= ========= α b c d ======== ========

...

Does the error also occur if you use the dev branch from github?

Yes.

...

$git checkout dev $context tests/rest

For the Git newbies. If you do this the first time, you need to do $ git checkout -b dev origin/dev beforehand. Thanks, Paul

Aditya Mahajan

5:10 p.m.

New subject: t-filter: no table created when Unicode characters are included

On Sat, 30 Apr 2011, Paul Menzel wrote:

...

Am Freitag, den 29.04.2011, 18:13 -0400 schrieb Aditya Mahajan:

...
On Fri, 29 Apr 2011, Paul Menzel wrote:

...
...
the filter module installed using does not work with the following example, which is also attached.

The output file `minimal-temp-rest.tex` is just empty.

Replacing »α« by »a« solves the problem.

Running `filtercommand` in a terminal produces the *correct* output though and no empty file.

Is that a known problem?

No. The content of the file should not affect whether the filter is run or not. Can you add \traceexternalfilters on the top and check the output (look for lines starting with t-filter).

...
The file works correctly on my machine.

That is weired.

...
Which version of pandoc are you using?

$ pandoc --version pandoc 1.5.1.1

I am using 1.6 (the latest version is 1.8), but ...

...

As already written in my last mail. Executing the command directly works fine.

I really have no idea why this is happening. Pandoc does not give too many debugging options. A couple of things to check: - What happens is you use pandoc -f rst -t native? (The updated example on github)? - If that fails, then try a simple rst file with a unicode letter (no table)? - If that also fails, then a simple markdown file with a unicode letter. - If that also fails, then some other program that reads a file and writes its contents to another file. - Manually execute the shell command from inside ConTeXt using \ctxlua{os.execute("....")}. Also, what OS you are on?

...

$ pandoc -f rst -t context -o rest-temp-rest.tex rest-temp-rest.tmp $ more rest-temp-rest.t*:::::::::::::: rest-temp-rest.tex :::::::::::::: \placetable[here]{none} \starttable[|l|l|] \HL \NC test \NC table \NC\AR \HL \NC α \NC b \NC\AR \NC c \NC d \NC\AR \HL \stoptable :::::::::::::: rest-temp-rest.tmp :::::::::::::: ========= ========= test table ========= ========= α b c d ======== ========

Since the .tex file is created, it means that the filter module is running correctly. It is calling pandoc with the right options, and pandoc is creating an empty rest-temp-rest.tex file. Now I have no idea why that file is empty. IIRC, pandoc sometimes misbehaves due to locale settings (but I don't see why locale should be different when a program is run from inside context then when it is run from a shell). I am stymied. Aditya

Paul Menzel

1 May 1 May

12:48 p.m.

New subject: t-filter: no table created when Unicode characters are included

Am Samstag, den 30.04.2011, 11:10 -0400 schrieb Aditya Mahajan:

...

On Sat, 30 Apr 2011, Paul Menzel wrote:

...
Am Freitag, den 29.04.2011, 18:13 -0400 schrieb Aditya Mahajan:

...
On Fri, 29 Apr 2011, Paul Menzel wrote:

...
...
the filter module installed using does not work with the following example, which is also attached.

The output file `minimal-temp-rest.tex` is just empty.

Replacing »α« by »a« solves the problem.

Running `filtercommand` in a terminal produces the *correct* output though and no empty file.

Is that a known problem?

No. The content of the file should not affect whether the filter is run or not. Can you add \traceexternalfilters on the top and check the output (look for lines starting with t-filter).

...
The file works correctly on my machine.

That is weired.

...
Which version of pandoc are you using?

$ pandoc --version pandoc 1.5.1.1

I am using 1.6 (the latest version is 1.8), but ...

Version 1.8 is not yet uploaded to the Debian repositories since some Haskell packages are not yet available. See ticket #613256 [1] in the Debian BTS.

...

...
As already written in my last mail. Executing the command directly works fine.

I really have no idea why this is happening. Pandoc does not give too many debugging options. A couple of things to check:

- What happens is you use pandoc -f rst -t native? (The updated example on github)?

The output file is also empty. Running the command from console works again.

...

- If that fails, then try a simple rst file with a unicode letter (no table)?

The output file is still empty. $ pandoc -f rst -t native -o rest-temp-rest.tex rest-temp-rest.tmp $ more rest-temp-rest.t{mp,ex} :::::::::::::: rest-temp-rest.tmp :::::::::::::: α :::::::::::::: rest-temp-rest.tex :::::::::::::: Pandoc (Meta {docTitle = [], docAuthors = [], docDate = []}) [ Para [Str "\945"] ] $ Is the α also encoded by `Str "\945"` in your setup?

...

- If that also fails, then a simple markdown file with a unicode letter.

That fails too. $ more rest-temp-rest.tex $ pandoc -f markdown -t native -o rest-temp-rest.tex rest-temp-rest.tmp $ more rest-temp-rest.t{mp,ex} :::::::::::::: rest-temp-rest.tmp :::::::::::::: α :::::::::::::: rest-temp-rest.tex :::::::::::::: Pandoc (Meta {docTitle = [], docAuthors = [], docDate = []}) [ Para [Str "\945"] ] $

...

- If that also fails, then some other program that reads a file and writes its contents to another file.

Usinng the following \defineexternalfilter [rest] [filter={cat \externalfilterinputfile\space | tee \externalfilteroutputfile}] works. $ more rest-temp-rest.t{mp,ex} :::::::::::::: rest-temp-rest.tmp :::::::::::::: α :::::::::::::: rest-temp-rest.tex :::::::::::::: α

...

- Manually execute the shell command from inside ConTeXt using \ctxlua{os.execute("....")}.

This does not work either. $ more tests/rest.tex % LM does not appear to have Greek letters in unicode slots \setupbodyfont[times] \starttext Start. \ctxlua{os.execute("pandoc -f rst -t context -o rest-temp-rest.tex rest-temp-rest.tmp")} \stoptext $ context tests/rest.tex $ more rest-temp-rest.t{mp,ex} :::::::::::::: rest-temp-rest.tmp :::::::::::::: α :::::::::::::: rest-temp-rest.tex :::::::::::::: $

...

Also, what OS you are on?

I am using Debian Sid/unstable.

...

...
$ pandoc -f rst -t context -o rest-temp-rest.tex rest-temp-rest.tmp $ more rest-temp-rest.t*:::::::::::::: rest-temp-rest.tex :::::::::::::: \placetable[here]{none} \starttable[|l|l|] \HL \NC test \NC table \NC\AR \HL \NC α \NC b \NC\AR \NC c \NC d \NC\AR \HL \stoptable :::::::::::::: rest-temp-rest.tmp :::::::::::::: ========= ========= test table ========= ========= α b c d ======== ========

Since the .tex file is created, it means that the filter module is running correctly. It is calling pandoc with the right options, and pandoc is creating an empty rest-temp-rest.tex file. Now I have no idea why that file is empty. IIRC, pandoc sometimes misbehaves due to locale settings (but I don't see why locale should be different when a program is run from inside context then when it is run from a shell).

I am using the following locale. $ locale LANG=de_DE.UTF-8 LC_CTYPE="de_DE.UTF-8" LC_NUMERIC="de_DE.UTF-8" LC_TIME="de_DE.UTF-8" LC_COLLATE="de_DE.UTF-8" LC_MONETARY="de_DE.UTF-8" LC_MESSAGES="de_DE.UTF-8" LC_PAPER="de_DE.UTF-8" LC_NAME="de_DE.UTF-8" LC_ADDRESS="de_DE.UTF-8" LC_TELEPHONE="de_DE.UTF-8" LC_MEASUREMENT="de_DE.UTF-8" LC_IDENTIFICATION="de_DE.UTF-8" LC_ALL=

...

I am stymied.

Well, I would say to not waste your time any longer and to suspect Pandoc at fault here. If a newer Pandoc version is available in the Debian archive I can retest to check the Pandoc version as the possible culprit. Thank you very much for your help, Paul [1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=613256

pmlists＠free.fr

2:12 p.m.

New subject: t-filter: no table created when Unicode characters are included

Aditya Mahajan writes:

...

(but I don't see why locale should be different when a program is run from inside context then when it is run from a shell).

Taco can surely better explain why. I think somewhere in the luatex manual, he explains why luatex sets locale to "C". To see the differences in the environment, just do "set >/tmp/set-shell" and run context on: --8<---------------cut here---------------start------------->8--- \starttext \ctxlua{os.execute"set >/tmp/set-context"} \stoptext --8<---------------cut here---------------end--------------->8--- I don't have pandoc, but I'm sure that you can reproduce the problem with "LANG=C pandoc ..." Thus, the solution would be something like: "filtercommand={LANG=en_GB.utf8 pandoc ...}" -- Peter

Aditya Mahajan

4:29 p.m.

New subject: t-filter: no table created when Unicode characters are included

On Sun, 1 May 2011, Peter Münster wrote:

...

Aditya Mahajan writes:

...
(but I don't see why locale should be different when a program is run from inside context then when it is run from a shell).

Taco can surely better explain why. I think somewhere in the luatex manual, he explains why luatex sets locale to "C".

To see the differences in the environment, just do "set >/tmp/set-shell" and run context on:

--8<---------------cut here---------------start------------->8--- \starttext \ctxlua{os.execute"set >/tmp/set-context"} \stoptext --8<---------------cut here---------------end--------------->8---

Hmm... I get $context --pipe ... ... *\directlua{os.execute("locale")} LANG=en_US.UTF-8 LC_CTYPE="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_PAPER="en_US.UTF-8" LC_NAME="en_US.UTF-8" LC_ADDRESS="en_US.UTF-8" LC_TELEPHONE="en_US.UTF-8" LC_MEASUREMENT="en_US.UTF-8" LC_IDENTIFICATION="en_US.UTF-8" LC_ALL=en_US.UTF-8 so context no longer changes the locale. Aditya

pmlists＠free.fr

8:27 p.m.

New subject: t-filter: no table created when Unicode characters are included

Aditya Mahajan writes:

...

$context --pipe

Oh, nice option, that I haven't known yet! :)

...

*\directlua{os.execute("locale")} LANG=en_US.UTF-8 LC_CTYPE="en_US.UTF-8"

Strange, here it's: LANG= LC_CTYPE=C LC_NUMERIC=C LC_TIME="POSIX" LC_COLLATE=C LC_MONETARY="POSIX" LC_MESSAGES="POSIX" LC_PAPER="POSIX" LC_NAME="POSIX" LC_ADDRESS="POSIX" LC_TELEPHONE="POSIX" LC_MEASUREMENT="POSIX" LC_IDENTIFICATION="POSIX" LC_ALL= And in my environment: LC_CTYPE=en_GB.utf8 When I add to my environment "LANG=en_GB.utf8", then the context output is: LANG=en_GB.utf8 LC_CTYPE=C LC_NUMERIC=C LC_TIME="en_GB.utf8" LC_COLLATE=C LC_MONETARY="en_GB.utf8" LC_MESSAGES="en_GB.utf8" LC_PAPER="en_GB.utf8" LC_NAME="en_GB.utf8" LC_ADDRESS="en_GB.utf8" LC_TELEPHONE="en_GB.utf8" LC_MEASUREMENT="en_GB.utf8" LC_IDENTIFICATION="en_GB.utf8" LC_ALL= So luatex and/or context changes at least LC_CTYPE to C in my case. When I set LC_ALL too, I get the same result as yours.

...

so context no longer changes the locale.

Just a little bit in some cases ... ;) -- Peter

pmlists＠free.fr

8:52 p.m.

New subject: t-filter: no table created when Unicode characters are included

Aditya Mahajan writes:

...

*\directlua{os.execute("locale")}

Ok, locale outputs something depending on the enviroment. Lets look directly at the enviroment: $ CMD="set | grep -e ^LC_ -e ^LANG" --8<---------------cut here---------------start------------->8--- $ eval $CMD LC_CTYPE=en_GB.utf8 $ echo '\directlua{os.execute"'$CMD'"}' | context --pipe | grep -e LC_ -e LANG *LC_COLLATE=C LC_CTYPE=C LC_NUMERIC=C <*> ...ua{os.execute"set | grep -e ^LC_ -e ^LANG"} --8<---------------cut here---------------end--------------->8--- --8<---------------cut here---------------start------------->8--- $ eval $CMD LANG=en_GB.utf8 LC_CTYPE=en_GB.utf8 $ echo '\directlua{os.execute"'$CMD'"}' | context --pipe | grep -e LC_ -e LANG *LANG=en_GB.utf8 LC_COLLATE=C LC_CTYPE=C LC_NUMERIC=C <*> ...ua{os.execute"set | grep -e ^LC_ -e ^LANG"} --8<---------------cut here---------------end--------------->8--- --8<---------------cut here---------------start------------->8--- $ eval $CMD LANG=en_GB.utf8 LC_ALL=en_GB.utf8 LC_CTYPE=en_GB.utf8 $ echo '\directlua{os.execute"'$CMD'"}' | context --pipe | grep -e LC_ -e LANG FontForge does not support your encoding (utf8), it will pretend the local encoding is latin1 FontForge does not support your encoding (utf8), it will pretend the local encoding is latin1 *LANG=en_GB.utf8 LC_ALL=en_GB.utf8 LC_COLLATE=C LC_CTYPE=C LC_NUMERIC=C <*> ...ua{os.execute"set | grep -e ^LC_ -e ^LANG"} --8<---------------cut here---------------end--------------->8--- So, whatever the original enviroment is, context and/or luatex sets LC_COLLATE=C LC_CTYPE=C LC_NUMERIC=C -- Peter

Aditya Mahajan

9:05 p.m.

New subject: t-filter: no table created when Unicode characters are included

On Sun, 1 May 2011, Peter Münster wrote:

...

So, whatever the original enviroment is, context and/or luatex sets LC_COLLATE=C LC_CTYPE=C LC_NUMERIC=C

Confirmed. Now the question is, why does this cause pandoc to behave strangely. Aditya

Aditya Mahajan

9:09 p.m.

New subject: t-filter: no table created when Unicode characters are included

On Sun, 1 May 2011, Aditya Mahajan wrote:

...

On Sun, 1 May 2011, Peter Münster wrote:

...
So, whatever the original enviroment is, context and/or luatex sets LC_COLLATE=C LC_CTYPE=C LC_NUMERIC=C

Confirmed. Now the question is, why does this cause pandoc to behave strangely.

Perhaps due to a old bug in pandoc http://code.google.com/p/pandoc/issues/detail?id=233 http://www.readmespot.com/question/f/163755/pandoc-locale-problems This was fixed in pandoc 1.6 that is why I don't see it. Paul, you will have to manually update pandoc. Aditya

Paul Menzel

2 May 2 May

5:30 p.m.

New subject: t-filter: no table created when Unicode characters are included

Am Sonntag, den 01.05.2011, 15:09 -0400 schrieb Aditya Mahajan:

...

On Sun, 1 May 2011, Aditya Mahajan wrote:

...
On Sun, 1 May 2011, Peter Münster wrote:

...
So, whatever the original enviroment is, context and/or luatex sets LC_COLLATE=C LC_CTYPE=C LC_NUMERIC=C

Confirmed. Now the question is, why does this cause pandoc to behave strangely.

Perhaps due to a old bug in pandoc http://code.google.com/p/pandoc/issues/detail?id=233 http://www.readmespot.com/question/f/163755/pandoc-locale-problems

This was fixed in pandoc 1.6 that is why I don't see it. Paul, you will have to manually update pandoc.

Thank you very much for all your help. I will wait for the updated Pandoc and retest. In the mean time I am going to use t-rst. Thanks, Paul

Paul Menzel

27 May 27 May

10:53 a.m.

New subject: [solved] t-filter: no table created when Unicode characters are included

Am Montag, den 02.05.2011, 17:30 +0200 schrieb Paul Menzel:

...

Am Sonntag, den 01.05.2011, 15:09 -0400 schrieb Aditya Mahajan:

...
On Sun, 1 May 2011, Aditya Mahajan wrote:

...
On Sun, 1 May 2011, Peter Münster wrote:

...
So, whatever the original enviroment is, context and/or luatex sets LC_COLLATE=C LC_CTYPE=C LC_NUMERIC=C

Confirmed. Now the question is, why does this cause pandoc to behave strangely.

Perhaps due to a old bug in pandoc http://code.google.com/p/pandoc/issues/detail?id=233 http://www.readmespot.com/question/f/163755/pandoc-locale-problems

This was fixed in pandoc 1.6 that is why I don't see it. Paul, you will have to manually update pandoc.

Thank you very much for all your help. I will wait for the updated Pandoc and retest. In the mean time I am going to use t-rst.

Pandoc 1.8.1.1 is available in the Debian Sid/unstable repositories [1] now. I tested the minimal example and after adding the \setupbodyfont[times] as you did, everything works as expected. Thank you very much and sorry for noise, because this was an external problem. Thanks, Paul [1] http://packages.qa.debian.org/p/pandoc/news/20110526T214713Z.html

yoraxe

23 Jun 23 Jun

4:31 p.m.

New subject: [solved] t-filter: no table created when Unicode characters are included

Am 27.05.2011 10:53, schrieb Paul Menzel:

...

Am Montag, den 02.05.2011, 17:30 +0200 schrieb Paul Menzel:

...
Am Sonntag, den 01.05.2011, 15:09 -0400 schrieb Aditya Mahajan:

...
On Sun, 1 May 2011, Aditya Mahajan wrote:

...
On Sun, 1 May 2011, Peter Münster wrote:

...
So, whatever the original enviroment is, context and/or luatex sets LC_COLLATE=C LC_CTYPE=C LC_NUMERIC=C

Confirmed. Now the question is, why does this cause pandoc to behave strangely.

Perhaps due to a old bug in pandoc http://code.google.com/p/pandoc/issues/detail?id=233 http://www.readmespot.com/question/f/163755/pandoc-locale-problems

This was fixed in pandoc 1.6 that is why I don't see it. Paul, you will have to manually update pandoc.

Thank you very much for all your help. I will wait for the updated Pandoc and retest. In the mean time I am going to use t-rst.

Pandoc 1.8.1.1 is available in the Debian Sid/unstable repositories [1] now. I tested the minimal example and after adding the

\setupbodyfont[times]

as you did, everything works as expected.

How can I use math-mode in such a table? Using $a_b$ it just prints $a_b$ instead of identifying this as a math-input. I'm using pandoc 1.8.1.1 and context 2011.06.19. Thanks, Yoraxe

Aditya Mahajan

6:30 p.m.

New subject: [solved] t-filter: no table created when Unicode characters are included

On Thu, 23 Jun 2011, yoraxe wrote:

...

How can I use math-mode in such a table? Using $a_b$ it just prints $a_b$ instead of identifying this as a math-input.

I'm using pandoc 1.8.1.1 and context 2011.06.19.

Can you give a complete minimal example? Aditya

yoraxe

9:59 p.m.

New subject: t-filter: how to use math-mode

Am 23.06.2011 18:30, schrieb Aditya Mahajan:

...

On Thu, 23 Jun 2011, yoraxe wrote:

...
How can I use math-mode in such a table? Using $a_b$ it just prints $a_b$ instead of identifying this as a math-input.

I'm using pandoc 1.8.1.1 and context 2011.06.19.

Can you give a complete minimal example?

Find all files attached. I see, it is the line \NC \$a\letterunderscore{}b\$ that is not correctly translated. What do I have to type instead of _?

yoraxe

11:08 p.m.

New subject: t-filter: how to use math-mode

Am 23.06.2011 21:59, schrieb yoraxe:

...

Am 23.06.2011 18:30, schrieb Aditya Mahajan:

...
On Thu, 23 Jun 2011, yoraxe wrote:

...
How can I use math-mode in such a table? Using $a_b$ it just prints $a_b$ instead of identifying this as a math-input.

I'm using pandoc 1.8.1.1 and context 2011.06.19.

Can you give a complete minimal example?

Find all files attached. I see, it is the line

\NC \$a\letterunderscore{}b\$

that is not correctly translated. What do I have to type instead of _?

Maybe this could be helpful: http://hackage.haskell.org/packages/archive/pandoc/1.8.1.1/doc/html/src/Text... Does someone know what all this means? I guess especially interesting are the following lines: inlineToConTeXt (Math InlineMath str) = return $ char '$' <> text str <> char '$' inlineToConTeXt (Math DisplayMath str) = return $ text "\\startformula " <> text str <> text " \\stopformula"

Aditya Mahajan

24 Jun 24 Jun

1:23 a.m.

New subject: t-filter: how to use math-mode

On Thu, 23 Jun 2011, yoraxe wrote:

...

Am 23.06.2011 18:30, schrieb Aditya Mahajan:

...
On Thu, 23 Jun 2011, yoraxe wrote:

...
How can I use math-mode in such a table? Using $a_b$ it just prints $a_b$ instead of identifying this as a math-input.

I'm using pandoc 1.8.1.1 and context 2011.06.19.

Can you give a complete minimal example?

Find all files attached. I see, it is the line

\NC \$a\letterunderscore{}b\$

that is not correctly translated. What do I have to type instead of _?

It seems that pandoc's rst reader does not handle inline mathematics (at least with pandoc 1.6; I don't know if Pandoc 1.8 fixes this). $echo "\$a_b\$" | pandoc -f rst -t native Pandoc (Meta {docTitle = [], docAuthors = [], docDate = []}) [ Para [Str "$",Str "a",Str "_",Str "b",Str "$"] ] In contrast, markdown reader handles inline mathematics correctly. $echo "\$a_b\$" | pandoc -f markdown -t native Pandoc (Meta {docTitle = [], docAuthors = [], docDate = []}) [ Para [Math InlineMath "a_b"] ] So, it would be better to discuss this on the pandoc mailing list. Aditya

yoraxe

25 Jun 25 Jun

12:58 p.m.

New subject: t-filter: how to use math-mode

Am 24.06.2011 01:23, schrieb Aditya Mahajan:

...

On Thu, 23 Jun 2011, yoraxe wrote:

...
Am 23.06.2011 18:30, schrieb Aditya Mahajan:

...
On Thu, 23 Jun 2011, yoraxe wrote:

...
How can I use math-mode in such a table? Using $a_b$ it just prints $a_b$ instead of identifying this as a math-input.

I'm using pandoc 1.8.1.1 and context 2011.06.19.

Can you give a complete minimal example?

Find all files attached. I see, it is the line

\NC \$a\letterunderscore{}b\$

that is not correctly translated. What do I have to type instead of _?

It seems that pandoc's rst reader does not handle inline mathematics (at least with pandoc 1.6; I don't know if Pandoc 1.8 fixes this).

$echo "\$a_b\$" | pandoc -f rst -t native

Pandoc (Meta {docTitle = [], docAuthors = [], docDate = []}) [ Para [Str "$",Str "a",Str "_",Str "b",Str "$"] ]

In contrast, markdown reader handles inline mathematics correctly.

$echo "\$a_b\$" | pandoc -f markdown -t native

Pandoc (Meta {docTitle = [], docAuthors = [], docDate = []}) [ Para [Math InlineMath "a_b"] ]

So, it would be better to discuss this on the pandoc mailing list.

Ok, thanks. I'm using now markdown instead of rest. If someone is interested, it looks fine using: \usemodule[filter] \defineexternalfilter[markdown][filtercommand={pandoc -f markdown -t context -o \externalfilteroutputfile\space \externalfilterinputfile}] \starttext \startmarkdown Right Left Center Default ------- ------ ---------- ------- $h_1$ 12 12 12 123 123 123 123 1 1 1 1 \stopmarkdown \stoptext Sorry for the noise. Yoraxe

Philipp Gesang

30 Apr 30 Apr

11 p.m.

New subject: t-filter: no table created when Unicode characters are included

On 2011-04-29 <23:09:42>, Paul Menzel wrote:

...

Dear ConTeXt folks,

the filter module installed using

$ ./first-setup.sh --extras="t-filter"

does not work with the following example, which is also attached.

\usemodule[filter] \defineexternalfilter[rest][filtercommand={pandoc -f rst -t context -o \externalfilteroutputfile\space \externalfilterinputfile}] \starttext There should be a table here. \startrest ========= ========= test table ========= ========= α b c d ======== ======== \stoprest \stoptext

Hi Paul, this should now work ootb as inline code with the reST module[1]. (There was some catcode inconsistency concerning spaces that I just wiped out.) Usage example appended. Best regards Philipp [1] https://bitbucket.org/phg/context-rst

...

The output file `minimal-temp-rest.tex` is just empty.

Replacing »α« by »a« solves the problem.

Running `filtercommand` in a terminal produces the *correct* output though and no empty file.

Is that a known problem?

I also include a patch adding this example into `tests/` in [1].

Thanks,

Paul

[1] https://github.com/adityam/filter

...

\usemodule[filter] \defineexternalfilter[rest][filtercommand={pandoc -f rst -t context -o \externalfilteroutputfile\space \externalfilterinputfile}] \starttext There should be a table here. \startrest ========= ========= test table ========= ========= α b c d ======== ======== \stoprest \stoptext

...

___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://tex.aanhet.net archive : http://foundry.supelec.fr/projects/contextrev/ wiki : http://contextgarden.net ___________________________________________________________________________________

-- () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments

Paul Menzel

20 May 20 May

11:24 a.m.

New subject: t-rst: Unicode character not printed (was: t-filter: no table created when Unicode characters are included)

Dear Philipp, Am Samstag, den 30.04.2011, 23:00 +0200 schrieb Philipp Gesang:

...

On 2011-04-29 <23:09:42>, Paul Menzel wrote:

...

...
the filter module installed using

$ ./first-setup.sh --extras="t-filter"

does not work with the following example, which is also attached.

\usemodule[filter] \defineexternalfilter[rest][filtercommand={pandoc -f rst -t context -o \externalfilteroutputfile\space \externalfilterinputfile}] \starttext There should be a table here. \startrest ========= ========= test table ========= ========= α b c d ======== ======== \stoprest \stoptext

...

this should now work ootb as inline code with the reST module[1]. (There was some catcode inconsistency concerning spaces that I just wiped out.) Usage example appended.

I finally had some time to try it out. Unfortunately the α is not printed either with the test file. Please find the resulting PDF file attached. • ConTeXt ver: 2011.05.06 16:52 MKIV fmt: 2011.5.7 int: english/english • context-rst from latest Mercurial branch Does it work on your system? […] Thanks, Paul

...

[1] https://bitbucket.org/phg/context-rst

Philipp Gesang

11:54 a.m.

New subject: t-rst: Unicode character not printed (was: t-filter: no table created when Unicode characters are included)

Hi Paul, On 2011-05-20 <11:24:31>, Paul Menzel wrote:

...

I finally had some time to try it out. Unfortunately the α is not printed either with the test file. Please find the resulting PDF file attached.

of course latin modern doesn’t provide Greek letters yet -- you’ll need to load another font. Closest to lm would be cmu.[1] ··8<····························································· \usetypescriptfile[type-cmu] \usetypescript[computer-modern-unicode] \setupbodyfont[computer-modern-unicode,11pt] \usemodule[rst] \setupTABLE[column][1][background=color,backgroundcolor=gray:2] \setupbodyfontenvironment [default] [em=italic] \starttext There will be a table here. \startRST ========= ========= test table ========= ========= α b c d ========= ========= \stopRST There was a table before this line. \stoptext ··8<····························································· Thanks anyways for the report! Philipp [1] http://canopus.iacp.dvo.ru/~panov/cm-unicode/

...

• ConTeXt ver: 2011.05.06 16:52 MKIV fmt: 2011.5.7 int: english/english • context-rst from latest Mercurial branch

Does it work on your system?

...

[…]

Thanks,

Paul

...
[1] https://bitbucket.org/phg/context-rst

...

___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://tex.aanhet.net archive : http://foundry.supelec.fr/projects/contextrev/ wiki : http://contextgarden.net ___________________________________________________________________________________

-- () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments

4876

Age (days ago)

4933

Last active (days ago)

List overview

Download

22 comments

5 participants

participants (5)

Aditya Mahajan
Paul Menzel
Philipp Gesang
pmlists＠free.fr
yoraxe

t-filter: no table created when Unicode characters are included

tags

participants (5)