Bibliography, unicode strings, @ELECTRONIC, sorting and bibtex

Marco Patzer

18 Sep 2012 18 Sep '12

12:28 p.m.

Hi, I have some questions regarding the bibliography. 1) The style files bibl-ams.tex, bibl-ssa.tex, etc contain strings like ``, ~. This does not make sense to me. The `` ends up like that in the output and the tilde might interfere with the \asciimode setting. Wouldn't it be better to use the proper Unicode glyphs (U+201C, U+00A0, etc.) nowadays? I assume the files are old MkII files and not being updated since. Example: \setupbibtex [database=sample] \setuppublications [alternative=ssa] \starttext \cite[hh2010a] \cite[Eijkhout1991] \cite[hh2010] \placepublications [criterium=cite] \stoptext 2) The @ELECTRONIC type does not work (already mentioned in the wiki). It does not show up in the references list. I guess no one bothered adding them and it's up to me to add the corresponding entries. Maybe someone has already prepared a working file for the APA style? 3) How to sort the bibliography by author? According to the bib manual I expected this to work: \setupbibtex [database=sample] \setuppublications [alternative=apa, criterium=cite, sort=author] \starttext \cite[hh2010a] \cite[Eijkhout1991] \placepublications \stoptext 4) \completepublications produces an empty page. Is that intended? \setupbibtex [database=sample] \setuppublications [alternative=ssa, criterium=cite] \starttext \cite[hh2010a] \completepublications \stoptext 5) (side question) What is BibTeX used for in ConTeXt? As far as I can see, it reads the .bib database and generates a .bbl file which then is read in by ConTeXt. Why do we need an external tool for this? Why does ConTeXt not read in the .bib database and directly save it as a Lua table? Marco

Show replies by date

Schmitz Thomas A.

18 Sep 18 Sep

1:41 p.m.

On Sep 18, 2012, at 12:28 PM, Marco Patzer wrote:

...

Hi,

I have some questions regarding the bibliography.

1) The style files bibl-ams.tex, bibl-ssa.tex, etc contain strings like ``, ~. This does not make sense to me. The `` ends up like that in the output and the tilde might interfere with the \asciimode setting. Wouldn't it be better to use the proper Unicode glyphs (U+201C, U+00A0, etc.) nowadays? I assume the files are old MkII files and not being updated since. Example:

\setupbibtex [database=sample] \setuppublications [alternative=ssa] \starttext \cite[hh2010a] \cite[Eijkhout1991] \cite[hh2010] \placepublications [criterium=cite] \stoptext

You're right, this is old code that needs to be fixed. The problem is that users of bibtex in ConTeXt are a minority, and users of such precooked styles a tiny minority - I use bibtex quite often, but wouldn't know how to fix these styles and have never used them. So if you're sure about what to do, fix them and send Hans a patch.

...

2) The @ELECTRONIC type does not work (already mentioned in the wiki). It does not show up in the references list. I guess no one bothered adding them and it's up to me to add the corresponding entries.

Maybe someone has already prepared a working file for the APA style?

Possible. I have @ELECTRONIC definitions in my own private style files, but have never used APA.

...

3) How to sort the bibliography by author? According to the bib manual I expected this to work:

\setupbibtex [database=sample] \setuppublications [alternative=apa, criterium=cite, sort=author]

You haven't read carefully enough, it's sorttype=author

...

\starttext \cite[hh2010a] \cite[Eijkhout1991] \placepublications \stoptext

4) \completepublications produces an empty page. Is that intended?

\setupbibtex [database=sample] \setuppublications [alternative=ssa, criterium=cite] \starttext \cite[hh2010a] \completepublications \stoptext

http://archive.contextgarden.net/message/20110923.091734.6b388a11.en.html

...

5) (side question) What is BibTeX used for in ConTeXt? As far as I can see, it reads the .bib database and generates a .bbl file which then is read in by ConTeXt. Why do we need an external tool for this? Why does ConTeXt not read in the .bib database and directly save it as a Lua table?

Because nobody has coded that part yet? This may not be as easy as you make it sound because quite a few heuristics go into parsing bibtex files (e.g., look at the way in which bibtex divides names into first name, last name, von-part, jr-part). Over the years, I have read and heard many complaints about bibtex and many announcements that something infinitely better is just around the corner, to be delivered in all its glory "real soon now." For the time being, I don't see any real replacement, but I may be wrong. Thomas

Marco Patzer

2:25 p.m.

New subject: Bibliography, unicode strings, @ELECTRONIC, sorting and bibtex

2012-09-18 "Schmitz Thomas A." : Hi Thomas

...

The problem is that users of bibtex in ConTeXt are a minority, and users of such precooked styles a tiny minority

With a handful of up-to-date working common citation styles citations would be much more easy. But I get your point.

...

I use bibtex quite often, but wouldn't know how to fix these styles and have never used them. So if you're sure about what to do, fix them and send Hans a patch.

Frankly, I'm no expert in those things. The bibliographies I created so far were quick and dirty, I didn't have the time to create proper styles or repair the shipped ones. I will see what I can do to at least improve the existing ones after the current project is done.

...

...
Maybe someone has already prepared a working file for the APA style?

Possible. I have @ELECTRONIC definitions in my own private style files, but have never used APA.

Same here.

...

...
\setuppublications [alternative=apa, criterium=cite, sort=author]

You haven't read carefully enough, it's sorttype=author

This doesn't work, either: \setupbibtex [database=sample] \setuppublications [alternative=apa, criterium=cite, sorttype=author] \starttext \cite[hh2010a] \cite[Eijkhout1991] \placepublications \stoptext

...

...
\completepublications produces an empty page. Is that intended?

http://archive.contextgarden.net/message/20110923.091734.6b388a11.en.html

Using criterium=text fixed it.

...

...
What is BibTeX used for in ConTeXt? As far as I can see, it reads the .bib database and generates a .bbl file which then is read in by ConTeXt. Why do we need an external tool for this? Why does ConTeXt not read in the .bib database and directly save it as a Lua table?

Because nobody has coded that part yet? This may not be as easy as you make it sound because quite a few heuristics go into parsing bibtex files (e.g., look at the way in which bibtex divides names into first name, last name, von-part, jr-part).

I'm aware of that. So it basically boils down to the fact that bibliographies are not popular among ConTeXt users (including Hans) and therefore much functionality is not properly implemented or cared for. And BibTeX is used since it understands the semantics of bib files, although a pure ConTeXt/Lua solution would be possible. Without BibTeX this functionality would be missing since no one is willing to implement a parser for .bib databases. If I only had time… Marco

Philipp Gesang

3:28 p.m.

New subject: Bibliography, unicode strings, @ELECTRONIC, sorting and bibtex

Hi all, ·········

...

2012-09-18 "Schmitz Thomas A." :

...
...
What is BibTeX used for in ConTeXt? As far as I can see, it reads the .bib database and generates a .bbl file which then is read in by ConTeXt. Why do we need an external tool for this? Why does ConTeXt not read in the .bib database and directly save it as a Lua table?

Because nobody has coded that part yet? This may not be as easy as you make it sound because quite a few heuristics go into parsing bibtex files (e.g., look at the way in which bibtex divides names into first name, last name, von-part, jr-part).

I'm aware of that. So it basically boils down to the fact that bibliographies are not popular among ConTeXt users (including Hans) and therefore much functionality is not properly implemented or cared for.

tl;dr: It’s not unpopular, it’s a Hard Problem™. I guess that is so because bibliographies and citation rules are a hard problem to solve generally. As a recent thread on this list revealed, most of us are content to instead solve the much easier problem of creating some bib functionality themselves, tailored to their own needs.[0] A given cite/bib ruleset is easy to implement (as long as you don’t put too much weight on sorting) -- we have Lua, after all. Developing a framework for bibliographies, where everything needs to be adjustable and parameterized (by non-technical people) on demand while remaining stable over a long time, is however a totally different matter. Just have a look at the biber/biblatex codebase and decide yourself. (Now try to imagine the same without Perl and XML to get the style bonus ;-) ) Context, as opposed to LaTeX, lacks the consistent formatting requirements by journals and editors, simply because they don’t usually accept it as an input format. [0] http://www.mail-archive.com/ntg-context@ntg.nl/msg62855.html

...

And BibTeX is used since it understands the semantics of bib files, although a pure ConTeXt/Lua solution would be possible. Without BibTeX this functionality would be missing since no one is willing to implement a parser for .bib databases.

Context happens to have such a parser, written in Lua. Probably the best one around: ······································································· \starttext \startluacode local db = bibtex.new() bibtex.load(db, "filename.bib") table.print(db) \stopluacode \stoptext ······································································· Regards Philipp

...

If I only had time…

Marco

___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://tex.aanhet.net archive : http://foundry.supelec.fr/projects/contextrev/ wiki : http://contextgarden.net ___________________________________________________________________________________

-- () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments

Marco Patzer

4:19 p.m.

New subject: Bibliography, unicode strings, @ELECTRONIC, sorting and bibtex

2012-09-18 Philipp Gesang :

...

[0] http://www.mail-archive.com/ntg-context@ntg.nl/msg62855.html

Thanks for the link. Since I usually don't deal much with different bibliography styles I tend to skip those threads.

...

...
And BibTeX is used since it understands the semantics of bib files, although a pure ConTeXt/Lua solution would be possible. Without BibTeX this functionality would be missing since no one is willing to implement a parser for .bib databases.

Context happens to have such a parser, written in Lua. Probably the best one around:

······································································· \starttext \startluacode local db = bibtex.new() bibtex.load(db, "filename.bib") table.print(db) \stopluacode \stoptext

Interesting, I didn't know that. But the values are only parsed, not interpreted. That means the only thing left for BibTeX is to do is to interpret the ugly “author” field? Marco

Philipp Gesang

5:34 p.m.

New subject: Bibliography, unicode strings, @ELECTRONIC, sorting and bibtex

·········

...

2012-09-18 Philipp Gesang :

...
[0] http://www.mail-archive.com/ntg-context@ntg.nl/msg62855.html

Thanks for the link. Since I usually don't deal much with different bibliography styles I tend to skip those threads.

...
...
And BibTeX is used since it understands the semantics of bib files, although a pure ConTeXt/Lua solution would be possible. Without BibTeX this functionality would be missing since no one is willing to implement a parser for .bib databases.

Context happens to have such a parser, written in Lua. Probably the best one around:

······································································· \starttext \startluacode local db = bibtex.new() bibtex.load(db, "filename.bib") table.print(db) \stopluacode \stoptext

Interesting, I didn't know that. But the values are only parsed, not interpreted. That means the only thing left for BibTeX is to do is to interpret the ugly “author” field?

From my bibliography (this assumes authors are separated by “ and ”; *warning*: ashamingly ugly code ahead): ······································································· -- adapted from Roberto -- www.inf.puc-rio.br/~roberto/lpeg.html function citator.split (s, sep) if type(sep) == "string" then sep = P(sep) end local elem = C((1 - sep)^0) local p = Ct(elem * (sep * elem)^0) return lpegmatch(p, s) end local split = citator.split -- Return a list of authors' names from a string separated by "and". local _p_spaces = S" \n\t\v"^1 local _p_and = _p_spaces * P"and" * _p_spaces function citator.get_author_list (rawaut) if not stringfind(rawaut, "and") then return { rawaut } end return split(rawaut, _p_and) end local get_author_list = citator.get_author_list do local wl = P{ [1] = "words", left = P"{", right = P"}", space = P" ", tabs = S"\v\t", eol = P"\n", whitespace = V"space" + V"tabs" + V"eol", inbrace = V"left" * (1 - V"right")^1 * V"right", other = (1 - V"inbrace" - V"whitespace")^1, elm = V"inbrace" + V"other", words = Ct((V"whitespace"^0 * C(V"elm"))^0) } -- Takes a string and splits it into words, returning a list of words. function citator.get_word_list(s) return lpegmatch(wl, s) end end local get_word_list = citator.get_word_list -- from http://osdir.com/ml/lua@bazar2.conectiva.com.br/2009-12/msg00910.html do local space = S" \t\v\n" local nospace = 1 - space local ptrim = space^0 * C((space^0 * nospace^1)^0) function citator.strip (s) return lpegmatch(ptrim, s) end end -- Return the formatted author field for one author string. function citator.reverse_one_author (rawaut, form) local listaut = get_word_list(rawaut) local formaut, tmpaut = "", {} if (#listaut > 1) then for i,j in next, listaut do listaut[i] = citator.strip(j) end lastname = listaut[#listaut] .. "," tableremove(listaut, #listaut) tmpaut[#tmpaut+1] = lastname for i,j in next, listaut do tmpaut[#tmpaut+1] = j end for i,j in next, tmpaut do formaut = formaut .. " " .. j end else formaut = listaut[1] end return formaut end local reverse_one_author = citator.reverse_one_author -- Take a string of authors' names rawaut and return a list that is built -- according to the global citator.cite_inv_author. -- <string> ‘resultformat’: if it has the value ‘string’ then the function will -- return a string instead of a table. function citator.format_author_list (rawaut, resultformat) warn("author list", rawaut) local max = citator.compress_authors -- <int>, default=3 local authorlist = get_author_list(rawaut) local cnt = 1 local tmplist = {} local citestyle = citator.styles[citator.cite_style] or fancy2 local etal = citestyle.cite_etal_string repeat if cnt == 1 then if citator.cite_author_form == "allinv" or citator.cite_author_form == "firstinv" then tmplist[#tmplist+1] = reverse_one_author(authorlist[cnt]) warn("num: "..cnt, authorlist[cnt]) else -- don’t reverse anything tmplist[#tmplist+1] = authorlist[cnt] end elseif cnt > max then tmplist[#tmplist+1] = etal break else warn("num: "..cnt, authorlist[cnt]) if citator.cite_author_form == "allinv" then tmplist[#tmplist+1] = reverse_one_author(authorlist[cnt]) elseif citator.cite_author_form == "firstinv" then tmplist[#tmplist+1] = citestyle.cite_author_separator tmplist[#tmplist+1] = authorlist[cnt] else tmplist[#tmplist+1] = citestyle.cite_author_separator tmplist[#tmplist+1] = authorlist[cnt] end end cnt = cnt + 1 until authorlist[cnt] == nil warn(#tmplist, tmplist[1]) if resultformat == "string" then return tableconcat(tmplist) end return tmplist end local format_author_list = citator.format_author_list ······································································· As you can see, all I have to offer is spaghetti :P And the formatting rules for names (the fields author, bookauthor, translator, editor, bookeditor, commentator, etc. pp.) are by no means everything that bibtex handles. The hard part is the formatting of entries according to cite style (apa etc.) and method (short, number, full). Then strings (ibidem, et. al) need to respect i18n. Sorting of the bib has to take place on a certain set of fields in a certain order depending on whether the entry has an author field or only an editor or both ... and then there is the problem with names in general: http://www.kalzumeus.com/2010/06/17/falsehoods-programmers-believe-about-nam... I don’t want to be spreading pessimism, but these problems are easily understimated. Philipp

...

Marco

___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://tex.aanhet.net archive : http://foundry.supelec.fr/projects/contextrev/ wiki : http://contextgarden.net ___________________________________________________________________________________

-- () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments

Schmitz Thomas A.

3:48 p.m.

On Sep 18, 2012, at 2:25 PM, Marco Patzer wrote:

...

This doesn't work, either:

\setupbibtex [database=sample] \setuppublications [alternative=apa, criterium=cite, sorttype=author] \starttext \cite[hh2010a] \cite[Eijkhout1991] \placepublications \stoptext

This works for me, I get Eijkhout sorted before Hagen.

...

I'm aware of that. So it basically boils down to the fact that bibliographies are not popular among ConTeXt users (including Hans) and therefore much functionality is not properly implemented or cared for. And BibTeX is used since it understands the semantics of bib files, although a pure ConTeXt/Lua solution would be possible. Without BibTeX this functionality would be missing since no one is willing to implement a parser for .bib databases.

If I only had time…

Yes, had we but world enough and time… Anyway, Hans began playing with transforming bibtex to xml and loading this. There is no user interface yet, but if you want to see what's possible, have a look at bibl-bib.lua and bibl-tst.lua Thomas

Marco Patzer

4:11 p.m.

New subject: Bibliography, unicode strings, @ELECTRONIC, sorting and bibtex

2012-09-18 "Schmitz Thomas A." : Hi Thomas

...

On Sep 18, 2012, at 2:25 PM, Marco Patzer wrote:

...
This doesn't work, either:

\setupbibtex [database=sample] \setuppublications [alternative=apa, criterium=cite, sorttype=author] \starttext \cite[hh2010a] \cite[Eijkhout1991] \placepublications \stoptext

This works for me, I get Eijkhout sorted before Hagen.

I deleted all temp files, but still get “Hagen” before “Eijkhout”. My test file is attached.

...

...
If I only had time…

Yes, had we but world enough and time… Anyway, Hans began playing with transforming bibtex to xml and loading this. There is no user interface yet, but if you want to see what's possible, have a look at bibl-bib.lua and bibl-tst.lua

Thanks, I will have a look. Marco

Alan BRASLAU

5:15 p.m.

New subject: Bibliography, unicode strings, @ELECTRONIC, sorting and bibtex

On Tue, 18 Sep 2012 13:41:56 +0200 Schmitz Thomas A. wrote:

...

On Sep 18, 2012, at 12:28 PM, Marco Patzer wrote:

...
5) (side question) What is BibTeX used for in ConTeXt? As far as I can see, it reads the .bib database and generates a .bbl file which then is read in by ConTeXt. Why do we need an external tool for this? Why does ConTeXt not read in the .bib database and directly save it as a Lua table?

Because nobody has coded that part yet? This may not be as easy as you make it sound because quite a few heuristics go into parsing bibtex files (e.g., look at the way in which bibtex divides names into first name, last name, von-part, jr-part). Over the years, I have read and heard many complaints about bibtex and many announcements that something infinitely better is just around the corner, to be delivered in all its glory "real soon now." For the time being, I don't see any real replacement, but I may be wrong.

Do not forget that there are many databases and tools that more or less know how to deal with bibtex files. And many web sites can output bibtex data, more or less correctly. Of course, one could also consider using other "standards", such as EndNote... but we would not want to do that, would we? Alan

4413

Age (days ago)

4413

Last active (days ago)

List overview

Download

8 comments

4 participants

participants (4)

Alan BRASLAU
Marco Patzer
Philipp Gesang
Schmitz Thomas A.

Bibliography, unicode strings, @ELECTRONIC, sorting and bibtex

tags

participants (4)