Re: [NTG-context] counting the words in a TeX document

6 Aug 2006

      On 8/5/06, Hans Hagen wrote:
...
...
Hello,
I would like to ask how difficult it would be to count the number of
words in a TeX/ConTeXt document. If it's too complex, please ignore
the rest of the message.
Mojca Miklavec wrote:
the way i do such things (and worse trickery) is using pdftotext
you can of course use tex, but then ther ecan be generated words and so and it is insane to use tex (or adapt a tex style) for that; it may help to run with (nondestructive)
\setupalign[nothyphenated]
anyhow, here is a script (i could not locate my normal one)
=== wordcount.rb ===
if (file = ARGV[0]) && file && FileTest.file?(file) then
    begin
        system("pdftotext #{ARGV[0]} wc.log")
        data = IO.read("wc.log")
        data.gsub!(/\d[\.\:]*\w+/o) do ' ' end  # remove suffixes
        data.gsub!(/\d/o)           do ' ' end  # remove numbers
        data.gsub!(/\-\s+/mo)       do ' ' end  # remove hyphenation
        data.gsub!(/\-/mo)          do ' ' end  # split compound words
        data.gsub!(/[\.\,\<\>\/\?\\\|\'\"\;\:\]\{\}\{\+\=\-\_\)\(\*\&\^\%\$\#\@\!\~\`]/mo) do ' ' end
        words = data.split(/\s+/)
        count = Hash.new
        words.each do |w|
            count[w] = (count[w] || 0) + 1
        end
    rescue
        puts("some error #{$!}")
    else
        puts("words  : #{words.size}")
        puts("unique : #{count.size}")
    end
    if ARGV[1] =~ /list/ then
        puts("\n")
        count.sort.each do |k,v|
            puts("#{k} : #{v}")
        end
    end
end
usage: wc filename.pdf [list]
it this kind of stuff is usefull, we can add it to one of the scripts that come with context
Thanks a lot! I guess that's *it*! I always forget about the most
powerful feature of ConTeXt in comparison to LaTeX - scripting can be
added to almost any place (and the user doesn't need to install any
additional executables, such as "detex" mentioned by Aditya).

Here's some of my feedback:
- pdftotext is far from being useful for pdf to text conversion
(doesn't handle any accents), but is perfectly suitable for wordcount
- \[ is missing in the last gsub (only the right bracket is deleted)
- something strange (but not critical) happens to en-dashes

But everything else looks like a perfect functionality for ctxtools --wordcount.

On 8/5/06, Aditya Mahajan wrote:
...
A very crude approach. There is a program called detex
http://ctan.org/tex-archive/support/detex/ I have not used it, but I
think that it strips off every command \something from the tex file.
Then you can filter the file through wc to get a rough estimate of
the number of words. One approach that will work is
\startstatistics[filename][words|letters|lines]
maps to
\startbuffer[\jobname-statistics-filename]
and
\stopstatistics maps to
\stopbuffer
\getbuffer[\jobname-statistics-filename]
\executesystemcommand{detex \jobname-statistics-filename.tmp | wc
 }
I took a look, but it merely looks like a parser for hardcoded (La)TeX
(someone should correct me if I'm wrong).

However, the fact that abstracts for which one might need wordcount
usually don't have too much trickery involved (they're usually olmost
pure plain text), doing the same, only with a simple ruby script
instead of compiling/installing some external LaTeX-aware C program
might already lead to satisfactory results.
...
It wasn't too complex for Michael Downes using LaTeX:
\ProvidesFile{wordcount.tex}[2000/09/27 v1.5 Michael Downes]
% Copyright 2000 Michael John Downes
% This file has no restrictions on its use, distribution, or sale.
%
% If you run LaTeX on wordcount.tex it will prompt you for the name of a
% document to be counted. For most people, however, it will be more
This solution is more likely to produce better results (just that it
includes slightly more work). It actually runs (La)TeX, just redefines
a few commands before, so that counting the words is then a
straightforward parsing of log files based on the number of some
boxes.

Base on those three answers I got a more clear idea of two (different,
but complementary) methods that might be sensible:

a) ctxtools --wordcount filename[tex|pdf]
to do the wordcount for the whole document using pdftotext + ruby regexp

b)
\usemodule[wordcount]

whatever

\startstatistics[name][words|letters|lines]
some more-or-less plain text
\stopstatistics

whatever

and according to Aditya's idea, run a (ruby) regular expression
(insead of detex) on it which would write the nicely formatted desired
number to the output/log file. (I don't know if it's possible to use
the first approach for the second problem, but it doesn't make sense
to complicate things too much.)

As long as the command names are carefully chosen (and extensible if
the need for more complex behaviour arises in the future), that should
be about everything and it doesn't seem so difficult to implement
after all. (But I would write to the documentation that the resulting
numbers might change slightly in the future if the algorithm for
counting the words is improved.)

Any thoughts?

Thanks a lot,
     Mojca