[NTG-context] CSV scanners built in ConTeXt - feature or bug?

26 Feb 2015

      Hi ConTeXist
A few days ago Hans pointed me builtin CSV splitter. I tried to test it 
sure will come in handy for my needs. I found that if the CSV file 
contain a blank line, then it stops processing the file (see my minimal 
example). It is clear to me that the incorrectness of file format (eg. 
different number of columns in rows etc.) may cause interruption of 
processing, however, I want to ask whether there is an opportunity to 
process the CSV file with blank lines until the end of CSV file. I 
noticed that when I exporting data from Excel sometimes happens that in 
the export file will appear blank line. Is it interrupt processing a 
feature of a buildin splitter or is it a bug? Can it possibly somehow 
fix or add new functionality?

Thanx
Jaroslav Hajtmar

Here is minimal example:

\starttext

\startluacode
local mycsvsplitter = utilities.parsers.rfc4180splitter{
     separator = ",",
     quote = '"',
}

local crap = io.loaddata("data.txt")

-- with header variant
local tablerows, columnname = mycsvsplitter(crap,true)
inspect(tablerows)
inspect(columnname)

-- without header variant
-- local tablerows = mycsvsplitter(crap)
-- inspect(tablerows)

for i=1,#tablerows do
     local l = tablerows[i]
      for j=1,#l do context(l[j]..", ")
     end
     context('\\crlf')
end

\stopluacode

\stoptext

% <-------------- here start data.txt file ---------------------->
first,second,third,fourth
1,"2","3","4"
"a","b","c","d"
"foo","bar""baz","boogie","xyzzy"
"    ","    ","    ","     "
"And now","followed by","several","blank lines"

"After several","empty rows","data continues","here"
11,"22","33","44"
"aa","bb","cc","dd"
% <-------------- and here stop data.txt file ---------------------->