I would be satisfied when a returned 404 error code will be handled within a reasonable (configurable) time delay. As for redirection, there I would not mind if it is not included. Restrictions like that are not a problem for me, because this is for building a pdf that accesses many internet located files. The check on existence is therefore especially meant for spotting missing or misnamed files. Hans van der Meer

On 16 May 2018, at 12:41, Henri Menke wrote:

On 05/16/2018 09:31 PM, Hans van der Meer wrote:

...

I tried to determine the existence of a file on the internet. See the following macro call:

% Test if file exists. \edef\theurl{\linkprotocol://\urlbase\xmlatt{#1}{link}\thesuffix} \doiffileelse {\theurl} {\verbose{HVDM-PEV-TEST}{file exists: \theurl}} {\verbose{HVDM-PEV-TEST}{file does not exist: \theurl}}

It seems that \doiffileelse always results in true except for local files where nonexistence is correctly handled.

For example the following url definitely does not exist but nevertheless \doiffileelse reports its existence: HVDM-PEV-TEST > file exists: http://hansvandermeer.myqnapcloud.com/archive/denhaag/hga-dtb-1869-6040.pdf

Is it possible that the \doiffile macros are extended for files on the internet or a correponding set \doifurl is developed?

I'm not speaking for the devs here, but I don't think that this is going to be included. It would require that all webpages send proper 404 codes and timeout after reasonable delay. Also how would you handle redirects? Should the ConTeXt run be stalled because checking for a files existence on the web is caught in a redirect loop?

...

Hans van der Meer

___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://context.aanhet.net archive : https://bitbucket.org/phg/context-mirror/commits/ wiki : http://contextgarden.net ___________________________________________________________________________________

___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl mailto:ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl http://www.pragma-ade.nl/ / http://context.aanhet.net http://context.aanhet.net/ archive : https://bitbucket.org/phg/context-mirror/commits/ https://bitbucket.org/phg/context-mirror/commits/ wiki : http://contextgarden.net http://contextgarden.net/ ___________________________________________________________________________________

Or use luasocket, which is included in the luatex binary: \startluacode content, status, authinfo = socket.http.request{ method = "HEAD", url = "http://hansvandermeer.myqnapcloud.com/archive/denhaag/hga-dtb-1869-6040.pdf” } print (status) \stopluacode prints ‘404’ in this case. Taco

On 16 May 2018, at 13:45, Henri Menke wrote:

On 05/16/2018 11:23 PM, Hans van der Meer wrote:

...

I would be satisfied when a returned 404 error code will be handled within a reasonable (configurable) time delay. As for redirection, there I would not mind if it is not included. Restrictions like that are not a problem for me, because this is for building a pdf that accesses many internet located files. The check on existence is therefore especially meant for spotting missing or misnamed files.

You can use libcurl via ffi: https://curl.haxx.se/libcurl/

\startluacode local ffi = require("ffi")

ffi.cdef[[ typedef struct Curl_easy CURL; typedef int CURLcode; typedef int CURLoption; typedef int CURLINFO; typedef size_t(*callback)(void *buffer, size_t size, size_t nmemb, void *userp);

CURLcode curl_global_init(long flags); CURL *curl_easy_init(void); CURLcode curl_easy_setopt(CURL *curl, CURLoption option, ...); CURLcode curl_easy_perform(CURL *curl); CURLcode curl_easy_getinfo(CURL *curl, CURLINFO info, ...); void curl_easy_cleanup(CURL *curl); void curl_global_cleanup(void); ]]

-- magic numbers local CURL_GLOBAL_DEFAULT = 3 local CURLOPT_URL = 10002 local CURLOPT_NOBODY = 44 local CURLOPT_HEADER = 42 local CURLOPT_WRITEFUNCTION = 20011 local CURLINFO_RESPONSE_CODE = 0x200000 + 2

local lcurl = ffi.load("curl")

local function discard(buffer, size, nmemb, userp) return size * nmemb end

function check_status(url) lcurl.curl_global_init(CURL_GLOBAL_DEFAULT)

local curl = lcurl.curl_easy_init()

lcurl.curl_easy_setopt(curl, CURLOPT_URL, url) lcurl.curl_easy_setopt(curl, CURLOPT_NOBODY, ffi.cast("long",1)) lcurl.curl_easy_setopt(curl, CURLOPT_HEADER, ffi.cast("long",1)) lcurl.curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, ffi.cast("callback",discard));

local res = lcurl.curl_easy_perform(curl) local c_http_code = ffi.new("long[1]") lcurl.curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, c_http_code);

lcurl.curl_easy_cleanup(curl) lcurl.curl_global_cleanup()

return tonumber(c_http_code[0]) end \stopluacode

\starttext

\ctxlua{context(check_status("http://example.com/"))}

\ctxlua{context(check_status("https://example.com/"))}

\ctxlua{context(check_status("https://example.com/xyz"))}

\stoptext

...

Hans van der Meer

...

On 16 May 2018, at 12:41, Henri Menke mailto:henrimenke@gmail.com> wrote:

On 05/16/2018 09:31 PM, Hans van der Meer wrote:

...

I tried to determine the existence of a file on the internet. See the following macro call:

% Test if file exists. \edef\theurl{\linkprotocol://\urlbase\xmlatt{#1}{link}\thesuffix} \doiffileelse {\theurl} {\verbose{HVDM-PEV-TEST}{file exists: \theurl}} {\verbose{HVDM-PEV-TEST}{file does not exist: \theurl}}

It seems that \doiffileelse always results in true except for local files where nonexistence is correctly handled.

For example the following url definitely does not exist but nevertheless \doiffileelse reports its existence: HVDM-PEV-TEST > file exists: http://hansvandermeer.myqnapcloud.com/archive/denhaag/hga-dtb-1869-6040.pdf

Is it possible that the \doiffile macros are extended for files on the internet or a correponding set \doifurl is developed?

I'm not speaking for the devs here, but I don't think that this is going to be included. It would require that all webpages send proper 404 codes and timeout after reasonable delay. Also how would you handle redirects? Should the ConTeXt run be stalled because checking for a files existence on the web is caught in a redirect loop?

...

Hans van der Meer

___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl mailto:ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://context.aanhet.net archive : https://bitbucket.org/phg/context-mirror/commits/ wiki : http://contextgarden.net ___________________________________________________________________________________

___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl mailto:ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl http://www.pragma-ade.nl/ / http://context.aanhet.net http://context.aanhet.net/ archive : https://bitbucket.org/phg/context-mirror/commits/ wiki : http://contextgarden.net http://contextgarden.net/ ___________________________________________________________________________________

___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://context.aanhet.net archive : https://bitbucket.org/phg/context-mirror/commits/ wiki : http://contextgarden.net ___________________________________________________________________________________

___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://context.aanhet.net archive : https://bitbucket.org/phg/context-mirror/commits/ wiki : http://contextgarden.net ___________________________________________________________________________________

Taco Hoekwater Elvenkind BV

On 5/16/2018 2:58 PM, Hans van der Meer wrote:

Beste Taco,

Ik probeer jouw oplossing maar er gebeurt nog iets raars. Ik krijg inderdaad 200 resp. 404 terug bij bestaande niet-bestaande Uri’s. Maar ik krijg wel een error die aan het eind van de run gegenereerd wordt. Heb jij enig idee waar ik dat zou moeten zoeken? Als ik de call naar \verbose uitschakel komt er geen foutmelding. Blijft er na de socket call iets achter wat nog opgeruimd had moeten worden?

how about adding local ... \startluacode local content, status, authinfo = socket.http.request { method = "HEAD", url = "http://hansvandermeer.myqnapcloud.com/archive/denhaag/hga-dtb-1869-6040.pdf", } print (status) \stopluacode

Hier een deel van de context-log en de code fragmenten in kwestie. <takepersons> is de root node van het bewerkte bestand.

HVDM-PEV-TEST   > return code: 200 HVDM-PEV        > number of events processed: 1 HVDM-PEV        > ============= end of takepersons ============= backend         > xmp > using file '/Users/hansm/TeX/context-45/tex/texmf-context/tex/context/base/mkiv/lpdf-pdx.xml' pages           > flushing realpage 1, userpage 1, subpage 1 close source    > level 2, order 13, name '/Users/hansm/Documents/Genealogie/familie-event.tex' close source    > level 1, order 13, name '/Users/hansm/TeX/context-45/tex/texmf-context/tex/context/base/mkiv/cont-yes.mkiv'

lua error       > lua error on line 0 in file :

...-45/tex/texmf-context/tex/context/base/mkiv/core-uti.lua:92: attempt to index a number value (global 'status')

<empty file>

? Process aborted

Code in tex: \def\URIReturnCode#1{\ctxlua{tex.print(hvdm.urireturncode("#1"))}} \verbose{HVDM-PEV-TEST}{return code: \URIReturnCode{\theurl}} Code in lua: hvdm.urireturncode = function (theuri) content, status, authinfo = socket.http.request{ method = "HEAD", url = theuri, } return tostring(status) end

Groet Hans

...

On 16 May 2018, at 14:23, Taco Hoekwater mailto:taco@elvenkind.com> wrote:

Or use luasocket, which is included in the luatex binary:

\startluacode content, status, authinfo = socket.http.request{    method = "HEAD",    url = "http://hansvandermeer.myqnapcloud.com/archive/denhaag/hga-dtb-1869-6040.pdf”

}

print (status)

\stopluacode

prints ‘404’ in this case.

Taco

...

On 16 May 2018, at 13:45, Henri Menke mailto:henrimenke@gmail.com> wrote:

On 05/16/2018 11:23 PM, Hans van der Meer wrote:

...

I would be satisfied when a returned 404 error code will be handled within a reasonable (configurable) time delay. As for redirection, there I would not mind if it is not included. Restrictions like that are not a problem for me, because this is for building a pdf that accesses many internet located files. The check on existence is therefore especially meant for spotting missing or misnamed files.

You can use libcurl via ffi: https://curl.haxx.se/libcurl/

\startluacode local ffi = require("ffi")

ffi.cdef[[ typedef struct Curl_easy CURL; typedef int CURLcode; typedef int CURLoption; typedef int CURLINFO; typedef size_t(*callback)(void *buffer, size_t size, size_t nmemb, void *userp);

CURLcode curl_global_init(long flags); CURL *curl_easy_init(void); CURLcode curl_easy_setopt(CURL *curl, CURLoption option, ...); CURLcode curl_easy_perform(CURL *curl); CURLcode curl_easy_getinfo(CURL *curl, CURLINFO info, ...); void curl_easy_cleanup(CURL *curl); void curl_global_cleanup(void); ]]

-- magic numbers local CURL_GLOBAL_DEFAULT = 3 local CURLOPT_URL = 10002 local CURLOPT_NOBODY = 44 local CURLOPT_HEADER = 42 local CURLOPT_WRITEFUNCTION = 20011 local CURLINFO_RESPONSE_CODE = 0x200000 + 2

local lcurl = ffi.load("curl")

local function discard(buffer, size, nmemb, userp)   return size * nmemb end

function check_status(url)   lcurl.curl_global_init(CURL_GLOBAL_DEFAULT)

  local curl = lcurl.curl_easy_init()

  lcurl.curl_easy_setopt(curl, CURLOPT_URL, url)   lcurl.curl_easy_setopt(curl, CURLOPT_NOBODY, ffi.cast("long",1))   lcurl.curl_easy_setopt(curl, CURLOPT_HEADER, ffi.cast("long",1))   lcurl.curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, ffi.cast("callback",discard));

  local res = lcurl.curl_easy_perform(curl)   local c_http_code = ffi.new("long[1]")   lcurl.curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, c_http_code);

  lcurl.curl_easy_cleanup(curl)   lcurl.curl_global_cleanup()

  return tonumber(c_http_code[0]) end \stopluacode

\starttext

\ctxlua{context(check_status("http://example.com/"))}

\ctxlua{context(check_status("https://example.com/"))}

\ctxlua{context(check_status("https://example.com/xyz"))}

\stoptext

...

Hans van der Meer

...

On 16 May 2018, at 12:41, Henri Menke mailto:henrimenke@gmail.com mailto:henrimenke@gmail.com> wrote:

On 05/16/2018 09:31 PM, Hans van der Meer wrote:

...

I tried to determine the existence of a file on the internet. See the following macro call:

       % Test if file exists.        \edef\theurl{\linkprotocol://\urlbase\xmlatt{#1}{link}\thesuffix}        \doiffileelse        {\theurl}        {\verbose{HVDM-PEV-TEST}{file exists: \theurl}}        {\verbose{HVDM-PEV-TEST}{file does not exist: \theurl}}

It seems that \doiffileelse always results in true except for local files where nonexistence is correctly handled.

For example the following url definitely does not exist but nevertheless \doiffileelse reports its existence: HVDM-PEV-TEST   > file exists: http://hansvandermeer.myqnapcloud.com/archive/denhaag/hga-dtb-1869-6040.pdf

Is it possible that the \doiffile macros are extended for files on the internet or a correponding set \doifurl is developed?

I'm not speaking for the devs here, but I don't think that this is going to be included.  It would require that all webpages send proper 404 codes and timeout after reasonable delay.  Also how would you handle redirects?  Should the ConTeXt run be stalled because checking for a files existence on the web is caught in a redirect loop?

...

Hans van der Meer

___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl mailto:ntg-context@ntg.nl mailto:ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage  : http://www.pragma-ade.nl / http://context.aanhet.net archive  : https://bitbucket.org/phg/context-mirror/commits/ wiki     : http://contextgarden.net ___________________________________________________________________________________

___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl mailto:ntg-context@ntg.nl mailto:ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage  : http://www.pragma-ade.nl http://www.pragma-ade.nl/ / http://context.aanhet.net http://context.aanhet.net/ archive  : https://bitbucket.org/phg/context-mirror/commits/ wiki     : http://contextgarden.net http://contextgarden.net/ ___________________________________________________________________________________

___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl mailto:ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage  : http://www.pragma-ade.nl / http://context.aanhet.net archive  : https://bitbucket.org/phg/context-mirror/commits/ wiki     : http://contextgarden.net ___________________________________________________________________________________

___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl mailto:ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage  : http://www.pragma-ade.nl / http://context.aanhet.net archive  : https://bitbucket.org/phg/context-mirror/commits/ wiki     : http://contextgarden.net ___________________________________________________________________________________

Taco Hoekwater Elvenkind BV

___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl mailto:ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage  : http://www.pragma-ade.nl / http://context.aanhet.net archive  : https://bitbucket.org/phg/context-mirror/commits/ wiki     : http://contextgarden.net ___________________________________________________________________________________

___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://context.aanhet.net archive : https://bitbucket.org/phg/context-mirror/commits/ wiki : http://contextgarden.net ___________________________________________________________________________________

-- ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------

Am Wed, 16 May 2018 14:23:42 +0200 schrieb Taco Hoekwater:

Or use luasocket, which is included in the luatex binary:

But I'm right that this works only with http and not with https? -- Ulrike Fischer http://www.troubleshooting-tex.de/

I too have the impression it is working for both. Hans van der Meer

On 17 May 2018, at 09:39, Taco Hoekwater wrote:

...

On 17 May 2018, at 09:32, Ulrike Fischer wrote:

Am Wed, 16 May 2018 14:23:42 +0200 schrieb Taco Hoekwater:

...

Or use luasocket, which is included in the luatex binary:

But I'm right that this works only with http and not with https?

Works ok for me. Did you test?

Taco

___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://context.aanhet.net archive : https://bitbucket.org/phg/context-mirror/commits/ wiki : http://contextgarden.net ___________________________________________________________________________________

Am Thu, 17 May 2018 09:54:25 +0200 schrieb Taco Hoekwater:

...

Works ok for me. Did you test?

Oh, sorry. It seems it is sneakily rewriting the https:// to http://, and so does not _actually_ work.

Ah. This explains why it seemed to work for https://httpbin.org/html, http://httpbin.org/html exists and why I get a lot of redirection answers. -- Ulrike Fischer http://www.troubleshooting-tex.de/

On Thu, May 17, 2018 at 9:54 AM, Taco Hoekwater wrote:

Doing https would need openssl support, which is unlikely to

ever be built into luatex.

We could provide a ffi wrapper (once ffi is stable) or a swiglib wrapper as module. Something to discuss at meeting.

-- luigi

On 17 May 2018, at 10:20, Ulrike Fischer wrote:

Am Thu, 17 May 2018 09:39:47 +0200 schrieb Taco Hoekwater:

...

...

...

Or use luasocket, which is included in the luatex binary:

...

...

But I'm right that this works only with http and not with https?

...

Works ok for me. Did you test?

I tried a few links and always got 301 back regardless if they exist or not and thought due to the discussion two month ago on the luatex list about https content that it doesn't work at all.

I know found https://httpbin.org/ and with it the tests worked: https://httpbin.org/html gave 200 and https://httpbin.org/htmlxxxx gave 404.

It only works because it silently discards the https: and replaces it with http: . Actually, it disregards what is in front of // completely, whatever it is. Even: url = "foobar://www.ntg.nl/watistex.html" works just 'fine'. Anyway, perhaps someone can answer me this? I tried the ffi/curl code, and the network stuff works, but only when I comment out the write callback: lcurl.curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, ffi.cast("callback",discard)); If I dont, I get an error message from ffi that is: unable to convert argument 3 from cdata to cdata<vararg> stack traceback: [C]: in function 'curl_easy_setopt' [ctxlua]:42: in function 'check_status' [ctxlua]:1: in main chunk And I have no idea what that actually means or how to fix it? Best wishes, Taco

On 05/17/2018 09:03 PM, Taco Hoekwater wrote:

...

On 17 May 2018, at 10:31, Taco Hoekwater wrote:

Anyway, perhaps someone can answer me this? I tried the ffi/curl code, and the network stuff works, but only when I comment out the write callback:

lcurl.curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, ffi.cast("callback",discard));

If I dont, I get an error message from ffi that is:

unable to convert argument 3 from cdata to cdata<vararg> stack traceback: [C]: in function 'curl_easy_setopt' [ctxlua]:42: in function 'check_status' [ctxlua]:1: in main chunk

And I have no idea what that actually means or how to fix it?

FYI: fixed by updating luatex.

Hi Taco, Actually you don't really need to set the WRITEFUNCTION callback of libcurl. I only set it to the "discard" function which simply return the number of bytes it got, such that libcurl doesn't print the header information to stdout. You could also plug a different function in there if you wanted to process the headers. I will soon post an annotated version of the code with more proper error-checking on the list. Cheers, Henri

Best wishes, Taco

___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://context.aanhet.net archive : https://bitbucket.org/phg/context-mirror/commits/ wiki : http://contextgarden.net ___________________________________________________________________________________