On Mon, Apr 11, 2022 at 1:01 PM Norbert Preining wrote:

Hi Hans, hi Roland,

thanks for your answer.

...

it actually defeats one of the security properties of lua (which was explicitly introduced at some point: make sure that hashes have random order each run so that it's harder to retrieve sensitive data from mem)

Well, that is a good point to *not* implement the change.

Roland, do you have any comments? I guess the reproducability strive is not as important as security.

So if something in this way should be done, it would need to changes sort order if and only if FORCE_SOURCE_DATE=1 in the env (this is what has required for tex engines to obey SOURCE_DATE_EPOCH settings).

not only fmt, every output could suffer from the same problem if it depends on a lua table that is not an array -- temp data, log and pdf . The format should serialize only array, or use a metatable (e.g. https://stackoverflow.com/questions/30970034/lua-in-pairs-with-same-order-as... ) Even if we hard code in some way an ordered table data structure, it's still the responsibility of the format to use it -- but then metatables are more flexible. -- luigi

Hello Hans, Norbert, Thanks for your answers. On 11/04/2022 13:01, Norbert Preining wrote:

...

it actually defeats one of the security properties of lua (which was explicitly introduced at some point: make sure that hashes have random order each run so that it's harder to retrieve sensitive data from mem)

Well, that is a good point to *not* implement the change.

Roland, do you have any comments? I guess the reproducability strive is not as important as security.

Well, reproducibility is *another* aspect of security; this time not for the regular environments that users will use, but for build environments. Reproducibility (as enforced by SOURCE_DATE_EPOCH) is typically enabled in an environment that generates binaries from source code for redistribution. It will guarantee that the build environment has not been tampered with, because you can (if you have made a similar build environment yourself) generate the binary files bit-for-bit identical. For a regular, production environment you should not have SOURCE_DATE_EPOCH set. Other programming languages also have solved the security risks associated with the randomness of the hashes and reproducibility, see [1]. For Perl, the hashes can be de-randomized with PERL_HASH_SEED. Python uses PYTHONHASHSEED. For Lua an environment variable LUA_HASH_SEED could be introduced, or per default the value of SOURCE_DATE_EPOCH (if set) instead of time(NULL) could be used to seed the hashes. The texlive-binaries in Debian contain an embedded copy of Lua 5.3. The Lua 5.4 version of luai_makeseed is more complex, see [2]. I'll write a feature request for Lua later, that is out-of-scope for this scenario.

So if something in this way should be done, it would need to changes sort order if and only if FORCE_SOURCE_DATE=1 in the env (this is what has required for tex engines to obey SOURCE_DATE_EPOCH settings).

Roland, if you have time, please adjust the patch to work within the above constraints.

Ack. Thanks for the pointer to luai_makeseed, that was some missing information that I needed. I'll post an updated patch soon (most probably much smaller and more elegant). As written above, the hash seed will be de-randomized only when both FORCE_SOURCE_DATE=1 and SOURCE_DATE_EPOCH are set. With kind regards, Roland Clobus [1] https://reproducible-builds.org/docs/stable-outputs/ [2] https://sources.debian.org/src/lua5.4/5.4.4-1/src/lstate.c/?hl=73#L73

On Mon, Apr 11, 2022 at 5:52 PM Roland Clobus wrote:

Hello Hans, Norbert,

Thanks for your answers.

On 11/04/2022 13:01, Norbert Preining wrote:

...

...

it actually defeats one of the security properties of lua (which was explicitly introduced at some point: make sure that hashes have random order each run so that it's harder to retrieve sensitive data from mem)

Well, that is a good point to *not* implement the change.

Roland, do you have any comments? I guess the reproducability strive is not as important as security.

Well, reproducibility is *another* aspect of security; this time not for the regular environments that users will use, but for build environments.

Reproducibility (as enforced by SOURCE_DATE_EPOCH) is typically enabled in an environment that generates binaries from source code for redistribution. It will guarantee that the build environment has not been tampered with, because you can (if you have made a similar build environment yourself) generate the binary files bit-for-bit identical. For a regular, production environment you should not have SOURCE_DATE_EPOCH set.

Other programming languages also have solved the security risks associated with the randomness of the hashes and reproducibility, see [1]. For Perl, the hashes can be de-randomized with PERL_HASH_SEED. Python uses PYTHONHASHSEED. For Lua an environment variable LUA_HASH_SEED could be introduced, or per default the value of SOURCE_DATE_EPOCH (if set) instead of time(NULL) could be used to seed the hashes.

The texlive-binaries in Debian contain an embedded copy of Lua 5.3. The Lua 5.4 version of luai_makeseed is more complex, see [2]. I'll write a feature request for Lua later, that is out-of-scope for this scenario.

...

So if something in this way should be done, it would need to changes sort order if and only if FORCE_SOURCE_DATE=1 in the env (this is what has required for tex engines to obey SOURCE_DATE_EPOCH settings).

Roland, if you have time, please adjust the patch to work within the above constraints.

Ack. Thanks for the pointer to luai_makeseed, that was some missing information that I needed. I'll post an updated patch soon (most probably much smaller and more elegant). As written above, the hash seed will be de-randomized only when both FORCE_SOURCE_DATE=1 and SOURCE_DATE_EPOCH are set.

I am perplexed, perhaps I misunderstood something. The distinction among "the regular environments that users will use" and the "build environments" seem to be done at runtime for the same binary by setting an env. variable -- but in this case a malicious "regular" user could also set LUA_HASH_SEED, breaking the security property. In this *specific* case, one can check by sorting -- as done by the patch: #!/bin/sh export FORCE_SOURCE_DATE=1 export SOURCE_DATE_EPOCH=$(date +%s) for i in `seq 1 10`; do luahbtex -ini -jobname=luahbtex -progname=luabhtex luatex.ini 1>/dev/null; gunzip -d -c luahbtex.fmt|tail -1 |xxd -i |perl -pe 's{,\s*}{\n}g;s{^\s*}{}g;'|sort|md5sum ; md5sum luahbtex.log; done because *in this case* two distinct fmt differ only at the last line -- but perhaps choosing another format (lualatex) could make more sense. -- luigi

Hello list, On 12/04/2022 08:44, Roland Clobus wrote:

I'll follow-up soon with an updated patch.

As discussed, I've updated the patch. For Lua-based TeX binaries, only when FORCE_SOURCE_DATE=1 and SOURCE_DATE_EPOCH are set, this will initialise the Lua seed to the value of SOURCE_DATE_EPOCH instead of a random value. With this patch, the .fmt files can be generated bit-for-bit identical. Regarding the patch: * This patch is intended only for Lua 5.3 that is embedded in texlive-binaries * A re-definition of `luai_makeseed` is unfortunately not sufficient for Lua 5.3, for 5.4.4 and later it would be. [1] * I've added no validation for the content of SOURCE_DATE_EPOCH: ** 1) That happens in other code locations already ** 2) Even if the value would be incorrect, the Lua seed will still be de-randomized * Do you want some comment lines? * The sorting from by previous patch is no longer required. Only lstate.c needs to be modified. With kind regards, Roland Clobus PS: If you later intend to upgrade to another version of Lua, the fixed seed value can help you in automated tests to see different behaviour due to the upgrade. [1] https://github.com/lua/lua/commit/97e394ba1805fbe394a5704de660403901559e54

Hello list, On 19/04/2022 09:52, luigi scarso wrote:

Thank you very much for your patch, I will check it this weekend.

Another note: While preparing for a generic change request for Lua, I found a mail by Hans Hagen [1], stating that all cases have been found in luatex. Sorting the table (as in my original patch) is also a solution, but my proposed patch in lstate.c will fix the root cause. I would rather fix the root cause. If you prefer the sorting patch, I'll adapt it to activate only when FORCE_SOURCE_DATE=1 and SOURCE_DATE_EPOCH are set. With kind regards, Roland Clobus [1] http://lua-users.org/lists/lua-l/2014-07/msg00564.html

On Wed, May 4, 2022 at 3:09 PM Roland Clobus wrote:

Hello luigi, list,

...

On 19/04/2022 09:52, luigi scarso wrote: >> Thank you very much for your patch, I will check it this weekend. Have you found the time already to review my patch? [1]

Yes, Hans and I are discussing. If possible, I would like to use a --reproducible switch at the command line. -- luigi