Re: [dev-context] local allocation in ConTeXt
Dear Hans, thank you very much for helping me out here. Aditya's rephrasal of the question might have shown a tree but hidden the forest ... ;-) Let me try to explain below, but please also take a look at my original post, https://mailman.ntg.nl/pipermail/dev-context/2016/003346.html. As far as I can see, in your 10 million example I can only work on one of the local boxes at the time, whereas I need to save an undetermined number of boxes, do some computation which uses dimensions of all these boxes (this computation potentially invokes user's code, so I don't have control over it), and then use all the saved boxes. (Btw, saving just the dimensions is not an acceptable solution, both because of speed and because in general, setting the box with same content twice won't yield the same result.) As the number of needed boxes cannot be known in advance, I'd like to be nice and "give them back" to the allocation system after using them. And I *do* know how to do that (see localloc, elocalloc, or etex), but I'm not sure if it would cause any trouble in ConTeXt: I'd prefer "not to mess with allocation" indeed and have the issue fixed systemically.
The overhead is just one box per name and you probably don't allocate a million unique tempboxes as you will then already ran out of names. I believe this overhead is precisely what I'm trying to avoid. A further detail of my situation might help. I've got, for each node of the tree: \locbox\tempbox \setbox\tempbox\hbox{...} \expandafter\let\csname node<id>@box\endcsname\tempbox
Messing with the allocation can fail because when the content of the boxes triggers a new box allocation itself that one might be forgotten as well. I'm not sure I get this. Even if the new box allocation was global (so, the usual \newbox)?
Best, Sašo P.S. And I did test it! Albeit with two iterations, not ten million. ;-) On 24. 01. 2017 13:48, Hans Hagen wrote:
On 1/24/2017 10:47 AM, Sašo Živanović wrote:
I'm afraid this doesn't do the trick: \c_syst_last_allocated_box is definitely incremented.
Let me try explaining again. This is essentially what I do:
{ \newbox\TempBox \setbox\TempBox\hbox{Box One} \let\BoxOne\TempBox \global\let\TempBox\relax
\newbox\TempBox \setbox\TempBox\hbox{Box Two} \let\BoxTwo\TempBox \global\let\TempBox\relax
\the\numexpr\BoxOne: [\box\BoxOne] \the\numexpr\BoxTwo: [\box\BoxTwo] }
But I want the number of available box registers at entering and exiting the group to be the same, and substituting \newbox for \locbox is what does the trick in my package. So I can run the following a million times:
{ \newlocalbox\TempBox \setbox\TempBox\hbox{Box One} \let\BoxOne\TempBox \global\let\TempBox\relax
\newlocalbox\TempBox \setbox\TempBox\hbox{Box Two} \let\BoxTwo\TempBox \global\let\TempBox\relax
\the\numexpr\BoxOne: [\box\BoxOne] \the\numexpr\BoxTwo: [\box\BoxTwo] }
Each of the million "\the\numexpr\BoxOne" should produce the same number, whereas in the \newbox-based example these numbers increment.
I suppose you didn't test it ...
\starttext
% \unprotect % % \installcorenamespace{localbox} % % \unexpanded\def\newlocalbox#1% % {\expandafter\let\expandafter#1\csname\??localbox\string#1\endcsname % \ifx#1\relax % \syst_aux_new_localbox#1% % \fi} % % \def\syst_aux_new_localbox#1% % {\expandafter\newbox\csname\??localbox\string#1\endcsname % \newlocalbox#1} % % \protect
\newbox\MyBoxA \number\MyBoxA
\dorecurse{10} { \dorecurse{1000000} { \newlocalbox\TempBox } \writestatus{!!!}{#1 million done} #1: \number\TempBox\par }
\newbox\MyBoxB \number\MyBoxB
\stoptext
The overhead is just one box per name and you probably don't allocate a million unique tempboxes as you will then already ran out of names.
Messing with the allocation can fail because when the content of the boxes triggers a new box allocation itself that one might be forgotten as well.
Best, Sašo
On 24. 01. 2017 09:52, Hans Hagen wrote:
You are asking how to define a macro, \locbox, such that the following code should not change the value of \c_syst_last_allocated_box.
\bgroup \locbox\BoxOne \locbox\BoxTwo
\BoxOne\hbox{Box One} \BoxTow\hbox{Box Two} % Do some measurement based on the size of the boxes \egroup
The fact that a LaTeX package implements this behaviour by allocating local box numbers from the end seems to be an implementation detail.
I don't know the best way to implement such a macro in ConTeXt, but I want to make sure that the user-level behavior that you want is properly understood. Given the details in your question, one can easily lose the forest for the trees (pun intended :-)
\unprotect
\installcorenamespace{localbox}
\unexpanded\def\newlocalbox#1% {\expandafter\let\expandafter#1\csname\??localbox\string#1\endcsname \ifx#1\relax \syst_aux_new_localbox#1% \fi}
\def\syst_aux_new_localbox#1% {\expandafter\newbox\csname\??localbox\string#1\endcsname \newlocalbox#1}
\protect
\starttext
\newlocalbox\BoxOne \newlocalbox\BoxTwo
\setbox\BoxOne\hbox{Box One} \setbox\BoxTwo\hbox{Box Two}
[\box\BoxTwo] [\box\BoxOne]
\let\locbox\newlocalbox
\stoptext
----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl ----------------------------------------------------------------- _______________________________________________ dev-context mailing list dev-context@ntg.nl https://mailman.ntg.nl/mailman/listinfo/dev-context
On 1/24/2017 3:05 PM, Sašo Živanović wrote:
Dear Hans,
thank you very much for helping me out here.
Aditya's rephrasal of the question might have shown a tree but hidden the forest ... ;-) Let me try to explain below, but please also take a look at my original post, https://mailman.ntg.nl/pipermail/dev-context/2016/003346.html.
As far as I can see, in your 10 million example I can only work on one of the local boxes at the time, whereas I need to save an undetermined number of boxes, do some computation which uses dimensions of all these boxes (this computation potentially invokes user's code, so I don't have control over it), and then use all the saved boxes. (Btw, saving just the dimensions is not an acceptable solution, both because of speed and because in general, setting the box with same content twice won't yield the same result.)
Just try it ... \newbox\MyBoxA \number\MyBoxA \dorecurse{2} { \dorecurse{1000000} { \newlocalbox\TempBoxA \newlocalbox\TempBoxB \newlocalbox\TempBoxC \newlocalbox\TempBoxD \newlocalbox\TempBoxE } \writestatus{!!!}{#1 million done} #1: \number\TempBoxA\par #1: \number\TempBoxB\par #1: \number\TempBoxC\par #1: \number\TempBoxD\par #1: \number\TempBoxE\par } \newbox\MyBoxB \number\MyBoxB \stoptext overhead: 5 boxes in 2*1 million (or more) allocations so, if a user needs 15.000 boxes locally you have an overhead of 15.000 but the second time if he/she uses the same names the overhead doesn't grow ... in fact, if he/she uses 15K new names also the hash will get them (even if they get undefined later)
As the number of needed boxes cannot be known in advance, I'd like to be nice and "give them back" to the allocation system after using them. And I *do* know how to do that (see localloc, elocalloc, or etex), but I'm not sure if it would cause any trouble in ConTeXt: I'd prefer "not to mess with allocation" indeed and have the issue fixed systemically.
no need to give them back ..
The overhead is just one box per name and you probably don't allocate a million unique tempboxes as you will then already ran out of names. I believe this overhead is precisely what I'm trying to avoid. A further detail of my situation might help. I've got, for each node of the tree: \locbox\tempbox \setbox\tempbox\hbox{...} \expandafter\let\csname node<id>@box\endcsname\tempbox
Messing with the allocation can fail because when the content of the boxes triggers a new box allocation itself that one might be forgotten as well. I'm not sure I get this. Even if the new box allocation was global (so, the usual \newbox)?
sure ... with 2 million: 452 1: 453 1: 454 1: 455 1: 456 1: 457 2: 453 2: 454 2: 455 2: 456 2: 457 458 so, only 5 slots used and lots left
Best, Sašo
P.S. And I did test it! Albeit with two iterations, not ten million. ;-)
On 24. 01. 2017 13:48, Hans Hagen wrote:
On 1/24/2017 10:47 AM, Sašo Živanović wrote:
I'm afraid this doesn't do the trick: \c_syst_last_allocated_box is definitely incremented.
Let me try explaining again. This is essentially what I do:
{ \newbox\TempBox \setbox\TempBox\hbox{Box One} \let\BoxOne\TempBox \global\let\TempBox\relax
\newbox\TempBox \setbox\TempBox\hbox{Box Two} \let\BoxTwo\TempBox \global\let\TempBox\relax
\the\numexpr\BoxOne: [\box\BoxOne] \the\numexpr\BoxTwo: [\box\BoxTwo] }
But I want the number of available box registers at entering and exiting the group to be the same, and substituting \newbox for \locbox is what does the trick in my package. So I can run the following a million times:
{ \newlocalbox\TempBox \setbox\TempBox\hbox{Box One} \let\BoxOne\TempBox \global\let\TempBox\relax
\newlocalbox\TempBox \setbox\TempBox\hbox{Box Two} \let\BoxTwo\TempBox \global\let\TempBox\relax
\the\numexpr\BoxOne: [\box\BoxOne] \the\numexpr\BoxTwo: [\box\BoxTwo] }
Each of the million "\the\numexpr\BoxOne" should produce the same number, whereas in the \newbox-based example these numbers increment.
I suppose you didn't test it ...
\starttext
% \unprotect % % \installcorenamespace{localbox} % % \unexpanded\def\newlocalbox#1% % {\expandafter\let\expandafter#1\csname\??localbox\string#1\endcsname % \ifx#1\relax % \syst_aux_new_localbox#1% % \fi} % % \def\syst_aux_new_localbox#1% % {\expandafter\newbox\csname\??localbox\string#1\endcsname % \newlocalbox#1} % % \protect
\newbox\MyBoxA \number\MyBoxA
\dorecurse{10} { \dorecurse{1000000} { \newlocalbox\TempBox } \writestatus{!!!}{#1 million done} #1: \number\TempBox\par }
\newbox\MyBoxB \number\MyBoxB
\stoptext
The overhead is just one box per name and you probably don't allocate a million unique tempboxes as you will then already ran out of names.
Messing with the allocation can fail because when the content of the boxes triggers a new box allocation itself that one might be forgotten as well.
Best, Sašo
On 24. 01. 2017 09:52, Hans Hagen wrote:
You are asking how to define a macro, \locbox, such that the following code should not change the value of \c_syst_last_allocated_box.
\bgroup \locbox\BoxOne \locbox\BoxTwo
\BoxOne\hbox{Box One} \BoxTow\hbox{Box Two} % Do some measurement based on the size of the boxes \egroup
The fact that a LaTeX package implements this behaviour by allocating local box numbers from the end seems to be an implementation detail.
I don't know the best way to implement such a macro in ConTeXt, but I want to make sure that the user-level behavior that you want is properly understood. Given the details in your question, one can easily lose the forest for the trees (pun intended :-)
\unprotect
\installcorenamespace{localbox}
\unexpanded\def\newlocalbox#1% {\expandafter\let\expandafter#1\csname\??localbox\string#1\endcsname \ifx#1\relax \syst_aux_new_localbox#1% \fi}
\def\syst_aux_new_localbox#1% {\expandafter\newbox\csname\??localbox\string#1\endcsname \newlocalbox#1}
\protect
\starttext
\newlocalbox\BoxOne \newlocalbox\BoxTwo
\setbox\BoxOne\hbox{Box One} \setbox\BoxTwo\hbox{Box Two}
[\box\BoxTwo] [\box\BoxOne]
\let\locbox\newlocalbox
\stoptext
----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl ----------------------------------------------------------------- _______________________________________________ dev-context mailing list dev-context@ntg.nl https://mailman.ntg.nl/mailman/listinfo/dev-context
-- ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------
On 24. 01. 2017 16:36, Hans Hagen wrote:
On 1/24/2017 3:05 PM, Sašo Živanović wrote:
Dear Hans,
thank you very much for helping me out here.
Aditya's rephrasal of the question might have shown a tree but hidden the forest ... ;-) Let me try to explain below, but please also take a look at my original post, https://mailman.ntg.nl/pipermail/dev-context/2016/003346.html.
As far as I can see, in your 10 million example I can only work on one of the local boxes at the time, whereas I need to save an undetermined number of boxes, do some computation which uses dimensions of all these boxes (this computation potentially invokes user's code, so I don't have control over it), and then use all the saved boxes. (Btw, saving just the dimensions is not an acceptable solution, both because of speed and because in general, setting the box with same content twice won't yield the same result.)
Just try it ...
\newbox\MyBoxA \number\MyBoxA
\dorecurse{2} { \dorecurse{1000000} { \newlocalbox\TempBoxA \newlocalbox\TempBoxB \newlocalbox\TempBoxC \newlocalbox\TempBoxD \newlocalbox\TempBoxE } \writestatus{!!!}{#1 million done} #1: \number\TempBoxA\par #1: \number\TempBoxB\par #1: \number\TempBoxC\par #1: \number\TempBoxD\par #1: \number\TempBoxE\par }
\newbox\MyBoxB \number\MyBoxB
\stoptext
overhead: 5 boxes in 2*1 million (or more) allocations
But not if: \def\csnewlocalbox#1{\expandafter\newlocalbox\csname #1\endcsname} \def\csboxnumber#1{\expandafter\number\csname #1\endcsname} \starttext \newbox\MyBoxA \number\MyBoxA \dorecurse{2} { \dorecurse{1000} { \csnewlocalbox{TempBox#1-1}% \csnewlocalbox{TempBox#1-2}% \csnewlocalbox{TempBox#1-3}% \csnewlocalbox{TempBox#1-4}% \csnewlocalbox{TempBox#1-5}% } \writestatus{!!!}{#1 million done} #1: \csboxnumber{TempBox#1-1}\par #1: \csboxnumber{TempBox#1-2}\par #1: \csboxnumber{TempBox#1-3}\par #1: \csboxnumber{TempBox#1-4}\par #1: \csboxnumber{TempBox#1-5}\par } \newbox\MyBoxB \number\MyBoxB \stoptext However, I see your point now. If I use a consistent naming scheme (and I do) and allocate using \csnewlocalbox{node<id>@box}, the boxes will be reused.
so, if a user needs 15.000 boxes locally you have an overhead of 15.000 but the second time if he/she uses the same names the overhead doesn't grow ... in fact, if he/she uses 15K new names also the hash will get them (even if they get undefined later)
As the number of needed boxes cannot be known in advance, I'd like to be nice and "give them back" to the allocation system after using them. And I *do* know how to do that (see localloc, elocalloc, or etex), but I'm not sure if it would cause any trouble in ConTeXt: I'd prefer "not to mess with allocation" indeed and have the issue fixed systemically.
no need to give them back ..
Well, imho, that's bad programming practice, but the fault goes all the way back to plain TeX ... Thanks for being patient with me! Sašo
On 1/24/2017 5:04 PM, Sašo Živanović wrote:
On 24. 01. 2017 16:36, Hans Hagen wrote:
On 1/24/2017 3:05 PM, Sašo Živanović wrote:
Dear Hans,
thank you very much for helping me out here.
Aditya's rephrasal of the question might have shown a tree but hidden the forest ... ;-) Let me try to explain below, but please also take a look at my original post, https://mailman.ntg.nl/pipermail/dev-context/2016/003346.html.
As far as I can see, in your 10 million example I can only work on one of the local boxes at the time, whereas I need to save an undetermined number of boxes, do some computation which uses dimensions of all these boxes (this computation potentially invokes user's code, so I don't have control over it), and then use all the saved boxes. (Btw, saving just the dimensions is not an acceptable solution, both because of speed and because in general, setting the box with same content twice won't yield the same result.)
Just try it ...
\newbox\MyBoxA \number\MyBoxA
\dorecurse{2} { \dorecurse{1000000} { \newlocalbox\TempBoxA \newlocalbox\TempBoxB \newlocalbox\TempBoxC \newlocalbox\TempBoxD \newlocalbox\TempBoxE } \writestatus{!!!}{#1 million done} #1: \number\TempBoxA\par #1: \number\TempBoxB\par #1: \number\TempBoxC\par #1: \number\TempBoxD\par #1: \number\TempBoxE\par }
\newbox\MyBoxB \number\MyBoxB
\stoptext
overhead: 5 boxes in 2*1 million (or more) allocations
But not if:
\def\csnewlocalbox#1{\expandafter\newlocalbox\csname #1\endcsname} \def\csboxnumber#1{\expandafter\number\csname #1\endcsname}
\starttext
\newbox\MyBoxA \number\MyBoxA
\dorecurse{2} { \dorecurse{1000} { \csnewlocalbox{TempBox#1-1}% \csnewlocalbox{TempBox#1-2}% \csnewlocalbox{TempBox#1-3}% \csnewlocalbox{TempBox#1-4}% \csnewlocalbox{TempBox#1-5}% } \writestatus{!!!}{#1 million done} #1: \csboxnumber{TempBox#1-1}\par #1: \csboxnumber{TempBox#1-2}\par #1: \csboxnumber{TempBox#1-3}\par #1: \csboxnumber{TempBox#1-4}\par #1: \csboxnumber{TempBox#1-5}\par }
\newbox\MyBoxB \number\MyBoxB
\stoptext
even then you only get 5000 boxes defined ... you can reset the storag of course but you still have 5000 hash entries (mening \undefined then) with names like TempBox*-* \starttext \dorecurse{100} {\bgroup \dorecurse{10000} {\expandafter\def\csname foo:#1:##1\endcsname{}} \egroup} \stoptext so in the end it doesn't matter much that the boxes are allocated given that one reuses names
However, I see your point now. If I use a consistent naming scheme (and I do) and allocate using \csnewlocalbox{node<id>@box}, the boxes will be reused.
so, if a user needs 15.000 boxes locally you have an overhead of 15.000 but the second time if he/she uses the same names the overhead doesn't grow ... in fact, if he/she uses 15K new names also the hash will get them (even if they get undefined later)
As the number of needed boxes cannot be known in advance, I'd like to be nice and "give them back" to the allocation system after using them. And I *do* know how to do that (see localloc, elocalloc, or etex), but I'm not sure if it would cause any trouble in ConTeXt: I'd prefer "not to mess with allocation" indeed and have the issue fixed systemically.
no need to give them back ..
Well, imho, that's bad programming practice, but the fault goes all the way back to plain TeX ...
and with the fact that we have a macro language with a grouping model and even worse, users who can do unexpected things like \starttext \bgroup \newlocalbox\foo \setbox\foo\hbox\bgroup bar \ifdefined\oof\else\newbox\oof\fi \global\setbox\oof\hbox{foo} \egroup \egroup \box\oof \stoptext when we would reset the allocator count then \oof would still be ok but its slot would be reallocated at some point by \newlocalbox ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------
On 24. 01. 2017 17:33, Hans Hagen wrote:
\def\csnewlocalbox#1{\expandafter\newlocalbox\csname #1\endcsname} \def\csboxnumber#1{\expandafter\number\csname #1\endcsname}
\starttext
\newbox\MyBoxA \number\MyBoxA
\dorecurse{2} { \dorecurse{1000} { \csnewlocalbox{TempBox#1-1}% \csnewlocalbox{TempBox#1-2}% \csnewlocalbox{TempBox#1-3}% \csnewlocalbox{TempBox#1-4}% \csnewlocalbox{TempBox#1-5}% } \writestatus{!!!}{#1 million done} #1: \csboxnumber{TempBox#1-1}\par #1: \csboxnumber{TempBox#1-2}\par #1: \csboxnumber{TempBox#1-3}\par #1: \csboxnumber{TempBox#1-4}\par #1: \csboxnumber{TempBox#1-5}\par }
\newbox\MyBoxB \number\MyBoxB
\stoptext
even then you only get 5000 boxes defined ... Well, that's because I was impatient and have changed a 1000000 to a 1000 ;-) you can reset the storag of course but you still have 5000 hash entries (mening \undefined then) with names like TempBox*-* That's very good info, thanks!
\starttext
\bgroup \newlocalbox\foo \setbox\foo\hbox\bgroup bar \ifdefined\oof\else\newbox\oof\fi \global\setbox\oof\hbox{foo} \egroup \egroup
\box\oof
\stoptext
when we would reset the allocator count then \oof would still be ok but its slot would be reallocated at some point by \newlocalbox I hope you don't mind explaining this further, as I'm lost.
1. I was unable to produce this effect. Should it happen with the \newlocalbox defined in this thread, or some hypothetical \newlocalbox (which would share the allocation counter with \newbox?). I tried both your \newlocalbox and elocalloc's \locbox, and didn't manage to reallocate \oof's register. But maybe this question is bogus, as: 2. I don't understand why it is important to use \newbox within \setbox\foo\hbox{...} Don't we get the same result in the following? \starttext \bgroup \newlocalbox\foo \setbox\foo\hbox\bgroup bar \egroup \ifdefined\oof\else\newbox\oof\fi \global\setbox\oof\hbox{foo} \egroup \box\oof \stoptext Best, Sašo
On 1/26/2017 5:02 PM, Sašo Živanović wrote:
On 24. 01. 2017 17:33, Hans Hagen wrote:
\def\csnewlocalbox#1{\expandafter\newlocalbox\csname #1\endcsname} \def\csboxnumber#1{\expandafter\number\csname #1\endcsname}
\starttext
\newbox\MyBoxA \number\MyBoxA
\dorecurse{2} { \dorecurse{1000} { \csnewlocalbox{TempBox#1-1}% \csnewlocalbox{TempBox#1-2}% \csnewlocalbox{TempBox#1-3}% \csnewlocalbox{TempBox#1-4}% \csnewlocalbox{TempBox#1-5}% } \writestatus{!!!}{#1 million done} #1: \csboxnumber{TempBox#1-1}\par #1: \csboxnumber{TempBox#1-2}\par #1: \csboxnumber{TempBox#1-3}\par #1: \csboxnumber{TempBox#1-4}\par #1: \csboxnumber{TempBox#1-5}\par }
\newbox\MyBoxB \number\MyBoxB
\stoptext
even then you only get 5000 boxes defined ... Well, that's because I was impatient and have changed a 1000000 to a 1000 ;-) you can reset the storag of course but you still have 5000 hash entries (mening \undefined then) with names like TempBox*-* That's very good info, thanks!
\starttext
\bgroup \newlocalbox\foo \setbox\foo\hbox\bgroup bar \ifdefined\oof\else\newbox\oof\fi \global\setbox\oof\hbox{foo} \egroup \egroup
\box\oof
\stoptext
when we would reset the allocator count then \oof would still be ok but its slot would be reallocated at some point by \newlocalbox I hope you don't mind explaining this further, as I'm lost.
1. I was unable to produce this effect. Should it happen with the \newlocalbox defined in this thread, or some hypothetical \newlocalbox (which would share the allocation counter with \newbox?). I tried both your \newlocalbox and elocalloc's \locbox, and didn't manage to reallocate \oof's register.
But maybe this question is bogus, as:
2. I don't understand why it is important to use \newbox within \setbox\foo\hbox{...} Don't we get the same result in the following?
\starttext
\bgroup \newlocalbox\foo \setbox\foo\hbox\bgroup bar \egroup \ifdefined\oof\else\newbox\oof\fi \global\setbox\oof\hbox{foo} \egroup
\box\oof
\stoptext
the difference is that a next \newbox\oof will report that \off is already defined (so no new box will be allocated unless you \relax \foo) while \newlocalbox\foo will happily reuse \foo and not report a message (as you use it local \foo is also defined as local while \oof is global) Hans -- ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------
participants (2)
-
Hans Hagen
-
Sašo Živanović