Le 8 févr. 08 à 13:52, Taco Hoekwater a écrit :

LAURENS Jérôme wrote:

...

Hi, I'd like to know the opinion of the developer team about synchronization and the patch I sent.

I looked at Synchronize4.zip a while back and liked the general idea well enough, but somehow never got around to writing up my feelings.

Here are a few remarks and questions:

* how does this patch relate to the --src-specials switch from web2c,

No common code better design stronger design better result

and what editor/previewer combinations understand your files format? (and/or are likely to start doing that in the future)

I understand your question: you are wondering if this feature would be used by a viewer or an editor. Anyone who ever used synchronization does not ask this question because he knows that such a synchronization is a "must have". But let me give more technical answers. - editors/viewers currently supporting .pdfsync files will most certainly support this new file format, because from the editor/viewer developer point of view, supporting this new format is much easier than supporting the old pdfsync. - Integrated TeX environments like iTeXMac2 or TeXShop on Mac OS X will support this feature because they want to provide the user a really efficient experience. - this new synchronization scheme is conceptually well designed, and won't break where src-ltx, src-specials and pdfsync would break - this synchronization scheme works the same for dvi or pdf Next question is: how hard is it to support this new feature? There are 2 possibilities, first, we can let things evolve according to nature's law and see what happens. Second, we can help things go the right way. More precisely, I see 2 possible designs, both relying on a synchronization controller, named "synchronizer" Consider 1 - editor 2 - synchronizer 3 - viewer forwards sync: design 1: - given a source file and a line number, editor asks the synchronizer to display the output for that location in the source - The synchronizer then parses the .sync file, finds out the real page and location to display, then asks the viewer to display it design 2: given a source file and a line number, editor asks the synchronizer for the corresponding page number and location The synchronizer then parses the .sync file, finds out the real page and location to display, and returns the result to the editor Then the editor asks the viewer to display the result backwards sync: design 1: given a pdf/dvi page and location, the viewer asks the synchronizer to display the intput for that location The synchronizer then parses the .sync file, finds out the real line number and input source file to edit, then asks the editor to edit it design 2: given a pdf page and location, the viewer asks the synchronizer for the intput corresponding to that location The synchronizer then parses the .sync file, finds out the real line number and input source file to edit, tand returns the result to the viewer Then the viewer asks the editor to edit the result The question is: where does the synchronizer live? For integrated TeX environments, this controller should be integrated, just like the editor and the viewer are integrated. But in general, the controller should be an external component(I vote for a ruby script) to which the viewer and the editor would send messages. And this controller would be the same for all editors and viewers! This is nothing but a MVC design: the model is the .sync file, the view is either the viewer or the editor, and both don't care about what is really inside the .sync file. Once the synchronization controller is available as an external script, xpdf (forthcoming 3.0.3) would support synchronization with an appropriate config file. I guess that auctex support would take just a few minutes.

* I consider the |if s>3| test in get_node() pretty bad style. There may very well be a conflicting extension in the future, and then it may turn out that your code suddenly needs rewriting. It is better to adjust the node-specific functions like new_null_box() etc.

This will result in more lines of web changes, but it will be easier to maintain the code by someone that is not you.

If you are talking about code design, I would follow you if you could explain what kind of extensions you are thinking of. The actual synchronization rules are simple - only nodes with size>3 are used for synchronization - when a node is used for synchronization, the last 2 words are reserved for synchronization - medium_node_size'd and box_node_size'd nodes are synchronized - every type of node is a potential candidate for forthcoming enhancements in synchronization By spreding initialization to many different places, you propose to create a set of rules and code for each type of node, mutliplying the synchronization hook points. I have the impression that maintaining such a code would be a pain, even for the programmer! However, if we want to initialize the nodes properly, we can do so in the node-specific function. Unfortunately, those functions are not always used to create new nodes and one must take care of other routines. But, how can we be sure that all the locations where initialization should take place are found? Any clue? Another important problem concerns memory management. I would not swear there is no memory leak.

* the web glue code in synchronize.c is not very pretty, but I doubt there is a nice solution to that, so just let it be.

yes, this is not a strong design. I guess that everything could be made in web but I don't know how (I don't want to;!), and I think it would have a non negligible computational cost when typesetting.

* you should use actual C comments, though. Not the C++ // things. (Martin may have something more to say about C indentation style)

* you licensed the C file as GPL2, yes?

Open source, whether GPL2 or something else, I don't know for sure because I was not given any hint about pdftex development rules.

* the attachments seem missing in sarovar.org at the moment

All the attachments have been removed, there might be a good reason, should I upload once again?

Best wishes, Taco

I myself never used synchronization, however I believe this is a very useful feature for a lot of users and if this is supported by pdftex/luatex, many editors/viewers will make use of it. so, here is my vote for accepting Jerome's patch. Thanh

2008/2/9, Taco Hoekwater :

The Thanh Han wrote:

...

so, here is my vote for accepting Jerome's patch.

I never explicitly said so, but I am in favor as well.

Me too.

The // comments really need fixing, though. The language is C, not C++, and pdftex is not always compiled with gcc.

Once I get the code again, I'll polish it up. Jerome, just send me the code per mail or try to attach it to #871 again. Best Martin

LAURENS Jérôme wrote:

I forgot to say that the command line option is now -synctex=1 and the primitive is \synctex

the primitive probably will be changes into something \pdfsynchronize=<number> Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

LAURENS Jérôme wrote:

Le 10 févr. 08 à 20:22, Hans Hagen a écrit :

...

LAURENS Jérôme wrote:

...

I forgot to say that the command line option is now -synctex=1 and the primitive is \synctex the primitive probably will be changes into something \pdfsynchronize=<number>

I don't think this is a good idea because synctex does not use any pdf related feature, unlike former pdfsync. It would be misleading to include "pdf" in the name. Moreover, one can see in synctex a common technology shared by all tex.web based engines, including xetex.

I agree that a name like \pdfsynchronize is more user friendly, but the advantage of synctex is that it can also be used in the CLI... as I said the string "synctex" is also some kind of keyword.

the policy for pdftex is not to introduce new primitives unless preceded by \pdf ... (not that i care that much because macro packages can deal with it but for latex it was considered a problem as there are many packages out there we don't know of), so such is life ... anyhow, even if we use a non prefixed primitive, i'd like a more verbose one \synchronizeoutputstate or, actually, it belongs to the trace family \traceoutputpositions or so; since we're talking tex here, the tex in \synctex is somewhat strange; of course you can call your mechanism as such, but that;s a different matter Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

Martin Schröder wrote:

2008/2/11, Hans Hagen :

...

...

I agree that a name like \pdfsynchronize is more user friendly, but the advantage of synctex is that it can also be used in the CLI... as I said the string "synctex" is also some kind of keyword.

LAURENS Jérôme wrote: the policy for pdftex is not to introduce new primitives unless preceded by \pdf ... (not that i care that much because macro packages can deal with it but for latex it was considered a problem as there are many packages out there we don't know of), so such is life ...

This isn't really an issue anymore with \pdfprimitive. :-)

huh? that's not so much different from a bunch of \let \mysavedwhatever \whatever and that was rejected (\primitive is not something that end users will use and i never could convince the team to not use the \pdf primitive als, if so, we should start rename the existing \pdfwhatevers

...

anyhow, even if we use a non prefixed primitive, i'd like a more verbose one

\synchronizeoutputstate

or, actually, it belongs to the trace family

\traceoutputpositions

or so; since we're talking tex here, the tex in \synctex is somewhat strange; of course you can call your mechanism as such, but that;s a different matter

Agreed. And we shouldn't forget to include Johnathan in the discussion, since this should also end up in XeTeX. :-)

(btw, is there an indication of the speed penalty and influence on the memory footprint when this feature is not enabled?) Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

Le 11 févr. 08 à 11:23, Hans Hagen a écrit :

the policy for pdftex is not to introduce new primitives unless preceded by \pdf ... (not that i care that much because macro packages can deal with it but for latex it was considered a problem as there are many packages out there we don't know of), so such is life ...

anyhow, even if we use a non prefixed primitive, i'd like a more verbose one

\synchronizeoutputstate

this one seems ambiguous

or, actually, it belongs to the trace family

\traceoutputpositions

or so; since we're talking tex here, the tex in \synctex is somewhat strange; of course you can call your mechanism as such, but that;s a different matter

Couldn't we adopt an intermediate position? \tracesynctexpositions where you consider synctex as a whole word, just like mutex, cortex, vortex, vertex and of course latex In fact, due to performance reasons, only approximate output positions are traced, the output position unit is tex sp, but the synctex position unit is 8192 sp. I would expect a \traceoutputpositions to really report the output position, not just something approximate. \tracingsynctex is another choice. More texish, but I would not choose it. JL

Le 11 févr. 08 à 19:22, Philip Taylor (Webmaster) a écrit :

If we /are/ going to abbreviate "synchroni{s|z}e" to "sync", could we please abbreviate it to "synch", otherwise there are going to be a lot of unintentional errors in coding ...

synctex is the way to go

** Phil. -------- [*] The "ch" group in "synchroni{s|z}e" is actually Greek "chi", whence indivisible.

Le 11 févr. 08 à 19:47, Hans Hagen a écrit :

LAURENS Jérôme wrote:

...

Couldn't we adopt an intermediate position? \tracesynctexpositions

i see no reason for 'tex' in the name, after all, we are running tex; also, nowadays there is no need to have short names (many new primitives have verbose names)

...

where you consider synctex as a whole word, just like mutex, cortex, vortex, vertex and of course latex In fact, due to performance reasons, only approximate output positions are traced, the output position unit is tex sp, but the synctex position unit is 8192 sp. I would expect a \traceoutputpositions to really report the output position, not just something approximate.

well, i'm more thinking along the line ...

\traceoutputpositions = 0 : disabled \traceoutputpositions = 1 : low resolution \traceoutputpositions = 2 : medium resolution (8192 resolution) \traceoutputpositions = 3 : high resolution

Unfortunately, this is not what synctex is designed for. It is more like \tracesynctexpositions=0:disabled \tracesynctexpositions=1:math, kern, glue, hbox \tracesynctexpositions=2:add noads \tracesynctexpositions=3:add some whatsits All this is dedicated to the Editor/Viewer synchronization, and it is not advisable to use it for something else. Roughly speaking, here is how synctex works: 1 - some nodes have been extended to -always- store synchronization information 2 - if some flag is enabled, this information is used to help synchronization. Point 1 belongs to the data model, it is public and can be used by any other tex extension. Point 2 is a controller, which means a piece of code that uses the data; it is -private- to synctex. The \synctex (or \tracesynctexpositions) primitive controls the behaviour of this synctex controller. So it is dedicated to point 2 and this is why it must be named accordingly. If someone plans to use point 1 for another tex extension, then it will be possible to use your \traceoutputpositions terminology if this is relevant.

although position are always approximations (take for instance complex ligatures orgenerated code) i wonder if we should divert from the full sp resolution; i'd say: output the full number (no devision) and provide resolution in the amount of sync points

ok, the question is about precision

(if this feature also ends up in luatex, then it will be full resolution if only because all dimensions accessible at the lua end are in scaled points)

no, you are confusing the data and the controller. Actually, the dimensions are available in full dimensions, but this is a deliberate choice of the controller to truncate them to a lower precision. If the controller is implemented in lua, it will also decide to truncate simply because it is a matter of efficiency. The purpose of the lower precision in synctex end is simply to reduce the size of the auxiliary output file. If we use full resolution, then the size increases by 40%, which is - very- significant because I/O operations are involved. 8192 is the best choice for that purpose. I hope this explanation will enlight the design of synctex, and make you understand how tex and synctex are tied together. JL

Hans Hagen wrote:

LAURENS Jérôme wrote:

...

\tracesynctexpositions=0:disabled \tracesynctexpositions=1:math, kern, glue, hbox \tracesynctexpositions=2:add noads \tracesynctexpositions=3:add some whatsits

so, variant 2 and 3 are influencing the node lists (I assume that this does not influence the outout i.e. no inhibiting of the \last... primitives etc (sometimes whatsits can get in the way)

I assume they will just be adding more output for other node types (read "add" as an abbreviation for "additionally, produce information for .."). The point of this patch, iiuc, is that there is no need for node list manupilations.

...

The purpose of the lower precision in synctex end is simply to reduce the size of the auxiliary output file. If we use full resolution, then the size increases by 40%, which is - very- significant because I/O operations are involved. 8192 is the best choice for that purpose.

It would be interesting to have the full precision as an option. A macro package for two-pass page break optimization may want to use the synctex file also , and such a package would really need full precision to avoid rounding errors. Best wishes, Taco

LAURENS Jérôme wrote:

Le 12 févr. 08 à 09:39, Taco Hoekwater a écrit :

...

...

...

The purpose of the lower precision in synctex end is simply to reduce the size of the auxiliary output file. If we use full resolution, then the size increases by 40%, which is - very- significant because I/O operations are involved. 8192 is the best choice for that purpose. It would be interesting to have the full precision as an option. A macro package for two-pass page break optimization may want to use the synctex file also , and such a package would really need full precision to avoid rounding errors.

I am not in favour of adding complexity to synctex out of its original purpose. I can't see how .synctex would benefit a page breaker. Conversely, basing extension on the synctex file will add limitations to both the extension and synctex.

can you elaborate?

I could, but no time. And it was not important anyway. Best wishes, Taco

LAURENS Jérôme wrote:

Le 12 févr. 08 à 09:26, Hans Hagen a écrit :

...

LAURENS Jérôme wrote:

...

\tracesynctexpositions=0:disabled \tracesynctexpositions=1:math, kern, glue, hbox \tracesynctexpositions=2:add noads \tracesynctexpositions=3:add some whatsits so, variant 2 and 3 are influencing the node lists (I assume that this does not influence the outout i.e. no inhibiting of the \last... primitives etc (sometimes whatsits can get in the way)

Actually, absolutely no new node is created, unlike srcltx, src- specials and pdfsync This is why there won't be compatibility problems with already existing macros.

ok, so 2 is not 'add noad's (which are nodes -) but 'add info to more nodes'

...

(btw, i then assume that there is also an implicit position insertion possible, like \injectsuchapositionhere or so)

not yet.

maybe it makes sense to add such a primitive

The idea would be something like \marksynctexposition{blah} or \synctexcomment{blah} where blah is something related to the content, but I do not know yet how it should be designed

hm, adding labels would add more info to the log file (which you want to keep small)

With synctex, you are absolutely sure that no new node has been created. This is what makes the difference with srcltx, src specials and pdfsync and this is what ensures that other packages won't break.

ok, good

I guess the size of the node is not something publicly available in luatex, otherwise there might be problems when changing the node size in the core of the engine.

luatex has reimplemented nodes completely so any sync feature in there would work differently: i.e. extra fields in each node, accessible as such (from lua) and even manipulateable; the file location aspect is even a bit more tricky because one can overload the file handler and as such positional information that tracks back to the source also needs to carry (probably indirect) information about that (for instance: input can come from zip files, or input sequences can be transformed into other sequences); in that respect the writing of tracking info to a file may depends on what the macro package does with the input handling (of course this can be hidden from users by appropriate functions) but there is no need to worry about that now (after all, this is the pdftex dev list) Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

LAURENS Jérôme wrote:

http://itexmac.sourceforge.net/synchronize+Frames.pdf

If you take a look at pages 5-9, you will see exactly the information the synctex auxiliary file provides. This is far better than former specials or pdfsync. I can't measure for sure, but to give just an idea, I would say that with this information, I can synchronize 99% of the characters, and 99.9% of the lines.

btw, this all depends on the kind of document, (e.g. xml input where data is filtered from the tree is trikcy), but for traditionally edited tex docs this is probably true (apart from tables of contents, lists, generated info, but ther one can disable the mechanism)

So I am not sure that a \injectsuchapositionhere would be of great help. Maybe adding meta information/comment would help.

it depends; i use \pdfsavepos cum suis lots of time (sometimes docs with many thousands of positions, and megabytes of multipass files) but this is different from a sync features (positional info carries much more info around) just wondering ... why do you use h: and g: ? i can also imagine that you just use the node id (hlists and glues are just a few types) ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

2008/2/9, Taco Hoekwater :

LAURENS Jérôme wrote:

...

Open source, whether GPL2 or something else, I don't know for sure because I was not given any hint about pdftex development rules.

GPL2 is fine, just making sure.

GPL2++, actually.

...

...

* the attachments seem missing in sarovar.org at the moment

All the attachments have been removed, there might be a good reason, should I upload once again?

I don't know why they were removed in the first place. Martin, is that something you did?

Seems to be a Sarovar problem. Note that it does not say that there are no attachements; it just doesn't display them. The last line of the page is "Existing Files:" -- wherever they are. :-( Best Martin