spirit_verbaasdIf you have been in one of my XyPerl presentations or even been sitting in my XyPerl course, you probably still vaguely remember that I’m not a big fan of the ‘p’ flag that one can set on the /Pb macro.
So far I always said that when you use XyPerl the right way (aka using subroutines), there is no need for the ‘p’ flag.
Well I was wrong.


Why this change of mind?

I was recently doing an XPP project that contained an index. In that index, the page numbers are generated using cross references. An index term can exist several time on the page, so I use XyPerl to dedupe the page numbers.
The index contains 15.000 index terms. Per index term I make several XyPerl calls. Which results in a massive amount of XyPerl calls during compose (119,896 in total to be exact)

First implementation

My first implemetation was using little xyperl calls like the following:
</Pb;;music.pl;Xpp::refStore()>
It ran at a decent speed. It took my laptop 1min30sec to compose the 34 4 column pages that make up the index. Not really fast, but considering the amount of work behind the scenes pretty decent.
Then, I can’t remember the reason why, I did something I normally never do. I launched compose from the pathfinder interface using the Process/Compose menu.
All of the sudden the same compose took 4min30sec and a zillion error messages showed up. These error messages were not showing up before during the Xyview interactive compose. And they were not written to the compose log. So up until now I had never seen these messages.
The messages were of the form:
Subroutine refStore redefined at (eval 83671) line 48

What was going on?

The error message told me that Xpp was reading in over and over my XyPerl music.pl program and Perl was interpreting it over and over again.
Hence the error message that Perl was putting out.
Because this error messages gets generated in the glue layer that connects compose with perl, it is never trapped by the compose log. Put just printing that same message a zillion times in the Pathfinder process window slowed down compose by a factor of 3. Now the Pathfinder process window is actually doing quite good when printing messages. Executing the same compose from a dos command window slowed compose down to 6min.
Anyhow enough of a speed problem to get worried. Plus the fact that I hate error messages.

The solution

That was not very hard to find. I just added the ‘p’ flag on all of the xyperl calls in the style setup (there were only 4 instances in the style setup). So things were now looking like:
</Pb;p;music.pl;Xpp::refStore()>
Running it again…hey the error messages are gone.
Let’s time this.
Running in the Pathfinder process window: 30 sec
Running compose in the Xyview: 32 sec
Just putting the ‘p’ flag there, made my compose 3 to 12 quicker.
I really liked that!

What took me so long?

So why, for so many years, did I think that I don’t need the ‘p’ flag. The ‘p’ stands for persistence, which indicates that a program and its variables stay loaded in memory all the time. This has 2 advantages:

  1. when you call the same program another time, it will get executed quicker. Just because the program does not need to be loaded and interpreted again.
  2. you can store things over multiple calls to your program. So if you use something like a counter, the counter will remember its state from the last time you run your program (in fact that is not always true, you would have to use an our variable, in case you wondered)

When using XyPerl in ‘f (file) mode, you can quickly see and time the difference whether you use the ‘p’ flag.
But when you use XyPerl in subroutine mode, the difference between switching on the ‘p’ flag, is not all that clear.
Even without the ‘p’ flag, you will get persistence for your our variables. For the tests I did in the past, I got almost the same timing for running things with the ‘fp’ flag as I got for running in subroutine mode without the ‘p’ flag as with the ‘p’ flag set.
So my conclusion was: when running in subroutine mode, you don’t need to set the ‘p’ flag, as persistence is there automatically. (quickly adding to that: and since using subroutines is the only real way to run XyPerl, the ‘p’ flag is useless).
I remember that Tony W. (the Xy father of XyPerl) was a bit amazed by my findings.
But hey the test results were there, so…

And if we look back at the error messages I got during my external compose:
Subroutine refStore redefined at (eval 83671) line 48
That also indicates that the music.pl program is still loaded in memory, otherwise how could we overwrite something unless it is already there.
What seems to be happening is that XPP compose keeps reloading the same program in memory over and over again, if you don’t set the ‘p’ flag.
When you set the ‘p’ flag, XPP will load your program once and that’s it. In fact it is a bit more complicated, as XPP will calculate a checksum of the content of your program and as long as your program does not change, XPP will not reload it. (so your program gets reloaded automatically for you when you change it, a handy feature during the development cycle)
Al of this make me think that ‘p’ is a bit misleading in the case of subroutines. It is more like a ‘l’ or ‘r’ flag, which stands for ‘load’ or ‘reload’.

Conclusion

Real XyPerl programming is done using subroutines in combination with the ‘p’ flag.
Mea Culpa for hiding the truth from you all these years.