Module talk:WikitextParser

This module should use Module:Plain text for parsing section names

Currently it just reads them naively. This causes an issue in Module:Excerpt which uses this to find section names. If the section has an anchor then it will not correctly find it. — Qwerfjkltalk 16:29, 14 May 2026 (UTC)

I think it's a good idea, but not as easy to implement as one might expect. Firstly, the plainText logic should probably be enhanced to strip files, categories, etc in languages other than English (though that's not too critical). Also, since the code is quite short and straightforward, I'll probably prefer to copy it rather than loading it, to avoid dependency complications. It can also be customized to better fit the needs of this particular use case. But more importantly, due to the way getSection currently works, it'll need to be completely rewritten to add plainText to it. It's not that straightforward. I started to do it in the sandbox, but with these changes, it no longer transcludes subsections, and apparently quite a few pages depend on that feature already. But I like the new sandbox version of getSection, so I'm thinking on writing a new getSubsections method that combined with getSection could re-implement the "subsections" feature. Furthermore, perhaps this set of changes is a good opportunity to implement this other good idea to increase/decrease the level of the subsections when needed. So anyway, it may take a bit of time to implement, but I think it's a good idea. Thanks for proposing it. Sophivorus (talk) 15:06, 15 May 2026 (UTC)

Iterator factory APIs

I think it is unfortunate so many of these API functions are written to return lists when they are internally implemented using things like: string.gmatch, mw.text.gsplit, etc. These already use iterators internally. Why not implement versions of the plural APIs to be iterator factories and then if necessary for compatibility reimplement the list APIs via those iterator factory functions? For example, one could make a local iterator function implemented something like:

local function iterTemplates(gmt)
	for template in gmt do
		if string.sub( template, 1, 3 ) ~= '{{#' then
			return template
		end
	end
end

and then have a new API function implemented something like:

function parser.ggetTemplates( wikitext )
	return iterTemplates, string.gmatch( wikitext, '{%b{}}' )
end

Then it would be easy to modify parser.getTemplate to use that instead, allowing it to parse only until it finds the template it wants and of course the compatibility function parser.getTemplates that returns the list could also easily be implemented via such as well. Then one can consider similar code for other plural APIs that return lists like: parser.getSections, parser.getLists, parser.getParagraphs, parser.getTemplateParameters, parser.getTags, parser.getGalleries, parser.getReferences, parser.getTables, parser.getLinks, parser.getFileLinks, parser.getCategories and parser.getExternalLinks. I am sure some of these will be a tad trickier to implement as iterator factories instead of lists than others but none of them should be that complex. The iterator factory interface seems like the natural choice, however, this module has instead chosen to implement APIs that return complete sequential list tables instead. It seriously makes me consider a rewrite (without all the list APIs) might be in order. —Uzume (talk) 08:31, 20 June 2026 (UTC)

Hi! I get your point. Iterator functions are the right choice, I just wasn't sufficiently aware of them. Lets do it! I think we can even adapt the (still few) modules that currently use WikitextParser (most notably Module:Excerpt) to use the new iterator functions, to avoid re-implementing the ones that return lists. Sophivorus (talk) 14:38, 23 June 2026 (UTC)

@Uzume I'm trying to get a proof of concept working in the sandbox but I'm having difficulty due to my lack of familiarity with iterator functions. Would you mind coding the first example? I can then study it and extend it to other cases. Also, while I like the idea of using the right tool for the job, could you explain what would be the actual benefit(s) of using iterator functions? Better performance? Cleaner interactions? Thanks! Sophivorus (talk) 15:10, 23 June 2026 (UTC)

@Sophivorus: Well, iterators iterate over some state, such as the place within a list, etc. In the task of parsing it would be most applicable to finding and returning matching tokens. In the code in this module, everything is done with completed lists, requiring one to traverse the complete list of something to create the list before attempting to use/identify the parts one is really interesting in. For example, in this module parser.getTemplate is implemented in terms of parser.getTemplates, traversing over the list of found template invocations stopping at the first one matching by name using parser.getTemplateName to identify the name of each template invocation. This is fine if one wants just the first template invocation of a template named name but suppose one wanted all of them and not just the first, or one wanted the one where parameter param matches some criteria, etc. The naive approach of this module is to pass around complete lists of things instead of iterators that generate such lists. Which do you think is a more performant method of writing parser.getTemplates? Using string.gmatch to create a list of all the double brace pairs found and then re-iterating over that list to filter out those that appear to be parser function calls that start with '#' creating a new filtered list? No, one filters the list at generation time so one does not have to generate one list just to traverse it and throw away the first. Suppose one wanted to optimize parser.getTemplateName which just returns the first matching template by name? Why does one have to use parser.getTemplates which identifies every template invocation first? I am sure you can imagine a version of parser.getTemplates that instead copies the code of parser.getTemplates, directly using string.gmatch and the same pattern but instead filters on the template name instead of being concerned about filtering out apparent parser function invocations starting with the pound symbol? Wouldn't that be faster since it never has to find every instance of the double brace matching—instead only needing and stopping after finding the first that matches the name of the template one is interested in? But then one has concerns of code maintenance. One cannot leverage the code in parser.getTemplates and has to maintain two copies of very similar code. If instead wouldn't it be nice if parser.getTemplates returned an iterator that only found the next matching brace pair that wasn't an obvious parser function call, instead of all of them? Then one could use this iterator to either iterator over all of them when needed, or to just find what one needs and stop iterating. That is what my parser.ggetTemplates above attempts to do (since I have not really tested it yet; it plays a little quick and dirty assuming string.gmatch returns a stated and not a stateless iterator). Anyway, my idea is to essentially redefine everything that return lists into things that return iterators. If one really wants the complete list, one can just run the iterator oneself and generate the complete list but the point is, iterators generate each item one at a time instead of requiring the complete list to be generated first. I can envision a wikitext parser that is implemented as iterators built upon iterators—where each iterator uses an earlier iterator to filter a more generic list into a more specific one. For example, instead of parser.getTemplate stopping at the first matching template by name, it could be (re)written as an iterator matching them all and a user could decide to call such and stop at the first, generate the complete list, or even build their own iterator that further filters that, etc. Think of this as a pipeline of iterators from generic to more specific. Maybe someone wants to find all the parser function calls. Why can't they use the same double brace matching iterator that the template call iterator also uses, etc.? This translates into better separation of concerns and in turn better code maintenance and performance even if the code might be a tad trickier to develop because one has to think in terms of iterators instead of loops and completed lists. I hope this explanation helps. —Uzume (talk) 03:55, 24 June 2026 (UTC)

@Sophivorus: Another thing iterators do, and why they are a tad trickier to think about, is that they provide an inversion of control. I just now noticed you are trying to do something similar in Module:Excerpt with all your "filters". I imagine many of those could be gotten rid of if the functionality above it was developed as an iterator and its factory. Then essentially one can think of parsing as a pipeline of filtering iterators. Instead of filtered looping over lists (which requires a filter function and a completed list) just iterate over iteration. Does that make sense? Lua provides the following iterator factories: ipairs, pairs (where the iterator next is the default), string.gmatch (compare to string.match which does the same but only returns the first match). The first two are designed for table iteration but the last is for string iteration. Likewise Scribunto expands on this with: mw.text.gsplit (which can be directly compared with the list function mw.text.split), mw.ustring.gcodepoint, mw.ustring.gmatch (compare to string.gmatch and mw.ustring.match). You can think of this module working like mw.text.split when it could be much optimized to instead be designed like mw.text.gsplit instead. If you are considering redesigning this, something to consider might be an object hierarchy, e.g., one could design an factory interface that generates parser objects over a specific piece of wikitext and it could in turn have an interface that returns a templates object. The returned templates object could then have its own iterator factor functions or be used with ipairs (by way of a metatable that defines __ipairs). And the templates object might have a object for specific templates by name which in turn might have parameters object, etc. I am not sure all the object overhead is really that interesting but maybe with the right design it might be good. I do like the idea of a parser object over a piece of wikitext. Then one does not need to consider passing such into everything and can use Lua object notation (i.e., "colon" function calls) or perhaps callable tables or something. —Uzume (talk) 05:38, 24 June 2026 (UTC)

As for creating some more developed examples for you, I shall attempt to get back to you later. Incidentally, I got here in a rather round about way—I was looking at another Scribunto module that was doing some somewhat naive wikitext parsing of its own and went looking for better solutions. I might work on s:Module:AuxTOCDetect, attempting to create a simplified version of the iterator design I am thinking of there and direct you there for examples. I am sure you know by now that a true wikitext parser is extremely non-trivial and even the code here is not really that robust. For example, even parser.getTemplateName will fail to find the first invocation of template name if name is not directly available in the provided wikitext but instead generated by another template or parser function, etc. because parser.getTemplateName and parser.getTemplateName do not expand such via frame:preprocess, etc. to search for such as a real parser like parsoid does. —Uzume (talk) 03:55, 24 June 2026 (UTC)

@Sophivorus: Incidentally, you might find s:Module:AuxTOCDetect of interest in that it is doing something I do not think your other code Module:Excerpt or mw:Module:Transcluder does or did. Like those it attempts to copy some wikitext source from one page to another. In the case of this module it is attempting to copy the first template invocation of a specific template (which has a few aliases via redirects) from one page to another, however, it also attempts to fix up wikitext links from relative subpage links to non-relative links (because the current page clearly lives at a different place so would not have the same subpages as the source page). I am not sure I have seen any of your earlier code that attempts to do such. —Uzume (talk) 04:32, 24 June 2026 (UTC)

Thanks for your detailed reply. I've read it all but I'll have to re-read it a few times and think about it before I can respond or act upon it, so it might take a little while. Cheers! Sophivorus (talk) 13:30, 24 June 2026 (UTC)

Optimizations

@Sophivorus: If you are interested in other performance optimizations, I notice this code is using mw.text.split and mw.text.gsplit in non-optimal ways. First, since every use of these functions uses a constant pattern with no actual Lua pattern matching characters, they should be converted to always specify the plain argument as true (the code only does this in one spot on line 366). Next, the only use of mw.text.split never uses anything but the first field which is immediately passed to mw.text.trim and the rest of the fields are pasted back together with table.concat. This is incredibly wasteful. This would be better handled by code like: name, value = string.match(param, '[^=]*'), string.match(param, '=(.*)') and then fixing up things if value is nil. FYI: I would be wary of turning name into a number just because tonumber succeeds, e.g., that conversion will go awry with constructs like |2e0=value, |.1=value, etc. Finally, because MediaWiki wikitext markup is all ASCII but supports UTF-8 Unicode, it would probably be considerably faster to write your own split function along the lines of the what is noted at: mw.text.gsplit (notice the 60× slower comment). —Uzume (talk) 08:11, 24 June 2026 (UTC)

Hi again! I just modified the module to use the plain argument when possible. As to using string.match instead of mw.text.string, I see your point and I'll get to it asap. Regarding tonumber though, I think constructs like |2e0=value or |.1=value are so unlikely that we shouldn't plan for them. Note taken though, thanks. Sophivorus (talk) 13:43, 24 June 2026 (UTC)

@Sophivorus: Well perhaps this will inspire you: Special:PermanentLink/1361011032#L-42--L-48. Notice this also applies to |01=value which MediaWiki will turn into a string parameter not a numeric one. —Uzume (talk) 16:24, 24 June 2026 (UTC)

@Sophivorus: I decided to implement some performance optimizations. FYI: I also integrated such with the changes you have in the sandbox that are related to functionality imported from Module:Plain text. —Uzume (talk) 16:01, 11 July 2026 (UTC)

@Uzume Hi, thanks for the effort! I reverted the experiment with getSection and updated the testcases to use the sandbox rather than the production module and all test cases seem to pass. However, before deploying, I'd like to say that I generally agree with Donald Knuth's saying that "premature optimization is the root of all evil". I value code simplicity, clarity, brevity and beauty much more than I value performance, because I think machines should work for us, not the other way round, and because I believe simple, clear, brief and beautiful code is much more likely to attract new developers (like yourself) and contributions, and that's much more valuable than any microseconds gained. With that in mind, I can understand the need for str_gsplit, str_gsplit2 and even str_trim per the 60× comment (though an operation that is 60 times slower than 0.001 secs is still very fast), but regarding str_find, str_sub, str_match, etc. can we do without them? Kind regards, Sophivorus (talk) 13:13, 13 July 2026 (UTC)

@Sophivorus: Well, you are welcome to remove those as "micro-optimizations", however, they do add up (when used in loops repeatedly all over, etc.) and in part serve to isolate external function usage which can be clustered at a single location near the top. That can actually improve documentation and understanding since there is not random global usage scattered all over. If you wanted better naming, I would be all for that but I would prefer if such things were kept. Consider them as local functions that just happen to have the exact same functionality as the global functions. If you later want to "patch" one of them it is a simple matter to rewrite that interface, possibly defined in terms of the original global and in fact those assignments could be considered as a simplification of things like: local function str_sub(...) return string.sub(...) end. —Uzume (talk) 14:34, 13 July 2026 (UTC)

While localising global methods like string.find is pretty much always faster, it's normally at a scale you'd struggle to measure unless your usage is incessant, and having to go out of your way to use fully-named functions as opposed to being able to namecall a string (some_str:find(...)) can be annoying for readability. Are there any major uses of this module that I could try benchmark in practice? I've got a sandbox module designed for exactly that, and it might be interesting to see some numbers on that (and it might help point out any functions specifically worth optimising). Aidan9382 _(talk) 16:04, 13 July 2026 (UTC)

@Sophivorus: If you really wanted to keep the same global-like references you could remove those micro-optimizations and do something like: local string = require('string') or even local string = string. Then each reference would look identical the traditional global reference and thus still need a table lookup but at the same time this would totally avoid a global lookup in each case providing some micro-optimization without compromising the naming which you seem to be opposed to. Those types of references are akin to the ones @Aidan9382 mentioned where one uses the colon operator like: some_str:find(...) as both would require a table lookup but without the global access. In the latter case the overhead is slightly lower as going through the __index metatable access is highly optimized. I did not measure these myself but Google's AI (if we can trust such) tells me relative times are along these lines:

Comparison of referencing methods

Rank	Method	Relative Time	VM Lookup Mechanism
1	#'abc'	1.0×	Zero lookups (LEN Direct internal opcode with no CALL overhead)
2	local strlen = string.len strlen('abc')	~3.5×	Zero lookups (GETLOCAL Direct register read)
3	('abc'):len()	~5.5×	1 Metatable lookup (SELF)
4	local string = string string.len('abc')	~7.5×	1 Table lookup (GETLOCAL + GETTABLE)
5	string.len('abc')	~11.0–13.0×	2 Table lookups (GETGLOBAL + GETTABLE)

Of course those explicitly ignore the one-time cost of caching in a local where applicable. Obviously we cannot always use method #1 as it is rare to have a direct operator for the functionality we want but it gives a useful baseline. You seem to be opposed to the renaming of method #2 so I won't say more about that. Method #3 is more than twice as fast as the general global usage. Method #4 requires no changes to references at least for functions in tables directly available as a global (e.g., string, table, etc.); it requires either some table reference renaming (e.g., local ustring = mw.ustring or local mw_ustring = mw.ustring, etc.) or just caching the top-level global only (e.g., local mw = mw) and of course if only that is done, reference cost is higher as multiple table lookups are needed. And #5 of course is the ultimate do nothing to optimize. I can easily implement method #3 to replace the objectionable method #2 references to string functions if you would prefer. I hope this overview helps. —Uzume (talk) 14:42, 15 July 2026 (UTC)

Since you mentioned that AI got the numbers on those performance tests, I thought I'd double check them, since the SELF call being as fast as it claimed didn't seem right to me intuitively. My version of those numbers looks more like the following (tested locally - this could vary depending on exact lua 5.1 build details, as this is such a low-level measurement):

• The localised version is actually ~5x in comparison to the raw operator
• localising the global string table gets you to ~6.5x
• doing a SELF call gets you ~7.5x
• The standard called form with no special optimisations gets you ~8.5x

So specifically comparing the local function version to the completely unoptimised version (since the operator one is a bit of a special case), the difference is about 1.7x, which sounds good, but this is being measured at the scale of a 2e7 loop, and the time taken is around 0.5-1s, so ~18 nanoseconds better per call. Unless this is calling string functions on the scale of at least thousands of times per use, I think you'd struggle to measure an in-practice performance difference. Aidan9382 _(talk) 15:34, 15 July 2026 (UTC)

@Aidan9382: Thank you for the empirical evidence! I asked the AI where it got is numbers from and it said a combination of benchmarks and data about the instructions used in the virtual machine. That said, I believe even the benchmark data was rather dated. On a contemporary machine and CPU the magnitudes of the pieces of these things can vary widely from earlier hardware implementations causing the numbers to skew much from dated data (e.g., a much better hardware cache architecture can cause these to flip around, etc.). If I am reading your report correctly that means some items did flip their positions. It sounds like method #3 from the table moved into a slower spot than method #4, however they are all closer together now so the differences start to become moot. It sounds like the global reference is still the largest factor though so something simplistic like methods #4 (very easy to implement and can be done in such a way as to not impact later code) and #3 (already a standard practice) sound like good options. And I agree these are all micro-optimizations to begin with that do very little unless used at very large scales. I prefer to isolate global references and tend to do such things as a matter of course but when I received the push back I thought a minor investigation was merited. Thanks, again, —Uzume (talk) 16:41, 15 July 2026 (UTC)

No problem. I will say that I actually do quite like the sound of solution #4 from a simplicity standpoint. It wouldn't be worth making it common practice, but in modules that are string-function heavy like this, I don't mind it much (infact, I did steal it during testing and tried it on a different module I maintain which uses string functions even more heavily than this module, and the difference was non-negligible, though only just). Aidan9382 _(talk) 17:20, 15 July 2026 (UTC)