Lately I've been trying to develop a layout engine for my own purposes, from scratch. Similar to CSS layout systems like flex, my layout system works on the basis of a tree of rectangles, with some constraints between them. For example, you might have a parent rectangle with a height of 100 pixels and a width of "fit content", and then inside that, two child rectangles with widths of 300 pixels each and heights of "fill parent". The point of the algorithm is then to resolve these constraints to calculate a width, height, x and y for each rectangle.
This is more complicated than it seems, because sometimes you can have circular or ambiguous dependencies. For instance, say a parent container has a width of "fit content", and the child has a width of "fill parent". Then in some sense, any width is valid for the parent, so long as we also assign it to the child.
What I'm really looking for here is some kind of mathematical model of layout algorithms that can help me clarify my thoughts on this problem. Of course I can come up with ad hoc solutions, but is this an actual field of research? Has anyone out there written a PhD thesis about mathematical models of layout algorithms?