@zkat my understanding (not a browser implementer so i could be wrong) is that all the optimization is in the "does this node match the selector string" bit and the actual querying is a straightforward traversal of nodes in document order, returning the ones which match
like you could maybe get a little faster by optimizing certain queries to not do a full traversal, keeping track of each node's position "in document order", and doing a sort at the end but i'm not sure it's worth it…
i would expect the more expensive operation in most documents is probably doing the "does this node match one of these selectors" boolean test