Does the serious historian refer to the author?
By itself, not necessarily.
"The person" is an idiom used in English, where "a" or "any" would make more grammatical sense, to place emphasis on the adjective. For example, papers about mathematics will often say things like "The interested reader can verify this by induction" (a mathematical technique). Presumably, the authors of such papers expect more than one reader to be interested in doing so.
In this context, the audience for the book being reviewed would also consist of historians. One could argue that any serious historian who saw the book would suspect that the work is inferior, because of its formatting.
However, given the complete sentence, it does appear intended to refer to the author. This is because the primary "danger" described of choosing this book layout is "the scornful reception... reserve[d] for books aimed at the popular market". The author is the person who would be affected by such scorn, not the readers (who are the people who would create the scornful reception).
The phrasing here is still probably not intended to be specific: "serious historian" is probably still meant generally, rather than as an epithet for the author. However, it still obliquely and snidely refers to the author. The idea is: any historian who is actually serious should be troubled by this; the author chose this formatting anyway; thus, the author comes across as incompetent.
I am also confused about not the least of which.
As previously noted, this is an expression that makes use of litotes (i.e., understatement). You can treat "not the least of which being", more or less, as a fancy way to say "in particular".