Ordo, I agree with your interpretation.
Can 'fair enough' be used in the literal sense?
Yes, and I've only heard it used literally. The comment to the question by @JasonBassford is still a literal usage. It would be possible to say "fair enough" when it clearly was not at all fair, but that is still a literal usage (it's literally a joke).
The expression "fair enough" means that it is "sufficiently fair", adequate, OK. Not "the best".
Example: Two people discuss something and after each has made one or more points to advance their position one of the two will say: "fair enough" - that means that the extent of the argument and counter argument have reached a point where one or both feel that further debate is not worthwhile - it doesn't absolutely mean that there is a complete agreement.
Example where there is no absolute agreement, only partial agreement:
Joe: I need my roof fixed for $1000.
Tom: Your roof is dangerous and risks falling in, there's no way it will cost any less than several thousand dollars. You'll need to come up with a lot more money.
Joe: The most I can afford is $1100 dollars.
Tom: Fair enough.
Joe: When can you start repairs?
Tom: When you can afford several thousand dollars.
Tom agrees with Joe that he can only afford a little over a thousand dollars, it's probably not a point that Tom can argue; he likely doesn't know Joe's finances. That he agrees with Joe that he can afford no more doesn't mean that he will do it for less. It is a literal usage.
Much as the comment to the question might mean that the person was "fair enough" for one date, that doesn't imply that she is the fairest in all the land.