Your statement about visiting her twice when she was at home:
I have been to France twice when she was at home.
is not ungrammatical. There is no time-phrase there that excludes the present. "when she was at home" can be understood as a situation or condition. Maybe you visited several times when she herself was traveling and not at home.
When the utterance is about a discrete event confined to the past then the present perfect is ungrammatical.
Napoleon has been exiled to Elba in 1814. ungrammatical
When the utterance merely mentions some past years yet those past years are not meant to confine an event to the past but to construct a pattern that amounts to a set of "life experiences", there is present relevance and those mentioned past years do not confine the events to the past and do not exclude the present:
I have been to France a number of times: in 1994, 2001, 2008, and 2015. grammatical