In the following context (From Breakfast at Tiffany's page 87)
I rang California again; the circuits were busy, stayed busy, and by the time O. J. Berman was on the line I’d emptied so many martinis he had to tell me why I was phoning him: “About the kid, is it? I know already. I spoke already to Iggy Fitelstein. Iggy’s the best shingle in New York. I said Iggy you take care of it, send me the bill, only keep my name anonymous, see. Well, I owe the kid something. Not that I owe her anything, you want to come down to it. She’s crazy. A phony. But a real phony, you know? Anyway, they only got her in ten thousand bail.”
I have checked the meaning of 'come down to something' on Webster on-line dictionaryDefinition of come down to it goes like this:
to have (something) as the most important part
What the meaning of the bold sentence in the above text? How should we understand it? who does ‘you’ represent here, O.J. Berman or I?