Does it mean, but that is not what I am trying to point out? That is only a trivia, what I am trying to say is something else.
"Beside the point" is an idiom meaning that what the person just said is irrelevant, out of context, or not important to the topic being discussed.
A: Those tacos have red shells. Yuck! B: That's beside the point. I ate the tacos earlier and they taste the same, even though their shells are a weird color.