In the sentence you're asking about, I would choose the phrasing "I called my mom to check on her." It doesn't imply that I necessarily believe something specific is wrong with her, or that she is in a vulnerable state, just that the purpose of the call is to find out how she is doing and talk about things she's been doing lately and such.
"Check in on" to me would not usually be used for a phone call, more likely for an in-person visit. It's not a very firm distinction so I wouldn't be confused or surprised by someone using them interchangeably. Note that while "call" or "call on" can be used to mean an in-person visit, that usage is very rare in modern American English compared to a phone call.
"Check in with" is somewhat of the opposite, "checking in" is when you tell someone about your condition and the person you check in "with" is who you are telling. "I called my mom to check in with her" would most likely involve telling her about how you are doing, though you might expect her to respond in kind.
"Ask after" is a phrase I have never heard, and if someone had ever said to me "I'm calling to ask after your mother" I wouldn't have had the slightest idea what that meant before reading this question. The closest phrasing I can think of would be "ask about", which is quite a bit broader but could be used on its own in context. For example for "I called my sister to ask about our mom" that would probably mean something similar to "ask after", but for "I called my sister to ask about Batman" I would expect answers like "The newest Batman movie is terrible" or "Batman's birthday is April 7" rather than information about Batman's current health or recent activities (unless my sister had a cat named Batman).
This information might be regional, so to be specific this answer reflects my personal experience living largely in the "Midwest" part of the United States. It's possible that "ask after" or "check in on" would be more common or preferred in other parts of the country.