The reason one of these sounds awkward and one does not has to do with the what "almost" is doing in each sentence.
(I am not talking about formal grammar rules, I don't know those very well.)
Let me use a slightly simplified example. All of these are normal-sounding:
I almost never talk to my brother.
I almost got the job.
I almost didn't get the job.
The first one is talking about a kind of event that is expected to happen on a regular basis. The second two are talking about a singular event that either happened or didn't.
What a native person is hearing in the first sentence is "almost" as a modification of "never". The meaning is: it's not quite never, but it's almost never.
In the sentences about the job, the native speaker is hearing "almost" as a modification of "got the job" or "didn't get the job". That entire event "almost happened" or "almost didn't happen". I don't know what the grammar rules are here but in our brain it's "almost->{never} talk to my brother" in the first sentence, but in the other sentences it's like "almost->{didn't get the job}".
So--comparing this to the awkward "I almost don't talk to my brother"--the reason it sounds awkward is because in our heads it's like "I almost->{don't talk to my brother}." That's not what you mean, what you are trying to express would be "I almost->{don't} talk to my brother," but we aren't going to hear it that way.
I'm not sure exactly why that is, but here's a guess (and maybe I can come up with a sort of rule about it). If you say "I don't talk to my brother," (without a modification like "often" or "regularly"), the implication is usually that you've had some kind of argument and you are no longer on speaking terms. So it's just idiomatically strange to say "I almost->{don't talk to my brother}."
Because of what a standalone "I don't talk to my brother" means, this ends up mapping in our heads to something like "I've almost come to the point where I have decided to stop speaking to my brother ever again," which is not what you mean, but it kind of goes that direction first before we parse it out and guess that you mean that you do talk, but not very often. (We might also wonder if you really did mean "I've almost stopped talking to my brother" but you weren't sure how to phrase it.)
As I referenced above, there is no such problem with
I don't talk to my brother often.
I don't talk to my brother very much, if at all.
I don't talk to my brother regularly.
Once it's clear that you are talking about the frequency of the event, it's very normal that it might be more or less frequent and the implication that you have stopped speaking altogether doesn't come up in our heads at all.
The same thing would happen with "I don't drive." It's a statement that you would use if you don't know how to, or don't have your license or you are no longer able to. You wouldn't say "I almost don't drive," you would say "I almost never drive," or "I very rarely drive."
I don't know if there is a general rule. "I don't golf," "I don't drink [meaning I don't consume alcohol]", "I don't listen to the radio." All of those would be awkward with "almost", because they have that binary, on/off quality.
There are some "I don't X" sentences where 'almost' does work:
I don't care.
I don't want to go there.
I don't think he likes me.
I'm not sure how to formally distinguish these situations. It has something to do with whether the thing you're talking about has that binary quality I mentioned earlier. You would not say "I almost golf," "I almost drive," "I almost listen to the radio," but you could say "I almost care," "I almost want to go there," "I almost think he likes me."
That test (replacing "don't" with "almost", and seeing if it makes sense) would have worked with "talk to my brother" because "I almost talk to my brother" doesn't really mean anything.
So if you were looking for a rule where "almost" can come before "don't", maybe see if "almost" works as a replacement for "don't."