Saying "most everyone" is much more popular in books than "mostly everyone".
To compute the distance between two coordinates most everyone/mostly everyone uses the Spherical Law of Cosines equation.
Which expression should I use?
You've left out the most important alternative: almost. Here's an expanded version of your Google Ngram: In most everyone, most is a contracted version of almost, an adverb modifying the every component of everyone. (Yes, I know everyone is written as one word, but syntactically it's apprehended as a 'pronominal' version of every. Any(one) works the same way.)
Most everyone and mostly everyone are colloquial variants; I advise you to avoid them in formal registers.
Most everyone is better than mostly, because everyone is a pronoun, which the quantifier most can modify. I should note, however, that a by far better option is almost, because most is a colloquial variant.
A pronoun works like a noun. See how the quantifier mostly modifies a noun:
Most desserts are sweet.
Mostly is not a quantifier but an adverb. An adverb usually modifies a verb, an adverb, or an adjective, so it would not modify the pronoun everyone (an adverb can modify a noun phrase, but it's very hard for an adverb to modify a pronoun).
An example from Cambridge Dictionaries:
We mostly stayed on the beach. (We stayed on the beach for the majority of the time.)
See, the adverb mostly modifies the verb stayed.
In your example, it's hard to imagine what exactly would mostly modify. We'd need to move it closer to the verb uses.
To compute the distance between two coordinates everyone mostly uses the Spherical Law of Cosines equation.
P.S. Thanks to the commenters for most gallantly reminding me that most is unwelcome in most situations where informal English is a no-no.
[Just to let you know that the word most is also an adverb!]
To answer your question,
WordWebOnline has an answer to this:
(of actions or states) slightly short of or not quite accomplished; all but
So, when it's something of actions/states, 'most' is fine.
The example follows:
"most everyone agrees"