The W in 'two' and 'sword' is silent because of a sound change that took place somewhere between Old & Middle English. The change applied to words in which the W was preceded by [s, t] and followed by a back vowel [ɒ ɔ o ɑ u] etc.
'Swore' and 'sworn' also lost their W's at one point, but were later on restored by analogy with swear.
We all know that English spelling is not congruous with English pronunciation. There are lots and lots of anomalies and idiosyncrasies in English. There's a sentence that illustrates one of the peculiarities of English: English can be learnt through tough thorough thought, though.
There are lots of sound changes and shifts that took place in Old English and Middle English. Most of which are responsible for the marked idiosyncrasies of English, for instance, 'colonel' is pronounced [ˈkɜː.nl̩], 'lieutenant' is pronounced with an F in British English, 'cycle' and 'bicycle' don't rhyme, 'south' and 'southern' don't rhyme, 'iron' is pronounced [ˈaɪən] and so forth.
The silent W in 'sword' and 'two'
Between Old English and Middle English, there was a sound change through which a /w/ was lost in the environment of a preceding [s] or [t] and a following back vowel ([ɒ ɔ o ɑ u] etc). So 'two' was twā and it was pronounced /twɑː/ in Early Middle English, it became /twoː/ and then it lost the /w/ and became /toː/ (the vowel /oː/ was shifted to /uː/ because of the Great Vowel Shift).
'Sword' was pronounced /sword/, it became /sord/ (or /sɔrd/). The conjunction/adverb 'so' used to be pronounced with a /w/, but it lost its /w/ due to the same reason. The change also affected 'who' as it was hwa before.
However, the change didn't occur when the /w/ was followed by a front vowel; for instance, 'twin' and 'swift' didn't lose their /w/'s because they were followed by a front vowel /ɪ/.
Now 'swore' and 'swollen' deviate from this rule. They do have back vowels after the /w/ and there is an /s/ before the /w/, yet the /w/ is preserved. Why? Now 'analogical change' comes into play.
Before explaining 'swore', I'd like to elucidate 'analogy' (or analogical change). It's a type of language change in which some forms are deliberately changed merely to make them look more like other forms (Trask). Larry Trask, a phenomenal linguist, gives a remarkable insight into 'analogy' in his book Trask's Historical Linguistics. I'll just explain it the way Trask has explained with another example.
Suppose I tell you that frumicate is a rare and obscure English verb meaning to put on airs. I'm fairly certain you've never heard it before (because you haven't, have you?). What do you suppose its past tense is? You've never heard its past tense before, but I'm pretty sure you'll say frumicated (agree?). How do you know its past tense is frumicated?
You do it by invoking analogy—that is, you assume that the required past tense is formed according to a very common pattern that is already familiar to you from large numbers of other English verbs.
In this case, the pattern for forming past tense is so widespread and regular that it actually constitutes a rule of English grammar for regular verbs.
Trask gives another example:
The past tense of 'catch' was formerly catched, but as a result of this analogy, it has become caught (which is now a standard past tense of 'catch').
Trask further says that analogy does not always operate on such a large scale, but often speakers create forms by invoking an analogy with a much smaller number of existing forms, perhaps only a dozen or two, perhaps even only a single form.
The /w/ in 'swore' and analogy with 'swear'
The same change (loss of /w/) should've happened in 'swore' and 'swollen', but these are nonetheless pronounced with a /w/ today.
It's possible though that the analogy of these forms—which always retained their /w/s because they had front vowels—prevented the regular sound change from affecting 'swore' and 'swollen'.
Or they did actually lose their /w/s at one point, but the /w/s were later restored by the analogy with swear and swell. [Analogy and levelling — Trask's Historical Linguistics pp101-102]
Wikipedia says the same: /w/ in swore is due to analogy with swear.
Another Wikipedia article says that an example of 'analogical maintenance' would the perseverance of /w/ in swollen by analogy with the present tense swell (contrast with sword, where the /w/ is lost by regular sound change.
The change didn't apply to swear and swell because the /w/ in those words was followed by a front vowel and the sound change didn't affect words in which the /w/ was followed by front vowels.