You are mixing up grammar and semantics.
There is the grammatical subject. This is the "noun phrase that goes before the verb". The grammatical subject in the passive sentence "The letter is being written by me" is the noun phrase "the letter".
There is the grammatical object. The is the noun phrase that follows the verb. In the active sentence "I am writing the letter", the noun phrase "the letter" is an object, and the subject is "I".
Then there is the semantic role. For example the "agent", "beneficiary" or "patient". An "agent" is the thing (or person) that carries out an action. A "patient" is (in semantics) the thing (or person) that is affected by an event or that undergoes an event. There are lots more semantic roles.
In an active sentence, the semantic role of "agent" is indicated by the grammatical "subject", and the semantic role of "patient" is usually the grammatical object.
In a passive sentence the "patient" is the grammatical subject, and the "agent" is indicated by the optional "by ..." prepositional phrase.