From Merriam-Webster, the word aroma is not defined as being only associated with food, although it has that as a common theme.
1 a : a distinctive, pervasive, and usually pleasant or savory smell • the aroma of freshly-baked bread ; broadly : ODOR
b : the odor of a wine imparted by the grapes from which it is made • The wine has a fruity aroma.
2 : a distinctive quality or atmosphere : FLAVOR • the sweet aroma of success
Note in particular the last sense where aroma is used metaphorically to describe success.
It goes on to mention the shades of meaning between the various synonyms:
SMELL, SCENT, ODOR, AROMA mean the quality that makes a thing perceptible to the olfactory sense. SMELL implies solely the sensation without suggestion of quality or character. • (an odd smell permeated the room) • SCENT applies to the characteristic smell given off by a substance, an animal, or a plant. • (the scent of lilacs) • ODOR may imply a stronger or more readily distinguished scent or it may be equivalent to smell. • (a cheese with a strong odor) • AROMA suggests a somewhat penetrating usually pleasant odor. • (the aroma of freshly ground coffee)
For its definition of perfume, it uses scent and odor, along with the generic smell:
1 : the scent of something sweet-smelling
2 : a substance that emits a pleasant odor; especially : a fluid preparation of natural essences (as from plants or animals) or synthetics and a fixative used for scenting
An analysis of perfume-related synonyms is also given:
FRAGRANCE, PERFUME, SCENT, REDOLANCE mean a sweet or pleasant odor. FRAGRANCE suggests the odors of flowers or other growing things. • (the fragrance of pine) • PERFUME may suggest a stronger or heavier odor. • (the perfume of lilacs) • SCENT is very close to perfume but of wider application because more neutral in connotation. • (scent-free soaps) • REDOLANCE implies a mixture of fragrant or pungent odors. • (the redolence of a forest after a rain)
Given all of that, I would say that aroma would not normally be associated with perfume more readily than one of the other synonyms. However, it could be used in such a way if, for instance, the scent were vanilla and it was being associated with the smell of that food ingredient:
This vanilla perfume has a pleasant aroma.
In other words, unless using a neutral word (such as scent or smell) pick the individual word that best matches the specific association.
Note, too, that while the other synonyms apply to the quality of the smell, odor applies to the quantity (or strength) of the smell. That's why perfumes have a strong odor to start (so that they are noticeable to others), and then are further defined by the nature of their scent.