The preposition used should be from, not of.
To make this clearer, compare the following sentences with the preposition moved to the end:
✘ The background they come of.
✔ The background they come from.
Alternatively, you can use different wording:
✘ They come of a certain background.
✔ They come from a certain background.
Once the correct preposition is identified, you can refer back to the original wording:
✘ The background of which they come.
✔ The background from which they come.
Note that I don't think of is strictly ungrammatical in every case. It's a preposition I've seen used in this way in some rare pieces of writing—and in a specifically different construction than I used here. However, it's very uncommon and wouldn't normally be considered idiomatic. (And at a certain point, something becomes effectively ungrammatical simply because it's so unidiomatic.) Unless you are certain of its use in the right context, it's best to stick to the far more common from.