I know a similar phrase which is "my circle of friends".
Thus, would this one apply as that phrase?
The sentence:
Who could that be? Although my circle of the people I know, is not that big.
Although my circle of the people I know, is not that big...
This is a little bit awkward because it says "my circle" and "I know" which is redundant. If it's your circle, you will know the people in it, right?
You could say:
Although the circle of people I know is not that big...
but it is still a little wordy. You might want to say,
Although my circle of acquaintances is not that big...
If you start your sentence with "although" I expect you to contrast your statement with something else after the comma. For example,
Although my circle of friends is small, they come from places all over the world.