The best version of this is
I decided to make the machine in a smaller size.
"Of" is simply not correct in this context, not matter how you torture the meaning. If we use the phrase "in a smaller size" we are suggesting that it this is a machine that could be made exactly the same in many sizes, similar to what is done with clothing. That seems reasonable, based on what you have written.
Reading the context, it's not clear to me that the smaller machine is actually identical to the original except for scale. In that case, you would be better saying
I decided to make a smaller machine.
I decided to make a smaller version of it.
I'm assuming here that there is previous context giving us an idea of the size of the original machine. If not, we need a more comprehensive rewrite of all of this. Consider something like this:
So I began the project of constructing my own helicopter. I found that parts for a machine of the size I originally wanted were hard to find, so I decided to make a smaller version of it.