It means that human beings should not try to study God, because God is beyond human understanding. To presume is to exceed limits undesirably. The first definition for presume on the Merriam-Webster page is:
to undertake without leave or clear justification : DARE
"Scan" means survey or examine. Pope says it is an unjustified act of daring to try to understand or study God. Instead, it is "proper" that we study our fellow humans.
The Merriam-Webster page for proper has some pertinent meanings:
belonging to one : OWN
strictly limited to a specified thing, place, or idea
marked by suitability, rightness, or appropriateness : FIT
In other words, the most suitable thing for humans to study is other humans, rather than God. To seek to understand God is a presumptuous act. And to study other humans, we have to being with ourselves, hence: "Know then thyself."
Edit, based on a couple of comments:
@EddieKal points out that "without leave" in the definition of "presume" given above means "without permission". Thanks for pointing that out.
"Presume not" is what's called a negative imperative and is a poetic or archaic way of saying Do not presume. As a deviation from the expected word order, it is an example of anastrophe. Thanks @jack-oflaherty for mentioning this in a comment, since moved to chat.
There is at least an argument to be made that "Presume not God to scan" is deliberately ambiguous: Presume to scan that which is not God, i.e., Man. In this interpretation, the relevant meaning of "presume", also from Merriam-Webster, would be "to expect or assume especially with confidence": Expect that you will be studying humans, not God. This would be a secondary way to read the line, a meaning layered on top of the primary meaning, which is simply "Don't take the liberty of trying to understand God."
Hope this helps.