Fill in is used when you tell someone to enter something (one field) on a form.
Example:
Fill in your name here.
It could possibly be more than one field, but generally not the entire form.
Example:
Fill in your name here, and your phone number over here.
Fill in means to supply something that's missing. So you fill in the blanks on a test, for example, or you can fill in a triangle with a color (change it from an outline to a solid triangle by coloring inside it), or you can fill in (substitute) for an absent colleague at work. And you can fill in a form because you're supplying missing information.
Fill out is generally used when you tell someone to enter all the fields on a form of more than one field. Example:
Please fill out this form.
Fill out means to complete by supplying requested information.