This is an extension of the onomatopoeic meaning of tick, as in the sound a clock makes: tick-tock. What makes someone or something tick is what makes them function, except this is generally even more metaphorical than just the tick being a metaphor. If you "know what makes Jeff tick", then you know how his mind works, how he will act, how to provoke him into doing what you want him to do, how to do things he will appreciate, and so on.
Essentially, it's a set phrase - "what makes ... tick". It refers to a deep and fundamental knowledge about how "..." works.
There are variations that go in different directions, of course. As fred2 noted in their answer, to "really make something tick" can be to make it work better, faster, more effectively. But to find out what "really makes someone/something tick" can be that, or can be a suggestion that a person has put on a false front and you're going to try to see "behind the mask", so to speak.
If something is "ticking over nicely", that's most directly a reference to an engine or other machine ticking over, which means running on idle - not doing anything in particular, but running. It's unclear exactly how we got that expression, whether it also comes from the noise of a clock, but I suspect so. Using that reference means something is functioning and continuing to function without any particular intervention. If it is "just about ticking over" it is barely continuing to function.