I'm trying to say "I see her coming": Is it "Je la vois venir." or "Je la vois en venant."?
Complex structure sentences contain more than one subject.
There are several possibilities including:
Je la vois venir
which has a double meaning:
Je la vois qui vient
She is coming and I see her.
On the other hand :
Je la vois en venant
means "while coming in, I see her."
There is no way of saying "Je la vois venant." in any case.
You can't use in this construction the "participe présent" without an adverbial of place.
If you add "en' to the "participe présent" obtaining thus a "gérondif" ("en venant"), then you can do away with the averbial if you want to; however the subject is changed: the locutor is now the person doing the action of coming (not the person for whom the pronoun "la" is used). When you omit the adverbial the person you are talking to must know beforehand where you are coming from, otherwise what you are saying makes no sense.
There are two meanings for "I see her coming.".
1/ (literal) "Je la vois venir." is not proper, not idiomatic; you'd say, rather "Je la vois arriver." or "Je la vois qui vient/arrive.". There is no way of saying "Je la vois venant.".
2/ (figurative) "Je la vois venir." is correct and corresponds to the English figurative "I see her coming.", which can also be expressed as "I see what she is getting/driving at." (Harrap's dictionary). In this case also, there is no way of saying "Je la vois venant.".
Note: The second part of the question (What are the rules for complex sentence structures?) involves too much grammar for an answer to be possible and I'd avise you to edit you question and remove that part so as to keep a neat formulation; you'll have to keep to specific points when you ask questions.
@jlliagre and @Lph have already provided excellent answers. As a side remark, note that one can use Deepl for questions of this type:
I see her coming = Je la vois venir
Je la vois en venant. = I see her on the way in.