Before anything, be careful that "monter" isn't an exact equivalent of "to walk up". Most of the time it means "to mount" and it can have a sexual meaning.
The rule is hard to grasp, but here are ways to help you get it right.
You can use avoir monté if the direct object is something that is being climbed upon. So the sentence:
X a monté Y.
Should have the same meaning at the passive voice:
Y est monté* par X. (you have to make the agreement with Y, so you'd write la fourmi est montée.)
For example:
J'ai monté un cheval. (I mounted a horse)
Un cheval a été monté par moi. (A horse was mounted by me)
But:
Je suis monté sur un cheval. (I climbed onto a horse)
Because you can't say:
Sur un cheval a été monté par moi. (Onto a horse was climbed by me?)
Likewise:
La fourmi est montée sur le poteau. (The ant climbed on the post)
or
La fourmi a monté le poteau.
This last one, depending on the context, could mean
"the ant mounted the post" (maybe it expected the post to start moving like a horse)
"the ant assembled the post" (it received a lego set of a post and built it up)
"the ant brought the post in a higher place" (it was needed upstairs for some reason)
"the ant crossed the post to move upward" (the ant path goes through climbing up this post)
Same goes for "la colline", "les cascades" and "les escaliers":
Elle est montée sur la colline. / Elle a monté la colline
Les saumons sont montés par les cascades. / Les saumons ont monté les cascades.
Il est monté par les escaliers. / Il a monté les escaliers.