As @Mat said it, you can use the direct translation "fonction d'ordre supérieur". There is another term that is commonly used, that is to say : "une fonctionnelle" which is the mathematical term for function which takes other functions in parameters.
Also, but very rare, the term "forme" is eventually used.
For higher order functions returning functions, there is the term "fonctions curryfiées", derived from the "curryfication". It comes from the Lambda-Calculus, where you can have functions taking multiple arguments, only use some of them and returning a function which will processed the other parameters left untouched. The concept exists in programming too.