Before getting on with my answer, I'll mention that I can't think of an instance for which a past participle would need to agree with a COI. So unless I'm missing something, that part of the statement in the question is wrong.
Nowadays, the past participle of a pronominal verb, even though it's quite generally using an être auxilary, would agree in gender and number with the COD, when the said COD comes before the verb (i.e. more or less the rule we follow for non-pronominal verbs using an avoir auxilary, though a case by case study MIGHT reveal some exceptions (I don't know...)).
Le bon usage, however, states that this rule is recent and not very well respected (one way of rewording this might be to say it hasn't established itself very well). To illustrate this point, the reference book proposes several examples of the historic rule of agreement with the subject by recent renowned authors:
Elle s'est lavée les mains. – Raymond Queneau
Elle s'est mise tout le monde à dos. – François Mauriac
(lavé and mis would have been expected from the rule)
Also, it is mentioned that the invariability, though less commonly, is a another way to break the rule, rather than opting for the former usage of agreeing with the subject.
For the sentences proposed in the question:
- Les vacances qu'il s'était octroyées. → What is being “octroyé”? Les vacances, feminine and plural, located before the verb, so agreement and therefore “octroyées”.
- Ils se sont octroyé une semaine de repos. → What is being “octroyé”? une semaine de repos, feminine and singular, but located after the verb. No agreement: “octroyé”.
Also, the tense being used is irrelevent: either past, present or future will behave the same. This is not specifically part of the original question post, but still part of its title as of the time of this answer.