“[Il est] impossible [pour X] de ⟨infinitive clause⟩” is another way of saying “Il est impossible que X ⟨subjunctive clause⟩”. In both cases, X is the semantic subject of the subordinate clause. With de + ⟨infinitive clause⟩, the subordinate clause has a verb in the infinitive mood, which does not have an explicit subject, so it's possible for X to be fully implicit in the sentence. With que + ⟨subjunctive clause⟩, the subordinate clause has an explicit subject, so X has to be explicit in the sentence.
In the infinitive case, there is no explicit subject, but the verb and other parts of the sentence still needs to agree with the semantic subject X. In your example, the clause consists of the verb être in the infinitive present and a predicative adjective (attribut du sujet) sidéré. (I analyze it as an adjective because sidéré here is likely about the end result, and not the action of sideration. But if it was a past participle, the clause would be using the verb sidéré in the infinitive present in the passive voice, and this would lead to the same agreement with the subject.)
So, if the implicit subject X is singular masculine (or non-gender-committing), it's sidéré. If X is singular feminine, it's sidérée. If X is plural masculine (or mixed or non-gender-committing), it's sidérés. If X is plural feminine, it's sidérées. Modern non-gender-committing forms also include sidéré(e) (singular) and sidéré(e)s (plural).
Presumably the speaker includes non only himself (or herself) but also his/her audience or other people. The implicit X is nous: “Impossible [pour nous] de ne pas être sidérés”. Therefore sidérés has a plural agreement.