"Une femme" and "Un homme", but not "Deuce femmes" and "Deux hommes"?
Is there a reason that the number one receives special treatment?
It's not really unusual at all for a French word to not have distinct forms for masculine and feminine. It's not obvious whether this is best analyzed as "lacking gender inflection" or "having the same form for both genders" (or different ways in different contexts).
But in any case, many adjectives such as jaune, rouge and orange don't change between masculine and feminine (and in many accents of French, the distinction between bleu and bleue is purely graphical), while vert(e), gris(e), blanc(he) do change. We see the same form for masculine and feminine with the plural articles les, des and the plural "possessive adjectives" mes, tes, ses, nos, vos, leurs.
In general, it's fairly common for languages with grammatical gender to only show gender agreement for a subset of cardinal numbers (usually only lower ones), and this is the case for Latin, the ancestor of French, even though Latin had gender agreement on a few more numbers than French. In Latin, the words for "one", "two" and "three" agreed in gender with the noun, but higher numbers did not (except for, oddly, multiples of 100).
So if French were to show gender agreement on four, five, six, seven, etc. this would represent an innovation from the situation in Latin. While there are some places where French has added inflection relative to Latin, the general trend has been a reduction rather than an increase in the amount of inflection of this kind.
The Latin word for "three" had the same form for masculine and feminine (tres); the distinct form tria was used with neuter-gender nouns, but as French lost the neuter gender, it may not be too surprising that it also lost gender inflection for the number 3.
In Latin, 2 in the accusative case (the main source of Romance forms) had the masculine form duōs and the feminine form duās. The modern French spelling with "x" is just from a spelling change of "us" to "ux"; this isn't a word that historically had a "c" sound in either the masculine or feminine. The "x" just represents the plural suffix that is spelled as "-s" in other contexts, and which has become silent (outside of liaison contexts) in modern French for both masculine and feminine words.
I'm not entirely sure about the phonetic development of duōs to deux /dø/, and whether duōs and the Latin feminine form duās would both regularly correspond to modern French /dø/. Maybe in theory there could have been a feminine form "deues" or something like that, but I don't see how it could have been anything as different from deux as your hypothetical form "deuce".
"un"/"une" is also the equivalent for "a" in English, so the undetermined article, whereas there is no other term for two, three...