1. Why is en Europe considered wrong?
You can write, answering to the question "What is the Portugal ?":
Le Portugal est un petit pays.
Être means "has the characteristic of being".
and, answering to the question "Where is the Portugal?":
Le Portugal est en Europe.
Être means "is located".
However,
Le Portugal est un petit pays en Europe.
despite being grammatically correct is unidiomatic and would be parsed somewhat as “Portugal is a small country when in Europe”. That might be understood to mean "Portugal is possibly a larger country elsewhere" which doesn't make sense. Countries are not known for traveling.
We do not juxtapose both complements, moreover without any conjunction, especially because être hasn't the same meaning.
Duolingo is right when considering the answer :
Le Portugal est un petit pays d'Europe
to be the right one. This is indeed the usual construction to refer to a country that is part of a continent or a larger group of countries.
Alternatives can make clear the complements are disjunct:
Le Portugal est un petit pays et il est en Europe.
Le Portugal est en Europe et c'est un petit pays.
or correctly link its location to the country:
Le Portugal est un petit pays européen.
Le Portugal est un petit pays situé en Europe.
Finally, un petit pays en Europe would be fine in this sentence:
Le Portugal est considéré comme un petit pays en Europe.
but the meaning is different:
Outside Europe, people might consider differently the Portugal size.
2. How can I know when to use de and when to use en when referring to countries or continents?
De is a preposition that mark a belonging relationship, similar to "of", e.g.:
Les États-Unis d'Amérique not Les États-Unis en Amérique
The United States of America not The United States in America
En is a preposition that mark a location like "in".
My life in America not My life of America
Ma vie en Amérique not Ma vie d'Amérique
PS: Whatever its size, Portugal is a great country :-)