The sentence:
Qu'est ce que c'est ?
Means "what is this?" (literally: "what is this that it is")
In older times, a simpler sentence would have been used:
Qu'est-ce ?
but it is now written/formal French.
The pronoun ça (a shortened form of cela) means this/that.
Here is is appended by the speaker to indicate what the question is referring to, usually by pointing the object.
It should be separated by a comma from the rest of the sentence:
Qu'est-ce que c'est, ça ?
or by another que:
Qu'est-ce que c'est que ça ? often colloquially pronounced /kɛseksa/
The next sentence is incorrect:
Qu'est ce que ça ?
Colloquial French can be:
C'est quoi, ça ?
Slang might be:
Quésaco / Kézaco ? (from Occitan Qu'es aquò)