It means "What's up?", "What's happening?". Translated literally, it means "what is there?": "il y a" means "there is" and you turn it into a question. In this context, the friend is asking what the other called about, stating that he just left work (explaining why he couldn't answer the phone).
Note that the meaning can change a little depending on the context. For example if you see someone looking upset and you ask "Qu'est-ce qu'il y a ?", you're asking if there is something making them upset (so it would be closer to "What's wrong?").
It's not rude, but I think it's a little bit informal. If you were to message back your CEO after you had missed her call, you probably wouldn't start your message with "Qu'est-ce qu'il y a ?" but rather something a bit more formal and explicit about what you're asking ("Pourquoi m'avez vous appelé ?" or something like that).