This is the distinction I would make (caveat: I am Belgian, I would not be surprised there would be other opinions in other French-speaking areas, esp. in Québec):
Un petit malin is someone smart, who uses their skills for finding subtle, practical solutions to tricky issues (and often with some mischief)
Un petit malin a réussi à entrer par derrière.
C'est un petit malin, il a trouvé comment payer moins cher.
Un gros malin is actually not very smart, even if they think they are; gros "negates" the quality of malin. Typically, someone who does more harm than good when trying to play it smart.
Ce gros malin a réussi à tomber en panne d'essence.
Mais tu te trompes complètement, gros malin !
So neither of the two phrases are really used the sense of "know-all"; you may want to use then eg monsieur/madame je-sais-tout (a person that knows it all and likes to show it).