JQuery :: :not Targeting A Child Element Not Working Correctly?

I'm trying to use a :not statement, so if you click anywhere else on the page BUT the element in the :not a div will disappear. When i use the same statement without the not, it works fine (click that one element and the div in the function hides).I'm assuming that the :not statement isn't working because the element is a child of its parent, so when you click it, essentially you are still clicking the parent which isn't included in the :not.

j(':not(.arrow)').live('click', function(e) {
j('.label').hide();
});