I'm allergic to ice cream...?

A while back I noticed that when I eat ice cream my lips become swollen. I thought it was just that brand, but I tried another brand and the same thing happened. Maybe it's something in the ice cream? I'm not lactose intolerant. I was never allergic to ice cream in my life and I guess it's a sudden outbreak of an allergy. BUT, I just can't GIVE IT UP. What can happen if I continue to eat it? Usually all that happens IS the swollen lips. So is that the worse case scenario since any other symptoms haven't really developed?