I believe there's a setting in Gmail to automatically enter contacts who have written to you. You have to uncheck that box.