You will need to go into the Login section of Keychain access and remove all of the URLs (If your CAC does not appear remove it from the reader and repeat)..