The button you refer to is actually the Phone or Contacts app. Simply access the application list and then long tap on the green button/text and drop it without raising the finger to the first screen where it belongs.
You shouldn't worry, because the app can't be deleted. You've just removed it from the main screen, but it's in the list with all the Applications.
