Viber depends on contact list to establish friends and get in contact with users who also use Viber. To remove a contact from Viber, access your contact list outside Viber and delete the contact directly then scan for updates in Viber and that's it. Everything should be removed.