Yes, that person will be able to send you text messages or call you on Viber. By deleting his contact information you affect only your contact list. If you do not wish to receive any message or calls from that person you have to block him in your address list.