Viber uses your contacts to scan for people who use the software. If those numbers appear in Viber, then it's possible they've installed the app on their phones. Also, when you switch to all contacts, Viber displays both contacts from phone and the ones who use the application. It's something normal because this is the way it's built. To delete those numbers from Viber, simply long tap on them and choose the REMOVE option.