You have to configure your Windows Live Mail to reject the old messages when new ones are delivered to your inbox. Usually, if you use a POP3/IMAP account, this needs to be set up in the web interface in the POP3/IMAP settings. You need to log-in to the email service and configure the IMAP options to delete the emails once you received them in the client, or at least archive them.
I know this works for Gmail since I somehow encountered the same issue as well.