Quicktime doesn't keep a library of the files opened, but it has a history with the files you have opened. To clear it simply go to File > Open Recent > Clear Menu and Quit the application. Next, open a MP4 or other file and the history should look empty (the only file is the one you've opened recently). This is the only way to clear the "database" with clips from the Quicktime Player.
