The first thing you have to do is to launch the application with Administrative permissions. Secondly, check the installed codecs. In most of the cases, XviD should be more than enough to have a proper encoder. This is needed to configure recording parameters, quality, etc.
You can download XviD from the following URL: https://www.xvid.com/download/
After you have met these requirements, I recommend checking the recording settings as well. Choose the Xvid codec if you can, and then set the correct quality, bitrate, resolution, etc.