In AutoRipNCompress, please go to Settings > DVD Profile and click the Plus sign. Once the window opens, please tweak the settings.
The extender supports all kind of video file formats. I'm not sure that the Xbox can play MKV, but since it's streaming it from the PC, I believe it will work because the processing is done on the PC side.