The RKS Fax doesn't work in Windows 10 because it's not compatible with this operating system. According to the system requirements, RKS Fax is compatible only with Microsoft Windows XP and Microsoft Windows Vista. There is nowhere specified about the compatibility with Windows 10. If you have the setup package, right click it and choose Properties > Compatibility. From the drop down menu, select Windows XP or Windows Vista then tick the Run as Administrator option.
These are the only solutions possible since the developer of the application is out of business.