If you have an international phone, Malayalam language might not be supported which means that you'll have to install a font. This way the system will use that font to display characters automatically. I recommend on installing Fontomizer from Play Store. Once the application is installed, use the built-in browser/search function and search for Malayalam. Install the font and reboot the phone, then access the SMS and WhatsApp apps and the characters should look normal in their native language.