It can be very confusing to users when plugins ask users to upload files and insert lines of code. The plugin linked to in the question does contain an option to automatically load the MathJax without the need to insert anything into the header.
1. Go to the plugin options page by going to Settings --> MathJax-Latex Plugin in your WordPress admin panel.
2. Check the box that says Use MathJax CDN Service.
