I have read a lot about it, but couldnt really decide which way is the best. I have a web app and a java rest application which serves to customers. What is the best way to prevent xss attacks using parameters in rest api and frontend?
When it comes to XSS only possible choice is to validate user input, any kind of user input, whether it is passed from the browser or in any other way (like from terminal client).
It depends on the scenario you are following. If it is just data without HTML content then you don't need to worry about XSS. Otherwise, just removing <,> symbols or casting them into character encoded string would be enough. Also you can avoid using innerHTML to append new content to the document, use innerText instead and even if there are XSS content it won't execute.
But it gets little bit complicated when api response returns HTML content as well which you need to display somewhere. In such cases avoid directly displaying user input inside HTML snippet - try to character encode or remove <, > symbols and it will be just fine
As with anything defense in depth is important, so validation and encoding should be done on any user provided input. Encoding is very important because what might be considered malicious is contextual. For example, what might be safe HTML might be an SQL Injection attack.
Parameters in a REST API may be saved which means they are returned from subsequent requests or the results may be reflected back to the user in the request. This means that you can get both reflected and stored XSS attacks. You also need to be careful about DOM Based XSS attacks. A more modern categorization that addresses overlap between stored, reflected, and DOM XSS is Server XSS and Client XSS.
OWASP has a great Cross Site Scripting Prevention Cheat Sheet that details out how to prevent cross site scripting. I find the XSS Prevention Rules Summary and the Output Encoding Rules Summary sections to be very handy.
The big take away is that browsers parse data differently depending on the context, so it is very important that you don't just HTML Entity Encode the data everywhere. This means it is important to do two things:
Rule #0 - Only insert untrusted (user provided) data in allowed locations. Only insert data into an HTML document into a "slot" defined by Rules #1-5.
When you insert data into one of the trusted slots follow the encoding rules for that specific slot. Again the rules are detailed in the previously linked Cross Site Scripting Prevention Cheat Sheet.
There is also a DOM based XSS Prevention cheat sheet. Like the server side XSS cheat sheet, it provies a set of rules to prevent DOM based XSS.
