How to Get Filter record in ROW_NUMBER() function and Union?

I have one Union Query for getting result from table. When only first query execute that time this working fine but if in second(union part) also return result that time it will not work. My query like

Select * from (
Select  ROW_NUMBER() OVER(ORDER BY EmpID DESC) as RowNo,
    Emp.EMPID, EMP.FirstName
    From Emp
Union
Select ROW_NUMBER() OVER(ORDER BY EMPID DESC) as RowNo,
    Emp.EMPID, EMP.FirstName
    From Emp Inner Join EMPDetail On Emp.EmpID = EMPDetail.EMPID 
    Where EMPDetail.IsActive=True
    ) as _EmpTable where RowNo between 1 and 20

Please help me for this. I want to add paging using Row number. is there any other solution for this?