Nextjs Link component reload 404 error when using as property

My nextjs Link works fine when the first load but it returns 404 error when it comes to reloading the page. Here is the code.

<Link
     href={link.href}
     as={link.href.replace('/blog', '')}
     passHref
                                        
>
{link.text}
</Link>

I don't want to show blog in the URL so I removed it bt adding as property. But when I try to reload, it doesn't work. I believe nextjs is looking for as path that does not contain blog so the content does not exist.

Is there anyway I can navigate nextjs to reload the original path, not as replaced path?