sveltekit routes not working when deployed to server

Here's my svelte.config.js and I'm using adapter-static :

const config = {
kit: {
    adapter: adapter({
        // default options are shown
        pages: '../backend/build',
        assets: '../backend/build',
        fallback: null,
        precompress: false,
    }),
    alias: {},
    appDir: '_app',
    browser: {
        hydrate: true,
        router: true,
    },
    files: {
        assets: 'static',
        hooks: 'src/hooks',
        lib: 'src/lib',
        params: 'src/params',
        routes: 'src/routes',
        serviceWorker: 'src/service-worker',
        template: 'src/app.html',
    },
    floc: false,
    methodOverride: {
        parameter: '_method',
        allowed: [],
    },

    paths: {
        assets: '',
        base: '',
    },
    trailingSlash: 'always',

    vite: {
        server: {
            proxy: {
                '/api': 'http://localhost:5555',
            },
        },
    },
},

preprocess: null,};

From the backend (Go lang) I'm serving build directory & index.html file. The homepage works fine but whenever I click on any route, it sends get request to the server instead of redirecting in the app itself.

Here's the go code to serve from backend:

router := gin.Default()

router.StaticFile("/", "./build/index.html")
router.StaticFS("/_app", http.Dir("build/_app"))

I have also tried with the following code:

router.NoRoute(func(c *gin.Context) {
    path := c.Request.URL.Path
    c.File("./build/index.html")
})

Note: Things work fine when I run npm run preview.