Angular app and API both behind NGINX reverse proxy

I have an Angular 4 app using RouterModule. It is served by one server and it makes API calls to another server on different port, hosted on the same machine. To cope with the cross-origin issue I use a NGINX reverse proxies and everything works fine.

The problem is when I directly enter URL in the browser - the Angular app's router does not get activated and I get error 404.

Ex. if I paste http://my-app/dashboard the app is not loaded and I get 404.

Here is the Angular config:

const routes: Routes = [
    { path: '', redirectTo: 'home', pathMatch: 'full' },
    { path: 'home', component: HomeComponent},
    { path: 'dashboard', component: DashboardComponent}
]
const routing: ModuleWithProviders = RouterModule.forRoot(routes);

@NgModule({
    imports: [
        routing,
        ...
    ],
    declarations: [AppComponent],
    providers: [...],
    bootstrap: [AppComponent]
})
export class AppModule { }

I use the http-server to serve the application and another application server (Tomcat) to serve to API calls.

Servers:

Server Type | Port  | Purpose
---------------------------------
http-server | 8081  | Angular App
tomcat      | 8080  | API

NGINX config:

upstream my_api_server {
  server localhost:8080;
}

upstream my_angular_app {
  server localhost:8081;
}

server {
  listen 80 default_server;

  location /api {
    proxy_pass http://my_api_server/api;

    proxy_http_version 1.1;
    proxy_set_header Host               $host;
    proxy_set_header X-Real-IP          $remote_addr;
    proxy_set_header X-Forwarded-For    $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto  $scheme;
  }

  location / {
    proxy_pass http://my_angular_app;

    proxy_http_version 1.1;
    proxy_set_header Host               $host;
    proxy_set_header X-Real-IP          $remote_addr;
    proxy_set_header X-Forwarded-For    $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto  $scheme;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_redirect off;
  }
}

So how do I make NGINX accept the /dashboard as part of the Angular app?

Note: I know the problem can be easily bypassed by just using NGINX to serve the Angular app instead of http-server but this is not the point.