Packages are missing after a successful dotnet publish

The publish command works fine but after it is run, I have to restore the project as there are missing packages...

So if I run the following command :

dotnet publish --runtime win-x64

Then the publish works but I have missing packages in my project right after.

BUT if I run publish without a runtime :

dotnet publish

Then the publish works fine and I don't have any missing package.

Is it a normal behaviour? What can I do to fix this? This is annoying.

Here is my csproj file :

<Project Sdk="Microsoft.NET.Sdk.Web">
<PropertyGroup>
    <TargetFramework>netcoreapp2.1</TargetFramework>
    <CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies>
    <LangVersion>7.1</LangVersion>
</PropertyGroup>
<ItemGroup>
    <PackageReference Include="AWSSDK.Extensions.NETCore.Setup">
        <Version>3.3.6</Version>
    </PackageReference>
    <PackageReference Include="Microsoft.AspNetCore.App">
    </PackageReference>
    <PackageReference Include="Microsoft.AspNetCore.StaticFiles">
        <Version>2.1.1</Version>
    </PackageReference>
    <PackageReference Include="NSwag.AspNetCore">
        <Version>12.0.13</Version>
    </PackageReference>
    <PackageReference Include="NSwag.MSBuild">
        <Version>12.0.13</Version>
    </PackageReference>
</ItemGroup>
<ItemGroup>
    <ProjectReference Include="..\MyProject.Analytics\MyProject.Analytics.csproj" />
    <ProjectReference Include="..\MyProject.ApiClient\MyProject.ApiClient.csproj" />
    <ProjectReference Include="..\MyProject.CommonApi\MyProject.CommonApi.csproj" />
    <ProjectReference Include="..\MyProject.Common\MyProject.Common.csproj" />
    <ProjectReference Include="..\MyProject.DbAccess\MyProject.DbAccess.csproj" />
    <ProjectReference Include="..\MyProject.Logging\MyProject.Logging.csproj" />
    <ProjectReference Include="..\MyProject.Messaging\MyProject.Messaging.csproj" />
    <ProjectReference Include="..\MyProject.Model\MyProject.Model.csproj" />
    <ProjectReference Include="..\MyProject.Settings\MyProject.Settings.csproj" />
</ItemGroup>
<ItemGroup>
    <Content Update="appsettings.Api.Development.json">
        <CopyToPublishDirectory>PreserveNewest</CopyToPublishDirectory>
        <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </Content>
    <Content Update="appsettings.Api.json">
        <CopyToPublishDirectory>PreserveNewest</CopyToPublishDirectory>
        <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </Content>
    <Content Update="appsettings.Api.Production.json">
        <CopyToPublishDirectory>PreserveNewest</CopyToPublishDirectory>
        <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </Content>
    <Content Include=".ebextensions\never-sleep-again.config">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </Content>
    <Content Update="appsettings.Api.PreProduction.json">
      <CopyToPublishDirectory>PreserveNewest</CopyToPublishDirectory>
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </Content>
</ItemGroup>
<ItemGroup>
  <Folder Include="MyProjectlogs\Development" />
</ItemGroup>
<ItemGroup>
    <ProjectCapability Include="SourceItemsFromImports" />
</ItemGroup>
<Target Name="NSwag" AfterTargets="Build">
    <Copy SourceFiles="@(Reference)" DestinationFolder="$(OutDir)References" />
    <Exec Command="$(NSwagExe_Core21) run nswag.json /variables" />
    <RemoveDir Directories="$(OutDir)References" />
</Target>

Edit : From the restore nugets logs, I get the following for each dependency of the project I publish

@   Project 'MyProject.Api' is affected (InstallCount = 0)

So it actually thinks something is different but doesn't seem to install anything.