Publish Azure appservice with auto-hotswap gives ERROR_SITE_SWAP_OPERATION_IN_PROGRESS error

I'm trying to publish an AppService using hot-swap with auto-swap enabled. I can do it without any issues via Visual Studio but not with CI which uses publishing progile and unable to finish publish with the following error:

C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v14.0\Web\Microsoft.Web.Publishing.targets(4276,5): msdeploy error ERROR_SITE_SWAP_OPERATION_IN_PROGRESS: Web deployment task failed. (A site swap operation is currently in progress. Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_SITE_SWAP_OPERATION_IN_PROGRESS.)

Even though I'm getting the error, publishing and auto-swap operations are performed without issues. The Error, however, prevents my CI job from finishing.

My publishing profile:

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup>
    <WebPublishMethod>MSDeploy</WebPublishMethod>
    <ADUsesOwinOrOpenIdConnect>False</ADUsesOwinOrOpenIdConnect>
    <PublishProvider>AzureWebSite</PublishProvider>
    <LastUsedBuildConfiguration>Release</LastUsedBuildConfiguration>
    <LastUsedPlatform>Any CPU</LastUsedPlatform>
    <SiteUrlToLaunchAfterPublish>https://my-cool-site-swap.azurewebsites.net</SiteUrlToLaunchAfterPublish>
    <LaunchSiteAfterPublish>False</LaunchSiteAfterPublish>
    <ExcludeApp_Data>False</ExcludeApp_Data>
    <MSDeployServiceURL>my-cool-site-swap.scm.azurewebsites.net:443</MSDeployServiceURL>
    <DeployIisAppPath>my-cool-site-swap</DeployIisAppPath>
    <RemoteSitePhysicalPath />
    <SkipExtraFilesOnServer>True</SkipExtraFilesOnServer>
    <MSDeployPublishMethod>WMSVC</MSDeployPublishMethod>
    <EnableMSDeployBackup>True</EnableMSDeployBackup>
    <UserName>$my-cool-site__swap</UserName>
    <Password>...password...</Password>
    <AllowUntrustedCertificate>True</AllowUntrustedCertificate>
    <_SavePWD>True</_SavePWD>
    <_DestinationType>AzureWebSite</_DestinationType>
  </PropertyGroup>
</Project>

And the CI command to start it:

C:\Program Files (x86)\MSBuild\14.0\Bin\MSBuild.exe" /p:Configuration=Release "/p:Platform=Any CPU" /p:DeployOnBuild=true /p:PublishProfile=my-cool-profile

UPDATE: I've discovered that website binaries are pushed correctly while webjobs libraries are outdated and not pushed with this any deployments. It seems that auto-swap is initiated earlier than msdeploy pushes webjobs part of binaries, that is why I have outdates webjobs and publication error.

What could be wrong with the publishing profile or the way I execute it?