Azure web app deployed to tmp folder instead of wwwroot

I found out what was the issue. There was a change in the Python build process on Azure App Services. The details can be found here.

I quote from the link above.

Change

Python build and runtime workflow is changing for the apps built on App Service.

Current build process

The app builds are carried out in a build container/agent by a Build System called Oryx. This container produces artifacts which are stored on an NFS volume and later run by a Runtime container defined by the user(eg: Python 3.8).

Oryx running in the Build Agent installs the dependencies listed in requirements.txt into a virtual environment named antenv. This build is carried out in a local build directory. This virtual environment produced by the above build is compressed and extracted by the runtime at the root of the container to enable the users to have an optimal build and startup time.

This caused a problem where the Virtual Environment could not resolve certain app dependencies. The reason for this is that during the creation of the Virtual Environment pip hardcodes certain paths for dependencies. Since, the builds are carried out in an Build Container and Run on a Runtime container at a different path, this causes a mismatch between the directory structure for the virtual environment.

Change/New Build+Runtime Process

The build workflow remains the same with respect to the use of a temporary build directory as described above. But now the whole app output(including the Virtual Environment) is compressed into a tarball and stored at /home/site/wwwroot with a manifest file oryx-manifest.toml. This manifest file describes how the build was carried out in the build container. The runtime container on startup reads this manifest and recreates a local copy of the folder structure on every instance. This change also ensures that the runtime during the app startup will already have the virtual environment activated and users can run commands like migrate without the need to activate the environment.

As a part of this change, the app would no longer run from /home/site/wwwroot but would have a dynamic runtime directory. This path can be retrieved by using the variable $APP_PATH.

If this change broke your app, your only option is to change your file path. There's nothing you can do to stop Azure App Service from running your Django app from tmp.

EDIT: I opened an issue with the Azure team on this and they are working to roll back the changes. More details can be found here

EDIT: Latest update (20201229)

We are in the process of reverting to the old deployment behavior across all regions. The new behavior would be an opt-in after this fix, we would send out notifications and give ample time to make changes when we change the default behavior.

You could pin to the old version by adding an app setting: KUDU_BUILD_VERSION=0.0.1 -> This would ensure the build behavior remains the same even when we change default behavior after the notification.

To try the new version, you can add the app setting: KUDU_BUILD_VERSION=1.0.0.

Please share your app names directly or indirectly if you are still experiencing the problem and we can investigate further

It is suggested to make the SCM_DO_BUILD_DURING_DEPLOYMENT= FALSE.

For more details, you can refer to this post(Azure App Service Getting error while deploying REACT JS application).

I read his post carefully and did not find the cause of this problem. If you are interested, you can raise a support ticket.

This post was a saver for me. Thanks for this. MS did a poor job updating folks on this change. It really bit us and was a wild goose chase until I ran into this. After I read this, it took very little time to pull things together and there's a few lessons learned I'll share as a thanks and in hopes others won't go through this pain.

1. this change happened on 3/1/21 but nothing will break until you do a deployment post that date. All will continue to work fine until then. That's when you'll see inexplicable things going on because MS is using a TAR file strategy and visualizing things, etc. BUT, these processes will NOT change your current setup nor will it delete anything. All creating a mess! So, if, for example, you look at wwwroot from bash or ftp, no files will change/update no matter how many times you redeploy. So, if you've learned about this rather radical change before a planned deployment, clean things up; then move to a CI/CD model and leverage GitHub is my recommendation, deleting everything beforehand - sort of a "start fresh" approach. Do that, and you're good to go. Else, you'll be chasing ghosts if you've already deployed. I was the later case and that just meant a redo from scratch after many headaches.
2. if you have any ref'd resources that are outside of you Git repo, $APP_PATH is key and while this is all virtual now, using "/home" as the user friendly way to point at you app path is the key tidbit. $APP_PATH is useful for Bash and FTP work. "/home" for code. Also NOTE, ONLY things saved under /home will persist. In my case, I moved my folder from wwwroot to /home/ and a slight change in code (added "/home/" was all that was needed. And using an app variable is probably better coding practice :)
3. Once you setup CI/CD, that rocks, so that's amazing. Make a code change, get an email things have changed and go monitor on GitHub the build/deploy process. If it turns "red", fix it; check it in; watch and wait.

Final point on #3. I know you can work around this with a variable. My suggestion is NOT to do that. MS' reasons for this change are way better than their ability to communicate it out. It's a good idea and works very well. I'm using a rather complex ML model whose performance improved by about 10-15%. Maybe that's luck, but it made me a believer in this change.

Just hope MS' learned how to keep us informed better next time :)

HTH.