After so many days, finally figured it out, I have VALID_ARCHS in build settings. By removing that and adding arm64 to excluded architectures solved my problem.