Error saying "Module Not Found" when adding SPM which uses other SPMs as dependencies within itself

I have been creating a Swift Package Manager. It uses 2 other SPMs within itself. SPM compiles fine when compiled independently. As soon as the project is imported into an Xcode project I get a compiler error saying that:

No such module 'ModuleName'

Note: The ModuleName in the above error corresponds to the package imported within the package that is being imported to my project.

I have been stuck on this for a pretty while now and have tried the following:

1. Removed and readded the SPMs to dependencies to my SPM, and then tried importing my SPM to my project (I did this before and after each of the other steps too).
2. Checked to see where these packages where being added as dependencies. It shows up in the SPM main target Module -> Build Phases -> Link binary with libraries. I additionally added it to the Dependencies section to see if it changes anything.
3. Tried adding SPMs to ModulePackageDescription target to Dependencies section.
4. Added the dependencies in the Package.swift file as follows.

dependencies: [
    // Dependencies declare other packages that this package depends on.
     .package(url: "package1_url", .branch("master")),
     .package(url: "package2_url", .branch("master"))
]

Adding this would import the other dependencies to my Xcode project. I don't exactly want this to happen because in case I try to use another version of the SPM that is being imported within my SPM, it would cause conflict between the two versions. But I'm willing to do this if it's the right way to go. But even adding dependencies in Package.swift didn't work for me. How would I resolve this issue? Let me know if anyone has faced the same issue.