Flutter Plugin -- build for iOS works in xCode, but doesn't work with flutter build because of package dependency

I am developing a Flutter plugin, which requires a third party library. This library is hosted on GitHub, and to add it to my project, they tell me to use XCode's package manager (File -> Add Packages). When I do it this way, I can use the package just fine, compile my code and run it on my iPad. However, when I try to use "flutter build ios", it doesn't work.

When building, I see a series of (with actual framework names and file names instead of ...):

    note: Removed stale file ...
    note: Removed stale file ...
    note: Removed stale file ...

Then eventually I will see that the compilation has failed due to being unable to import the library I need. Additionally, when I go back into XCode, I find that the package dependency I had installed is no longer there.

So, what I suspect is happening, is when Flutter builds it is cleaning up all the old frameworks, and running its own pod install. However, since my package is not part of my podspec file, it is being left behind. I am unsure how to get this to work. I've tried adding a line in my podspec file like:

s.dependency 'PackageName'
s.source { :git => 'https://linkToTheGitRepo' }

and then running pod repo update, but when I go to build I still get "Unable to find a specification for PackageName depended upon by my_plugin".

At this point, I just build in XCode and then don't touch it, however this has also caused issues when I go to use the plugin later in another project -- I also cannot use flutter build ios in that project, and I have to do it through XCode. This is an annoying workaround, since I lose the hot reload ability and other nice features of working with Flutter. Any suggestions are appreciated. Thanks.