So maybe it's better to explain this a little.
With Elliptic curve crypto, the private key is just a number (a big one).
The public key is actually a point on the curve (like actually x, y).
You must generate a private key first in order to obtain the corresponding public key coordinate, as you may know, the trap door function in ECC is predicated upon not being able to deduce a private key from a public key point.
The public key is obtained by scalar multiplication of the private key by a special point on the curve called the generator point.
So... that said, the only way to start with a public key that you know, would be to have (at some point previous), generated and stored the private key, calculated the corresponding public key by using the standard methods (which FYI, is actually a process where one scalar multiplies a private key number by a known generator (special publicly known x,y point) on the curve, and then store the public key for future use.
One last thing, what you're doing here (you may already know) is ECDH (Elliptic curve Diffie Hellman), generating a shared secret using the Diffie Hellman protocol.
In that protocol, generally you use ephemeral (temporary) key pairs every time. That way each session has a new session key, meaning forward secrecy isn't predicated upon one single private key.
Additionally remember that a derived ECDH secret is not ready to use as a symmetric key. It should be passed through a HKDF (key derivation function), as random secret != uniform secret suitable for cryptographic use.
Any questions pls ask below, great explanation as to why it's important to use ephemeral keys in ECDH here from @Maarten.
Please let me know if this is not clear.