I want to restrict access to azure keyvault using virtual Network. but when i configure it as below. and i expect to be able to manage secrets when i am on VPN(the selected Vnet). but it doesn't allow me.

I want to restrict access to azure keyvault using virtual Network. but when i configure it as below. and i expect to be able to manage secrets when i am on VPN(the selected Vnet). but it doesn't allow me.

When you allow access only from specific VNETs (and Subnets) you are leveraging Service Endpoints (https://docs.microsoft.com/en-us/azure/virtual-network/virtual-network-service-endpoints-overview).
This means that only VMs from the particular VNET/Subnet can access the service endpoints and thus the service. The corresponding FAQs state:
By default, Azure service resources secured to virtual networks are not reachable from on-premises networks. If you want to allow traffic from on-premises, you must also allow public (typically, NAT) IP addresses from your on-premises or ExpressRoute. These IP addresses can be added through the IP firewall configuration for the Azure service resources. Reference: https://docs.microsoft.com/en-us/azure/virtual-network/virtual-networks-faq#can-an-on-premises-devices-ip-address-that-is-connected-through-azure-virtual-network-gateway-vpn-or-expressroute-gateway-access-azure-paas-service-over-vnet-service-endpoints
Unless you would like to completely disable public access and leverage Private Endpoints, you would need to add your on-premises public IP address to the firewall rules. Please note that you will then connect to the key vault through the public internet - your network traffic is not going through the VPN tunnel.
The only way to fully leverage private network traffic is to use Private Endpoints.