Github action build fails on dotnet restore but local version works

I am using github to host private nuget packages, and the project I am trying to build an image for uses one of those packages. When image is being built via a github action I successfully add the nuget source, but restore fails to find it. However if I run it locally everything works just fine.

Action yaml:

    name: Docker
on:
  release:
    branches: [ master, qa ]
    tags: [ 'v*.*.*' ]
    types: [ published ]

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ph/logging-api  #<------------------------Container name
jobs:
  build:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
      id-token: write
    steps:
      - name: Checkout repository
        uses: actions/checkout@v2
      # Workaround: https://github.com/docker/build-push-action/issues/461
      - name: Setup Docker buildx
        uses: docker/setup-buildx-action@79abd3f86f79a9d68a23c75a09a9a85889262adf
      # Login against a Docker registry except on PR
      # https://github.com/docker/login-action

      - name: Log into registry ${{ env.REGISTRY }}
        if: github.event_name != 'pull_request'
        uses: docker/login-action@28218f9b04b4f3f62068d7b6ce6ca5b26e35336c
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      # Extract metadata (tags, labels) for Docker
      # https://github.com/docker/metadata-action
      - name: Extract Docker metadata
        id: meta
        uses: docker/metadata-action@98669ae865ea3cffbcbaa878cf57c20bbf1c6c38
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}

      # Build and push Docker image with Buildx (don't push on PR)
      # https://github.com/docker/build-push-action
      - name: Build and push Docker image
        id: build-and-push
        uses: docker/build-push-action@ad44023a93711e3deb337508980b4b5e9bcdc5dc
        with:
          file: api.Dockerfile #<----------------------------------------- Dockerfile
          push: ${{ github.event_name != 'pull_request' }}
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          build-args: |
            REGISTRY_USER=${{ github.actor }}
            REGISTRY_TOKEN=${{ secrets.GITHUB_TOKEN }}

Dockerfile:

FROM mcr.microsoft.com/dotnet/aspnet:6.0-focal AS base
WORKDIR /app
EXPOSE 5000

ENV ASPNETCORE_URLS=http://+:5000

# Creates a non-root user with an explicit UID and adds permission to access the /app folder
# For more info, please refer to https://aka.ms/vscode-docker-dotnet-configure-containers
RUN adduser -u 5678 --disabled-password --gecos "" appuser && chown -R appuser /app
USER appuser

FROM mcr.microsoft.com/dotnet/sdk:6.0-focal AS build
WORKDIR /src
ARG REGISTRY_USER
ARG REGISTRY_TOKEN
RUN dotnet nuget add source --username  $REGISTRY_USER --password $REGISTRY_TOKEN --store-password-in-clear-text --name github "https://nuget.pkg.github.com/ph/index.json"
COPY ["ph.LogApi/ph.LogApi.csproj", "ph.LogApi/"]
RUN dotnet restore "ph.LogApi/ph.LogApi.csproj"
COPY . .
WORKDIR "/src/ph.LogApi"
RUN dotnet build "ph.LogApi.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "ph.LogApi.csproj" -c Release -o /app/publish

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "ph.LogApi.dll"]

If I execute the docker file directly and pass in the build-args it works great. However in the github action it successfully adds the nuget source and then fails to find the package in either of them.

#10 [build 3/8] RUN dotnet nuget add source --username  h-w -*** --store-password-in-clear-text --name github "https://nuget.pkg.github.com/ph/index.json"
#10 0.305 Package source with Name: github added successfully.
#10 DONE 0.3s

#11 [build 4/8] COPY [ph.LogApi/ph.LogApi.csproj, ph.LogApi/]
#11 DONE 0.0s

#12 [build 5/8] RUN dotnet restore "ph.LogApi/ph.LogApi.csproj"
#12 0.792   Determining projects to restore...
#12 4.158 /src/ph.LogApi/ph.LogApi.csproj : error NU1101: Unable to find package ph.Configuration. No packages exist with this id in source(s): github, nuget.org
#12 4.343   Failed to restore /src/ph.LogApi/ph.LogApi.csproj (in 3.35 sec).
#12 ERROR: process "/bin/sh -c dotnet restore \"ph.LogApi/ph.LogApi.csproj\"" did not complete successfully: exit code: 1
------
 > [build 5/8] RUN dotnet restore "ph.LogApi/ph.LogApi.csproj":
#12 0.792   Determining projects to restore...
#12 4.158 /src/ph.LogApi/ph.LogApi.csproj : error NU1101: Unable to find package ph.Configuration. No packages exist with this id in source(s): github, nuget.org
#12 4.343   Failed to restore /src/ph.LogApi/ph.LogApi.csproj (in 3.35 sec).
------
api.Dockerfile:18
--------------------
  16 |     RUN dotnet nuget add source --username  $REGISTRY_USER -*** --store-password-in-clear-text --name github "https://nuget.pkg.github.com/PhoeniqsTech/index.json"
  17 |     COPY ["ph.LogApi/ph.LogApi.csproj", "ph.LogApi/"]
  18 | >>> RUN dotnet restore "ph.LogApi/ph.LogApi.csproj"
  19 |     COPY . .
  20 |     WORKDIR "/src/ph.LogApi"
--------------------
error: failed to solve: process "/bin/sh -c dotnet restore \"ph.LogApi/ph.LogApi.csproj\"" did not complete successfully: exit code: 1
Error: buildx call failed with: error: failed to solve: process "/bin/sh -c dotnet restore \"ph.LogApi/ph.LogApi.csproj\"" did not complete successfully: exit code: 1

Any input as to what the problem could be is greatly appreciated!