Not seeing why published artifacts are working in one case and not another for Azure DevOps Pipelines

Viewed 117

Overview

I'm using artifacts to persist some files and builds across stages. In one case it is working, and in another, it is not despite being able to see the artifact is published.

What Works

For example, this uploads my k8s manifests to an artifact so they can be access in the deployment stage to AKS and it works perfectly:

# publishStage.yaml
stages:
- stage: Publish
  displayName: Publish artifacts
  dependsOn: 
  - SeleniumTests
  - Changed
  condition: succeeded()
  variables:
    anyServicesChanged: $[ stageDependencies.Changed.Changes.outputs['detectChanges.anyServicesChanged'] ]
    anyConfigsChanged: $[ stageDependencies.Changed.Changes.outputs['detectChanges.anyConfigsChanged'] ]
  jobs:
  - job: Publish
    condition: or(eq(variables.anyServicesChanged, true), eq(variables.anyConfigsChanged, true), eq(variables['Build.Reason'], 'Manual'))
    displayName: Publishing artifacts...
    steps:
    - upload: k8s
      artifact: k8s
# deployStage.yaml
parameters:
- name: tag
  default: ''
- name: tagVersion
  default: ''

stages:
- stage: Deploy
  displayName: Deployment stage...
  dependsOn: 
  - Publish
  - Changed
  condition: succeeded()
  variables:
    anyServicesChanged: $[ stageDependencies.Changed.Changes.outputs['detectChanges.anyServicesChanged'] ]
    anyConfigsChanged: $[ stageDependencies.Changed.Changes.outputs['detectChanges.anyConfigsChanged'] ]
    servicesChanged: $[ stageDependencies.Changed.Changes.outputs['detectChanges.servicesChanged'] ]
  jobs:
  - deployment: Deploy
    condition: or(eq(variables.anyServicesChanged, true), eq(variables.anyConfigsChanged, true), eq(variables['Build.Reason'], 'Manual'))
    displayName: Deploying services...
    environment: 'App Production AKS'
    strategy:
      runOnce:
        deploy:
          steps:
          - task: KubernetesManifest@0
            displayName: Create imagePullSecret
            inputs:
              action: createSecret
              secretName: $(imagePullSecret)
              kubernetesServiceConnection: 'App Production AKS'
              dockerRegistryEndpoint: $(dockerRegistryServiceConnection)
          - template: deployStep.yaml
            parameters: 
              tag: ${{ parameters.tag }}
              tagVersion: ${{ parameters.tagVersion }}
              serviceName: api-v1
              pathName: api
# deployStep.yaml
parameters:
- name: tag
  default: ''
- name: tagVersion
  default: ''
- name: serviceName
  default: ''
- name: pathName
  default: ''

steps:
- task: KubernetesManifest@0
  condition: contains(variables['servicesChanged'], '${{ parameters.serviceName }}')
  displayName: Deploy to ${{ parameters.pathName }} Kubernetes cluster...
  inputs:
    action: deploy
    kubernetesServiceConnection: 'App Production AKS'
    manifests: |
      $(Pipeline.Workspace)/k8s/aks/${{ parameters.pathName }}.yaml
    imagePullSecrets: |
      $(imagePullSecret)
    containers: |
      $(containerRegistry)/$(imageRepository)-${{ parameters.pathName }}:${{ parameters.tag }}-${{ parameters.tagVersion }}

What Doesn't Work

However, now what I'm trying to do is build a service in the application, run unit tests, and take the exact same build that was tested and use it for building the Docker image.

I'm trying to do this with the following:

# unitTestsStage.yaml
stages:
- stage: UnitTests
  displayName: Run unit tests for services...
  dependsOn: Changed
  condition: succeeded()
  jobs:
  - template: ../secretsJob.yaml
  - template: pythonJob.yaml
    parameters:
      serviceName: api-v1
      pathName: api
# pythonJob.yaml
parameters:
- name: serviceName
  type: string
  default: ''
- name: pathName
  type: string
  default: ''

jobs:
- job: UnitTests
  displayName: Running unit tests for ${{ parameters.serviceName }}...
  variables:
    servicesChanged: $[ stageDependencies.Changed.Changes.outputs['detectChanges.servicesChanged'] ]
  condition: or(contains(variables['servicesChanged'], '${{ parameters.serviceName }}'), eq(variables['Build.Reason'], 'Manual'))
  dependsOn: Secrets
  steps:
  - task: UsePythonVersion@0
    inputs:
      versionSpec: '3.8'
  - script: |
      cd ${{ parameters.pathName }} && 
      python -m pip install --upgrade pip && 
      pip install -r requirements.txt
    displayName: Install requirements for ${{ parameters.pathName }}...
  - script: cd ${{ parameters.pathName }} &&  coverage run --omit='manage.py,config/*,.venv*,*/*__init__.py,*/tests.py,*/admin.py' manage.py test && coverage report
    displayName: Run unit tests and coverage for ${{ parameters.pathName }}...
    env:
      DJANGO_SECRET_KEY: $(PROD-DJANGOSECRETKEY)
      DJANGO_DEBUG: $(PROD-DJANGODEBUG)
      DOMAIN: $(PROD-DOMAIN)
      PGDATABASE: $(PROD-PGDATABASE)
      PGDATABASEV2: $(PROD-PGDATABASEV2)
      PGUSER: $(PROD-PGUSER)
      PGPASSWORD: $(PROD-PGPASSWORD)
      PGHOST: $(PROD-PGHOST)
      PGPORT: $(PROD-PGPORT)
  - upload: ${{ parameters.pathName }}
    artifact: ${{ parameters.pathName }}
    condition: succeeded()

# buildStage.yaml
parameters:
- name: tag
  default: ''
- name: tagVersion
  default: ''
#
stages:
- stage: BuildAndPush
  displayName: Build and Push Docker images of services...
  dependsOn: 
  - UnitTests
  - Changed
  condition: succeeded()
  variables:
    anyServicesChanged: $[ stageDependencies.Changed.Changes.outputs['detectChanges.anyServicesChanged'] ]
    servicesChanged: $[ stageDependencies.Changed.Changes.outputs['detectChanges.servicesChanged'] ]
  jobs:
  - job: BuildAndPush
    condition: or(eq(variables.anyServicesChanged, true), eq(variables['Build.Reason'], 'Manual'))
    displayName: Building and Push Docker images of services...
    steps:
    - template: buildStep.yaml
      parameters: 
        tag: ${{ parameters.tag }}
        tagVersion: ${{ parameters.tagVersion }}
        serviceName: api-v1
        pathName: api
# buildStep.yaml
parameters:
- name: tag
  default: ''
- name: tagVersion
  default: ''
- name: serviceName
  default: ''
- name: pathName
  default: ''

steps: 
- task: Docker@2
  condition: contains(variables['servicesChanged'], '${{ parameters.serviceName }}')
  displayName: Build and Push ${{ parameters.pathName }} Docker image
  inputs:
    command: buildAndPush
    repository: $(imageRepository)-${{ parameters.pathName }}
    dockerfile: $(Pipeline.Workspace)/${{ parameters.pathName }}/Dockerfile
    buildContext: $(Pipeline.Workspace)/${{ parameters.pathName }}
    containerRegistry: $(dockerRegistryServiceConnection)
    tags: |
      ${{ parameters.tag }}-${{ parameters.tagVersion }}

I can see the api artifact was published:

enter image description here

But then when it comes actually pulling it for building the Docker image I just get:

Starting: Build and Push api Docker image
==============================================================================
Task         : Docker
Description  : Build or push Docker images, login or logout, start or stop containers, or run a Docker command
Version      : 2.187.0
Author       : Microsoft Corporation
Help         : https://aka.ms/azpipes-docker-tsg
==============================================================================
##[error]Unhandled: No Dockerfile matching  /home/vsts/work/1/api/Dockerfile  was found.
Finishing: Build and Push api Docker image

If I had a script task to ls -la $(Pipeline.Workspace) I get:

drwxr-xr-x  6 vsts docker 4096 Jul  9 16:19 .
drwxr-xr-x  7 vsts root   4096 Jul  9 16:19 ..
drwxr-xr-x  2 vsts docker 4096 Jul  9 16:19 TestResults
drwxr-xr-x  2 vsts docker 4096 Jul  9 16:19 a
drwxr-xr-x  2 vsts docker 4096 Jul  9 16:19 b
drwxr-xr-x 12 vsts docker 4096 Jul  9 16:19 s

Question

So what am I doing wrong here that referencing the artifact is working in one case and not the other?

1 Answers

Deployment job in you first case automatically download all published artifacts.

In your second case you use regular job which doesn't download it. You have to do it excplicitly by adding download step before running your docker step..

Related