I am using step([$class: 'AnsibleAdHocCommandBuilder']): Invoke Ansible Ad-Hoc Command to invoke adhoc command in my pipeline, but somehow, it gives error when use credentialsId to pass the private key variable.
step([
$class: 'AnsibleAdHocCommandBuilder',
ansibleName: 'Run cvWatchBuilder',
inventory: [$class: 'InventoryPath', path: "myhost,"],
hostPattern: "all",
module: 'shell',
disableHostKeyChecking: true,
command: 'ls',
forks: 1,
credentialsId: 'ssh-private-key'
]);
When this step run, it gives error for private key.
Running on k8s-6930a022-327d-44a2-9065-cae0fde58c14-s8pbx-70c8d in /home/jenkins/agent/workspace/k8s-e2e
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Run shell)
[Pipeline] container
[Pipeline] {
[Pipeline] step
Executing sh script inside container ansible of pod k8s-6930a022-327d-44a2-9065-cae0fde58c14-s8pbx-70c8d
Executing command: "ansible" "all" "-i" "myhost," "-m" "shell" "-a" "ls" "-f" "1" "--private-key" "/tmp/ssh11730288813016418676.key" "-u" "devops"
exit
myhost | UNREACHABLE! => {
"changed": false,
"msg": "Failed to connect to the host via ssh: Warning: Permanently added 'myhost,172.17.203.19' (ECDSA) to the list of known hosts.\r\n\n\n__________ __ __ _________ .____ .___\n\\______ \\_____ ____ | | ____ ________ ____ ____ _____/ |_ ___________ \\_ ___ \\| | | |\n | | _/\\__ \\ _/ ___\\| |/ / | \\____ \\_/ ___\\/ __ \\ / \\ __\\/ __ \\_ __ \\ / \\ \\/| | | |\n | | \\ / __ \\\\ \\___| <| | / |_> > \\__\\ ___/| | \\ | \\ ___/| | \\/ \\ \\___| |___| |\n |______ /(____ /\\___ >__|_ \\____/| __/ \\___ >___ >___| /__| \\___ >__| \\______ /_______ \\___|\n \\/ \\/ \\/ \\/ |__| \\/ \\/ \\/ \\/ \\/ \\/\n\n\n\nno such identity: /tmp/ssh11730288813016418676.key: No such file or directory\r\ndevops@myhost: Permission denied (publickey,password).",
"unreachable": true
}
FATAL: command execution failed
hudson.AbortException: Ansible Ad-Hoc command execution failed
at org.jenkinsci.plugins.ansible.AnsibleAdHocCommandBuilder.perform(AnsibleAdHocCommandBuilder.java:224)
at jenkins.tasks.SimpleBuildStep.perform(SimpleBuildStep.java:123)
at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:100)
at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:70)
at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
But when use same key with sh function, it works.
withCredentials([sshUserPrivateKey(credentialsId: 'ssh-private-key', keyFileVariable: 'keyfile')]) {
sh("ANSIBLE_HOST_KEY_CHECKING=False ansible all -i myhost, -m shell -a ls --private-key $keyFile -u devops")
}
This works and print output as below.
Running on k8s-8f822fb4-b0bf-4c8d-9df2-9fa3da5e4a55-v9j9n-kdbtg in /home/jenkins/agent/workspace/k8s-e2e
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Run shell)
[Pipeline] container
[Pipeline] {
[Pipeline] withCredentials
Masking supported pattern matches of $keyfile
[Pipeline] {
[Pipeline] sh
Warning: A secret was passed to "sh" using Groovy String interpolation, which is insecure.
Affected argument(s) used the following variable(s): [keyfile]
See https://jenkins.io/redirect/groovy-string-interpolation for details.
+ ANSIBLE_HOST_KEY_CHECKING=False ansible all -i myhost, -m shell -a ls --private-key **** -u devops
myhost | CHANGED | rc=0 >>
EDIT
I debug little more, then sees that, it complain about /tmp/ssh11730288813016418676.key: No such file or directory, then I was thinking to see where this file is :)
We are using Jenkins-operator on kubernetes cluster to run the jenkins instance. What it will do is, it create 2 container in pod. One for jnlp to connect to jenkins master, and second to use for the execution.
My full pipeline is as below.
$ cat cicd/pipelines/k8s.jenkins
#!/usr/bin/env groovy
def label = "k8s-${UUID.randomUUID().toString()}"
def home = "/home/jenkins"
def workspace = "${home}/workspace/build-jenkins-operator"
def workdir = "${workspace}/src/github.com/jenkinsci/kubernetes-operator/"
podTemplate(label: label,
containers: [
containerTemplate(name: 'ansible', image: 'my registry/ansible', ttyEnabled: true, command: 'cat'),
// containerTemplate(name: 'ansible', image: 'python', ttyEnabled: true, command: 'cat'),
],
) {
node(label) {
stage('Run shell') {
container('ansible') {
// withCredentials([sshUserPrivateKey(credentialsId: 'ssh-private-key', keyFileVariable: 'keyfile')]) {
// sh("ANSIBLE_HOST_KEY_CHECKING=False ansible all -i myhost, -m shell -a ls --private-key $keyFile -u devops")
// }
// sh 'pip install ansible'
step([
$class: 'AnsibleAdHocCommandBuilder',
ansibleName: 'Run command',
inventory: [$class: 'InventoryPath', path: "myhost,"],
// inventory: [$class: 'InventoryPath', path: "localhost,"],
hostPattern: "all",
// module: 'shell',
module: 'pause',
disableHostKeyChecking: true,
command: 'minutes=5',
// command: 'ls',
forks: 1,
additionalParameters: '-vvv',
credentialsId: 'ssh-private-key'
]);
sleep time: 60
}
}
}
}
When I check the jenkins console log, it shows that which pod it created and what it runs under which container.
Running on k8s-d83c8383-beca-4cfb-ac61-5b8df98d76e7-t4s7w-chmhn in /home/jenkins/agent/workspace/k8s-e2e
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Run shell)
[Pipeline] container
[Pipeline] {
[Pipeline] step
Executing sh script inside container ansible of pod k8s-d83c8383-beca-4cfb-ac61-5b8df98d76e7-t4s7w-chmhn
Executing command: "ansible" "all" "-i" "myhost," "-m" "pause" "-a" "minutes=5" "-f" "1" "--private-key" "/tmp/ssh6159272872640967836.key" "-u" "devops" "-vvv"
exit
ansible 2.10.11
config file = /etc/ansible/ansible.cfg
configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/local/lib/python3.8/site-packages/ansible
executable location = /usr/local/bin/ansible
python version = 3.8.8 (default, Mar 31 2021, 12:38:11) [GCC 8.3.0]
Using /etc/ansible/ansible.cfg as config file
Parsed myhost, inventory source with host_list plugin
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.
META: ran handlers
Pausing for 300 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
myhost | SUCCESS => {
"changed": false,
"delta": 300,
"echo": true,
"rc": 0,
"start": "2021-12-15 19:45:35.186293",
"stderr": "",
"stdout": "Paused for 5.0 minutes",
"stop": "2021-12-15 19:50:35.186787",
"user_input": ""
}
META: ran handlers
META: ran handlers
[Pipeline] sleep
Sleeping for 1 min 0 sec
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
Finished: SUCCESS
from above message, it created k8s-d83c8383-beca-4cfb-ac61-5b8df98d76e7-t4s7w-chmhn pod in our kubernetes cluster. When I did describe on that pod, I see there are 2 containers running in this pod.
# kubectl describe pod k8s-d83c8383-beca-4cfb-ac61-5b8df98d76e7-t4s7w-chmhn -n jenkins
Name: k8s-d83c8383-beca-4cfb-ac61-5b8df98d76e7-t4s7w-chmhn
Namespace: jenkins
Priority: 0
...
...
Status: Running
IP: 192.168.221.84
IPs:
IP: 192.168.221.84
Containers:
ansible:
Container ID: containerd://400d9360a9b3a1415335d6e57148a58d2e64576649323e203a9b66a0de2b048a
Image: myrepo/ansible
Image ID: myrepo/ansible@sha256:6b797d6f80c7ec8c209bec8bde363b84c910a68b4b608c9db6a0411acff3b515
Port: <none>
Host Port: <none>
Command:
cat
State: Running
Started: Wed, 15 Dec 2021 19:45:30 +0000
Ready: True
Restart Count: 0
...
...
Mounts:
/home/jenkins/agent from workspace-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mrww8 (ro)
jnlp:
Container ID: containerd://f592bf372bf8fde5baca8a28918b275125fea9b66c9d6ea422172adb29c688ac
Image: jenkins/inbound-agent:4.3-4-jdk11
Image ID: docker.io/jenkins/inbound-agent@sha256:2ee9fb7405f135e15ea3c5efaa605b7c19bc0a93a306deec7a0826a489c1a364
Port: <none>
Host Port: <none>
State: Running
Started: Wed, 15 Dec 2021 19:45:31 +0000
Ready: True
Restart Count: 0
Requests:
cpu: 100m
memory: 256Mi
Environment:
JENKINS_SECRET: 0e0f75a0d4e42f0b6c0964f12a1bd1d331d75b0578d8b0e1fd2a7fd9d1175b9d
JENKINS_TUNNEL: jenkins-operator-slave-jenkins-instance.jenkins.svc.cluster.local:50000
JENKINS_AGENT_NAME: k8s-d83c8383-beca-4cfb-ac61-5b8df98d76e7-t4s7w-chmhn
JENKINS_NAME: k8s-d83c8383-beca-4cfb-ac61-5b8df98d76e7-t4s7w-chmhn
JENKINS_AGENT_WORKDIR: /home/jenkins/agent
...
...
Mounts:
/home/jenkins/agent from workspace-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mrww8 (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
workspace-volume:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
kube-api-access-mrww8:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: kubernetes.io/os=linux
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 3m13s default-scheduler Successfully assigned jenkins/k8s-d83c8383-beca-4cfb-ac61-5b8df98d76e7-t4s7w-chmhn to myworker
Normal Pulled 3m13s kubelet Container image "stepladder-docker-local.artifacts.rackspace.net/ansiblelint:latest" already present on machine
Normal Created 3m13s kubelet Created container ansible
Normal Started 3m13s kubelet Started container ansible
Normal Pulled 3m13s kubelet Container image "jenkins/inbound-agent:4.3-4-jdk11" already present on machine
Normal Created 3m13s kubelet Created container jnlp
Normal Started 3m12s kubelet Started container jnlp
In the console log it has log line as Executing sh script inside container ansible of pod k8s-d83c8383-beca-4cfb-ac61-5b8df98d76e7-t4s7w-chmhn means ansible command will run in ansible container of pod.
When I check the /tmp directory in ansible pod, I didn't see any files in that.
# kubectl exec -i -t k8s-d83c8383-beca-4cfb-ac61-5b8df98d76e7-t4s7w-chmhn -c ansible -n jenkins -- ls -al /tmp/
total 8
drwxrwxrwt 1 root root 4096 Dec 15 19:45 .
drwxr-xr-x 1 root root 4096 Dec 15 19:45 ..
While as per the ansible command, there should be file /tmp/ssh11730288813016418676.key exists in the /tmp folder. Then I check the jnlp container.
# kubectl exec -i -t k8s-d83c8383-beca-4cfb-ac61-5b8df98d76e7-t4s7w-chmhn -c jnlp -n jenkins -- ls -al /tmp/
total 20
drwxrwxrwt 1 root root 4096 Dec 15 19:45 .
drwxr-xr-x 1 root root 4096 Dec 15 19:45 ..
drwxr-xr-x 2 jenkins jenkins 4096 Dec 15 19:45 hsperfdata_jenkins
drwxr-xr-x 2 root root 4096 Mar 31 2020 hsperfdata_root
-r-------- 1 jenkins jenkins 3243 Dec 15 19:45 ssh11730288813016418676.key
I can see the sshXXXX.key file is created in jnlp container.
When ansible command runs on ansible container, it is not able to find the sshXXXXX.key file on that container, and it gives error that no such file or directory.