Monitoring kubernetes job

Question

I have a kubernetes job that takes quite some time to finish the job, almost around 4 minutes in best case and 8 mins in worse case.

Instead of just ideally waiting for the job to be completed, I tried creating a code that alerts me when the job is completed.

It does the following:

Submits the kubernetes job.

Waits for its completion.

Checks whether the job has had the expected affect.

But the problem is, no matter how quickly the job is completed I have to wait for 8 minutes as the worse case input is 8 mins.

Is there a way to make this code more efficient?

Answer 1

<kube master>/apis/batch/v1/namespaces/default/jobs 

endpoint lists status of the jobs. I have parsed this json and retrieved the name of the latest running job that starts with "deploy...".

Then we can hit

<kube master>/apis/batch/v1/namespaces/default/jobs/<job name retrieved above>

And monitor the status field value which is as below when the job succeeds

"status": {
    "conditions": [
      {
        "type": "Complete",
        "status": "True",
        "lastProbeTime": "2016-09-22T13:59:03Z",
        "lastTransitionTime": "2016-09-22T13:59:03Z"
      }
    ],
    "startTime": "2016-09-22T13:56:42Z",
    "completionTime": "2016-09-22T13:59:03Z",
    "succeeded": 1
  }

So we keep polling this endpoint till it completes.

Answer 2

you can do

watch 'kubectl get pods | grep <name of the pod>'

or

kubectl get pods -w

It will not be the full name of course as most of the time the pods get random names if you are running nginx replica or deployment your pods will end up with something like nginx-1696122428-ftjvy so you will want to do

watch 'kubectl get pods | grep nginx'

You can replace the pods with whatever job you are doing i.e (rc,svc,deployments....)

Even better you can use labels: watch 'kubectl get pods -l job=foobar' or kubectl get -w pods -l job=foobar