Running One-Off Batch Jobs

1. What is a Kubernetes Job?

At its core, a Job is designed for tasks with a clear start and end. Unlike a Deployment, which replaces Pods that fail or are terminated to maintain a desired state, a Job's goal is to ensure its Pods run to completion.

Common use cases for Jobs that you'll encounter include:

• Data processing: Running a script to transform and load a batch of data.
• Backups: Triggering an application or database backup.
• Administrative tasks: Running a database schema migration script before deploying a new application version.
• Testing: Executing an end-to-end test suite against a staging environment.

To get a clear overview of what Jobs are and how they differ from other Kubernetes objects, let's watch the beginning of a helpful tutorial.

2. Creating Your First Job

Like other Kubernetes resources, a Job is defined using a YAML manifest. The structure is similar to a Deployment, as it includes a Pod template (spec.template) that defines the container(s) to run.

Let's start with a basic Job that prints a message and then exits.

Here is the manifest from the article for easy reference. Save it as hello-job.yaml:

# hello-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: hello-job
spec:
  template:
    spec:
      containers:
      - name: hello
        image: busybox:1.28
        command: ["echo", "Hello, Kubernetes Job!"]
      restartPolicy: Never

Let's break this down:

• apiVersion: batch/v1: Jobs belong to the batch API group, signifying their role in batch processing.
• kind: Job: This specifies the resource type.
• spec.template: This is the standard Pod template. It defines the Pod that the Job will create.
• restartPolicy: Never: This is a crucial field. For Jobs, the Pod's restart policy can only be Never or OnFailure. It cannot be Always (the default for Pods created by Deployments), because a Job is expected to terminate. If a Pod in a Job were to restart Always, the Job would never complete.

Now, run the Job and observe its behavior:

1. Apply the manifest:

   kubectl apply -f hello-job.yaml
   

2. Check the Job's status:

   kubectl get jobs
   

   You'll see output like this, showing that 1 completion was desired and 1 was achieved.

   NAME        COMPLETIONS   DURATION   AGE
   hello-job   1/1           2s         10s
   

3. Check the Pod created by the Job:

   kubectl get pods
   

   You'll notice the Pod has a status of Completed. The Job controller doesn't delete the Pod, which is useful for inspecting logs and debugging.

   NAME              READY   STATUS      RESTARTS   AGE
   hello-job-abcde   0/1     Completed   0          25s
   

4. View the logs:

   # Replace hello-job-abcde with your actual Pod name
   kubectl logs hello-job-abcde
   

   You should see the output: Hello, Kubernetes Job!.

3. Controlling Job Execution and Parallelism

A Job can be configured to run a task multiple times, either sequentially or in parallel. This is controlled by two fields in the spec:

• completions: The number of Pods that must complete successfully for the Job to be considered complete. Defaults to 1.
• parallelism: The maximum number of Pods that can run concurrently. Defaults to 1.

Let's look at a manifest that uses these fields.

Here is a practical example. This Job will run 6 "tasks" in total, with up to 2 running at any given time.

# parallel-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: parallel-job
spec:
  completions: 6
  parallelism: 2
  template:
    spec:
      containers:
      - name: worker
        image: busybox:1.28
        command: ["sh", "-c", "echo 'Processing item...'; sleep 5"]
      restartPolicy: OnFailure

Apply this manifest (kubectl apply -f parallel-job.yaml) and then watch the Pods get created:

# Watch the pods associated with this job
kubectl get pods -l job-name=parallel-job -w

You will see two Pods start. As one completes, another one will be created to take its place, until a total of six Pods have successfully completed.

4. Handling Failures and Timeouts

In a real-world production environment, tasks can fail. Jobs provide two key mechanisms for building resilience and preventing runaway processes.

As the video explains:

• backoffLimit (Default: 6): The number of times a Job will retry a failing Pod before marking the entire Job as failed. The retries are performed with an exponential backoff delay (10s, 20s, 40s...).
• activeDeadlineSeconds: A hard time limit (in seconds) for the Job. If the Job runs for longer than this, it will be terminated with a DeadlineExceeded status, regardless of how many completions have been achieved. This takes precedence over backoffLimit.

Here is an example combining these concepts:

# resilient-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: resilient-job
spec:
  completions: 3
  parallelism: 1
  backoffLimit: 4 # Retry up to 4 times on failure
  activeDeadlineSeconds: 60 # Fail the job if it runs longer than 1 minute
  template:
    spec:
      containers:
      - name: unreliable-worker
        image: busybox:1.28
        # This command will fail occasionally
        command: ["sh", "-c", "echo 'Attempting task...'; exit $((RANDOM % 2))"]
      restartPolicy: OnFailure

5. Job Cleanup and Management

A key operational aspect of Jobs is that completed Jobs and their Pods are not automatically deleted. While this is great for debugging, it can lead to clutter in your cluster over time.

There are two ways to manage this:

1. Automatic Cleanup (Recommended)
You can use the ttlSecondsAfterFinished field in the Job's spec. This tells the controller to clean up the Job (and its Pods) a certain number of seconds after it completes.

# ttl-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: ttl-job
spec:
  ttlSecondsAfterFinished: 100 # Cleanup 100 seconds after completion
  template:
    spec:
      containers:
      - name: short-lived-task
        image: busybox:1.28
        command: ["echo", "This job will self-destruct!"]
      restartPolicy: Never

2. Manual Cleanup
As a Platform Engineer, you'll often need to perform manual cleanup. You can use kubectl with field selectors to delete Jobs based on their status.

Here are the commands from the article:

# Delete all successfully completed jobs
kubectl delete jobs --field-selector status.successful=1

# Delete all failed jobs
kubectl delete jobs --field-selector status.successful=0

These commands are perfect for incorporating into your own automation or cleanup scripts.

Conclusion

You have now learned how to create, configure, and manage Kubernetes Jobs for running one-off tasks. This is a fundamental workload type for handling many of the automation and operational duties you'll face as a Platform Engineer.

Key Takeaways:

• Jobs are for finite tasks that run to completion, unlike Deployments which are for continuous services.
• A Job's Pod template must have a restartPolicy of Never or OnFailure.
• You can control execution flow with spec.completions (how many successes) and spec.parallelism (how many run at once).
• Build resilient Jobs using spec.backoffLimit for retries and spec.activeDeadlineSeconds to prevent indefinite execution.
• Remember to clean up completed Jobs, either automatically with spec.ttlSecondsAfterFinished or manually with kubectl delete.

Next Up:
We've seen how to run a task once. But what if you need to run a task on a recurring schedule, such as a daily backup or a weekly report? In our next lesson, we will build directly on what we've learned today to explore CronJobs.