Call today

Our journey to managing jenkins on aws eks

•

blogcontent

Updated on

0
(0)

To streamline your Jenkins operations on AWS EKS, here are the detailed steps for a robust, scalable setup:

👉 Skip the hassle and get the ready to use 100% working script (Link in the comments section of the YouTube Video) (Latest test 31/05/2025)

Check more on: How to Bypass Cloudflare Turnstile & Cloudflare WAF – Reddit, How to Bypass Cloudflare Turnstile, Cloudflare WAF & reCAPTCHA v3 – Medium, How to Bypass Cloudflare Turnstile, WAF & reCAPTCHA v3 – LinkedIn Article

Phase 1: Setting the Foundation

  1. Prepare your AWS environment: Ensure you have an active AWS account, sufficient IAM permissions, and familiarity with AWS CLI and kubectl.
  2. Establish your EKS Cluster:
    • Tools: Utilize eksctl highly recommended for declarative cluster management or AWS Management Console.
    • Command Example eksctl: 
      eksctl create cluster \
  --name jenkins-eks-cluster \
  --region us-east-1 \
  --version 1.28 \
  --nodegroup-name jenkins-workers \
  --node-type t3.medium \
  --nodes 3 \
  --nodes-min 1 \
  --nodes-max 5 \
  --with-oidc \
 --vpc-nat-mode Disable \ # Consider this for cost or network needs, otherwise keep default
 --ssh-access --external-ssh-allow-cidrs 0.0.0.0/0 # Restrict this heavily in production
    • Verification: kubectl get nodes to confirm cluster readiness.

Phase 2: Jenkins Deployment on EKS

  1. Persistent Storage Setup EFS Recommended:

    • Why: Jenkins needs persistent storage for its home directory, configuration, and build artifacts. EFS CSI driver allows dynamic provisioning.

    • Install EFS CSI Driver: Follow the official AWS EFS CSI driver documentation: https://docs.aws.amazon.com/eks/latest/userguide/efs-csi.html

      Amazon

    • Create EFS File System:

      Aws efs create-file-system –performance-mode generalPurpose –throughput-mode bursting –encrypted –tags Key=Name,Value=jenkins-efs

    • Mount Targets: Create mount targets in your EKS VPC subnets.

    • StorageClass: Define a StorageClass in Kubernetes referencing your EFS.

      # jenkins-efs-sc.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: jenkins-efs-sc
provisioner: efs.csi.aws.com
parameters:
  provisioningMode: efs-ap
 fileSystemId: fs-XXXXXXXXX # Your EFS File System ID
  directoryPerms: "700"
mountOptions:
  - tls
`kubectl apply -f jenkins-efs-sc.yaml`
  2. Jenkins Helm Chart Deployment:

Phase 3: Post-Deployment Configuration & Optimization

  1. Access Jenkins:
    • Get LoadBalancer URL: kubectl get svc -n jenkins jenkins -o jsonpath='{.status.loadBalancer.ingress.hostname}'
    • Retrieve Admin Password:
      printf $kubectl get secret -n jenkins jenkins -o jsonpath=”{.data.jenkins-admin-password}” | base64 –decode.echo
    • Log in and complete initial setup.
  2. Configure Kubernetes Plugin: In Jenkins, go to Manage Jenkins > Manage Nodes and Clouds > Configure Clouds > Add a new cloud > Kubernetes.
    • Kubernetes URL: https://kubernetes.default.svc.cluster.local default in-cluster service account
    • Jenkins URL: The external URL of your Jenkins LoadBalancer URL.
    • Pod Templates: Define your build environments as Pod Templates, specifying Docker images, resource requests/limits, and volume mounts.
  3. Implement Jenkins Configuration as Code JCasC: Store your Jenkins configurations in a Git repository e.g., jenkins.yaml and point Jenkins to it. This ensures reproducibility and version control.
    • Example jenkins.yaml:
      jenkins:
      clouds:
      – kubernetes:
      name: “kubernetes”

      jenkinsUrl: “http://jenkins-lb-url.elb.amazonaws.com”
      containerCap: 10
      namespace: “jenkins”
      podLabels:
      “purpose”: “jenkins-agent”
      templates:
      – name: “maven”

      image: “maven:3.8.6-openjdk-11”
      # … other pod template configurations

  4. Monitoring & Logging:
    • CloudWatch Container Insights: Enable for EKS cluster to get metrics on pods, nodes, and more.
    • Prometheus & Grafana: Deploy a Prometheus operator and Grafana within your EKS cluster for detailed Jenkins metrics.
    • Centralized Logging: Configure Fluent Bit to send Jenkins controller and agent logs to CloudWatch Logs or an external logging solution.

Phase 4: Security & Best Practices

  1. IAM Roles for Service Accounts IRSA: Assign fine-grained AWS permissions to Jenkins pods and agents directly, rather than relying on node instance profiles. This is a critical security enhancement.
  2. Network Policies: Implement Kubernetes Network Policies to control ingress/egress traffic to Jenkins pods and agents.
  3. VPC Private Link/Endpoint: For enhanced security and lower latency, consider using VPC Private Link for accessing AWS services from within your EKS cluster where applicable.
  4. Regular Backups: Snapshot your EFS volume regularly.
  5. Upgrade Strategy: Plan for regular Jenkins and EKS upgrades.

The Strategic Shift: Why Jenkins on EKS is a Game-Changer

Moving Jenkins, the venerable automation server, from traditional virtual machines or self-managed EC2 instances to AWS Elastic Kubernetes Service EKS isn’t just an infrastructure migration. it’s a strategic embrace of cloud-native principles. For organizations that are serious about scalability, resilience, and operational efficiency in their CI/CD pipelines, EKS provides a robust foundation that’s hard to beat. The journey, while requiring a learning curve, ultimately delivers a system that can handle fluctuating build demands with grace, reduce infrastructure overhead, and integrate seamlessly with other AWS services. It’s about building a future-proof CI/CD platform where agility meets stability. This shift fundamentally transforms Jenkins from a monolithic application into a dynamic, containerized workload, leveraging the orchestration power of Kubernetes to ensure high availability and optimal resource utilization. Imagine a Jenkins instance that can automatically scale its build agents up and down based on demand, recover from failures with minimal intervention, and provide isolated, reproducible build environments for every job – that’s the promise of Jenkins on EKS.

The Problem with Traditional Jenkins Deployments

Traditional Jenkins setups often hit bottlenecks that can cripple a growing development team’s productivity. These issues become particularly acute as the number of developers, projects, and concurrent builds increases. One of the most common challenges is resource contention. When Jenkins controller and agents run on the same or limited VMs, heavy builds can starve other jobs of CPU and memory, leading to slow build times and unreliable outcomes. Scaling is another major headache. manually provisioning and configuring new VMs or EC2 instances for agents is a time-consuming, error-prone process that doesn’t react quickly to sudden spikes in demand. Furthermore, maintenance overhead is significant, involving manual OS patching, Jenkins upgrades, and plugin management across multiple servers. Lack of isolation between build environments can lead to dependency conflicts or “works on my machine” syndromes, as shared agent environments might not be clean. Finally, achieving high availability in a traditional setup often requires complex, custom solutions with significant operational overhead, making the Jenkins instance a single point of failure. Data indicates that organizations spend up to 30% of their DevOps budget on infrastructure and maintenance for CI/CD tools, a significant portion of which can be attributed to managing traditional Jenkins deployments.

The Lure of Kubernetes: Benefits for CI/CD

Kubernetes, in essence, is an orchestration powerhouse that transforms infrastructure management. For CI/CD, its benefits are profound and directly address the shortcomings of traditional setups. The primary advantage is elastic scalability. Kubernetes can automatically provision and de-provision Jenkins build agents as temporary pods, scaling up during peak hours and scaling down when idle, leading to significant cost savings and optimal resource utilization. This dynamic scaling contrasts sharply with the static, often over-provisioned nature of VM-based agents. Secondly, Kubernetes offers unparalleled resilience and self-healing capabilities. If a Jenkins agent pod crashes, Kubernetes automatically restarts it or schedules a new one on a healthy node, ensuring continuous operation. This drastically improves the reliability of your CI/CD pipeline. Thirdly, resource isolation is inherent. Each Jenkins agent runs in its own dedicated pod, providing a clean, consistent, and reproducible environment for every build, eliminating “noisy neighbor” problems. This isolation is further enhanced by Kubernetes’s network policies and namespaces. Fourthly, declarative configuration via YAML files allows for infrastructure as code, making environments reproducible and version-controlled. This simplifies disaster recovery and ensures consistency across development, staging, and production environments. Lastly, Kubernetes provides seamless integration with other cloud services and monitoring tools, offering a holistic view of your CI/CD health. According to a 2023 survey by the Cloud Native Computing Foundation CNCF, over 96% of organizations are using or evaluating Kubernetes, with CI/CD identified as a top workload.

Crafting Your EKS Foundation: The Cluster Build-Out

The very first step in this transformative journey is to establish a robust and secure EKS cluster.

This isn’t just about spinning up a few EC2 instances.

It’s about configuring your cloud environment to be a reliable host for your critical CI/CD infrastructure.

Think of it as laying the foundation for a high-rise building – you need stable ground, strong footings, and proper plumbing before you even think about the structure above.

The choices made here regarding networking, node types, and security will have a cascading effect on the performance, cost, and maintainability of your Jenkins deployment.

A well-planned EKS cluster ensures that Jenkins has the stable, scalable, and secure environment it needs to operate at its best.

Prerequisites: Getting Your AWS Environment Ready

Before you even think about typing eksctl create cluster, ensure your AWS environment is properly prepped. This stage is akin to ensuring you have all your tools and materials ready before starting a complex build. Ignoring these prerequisites can lead to frustrating roadblocks and security vulnerabilities down the line. Web application testing checklist

  • Active AWS Account: This seems obvious, but ensure your account is active and has sufficient service limits for EKS, EC2, and other necessary services. New accounts might have default limits that need to be increased.
  • IAM Permissions: You’ll need an IAM user or role with administrative privileges to create and manage EKS clusters, VPCs, EC2 instances, and other related resources. Specifically, permissions for eks:*, ec2:*, iam:*, vpc:*, and cloudformation:* are typically required for cluster creation. Best practice dictates using the principle of least privilege for ongoing operations, but for initial setup, broader permissions might be necessary.
  • AWS CLI Installed and Configured: The AWS Command Line Interface is your primary tool for interacting with AWS services. Ensure it’s installed pip install awscli, configured with your credentials aws configure, and pointing to the correct region. Verify installation with aws --version.
  • kubectl Installed: This is the command-line tool for interacting with Kubernetes clusters. Install it and ensure it’s in your system’s PATH. You can download it from the official Kubernetes documentation.
  • eksctl Installed Highly Recommended: While you can use the AWS console or CloudFormation, eksctl is the official CLI for Amazon EKS and significantly simplifies cluster creation and management. It handles much of the boilerplate, including VPCs, subnets, and node groups. Install it using Homebrew brew install eksctl on macOS/Linux or follow the official installation guide.
  • helm Installed for Jenkins Deployment: Helm is the package manager for Kubernetes. You’ll use it to deploy Jenkins easily. Install it via Homebrew brew install helm or by downloading the binary.
  • Networking Knowledge VPC, Subnets, Security Groups: While eksctl automates much of this, a basic understanding of AWS networking is crucial for troubleshooting and advanced configurations. You should know the difference between public and private subnets, NAT Gateways, and security group rules.
  • Resource Tagging Strategy: Plan your AWS resource tags from the start. Consistent tagging helps with cost allocation, resource identification, and automation.

eksctl: Your Go-To for EKS Cluster Creation

When it comes to provisioning an EKS cluster, eksctl is undoubtedly the most user-friendly and robust tool available. It abstracts away much of the complexity of CloudFormation templates and manual configurations, allowing you to define your cluster declaratively. This means you specify what you want, and eksctl handles how it gets built. Using eksctl promotes an Infrastructure-as-Code IaC approach, making your cluster setup repeatable, version-controlled, and auditable.

Amazon

Let’s break down a typical eksctl command and its implications:

eksctl create cluster \
 --name jenkins-eks-cluster \ # A descriptive name for your cluster
 --region us-east-1 \ # AWS region where the cluster will be deployed
 --version 1.28 \ # Kubernetes version. Always use a supported, recent version. EKS versions are typically n-2 for security.
 --nodegroup-name jenkins-workers \ # Name for the default node group
 --node-type t3.medium \ # EC2 instance type for your worker nodes. Start with t3.medium or m5.large for Jenkins.
 --nodes 3 \ # Initial desired number of worker nodes
 --nodes-min 1 \ # Minimum number of nodes for autoscaling
 --nodes-max 5 \ # Maximum number of nodes for autoscaling enables cluster autoscaler implicitly
 --with-oidc \ # Crucial for IAM Roles for Service Accounts IRSA, a major security feature
 --vpc-nat-mode Disable \ # Optional Disables NAT Gateways if private subnets don't need internet egress, saving costs. Reconsider if your agents need external access e.g., pulling images from Docker Hub. Default is `Enable` which creates NAT Gateways.
 --ssh-access --external-ssh-allow-cidrs 0.0.0.0/0 \ # CAUTION! Allows SSH access to worker nodes. VERY IMPORTANT: Restrict `0.0.0.0/0` to your specific IP ranges in production e.g., your office VPN CIDR.
 --managed # Optional Uses EKS Managed Node Groups for easier updates and patching

Key Parameters Explained:

  • --name: Choose a clear, identifiable name.
  • --region: Select a region close to your users or other AWS resources.
  • --nodegroup-name: Logical grouping for your worker nodes. You can create multiple node groups later for different workloads e.g., GPU nodes for ML builds, Graviton nodes for cost savings.
  • --node-type: Crucial for performance and cost. t3.medium is a good starting point for lighter loads, m5.large or c5.large are better for compute-intensive builds. Consider AWS Graviton instances e.g., m6g.large for significant cost-performance improvements if your workloads are ARM-compatible most Jenkins agents will be. A recent study showed Graviton instances can offer up to 40% better price-performance compared to x86 instances.
  • --nodes, --nodes-min, --nodes-max: These parameters enable the Cluster Autoscaler by default, allowing your node group to dynamically scale based on pending pods. This is fundamental for cost optimization and handling fluctuating build demands.
  • --with-oidc: This is a non-negotiable for modern EKS security. It sets up an OpenID Connect OIDC provider for your cluster, enabling IAM Roles for Service Accounts IRSA. IRSA allows you to grant fine-grained AWS permissions directly to Kubernetes service accounts and thus to your Jenkins pods without distributing AWS credentials or relying on broader node instance profiles. This is a massive security improvement.
  • --vpc-nat-mode Disable: If your worker nodes primarily communicate within your VPC or pull images from ECR which can be done via VPC Endpoints, you might disable NAT Gateways to save costs. Each NAT Gateway can cost around $32.40 per month plus data processing charges. However, if your agents need to pull images from public Docker Hub or access external APIs, you’ll need NAT Gateways or an alternative egress path.
  • --ssh-access --external-ssh-allow-cidrs: Exercise extreme caution here. Allowing SSH access to worker nodes is useful for debugging, but 0.0.0.0/0 is a massive security risk. Always restrict this to known, secure IP ranges e.g., your corporate VPN CIDR or specific office IP addresses. In production, ideally, you wouldn’t need direct SSH access at all, relying on Kubernetes logs and kubectl debug.
  • --managed: EKS Managed Node Groups automate many operational tasks, including node health management, scaling, and most importantly, node updates. This significantly reduces the patching burden and ensures your underlying EC2 instances are kept up-to-date with security patches. For production, Managed Node Groups are the preferred approach.

Post-Creation Verification:

Once eksctl create cluster completes which can take 15-25 minutes, verify your cluster is operational:

kubectl get nodes

You should see a list of your worker nodes with a Ready status.

This confirms that Kubernetes is up and running and your worker nodes are connected. Your EKS foundation is now complete.

Jenkins on EKS: The Helm Deployment Strategy

With your EKS cluster humming along, the next crucial step is to deploy Jenkins itself. While manual deployments are possible, the Kubernetes ecosystem thrives on package managers, and for Jenkins, Helm is the undisputed champion. Helm allows you to define, install, and upgrade even the most complex Kubernetes applications with a single command, using pre-configured charts. The official Jenkins Helm chart is incredibly flexible, enabling you to customize nearly every aspect of your Jenkins deployment, from resource requests to plugin installations and persistent storage configurations. This declarative approach, enabled by Helm, is central to managing Jenkins on EKS efficiently. Integration tests on flutter apps

Securing Jenkins Data: The Role of Persistent Storage

Jenkins is an inherently stateful application. Its configuration, build history, plugins, and workspace data all reside in its home directory /var/jenkins_home. Without persistent storage, any Jenkins pod restart or redeployment would wipe out all your work, making your CI/CD pipeline highly unreliable. In the Kubernetes world, this means you need a PersistentVolume PV and a PersistentVolumeClaim PVC. For Jenkins on EKS, Amazon Elastic File System EFS is often the superior choice over single-node solutions like EBS.

Amazon

Why EFS is Ideal for Jenkins on EKS:

  1. Shared Access ReadWriteMany: EFS supports ReadWriteMany access mode, meaning multiple pods e.g., Jenkins controller and potential future agents if you configure them to share a workspace can mount the same file system simultaneously. While the Jenkins controller is usually single-instance, ReadWriteMany simplifies disaster recovery and potential blue/green deployments where a new Jenkins instance needs to pick up existing data. For Jenkins, this is crucial.
  2. Scalability: EFS file systems automatically scale up or down as you add or remove files, eliminating the need to provision storage capacity beforehand. This is a significant advantage over EBS, where you need to pre-allocate storage.
  3. High Availability and Durability: EFS is a regional service, storing data redundantly across multiple Availability Zones AZs within a region, offering high availability and durability. This is critical for a core service like Jenkins.
  4. Managed Service: AWS manages the underlying file system infrastructure, so you don’t have to worry about patching, backups, or scaling the file system itself.

Steps to Integrate EFS with Jenkins on EKS:

  1. Install AWS EFS CSI Driver: The EFS Container Storage Interface CSI driver allows Kubernetes to communicate with EFS. This is a one-time setup per cluster.

    • Follow the official AWS EFS CSI driver documentation: https://docs.aws.amazon.com/eks/latest/userguide/efs-csi.html. This typically involves creating an IAM OIDC provider for the cluster which --with-oidc handles, creating an IAM policy, and deploying the CSI driver using helm or kubectl.

    • `helm install aws-efs-csi-driver jenkins/aws-efs-csi-driver
      –namespace kube-system \

      –set controller.serviceAccount.create=true \

      –set controller.serviceAccount.name=efs-csi-controller-sa \

      –set image.repository=registry.k8s.io/sig-storage/efs-csi-driver \ Test websites with screen readers

      –set image.tag=v1.5.0` adjust tag to latest stable version.

  2. Create an EFS File System:

    aws efs create-file-system \
  --performance-mode generalPurpose \
 --throughput-mode bursting \ # Bursting is cost-effective for typical Jenkins workloads
 --encrypted \ # Always encrypt your data at rest
  --tags Key=Name,Value=jenkins-efs \
  --query 'FileSystemId' --output text
# Output will be something like fs-XXXXXXXXX
    • Throughput Mode: bursting is generally sufficient for most Jenkins workloads. For extremely high concurrency or very large repositories, consider provisioned throughput.

  3. Create EFS Mount Targets: EFS needs mount targets in the same subnets where your EKS worker nodes reside. These mount targets define network interfaces for EFS.

    Get your EKS VPC ID and subnet IDs

    VPC_ID=$aws eks describe-cluster –name jenkins-eks-cluster –query ‘cluster.resourcesVpcConfig.vpcId’ –output text

    SUBNET_IDS=$aws ec2 describe-subnets –filters “Name=vpc-id,Values=$VPC_ID” “Name=tag:eks:cluster-name,Values=jenkins-eks-cluster” –query ‘Subnets.SubnetId’ –output text

    Create security group for EFS allowing NFS traffic from EKS worker nodes

    EFS_SG_ID=$aws ec2 create-security-group –group-name jenkins-efs-sg –description “EFS SG for Jenkins EKS” –vpc-id $VPC_ID –query ‘GroupId’ –output text

    Get security group of your worker nodes often named like eks-cluster-sg-jenkins-eks-cluster-XXXXXXXX

    WORKER_SG_ID=$aws ec2 describe-security-groups –filters “Name=tag:eks:cluster-name,Values=jenkins-eks-cluster” “Name=tag:eks:instance-type,Values=node” –query ‘SecurityGroups.GroupId’ –output text

    Aws ec2 authorize-security-group-ingress –group-id $EFS_SG_ID –protocol tcp –port 2049 –source-group $WORKER_SG_ID

    Create mount targets for each subnet

    for subnet in $SUBNET_IDS. do

    echo “Creating mount target for subnet: $subnet” Testcafe vs cypress

    aws efs create-mount-target –file-system-id fs-XXXXXXXXX –subnet-id $subnet –security-groups $EFS_SG_ID
    done

  4. Define a Kubernetes StorageClass: This StorageClass tells Kubernetes how to dynamically provision storage for PVCs.

    # jenkins-efs-sc.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
 name: jenkins-efs-sc # This name will be referenced in your Jenkins Helm values
provisioner: efs.csi.aws.com
parameters:
 provisioningMode: efs-ap # Use Access Point mode for better access control
 fileSystemId: fs-XXXXXXXXX # REPLACE with your EFS File System ID
 directoryPerms: "700" # Permissions for the root directory created by the CSI driver
 # `basePath` can be used to mount a specific subdirectory within the EFS volume
mountOptions:
 - tls # Always mount with TLS encryption for data in transit


Apply it: `kubectl apply -f jenkins-efs-sc.yaml`

With the StorageClass defined, Kubernetes can now automatically provision a PersistentVolume on your EFS file system when Jenkins requests a PersistentVolumeClaim.

This ensures that your Jenkins data is durable, highly available, and scalable.

Helm Chart for Jenkins: Your Deployment Blueprint

Helm charts are pre-packaged applications for Kubernetes, simplifying deployment significantly.

The official Jenkins Helm chart is incredibly well-maintained and configurable.

Deploying Jenkins with Helm offers several advantages:

  • Reproducibility: Your entire Jenkins setup is defined in a values.yaml file, which can be version-controlled.
  • Customization: You can override default settings to tailor Jenkins to your specific needs.
  • Easy Upgrades: Helm makes upgrading Jenkins and its dependencies straightforward.

Deployment Steps:

  1. Add Jenkins Helm Repository: This makes the Jenkins chart available to Helm.

    Helm repo add jenkins https://charts.jenkins.io
    helm repo update Esop buyback worth 50 million

    This command ensures you have the latest chart versions.

  2. Create a Dedicated Namespace: It’s good practice to isolate Jenkins into its own Kubernetes namespace.
    kubectl create namespace jenkins

  3. Customize Helm Values jenkins-values.yaml: This is where you configure Jenkins. The values.yaml file is critical. Here’s a highly detailed example with explanations for key settings relevant to EKS:

    jenkins-values.yaml

    controller:

    Image configuration for the Jenkins controller pod

    image: “jenkins/jenkins”
    tag: “2.426.2-lts-jdk17” # Always use a specific, stable LTS version. Avoid ‘latest’.

    Resource requests and limits for the Jenkins controller pod

    Crucial for performance and stability. Adjust based on your workload.

    resources:
    requests:
    cpu: “1000m” # 1 CPU core
    memory: “2Gi” # 2 Gigabytes
    limits:
    cpu: “2000m” # Max 2 CPU cores
    memory: “4Gi” # Max 4 Gigabytes

    Java arguments for Jenkins controller JVM. Tune for memory.

    javaOpts: “-Xms1536m -Xmx3072m -XX:+UseG1GC -XX:MaxRAMPercentage=70.0” # Example: 1.5GB initial, 3GB max heap. Use G1GC for modern JVMs. MaxRAMPercentage is useful in container environments.

    Plugins to install on Jenkins startup. This ensures consistency.

    Versions are important for reproducibility and compatibility.

    installPlugins:
    – kubernetes:1.31.2 # Essential for Kubernetes dynamic agents
    – workflow-aggregator:2.7 # Core for Pipeline builds
    – git:4.15.0 # For Git SCM
    – pipeline-github-lib:1.10 # For GitHub integration in Pipelines
    – configuration-as-code:1653.v75053bc8624c # For JCasC
    – blueocean:1.27.7 # Modern Jenkins UI optional
    – aws-credentials:1.35 # For AWS credentials management
    – durable-task:533.v6ca_0f6787ec3 # For durable task execution
    – timestamper:1.22 # Adds timestamps to console output

    Persistent storage configuration for Jenkins home directory

    This leverages the EFS StorageClass created earlier.

    persistentVolumeClaim:
    storageClass: jenkins-efs-sc # MUST match the name of your EFS StorageClass
    accessModes: # Crucial for EFS, allows shared access
    size: 50Gi # Initial size of the PVC. EFS scales, but this reserves space.
    # existingClaim: “jenkins-pv-claim” # Use this if you have an existing PVC to reuse

    Service exposure for the Jenkins controller

    serviceType: LoadBalancer # Exposes Jenkins via an AWS Classic Load Balancer or NLB if using annotation Introducing test university

    You can also use NodePort or ClusterIP + Ingress Controller e.g., NGINX Ingress or AWS ALB Ingress

    Ingress is generally preferred for production for features like WAF, SSL termination, path-based routing.

    If using Ingress, change serviceType to ClusterIP and configure ingress below.

    service:

    annotations:

    service.beta.kubernetes.io/aws-load-balancer-type: “nlb” # For NLB

    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: “tcp”

    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: “true”

    Administration settings

    adminUser: admin # Default admin username change post-install

    adminPassword: “your-secure-password” # Omit this to auto-generate password, then retrieve from secret

    Agent configuration: Enables Jenkins to launch dynamic agents as Kubernetes pods

    agent:
    enabled: true # Enable Jenkins agents
    kubernetes:
    enabled: true # Ensure the Kubernetes plugin is configured to use the cluster
    # Other Kubernetes agent settings can be defined here, or primarily via JCasC/Jenkins UI.
    # e.g., defaultAgentResources:
    # requests:
    # cpu: “250m”
    # memory: “512Mi”
    # limits:
    # cpu: “1000m”
    # memory: “1Gi”
    # You’ll define specific pod templates within Jenkins itself via JCasC, ideally.

    Ingress configuration if you choose to use an Ingress controller instead of LoadBalancer service

    ingress:

    enabled: true

    className: “nginx” # Or “alb” if using AWS ALB Ingress Controller

    annotations:

    # Example for ALB Ingress

    # alb.ingress.kubernetes.io/scheme: internet-facing

    # alb.ingress.kubernetes.io/target-type: ip

    # alb.ingress.kubernetes.io/healthcheck-path: /login

    host: jenkins.yourdomain.com

    paths:

    – path: /

    pathType: Prefix

    Key values.yaml Considerations:

    • controller.image & tag: Always pin to a specific, stable LTS Long Term Support version of Jenkins. This ensures predictable behavior and easier upgrades.
    • controller.resources: Crucial for performance. Start with generous requests and limits, then monitor and fine-tune. Under-provisioning leads to slow UI and build failures. Over-provisioning wastes resources. Use CloudWatch Container Insights or Prometheus/Grafana to monitor CPU/memory usage.
    • controller.javaOpts: Fine-tune the JVM for a container environment. MaxRAMPercentage is excellent for letting the JVM intelligently use available memory within the pod’s limits. UseG1GC is the recommended garbage collector for modern JVMs.
    • controller.installPlugins: A fundamental aspect of reproducible Jenkins. List all necessary plugins here with their versions. This ensures that every time Jenkins is deployed or re-deployed, it has the exact set of plugins.
    • controller.persistentVolumeClaim: This is the bridge to your EFS storage. Ensure storageClass matches your EFS StorageClass name and accessModes is ReadWriteMany.
    • controller.serviceType:
      • LoadBalancer: Simple and quick for initial setup. AWS will provision a Classic Load Balancer or NLB/ALB if specific annotations are added. This is typically internet-facing.
      • ClusterIP + Ingress: Recommended for production. An Ingress controller like NGINX Ingress Controller or AWS ALB Ingress Controller provides more advanced features: SSL termination, custom domains, path-based routing, WAF integration, and often more cost-effective if sharing an Ingress with other services.
    • agent.enabled: true & agent.kubernetes: enabled: true: These ensure Jenkins is set up to leverage Kubernetes for dynamic agent provisioning. The actual pod templates for agents e.g., Maven, Node.js, Python build environments will be defined within Jenkins itself, ideally using Jenkins Configuration as Code JCasC.

  4. Deploy Jenkins with Helm:

    Helm install jenkins -n jenkins -f jenkins-values.yaml jenkins/jenkins
    This command will:

    • Create the Jenkins deployment, service, PVC, and other Kubernetes resources.
    • Download and install the specified Jenkins image.
    • Start the Jenkins controller pod.
    • Attach the EFS PersistentVolume.

Once the deployment is complete, you can monitor the pod status: kubectl get pods -n jenkins -w. Wait until the Jenkins controller pod is in a Running state and its containers are Ready.

This Helm-based deployment not only gets Jenkins up and running but also lays the groundwork for a robust, scalable, and manageable CI/CD system on AWS EKS, leveraging best practices for container orchestration and persistent storage.

Post-Deployment: Initial Setup and Optimization

With Jenkins successfully deployed on your EKS cluster, the immediate next steps involve accessing it for the first time, completing its initial setup, and then configuring it to dynamically launch build agents within your Kubernetes environment.

This phase is about transforming a raw Jenkins instance into a fully functional CI/CD engine that leverages the power of EKS.

It also includes critical steps for ensuring the stability, performance, and maintainability of your Jenkins setup. Localization testing on websites and apps

Accessing Jenkins and Initial Configuration

Once the Jenkins controller pod is running, you need to access its UI to complete the initial setup.

  1. Retrieve Jenkins LoadBalancer URL:

    If you used serviceType: LoadBalancer in your values.yaml, AWS will provision an Elastic Load Balancer. Get its DNS hostname:

    Kubectl get svc -n jenkins jenkins -o jsonpath='{.status.loadBalancer.ingress.hostname}’

    This will output a URL like a1234567890abcdef.elb.us-east-1.amazonaws.com. Open this URL in your web browser.

    Amazon

  2. Retrieve Admin Password:

    Jenkins generates an initial administrator password. Retrieve it from the Kubernetes secret:
    printf $kubectl get secret -n jenkins jenkins -o jsonpath=”{.data.jenkins-admin-password}” | base64 –decode.echo
    Copy this password.

  3. Complete Initial Setup:

    • Paste the retrieved password into the “Unlock Jenkins” page.
    • On the next screen, choose “Install suggested plugins.” This installs a comprehensive set of common and useful plugins, including the Kubernetes plugin, which is essential for dynamic agents.
    • Create your first admin user. Crucially, do NOT use admin again or the auto-generated password. Create a new, strong administrator username and password immediately.
    • Configure your Jenkins URL. This should be the LoadBalancer URL or your custom domain if you’re using an Ingress controller. This URL is vital for agents to connect back to the controller.

Configuring the Kubernetes Plugin for Dynamic Agents

The real power of Jenkins on EKS comes from its ability to dynamically provision build agents as Kubernetes pods. This is managed by the Kubernetes Plugin. 10 must have chrome extensions

  1. Navigate to Cloud Configuration:

    In Jenkins UI, go to Manage Jenkins > Manage Nodes and Clouds > Configure Clouds.

    Click “Add a new cloud” and select “Kubernetes.”

  2. Kubernetes Cloud Details:

    • Name: kubernetes a common convention.
    • Kubernetes URL: https://kubernetes.default.svc.cluster.local This is the internal service account URL that Jenkins, running within the EKS cluster, uses to communicate with the Kubernetes API server.
    • Jenkins URL: This is the URL where your Kubernetes agents can reach the Jenkins controller. This must be the external LoadBalancer URL or your Ingress URL e.g., http://your-jenkins-lb-url.elb.amazonaws.com or https://jenkins.yourdomain.com.
    • Kubernetes Namespace: jenkins or whatever namespace your Jenkins controller is running in. This is the namespace where agents will be launched.
    • Connection Timeout/Read Timeout: Default values are usually fine.
    • Credentials: Leave as “No credentials default Kubernetes service account” if your Jenkins controller pod’s service account has the necessary permissions to create pods in the jenkins namespace which it typically does by default if installed via Helm. Alternatively, you can use IRSA IAM Roles for Service Accounts to assign a specific IAM role to the Jenkins controller’s service account with eks:DescribeCluster, eks:ListNodegroups, eks:DescribeNodegroup, and kubernetes:create/delete pods permissions.

  3. Pod Templates Crucial for Build Environments:

    This is where you define the blueprints for your Jenkins agents.

Each “Pod Template” specifies the Docker image, resource requirements, environment variables, and volumes needed for a specific type of build.

*   Click "Add Pod Template" under the Kubernetes cloud configuration.
*   Name: A descriptive name e.g., `maven-builder`, `nodejs-runner`.
*   Labels: Essential! These labels e.g., `maven`, `nodejs` are used in your Jenkins Pipelines to select the appropriate agent. `agent { label 'maven' }`
*   Containers:
    *   Name: `jnlp` this is the mandatory container name for the Jenkins agent connector.
    *   Docker Image: `jenkins/inbound-agent:4.11.2-1-jdk11` or a `jdk17` version. This is the base image that connects to Jenkins.
    *   Add Container for build tools: This is where you define your actual build environment.
        *   Name: `maven` or `node`, `python`, etc.
        *   Docker Image: `maven:3.8.6-openjdk-11`, `node:16-alpine`, `python:3.9-slim`, etc. Choose images that contain your build tools.
        *   Always pull image: Recommended to ensure you're using the latest version of your build image.
        *   Resource Requests/Limits: Define CPU and memory for this specific container. This is crucial for avoiding resource starvation and managing costs. Under-provisioning leads to slow builds and OOMKills.
    *   Volumes: You might need to mount volumes e.g., hostPath for Docker socket if building Docker images, or `PersistentVolumeClaim` for shared caches.

*   Example Pod Template for a Maven Build:
    *   Labels: `maven`
    *   Containers:
        *   Name: `jnlp`
        *   Docker Image: `jenkins/inbound-agent:4.11.2-1-jdk11`
        *   Name: `maven-tools`
        *   Docker Image: `maven:3.8.6-openjdk-11`
        *   Resource Requests: `cpu: 500m`, `memory: 1Gi`
        *   Resource Limits: `cpu: 1000m`, `memory: 2Gi`



Click "Save" and test by running a simple pipeline that uses the defined labels.

Jenkins Configuration as Code JCasC: The Modern Approach

Manually configuring Jenkins via the UI, especially for agents, plugins, and security settings, is prone to errors, not reproducible, and hard to audit. Jenkins Configuration as Code JCasC, using the configuration-as-code plugin, solves this by defining your Jenkins configuration in YAML files, typically stored in a Git repository.

Benefits of JCasC:

  • Reproducibility: Your Jenkins setup can be rebuilt identically anywhere.
  • Version Control: Track changes, revert if necessary, and collaborate on configurations.
  • Auditable: See exactly what changes were made and by whom.
  • Automation: Apply configurations automatically on Jenkins startup.

Implementation Steps: Open source spotlight dexie js david fahlander

  1. Ensure configuration-as-code plugin is installed: Included in installPlugins in jenkins-values.yaml.

  2. Create a jenkins.yaml file: This file will contain your Jenkins configurations. A simple structure might look like this:

    jenkins.yaml example

    jenkins:
    clouds:
    – kubernetes:
    name: “kubernetes”
    jenkinsUrl: “http://your-jenkins-lb-url.elb.amazonaws.com” # Your external Jenkins URL
    containerCap: 10 # Max number of concurrent pods for this cloud
    namespace: “jenkins”
    # Optional: custom configuration for jnlp agent
    # jnlpCredentialsId: “jenkins-sa-secret” # If using specific credentials
    # podRetention: “OnFailure” # Keep pod around for debugging on failure
    templates:
    – name: “maven”

    image: “jenkins/inbound-agent:4.11.2-1-jdk11”
    label: “maven”
    containers:
    – name: “maven-tool”

    imagePullPolicy: “IfNotPresent”
    command: “cat”
    args:
    tty: true
    resources:
    requests:
    cpu: “500m”
    memory: “1Gi”
    limits:
    cpu: “1000m”
    memory: “2Gi”
    # Add more pod templates here for other build types
    – name: “nodejs”

    label: “nodejs”
    – name: “nodejs-tool”
    image: “node:16-alpine”

    cpu: “250m”
    memory: “512Mi”

    local:
    users:
    – id: “newadmin”
    password: “a_very_strong_password_hashed_or_managed_externally” # Best to manage this via secrets management
    authorizationStrategy:
    loggedInUsersCanDoAnything:
    allowAnonymousRead: false
    systemMessage: “Welcome to Jenkins on EKS!”

    More configurations like global tool configurations, email notifiers, etc.

  3. Mount jenkins.yaml into Jenkins:

    • Option 1 Recommended for production: Store jenkins.yaml in a Git repository e.g., AWS CodeCommit, GitHub, GitLab. Configure Jenkins to pull this repository on startup and load the configuration. This is usually done by adding a JCasC configuration to the initial values.yaml for Helm or via the Jenkins UI though less “as code”. Browserstack has acquired percy

    • Option 2 Simpler for initial setup: Create a Kubernetes ConfigMap from your jenkins.yaml and mount it into the Jenkins controller pod.

      Kubectl create configmap jenkins-casc-config –from-file=jenkins.yaml -n jenkins

      Then, modify your jenkins-values.yaml to mount this ConfigMap:

      … inside controller section …

      configScripts:
      jenkins-casc: |
      # Include content of jenkins.yaml here or use existingConfigMap
      existingConfigMap: jenkins-casc-config

      Then helm upgrade jenkins -n jenkins -f jenkins-values.yaml jenkins/jenkins.

      Once configured, in Jenkins UI, go to Manage Jenkins > Configuration as Code to verify the configuration.

By leveraging JCasC, your Jenkins environment becomes a declarative artifact, easy to manage, scale, and recover.

This is a significant step towards a truly cloud-native CI/CD platform.

Monitoring, Logging, and Observability on EKS

In a dynamic, containerized environment like Jenkins on EKS, “set it and forget it” is a recipe for disaster. Effective monitoring, logging, and observability MLO are paramount to understanding the health, performance, and operational status of your CI/CD pipeline. Without MLO, you’re flying blind, unable to diagnose issues, optimize resource usage, or predict bottlenecks. This section outlines essential strategies for gaining deep insights into your Jenkins deployment and the underlying EKS infrastructure.

Leveraging AWS CloudWatch Container Insights

CloudWatch Container Insights is a fully managed monitoring service provided by AWS that collects, aggregates, and summarizes metrics and logs from containerized applications and microservices. It’s specifically designed for EKS, ECS, Fargate, and Kubernetes. 200 million series b funding

Benefits:

  • Automatic Discovery: Automatically discovers containerized resources and starts collecting performance metrics.
  • Comprehensive Metrics: Gathers metrics like CPU utilization, memory utilization, disk I/O, network performance, and process counts at the cluster, node, pod, container, and service levels.
  • Centralized Logging: Can collect logs from all containers in your cluster and send them to CloudWatch Logs.
  • Dashboards and Alarms: Provides pre-built dashboards and allows you to set up alarms based on thresholds, enabling proactive problem detection.
  • Cost-Effective: It’s integrated with CloudWatch, and you only pay for the metrics and logs ingested.

Implementation:

  1. Enable Container Insights for your EKS cluster: This can be done via the AWS Management Console, AWS CLI, or eksctl.
    Using eksctl is straightforward:
    eksctl utils enable-secrets-encryption –cluster jenkins-eks-cluster –region us-east-1 # Enable KMS for logs

    To enable Container Insights for an existing cluster:

    Aws eks update-addon –cluster-name jenkins-eks-cluster –addon-name amazon-cloudwatch-observability –addon-version v1.0.0-eksbuild.1 –resolve-conflicts OVERWRITE # Or use latest version

    Amazon

    If the addon is not available, you might need to manually deploy the CloudWatch agent.

    The recommended way is via the AWS Distro for OpenTelemetry ADOT Collector.

    For logs, deploy Fluent Bit as a DaemonSet:

    Kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cloudwatch-agent-rbac.yaml

    Kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cloudwatch-agent.yaml
    kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/fluentd.yaml # For Fluentd, though Fluent Bit is often preferred

    Or for Fluent Bit:

    Kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/fluent-bit.yaml

    Confirm Fluent Bit or Fluentd pods are running in kube-system namespace.

  2. Monitor via CloudWatch Console: Once enabled, navigate to the CloudWatch console, then “Container Insights” under “Metrics.” You’ll see aggregated dashboards for your EKS cluster, nodes, namespaces including jenkins, pods, and containers. This provides a high-level overview of resource utilization and performance. Breakpoint 2021 speaker spotlight julia pottinger

Prometheus and Grafana: Deep-Dive Observability

For a more granular, customizable, and open-source observability stack, Prometheus for metrics collection and alerting and Grafana for visualization are the de-facto standards in the Kubernetes ecosystem.

Prometheus Benefits:

  • Pull-based Metrics: Prometheus scrapes metrics from configured targets Kubernetes pods, nodes, services.
  • Powerful Query Language PromQL: Allows for complex queries, aggregations, and alerting rules.
  • Extensibility: Many exporters are available for various technologies node exporter for hosts, cAdvisor for containers, JMX exporter for Java apps like Jenkins.

Grafana Benefits:

  • Rich Dashboards: Create interactive and visually appealing dashboards from various data sources, including Prometheus.
  • Alerting: Configure alerts based on metric thresholds.
  • Versatility: Can integrate with many other data sources beyond Prometheus.

  1. Deploy Prometheus Operator: The Prometheus Operator simplifies deploying and managing Prometheus instances and related components in Kubernetes.

    Helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

    Helm install prometheus prometheus-community/kube-prometheus-stack -n monitoring –create-namespace \

    –set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \

    –set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false
    –set grafana.ingress.enabled=true
    –set grafana.ingress.hostname=grafana.yourdomain.com # Set your Grafana URL
    This single Helm chart typically installs:

    • Prometheus Operator
    • Prometheus itself
    • Alertmanager for handling alerts
    • Grafana
    • kube-state-metrics for Kubernetes API metrics
    • node-exporter for node-level metrics

  2. Expose Jenkins Metrics:

    The Jenkins controller exposes its metrics via the metrics endpoint usually /metrics or /jenkins/metrics if behind a path-based ingress. You’ll need to configure Prometheus to scrape these metrics. How to install ipa on iphone

    • JMX Exporter: For more detailed JVM metrics from Jenkins, you can use the Jenkins JMX Exporter plugin or integrate a sidecar JMX exporter container with your Jenkins pod.
    • ServiceMonitor/PodMonitor: The Prometheus Operator uses ServiceMonitor or PodMonitor custom resources to define scraping targets.

      jenkins-servicemonitor.yaml example

      apiVersion: monitoring.coreos.com/v1
      kind: ServiceMonitor
      name: jenkins-servicemonitor
      labels:
      app.kubernetes.io/name: jenkins
      release: prometheus # Match the release label of your Prometheus installation
      spec:
      selector:
      matchLabels:
      app.kubernetes.io/name: jenkins # Match the label on your Jenkins service
      namespaceSelector:
      matchNames:
      – jenkins # The namespace where Jenkins is deployed
      endpoints:
      – port: http # The name of the port exposed by Jenkins service e.g., ‘http’
      path: /metrics # The path to Jenkins metrics endpoint
      interval: 30s
      Apply this: kubectl apply -f jenkins-servicemonitor.yaml -n jenkins

  3. Grafana Dashboards:

    • Access Grafana via its Ingress URL e.g., grafana.yourdomain.com. Default credentials are often admin/prom-operator. Change immediately.
    • Add Prometheus as a data source.
    • Import pre-built Jenkins dashboards search Grafana Labs’ dashboard repository for “Jenkins Kubernetes” or create your own. Dashboards covering CPU, memory, concurrent builds, queue size, and agent status are essential.

Centralized Logging with Fluent Bit and CloudWatch Logs

Containerized applications produce logs, and these logs are critical for debugging, auditing, and understanding application behavior.

Relying solely on kubectl logs is not scalable for production.

Fluent Bit is a lightweight and highly efficient log processor and forwarder. It’s often deployed as a DaemonSet on each EKS node, collecting logs from all containers and forwarding them to a centralized logging solution. CloudWatch Logs is a common destination for logs in AWS.

  1. Deploy Fluent Bit as a DaemonSet:

    • Fluent Bit needs appropriate IAM permissions to put logs into CloudWatch Logs. Use IRSA IAM Roles for Service Accounts to assign an IAM role with logs:CreateLogGroup, logs:CreateLogStream, logs:PutLogEvents, and logs:DescribeLogStreams permissions to the Fluent Bit service account.

    • Deploy Fluent Bit, typically using a Helm chart or official Kubernetes manifests.

      Helm repo add fluent https://fluent.github.io/helm-charts
      helm repo update

      Helm install fluent-bit fluent/fluent-bit -n kube-system
      –set serviceAccount.create=true \

      –set serviceAccount.name=fluent-bit-sa \ Breakpoint highlights testing at scale

      –set serviceAccount.annotations.”eks.amazonaws.com/role-arn”=”\arn:aws:iam::YOUR_ACCOUNT_ID:role/FluentBitEKSIRSA”
      –set cloudWatch.region=us-east-1 \

      –set cloudWatch.logGroupName=/eks/jenkins-eks-cluster/container-logs \

      –set cloudWatch.logStreamPrefix=jenkins-
      –set resources.limits.cpu=”100m”
      –set resources.limits.memory=”100Mi”

      Ensure you create the IAM role FluentBitEKSIRSA with the necessary CloudWatch permissions and attach it to the fluent-bit-sa service account.

  2. View Logs in CloudWatch Logs:

    Once Fluent Bit is running, logs from your Jenkins controller and agent pods will appear in the specified CloudWatch Log Group e.g., /eks/jenkins-eks-cluster/container-logs, organized by log streams e.g., jenkins-jenkins-controller-pod-id, jenkins-jenkins-agent-pod-id. You can search, filter, and analyze these logs within the CloudWatch console.

By implementing these MLO strategies, you gain comprehensive visibility into your Jenkins on EKS environment, enabling you to proactively identify and resolve issues, optimize resource usage, and ensure the reliability and efficiency of your CI/CD pipelines.

Security Best Practices for Jenkins on EKS

Securing your CI/CD platform is paramount.

A compromised Jenkins instance can be a gateway to your source code, production environments, and sensitive data.

Running Jenkins on EKS provides a robust security foundation, but only if you adhere to Kubernetes and AWS security best practices. Handling tabs in selenium

This section covers critical security measures that go beyond basic setup.

IAM Roles for Service Accounts IRSA: Fine-Grained Permissions

IRSA is one of the most significant security enhancements for EKS. Traditionally, pods on an EKS node inherited the IAM permissions of the underlying EC2 instance role. This meant if one pod needed S3 access, all pods on that node implicitly had S3 access, leading to overly permissive roles and a larger attack surface. IRSA solves this by allowing you to associate a specific IAM role directly with a Kubernetes Service Account, which your pods then use.

How IRSA enhances security:

  • Least Privilege: Grant only the specific AWS permissions required by individual pods. For example, your Jenkins agent pod might need S3 permissions to store artifacts, while your Jenkins controller might need EC2 permissions to describe instances though ideally it shouldn’t be directly interacting with EC2.
  • Eliminates Credential Distribution: No need to hardcode AWS access keys/secrets into containers or configuration files. The credentials are automatically injected by the EKS Pod Identity Webhook.
  • Auditability: Easier to audit which specific pod or service account is performing which AWS API calls.

  1. Ensure OIDC Provider is Enabled for your Cluster: This was covered in the eksctl create cluster --with-oidc step. Verify it:

    Aws eks describe-cluster –name jenkins-eks-cluster –query ‘cluster.identity.oidc.issuer’ –output text

  2. Create IAM Policy: Define an IAM policy with the minimum necessary permissions. For example, for a Jenkins agent that needs to upload build artifacts to S3:

    # jenkins-agent-s3-policy.json
{
    "Version": "2012-10-17",
    "Statement": 
        {
            "Effect": "Allow",
            "Action": 
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:ListBucket"
            ,
            "Resource": 


               "arn:aws:s3:::your-jenkins-artifacts-bucket",
               "arn:aws:s3:::your-jenkins-artifacts-bucket/*"
            
        }
    
}


Create the policy: `aws iam create-policy --policy-name JenkinsAgentS3AccessPolicy --policy-document file://jenkins-agent-s3-policy.json`

  3. Create IAM Role and Associate with Kubernetes Service Account: eksctl makes this easy.
    eksctl create iamserviceaccount
    –cluster jenkins-eks-cluster
    –namespace jenkins
    –name jenkins-agent-sa \

    –attach-policy-arn arn:aws:iam::YOUR_ACCOUNT_ID:policy/JenkinsAgentS3AccessPolicy
    –override-existing-serviceaccounts \ # Use with caution, only if you know what you’re doing
    –approve
    This command does three things:

    • Creates a Kubernetes Service Account named jenkins-agent-sa in the jenkins namespace if it doesn’t exist.
    • Creates a corresponding IAM role in AWS.
    • Establishes the trust relationship between the IAM role and the OIDC provider, allowing the service account to assume the role.
    • Adds an annotation to the service account: eks.amazonaws.com/role-arn: arn:aws:iam::YOUR_ACCOUNT_ID:role/EKS-generated-role-name.

  4. Configure Pod to Use Service Account:

    Amazon

    In your Jenkins Pod Templates either via JCasC or Jenkins UI, specify the serviceAccountName:

    Part of your JCasC jenkins.yaml or Kubernetes Pod Template

    templates:

    • name: “s3-builder”
      # …
      serviceAccountName: jenkins-agent-sa # IMPORTANT!
      containers:

      • name: “builder”
        image: “my-custom-builder-image”

      …

    Now, any build job using the s3-builder agent will automatically have access to S3 based on the JenkinsAgentS3AccessPolicy, and only that specific agent.

Kubernetes Network Policies: Micro-Segmentation

Kubernetes Network Policies provide firewall-like rules for controlling communication between pods, namespaces, and external endpoints. They enable micro-segmentation, ensuring that pods can only communicate with what they explicitly need to.

Why Network Policies?

  • Reduced Attack Surface: Limit lateral movement of attackers within the cluster.
  • Compliance: Meet regulatory requirements for network segmentation.
  • Isolation: Prevent accidental or malicious communication between different application components or environments.

Implementation Example:

  1. Enable Network Policy Enforcement: Some CNI plugins like Calico or Cilium are required for Network Policy enforcement. AWS VPC CNI does not enforce Network Policies by default. you need to install a network policy engine like Calico.

    Install Calico example

    Kubectl apply -f https://docs.tigera.io/archive/v3.25/manifests/tigera-operator.yaml

    Kubectl apply -f https://docs.tigera.io/archive/v3.25/manifests/custom-resources.yaml

  2. Define a Network Policy:

    jenkins-network-policy.yaml

    apiVersion: networking.k8s.io/v1
    kind: NetworkPolicy
    name: jenkins-controller-policy
    namespace: jenkins # Apply to the Jenkins namespace
    spec:
    podSelector:
    matchLabels:
    app.kubernetes.io/component: controller # Selects the Jenkins controller pod
    policyTypes:
    – Ingress
    – Egress
    ingress:
    # Allow ingress from the LoadBalancer or Ingress Controller
    – from:
    – podSelector: {} # Allow from all pods if your ingress controller is in same namespace
    – namespaceSelector:
    matchLabels:
    name: ingress-nginx # Or your ingress controller namespace
    podSelector: {}
    ports:
    – protocol: TCP
    port: 8080 # Jenkins default port
    egress:
    # Allow egress to Kubernetes API server
    – to:
    – ipBlock:
    cidr: 0.0.0.0/0 # Restrict this to EKS control plane CIDR if possible
    port: 443 # Kubernetes API server port
    # Allow egress to EFS NFS port
    cidr: 10.0.0.0/8 # Or your specific VPC CIDR for EFS
    port: 2049 # NFS port for EFS
    # Allow egress to external Git repositories e.g., GitHub, GitLab
    cidr: 0.0.0.0/0 # Broad, but needed for external Git. Refine with specific IPs/CIDRs if possible.
    port: 443
    port: 22 # For SSH Git
    # Deny all other egress by default implied by policyTypes: Egress and limited rules
    Apply this: kubectl apply -f jenkins-network-policy.yaml

This policy ensures that the Jenkins controller can only communicate with the Kubernetes API, EFS, external Git repositories, and is only accessible from your ingress.

Similarly, define policies for Jenkins agents to limit their network access to only what’s necessary for builds e.g., pulling code, pushing artifacts, accessing internal services.

Container Security and Image Management

The security of your Jenkins agents starts with the Docker images they run.

  • Use Minimal Base Images: Start with slim or alpine-based images e.g., alpine, debian-slim to reduce the attack surface.
  • Scan Images for Vulnerabilities: Integrate image scanning tools e.g., AWS ECR image scanning, Clair, Trivy, Snyk into your image build pipeline. Scan images before they are pushed to your registry and before they are deployed.
  • Regularly Update Images: Keep your Jenkins controller and agent images up-to-date to patch known vulnerabilities. Automate this process using tools like Renovate or Dependabot.
  • Private Container Registry: Store your custom Jenkins agent images in a private, secure registry like AWS Elastic Container Registry ECR. Integrate ECR with your EKS cluster using IRSA for pulling images securely.
  • Immutable Infrastructure: Treat your Jenkins controller and agent pods as immutable. Don’t make manual changes inside running containers. If a change is needed, rebuild the image and redeploy the pod.

Jenkins-Specific Security Enhancements

Beyond Kubernetes and AWS layer security, Jenkins itself requires diligent security management.

  • Strong Authentication:
    • Disable default admin user: Create a new admin user with a strong password during initial setup.
    • Integrate with Identity Provider: Use plugins to integrate Jenkins with your corporate SSO e.g., AWS SSO, Okta, Azure AD, LDAP for centralized user management and stronger authentication mechanisms like MFA.
  • Role-Based Access Control RBAC:
    • Matrix Authorization Strategy: Use the Matrix Authorization Strategy plugin to define fine-grained permissions for different users or groups e.g., developers can only view builds, managers can approve deployments, administrators can manage Jenkins settings.
    • Folder-based Authorization: Organize jobs into folders and apply specific permissions to each folder.
  • Credential Management:
    • Jenkins Credentials Plugin: Use the built-in Jenkins Credentials plugin to store sensitive credentials API keys, SSH keys, passwords securely. Encrypt them.
    • Secrets Management with Kubernetes/Vault: For highly sensitive credentials, integrate Jenkins with a centralized secrets management solution like AWS Secrets Manager, AWS Parameter Store, or HashiCorp Vault. Use the Kubernetes Secrets Store CSI driver to mount secrets directly into pods.
  • Security Scans for Jenkins: Regularly scan your Jenkins instance for common vulnerabilities using tools like OWASP ZAP or specialized Jenkins security scanners.
  • Pipeline Security:
    • Input Steps: Use input steps in pipelines to require manual approval for sensitive operations e.g., production deployments.
    • Sandbox Mode: Run untrusted Groovy scripts in a sandbox environment if allowing non-admins to define pipelines.
    • Secrets Masking: Ensure sensitive information is masked in build logs.
  • Audit Logging: Enable comprehensive audit logging in Jenkins to track who did what and when. Send these logs to your centralized logging solution CloudWatch Logs.
  • Regular Backups: Implement a robust backup strategy for your Jenkins EFS volume to enable quick recovery in case of data loss or corruption.

By systematically applying these security best practices, you can significantly harden your Jenkins on EKS environment, protecting your CI/CD pipelines and the critical assets they manage.

Cost Optimization for Jenkins on EKS

Running Jenkins on AWS EKS offers immense benefits in scalability and resilience, but without careful planning, it can also lead to spiraling cloud costs. Optimizing your EKS infrastructure for Jenkins involves a combination of smart resource allocation, leveraging Kubernetes features, and utilizing AWS cost-saving mechanisms. The goal is to maximize performance while minimizing your AWS bill. Data from various cloud spend reports indicates that up to 30% of cloud spend is wasted due to inefficient resource provisioning.

Leveraging Cluster Autoscaler and Karpenter

Dynamic scaling is the cornerstone of cost optimization for burstable workloads like CI/CD.

  • Cluster Autoscaler: This Kubernetes component automatically adjusts the number of nodes in your EKS cluster based on the aggregate resource requirements of pending pods. If Jenkins needs more agents than available nodes, the Cluster Autoscaler requests new nodes from EC2. If nodes become underutilized, it safely drains and terminates them.

    • How it saves cost: Prevents over-provisioning nodes for peak demand, only paying for compute when actively needed.
    • Configuration: You’ve already enabled the basic Cluster Autoscaler with --nodes-min and --nodes-max in eksctl. Ensure the Cluster Autoscaler has the correct IAM permissions autoscaling:DescribeAutoScalingGroups, autoscaling:SetDesiredCapacity, autoscaling:TerminateInstanceInAutoScalingGroup, ec2:DescribeLaunchTemplates, etc..

  • Karpenter: A newer, highly efficient open-source node provisioning project built by AWS. Karpenter is designed to provision the right node at the right time based on the cluster’s needs. It often out-performs Cluster Autoscaler by:

    • Faster Scaling: Direct interaction with EC2, bypassing Auto Scaling Groups.
    • Optimal Node Selection: Chooses the most cost-effective instance type and size for pending pods. It can combine different instance types, families e.g., Graviton, and purchasing options Spot Instances dynamically.
    • Consolidation: More aggressively consolidates pods onto fewer, larger nodes to terminate underutilized ones.
    • How it saves cost: By provisioning precisely what’s needed and optimizing instance selection, Karpenter can lead to significant cost reductions, potentially 20-30% on compute.

    Implementing Karpenter:

    1. Install Karpenter: Follow the official Karpenter documentation for EKS. It involves creating an IAM role, service account, and deploying the Karpenter controller via Helm.
    2. Define Provisioners: Instead of node groups, you define Karpenter Provisioner custom resources.

      karpenter-provisioner.yaml

      apiVersion: karpenter.sh/v1beta1
      kind: Provisioner
      name: default
      requirements:
      – key: “karpenter.sh/capacity-type”
      operator: In
      values: # Prioritize Spot Instances for cost savings
      – key: “kubernetes.io/arch”
      values: # Allow both x86 and Graviton arm64
      – key: “kubernetes.io/os”
      values:

      – key: “karpenter.k8s.aws/instance-category”
      values: # General purpose, compute, memory optimized

      – key: “karpenter.k8s.aws/instance-generation”
      operator: Gt # Greater than
      values: # Prefer newer generation instances
      limits:
      resources:
      cpu: 100 # Total CPU limit for this provisioner
      providerRef:
      name: default # References the AWSNodeTemplate

      ttlSecondsAfterEmpty: 60 # Terminate nodes after 60 seconds if empty
      ttlSecondsAfterAllocation: 604800 # Terminate nodes after 7 days regardless of activity for regular refresh

      apiVersion: karpenter.k8s.aws/v1beta1
      kind: AWSNodeTemplate
      amiFamily: AL2 # Amazon Linux 2
      instanceProfile: KarpenterNodeInstanceProfile-jenkins-eks-cluster # IAM Instance Profile created by Karpenter setup
      subnetSelector:
      eks.amazonaws.com/cluster-name: jenkins-eks-cluster # Select subnets where your EKS worker nodes should run
      securityGroupSelector:
      eks.amazonaws.com/cluster-name: jenkins-eks-cluster # Select security groups for worker nodes
      tags:
      Name: KarpenterNode # Custom tags for your nodes
      # … other tags

    With Karpenter, you no longer define node-type, nodes-min, nodes-max in eksctl for your worker nodes, as Karpenter handles it.

    Amazon

You primarily rely on pod resource requests to drive node provisioning.

Right-Sizing Jenkins Controller and Agent Resources

This is the single most impactful cost optimization for any containerized application. Many over-provision resources “just in case,” leading to significant waste.

  • Jenkins Controller:
    • Monitor relentlessly: Use CloudWatch Container Insights or Prometheus/Grafana to track CPU and memory usage of your jenkins-controller pod over a period of high and low activity.
    • Set requests and limits wisely:
      • requests: The guaranteed minimum resources the pod will receive. Set this close to average usage.
      • limits: The hard maximum resources the pod can consume. Set this slightly above peak usage.
      • If requests are too low, the pod might get throttled or starve. If limits are too low, the pod might be OOMKilled. If both are too high, you pay for unused resources.
      • Typical starting point: cpu: 1000m 1 core, memory: 2Gi. Adjust based on your build concurrency and Jenkins UI activity. For example, a Jenkins managing 20-30 concurrent medium-sized builds might need 2-4 cores and 4-8Gi memory.
  • Jenkins Agents:
    • Heterogeneous Pod Templates: Different build types e.g., Java compile vs. Node.js linting have vastly different resource needs. Create specific Pod Templates for each build type and right-size their requests and limits for their primary container e.g., maven-tool, nodejs-tool.
    • Memory is often the bottleneck: Especially for Java-based builds Maven, Gradle or large Node.js projects. Monitor memory usage carefully.
    • CPU Bursting: Kubernetes handles CPU requests/limits slightly differently CPU is compressible. Ensure you provide enough requests to avoid throttling, but limits can be set higher to allow bursting when needed.
    • Cost Impact: If you have 50 agent pods running, and each is over-provisioned by 500m CPU and 1Gi memory, that’s equivalent to 25 wasted CPU cores and 50Gi of memory. This quickly adds up!

Leveraging Spot Instances for Agents

Spot Instances are unused EC2 capacity available at significantly discounted prices often 60-90% off On-Demand prices. While they can be interrupted with a two-minute warning, Jenkins agents are perfect candidates because:

  • Idempotent Builds: Jenkins builds should be restartable and idempotent. If a Spot Instance is interrupted, Kubernetes will simply reschedule the agent pod on another available node Spot or On-Demand.

  • Short-lived Nature: Jenkins agent pods are typically short-lived spin up for a build, spin down after.

  • Cost Savings: The potential savings are enormous, making Spot Instances a primary tool for cost optimization.

  • With Cluster Autoscaler: Configure your EKS Managed Node Groups or self-managed Auto Scaling Groups to use a mix of On-Demand and Spot Instances, or dedicate a separate node group purely for Spot.

  • With Karpenter Recommended: Karpenter natively supports prioritizing Spot Instances as shown in the Provisioner example above: karpenter.sh/capacity-type: . It will automatically try to provision Spot Instances first and only fall back to On-Demand if Spot capacity is unavailable or unreliable.

EFS Cost Optimization

While EFS is excellent for Jenkins, consider its cost implications:

  • Performance Mode: General Purpose is typically fine. Max I/O is for extreme throughput needs and is more expensive.
  • Throughput Mode: Bursting is usually cost-effective for Jenkins, which has periods of high I/O cloning repos, saving artifacts and idle times. Provisioned throughput is for consistent, high-throughput requirements and costs more. Only provision if you’ve hit performance bottlenecks with Bursting.
  • Lifecycle Management: Implement EFS lifecycle policies to automatically move infrequently accessed data to EFS Infrequent Access EFS IA storage class, which is cheaper. Jenkins data can grow significantly over time.
    • Data that hasn’t been accessed for X days e.g., 30 days can be automatically moved to EFS IA. This can save up to 92% on storage costs for cold data.
    • Configure this in the EFS console for your jenkins-efs file system.

Graviton Instances for Worker Nodes

AWS Graviton processors, based on ARM architecture, offer a significantly better price-performance ratio up to 40% better compared to comparable x86 instances.

Considerations:

  • Image Compatibility: Ensure your Jenkins agent Docker images and the applications they run are compatible with ARM64 architecture. Most common open-source tools Maven, Node.js, Python, Git have ARM64-compatible Docker images available.
  • Transition Strategy: Start by introducing a new node group if using Cluster Autoscaler or add arm64 to your Karpenter Provisioner requirements, then slowly migrate your agent workloads to these instances.
  • Multi-architecture Images: If you need to support both x86 and ARM, build multi-architecture Docker images using buildx.

By strategically applying these cost optimization techniques, you can ensure that your Jenkins on EKS platform is not only powerful and resilient but also fiscally responsible, maximizing your return on investment in cloud infrastructure.

High Availability and Disaster Recovery

For a mission-critical CI/CD platform like Jenkins, high availability HA and disaster recovery DR are non-negotiable.

An outage in your Jenkins environment can halt development, delay releases, and lead to significant financial losses.

Running Jenkins on EKS inherently provides some level of resilience, but achieving true HA and DR requires careful architectural planning and configuration, extending beyond just the core Jenkins application to its underlying infrastructure.

EKS Cluster Resilience Across Availability Zones

EKS is a regional service, and its control plane is inherently highly available, distributed across multiple Availability Zones AZs within a region.

However, the worker nodes where your Jenkins controller and agents run need to be explicitly configured for HA.

  • Multi-AZ Node Groups: When creating your EKS cluster with eksctl, ensure your node groups span multiple Availability Zones. eksctl typically does this by default if you don’t specify single subnets. This means if one AZ experiences an outage, your pods can be rescheduled onto nodes in another healthy AZ.

  • Example eksctl Configuration for Multi-AZ:

    eks-cluster-config.yaml

    apiVersion: eksctl.io/v1alpha5
    kind: ClusterConfig
    name: jenkins-eks-cluster
    region: us-east-1
    version: “1.28”
    vpc:

    If you specify subnets, ensure they are in different AZs

    You can let eksctl create them too, it will by default span AZs

    publicAccessCIDRs: # Restrict this

    privateAccessCIDRs: # If using private endpoints

    managedNodeGroups:

    • name: jenkins-workers
      instanceType: m5.large
      desiredCapacity: 3 # Spread across AZs
      minSize: 1
      maxSize: 5
      subnets:

      • subnet-0abcdef1234567890 # Subnet in AZ1
      • subnet-0fedcba9876543210 # Subnet in AZ2
      • subnet-0123456789abcdef0 # Subnet in AZ3

        … other configurations

  • Pod Anti-Affinity: For your Jenkins controller which is a single pod deployment, configure Pod Anti-Affinity to ensure it’s not scheduled on the same node or even the same AZ as other critical components.

    • Use topologyKey: kubernetes.io/hostname to prevent scheduling on the same node.
    • Use topologyKey: topology.kubernetes.io/zone to prevent scheduling in the same AZ requires multiple nodes per AZ.

    Part of your jenkins-values.yaml for controller

    affinity:
    podAntiAffinity:

      preferredDuringSchedulingIgnoredDuringExecution:
     - weight: 100
       podAffinityTerm:
         labelSelector:
           matchLabels:
            app.kubernetes.io/component: controller # The Jenkins controller pod label
        topologyKey: kubernetes.io/hostname # Ensure different nodes
     - weight: 50


            app.kubernetes.io/component: controller
        topologyKey: topology.kubernetes.io/zone # Spread across AZs

    …

Leveraging EFS for Jenkins Data Durability

As discussed, EFS is a cornerstone of Jenkins HA on EKS.

  • Regional Service: EFS data is stored redundantly across multiple AZs within a region, providing high durability. If an AZ goes down, your EFS volume is still available from other AZs.
  • ReadWriteMany: EFS supports ReadWriteMany access mode, which is crucial for potential future HA scenarios where Jenkins might be configured for active-passive or even limited active-active setups though active-active for Jenkins controller itself is complex due to its stateful nature.
  • Backups: Despite EFS’s durability, always implement a robust backup strategy.
    • AWS Backup: Use AWS Backup to create automated, scheduled backups of your EFS file system. This allows for point-in-time recovery. Configure retention policies e.g., daily backups for 7 days, weekly for 4 weeks, monthly for 1 year.
    • Cross-Region Replication: For true disaster recovery protection against regional outages, configure EFS cross-region replication to asynchronously replicate your Jenkins EFS data to a different AWS region.

Jenkins Controller Resiliency Single vs. Active-Passive

The Jenkins controller is typically a single point of failure in standard deployments.

  • Single-Instance Controller Highly Recommended for EKS:
    • Pros: Simpler to manage, avoids complexities of distributed state.
    • HA through EKS: EKS ensures the controller pod is always running. If the node fails, Kubernetes reschedules the pod to a healthy node, and since /var/jenkins_home is on EFS, the new pod mounts the existing data.
    • Recovery Time: Recovery time is limited to the time it takes for a new pod to start and initialize typically minutes.
  • Active-Passive Warm Standby/Cold Standby:
    • Manual Failover: In a true DR scenario e.g., regional outage, you would spin up a new EKS cluster in your DR region, deploy Jenkins, and restore the EFS backup or leverage cross-region replicated EFS. This is a manual process.
    • Automated Failover: Highly complex for Jenkins due to its state. It involves replicated databases if using one, shared storage, and a mechanism to detect failure and promote a standby. Generally not recommended for most organizations due to complexity.

Key DR Planning Steps:

  1. Define Recovery Point Objective RPO: How much data loss are you willing to tolerate? e.g., last 24 hours of data. This determines your backup frequency.
  2. Define Recovery Time Objective RTO: How quickly must Jenkins be operational again after an outage? e.g., 4 hours. This determines your DR strategy warm vs. cold standby.
  3. Regular DR Drills: Periodically test your disaster recovery procedures. Practice restoring Jenkins from a backup in a separate environment. This is crucial for verifying your RTO and RPO. According to industry statistics, over 50% of organizations that don’t conduct regular DR drills find their DR plans fail during an actual disaster.

Immutable Infrastructure and Configuration as Code

These practices are not just for efficiency. they are fundamental to HA and DR.

  • Immutable Infrastructure: Never manually modify a running Jenkins controller or agent pod. If a change is needed, build a new Docker image and deploy it. This ensures that your deployments are consistent and reproducible. If a disaster strikes, you can quickly spin up an identical environment.
  • Jenkins Configuration as Code JCasC: Store your entire Jenkins configuration plugins, credentials, agent templates, security settings in version control using JCasC.
    • DR Benefit: If your primary Jenkins instance is lost, you can redeploy a new Jenkins controller from scratch in any environment including your DR region and simply point it to your JCasC repository to quickly reconfigure itself. This significantly reduces RTO.
    • Secret Management: Integrate JCasC with an external secrets manager e.g., AWS Secrets Manager for sensitive data. Do not commit secrets to Git.

Externalizing Dependencies

Reduce the number of single points of failure within your Jenkins instance by externalizing components:

  • External Database if applicable: While Jenkins uses a built-in H2 database by default, for very large instances, some might migrate to an external database like PostgreSQL e.g., using AWS RDS. Ensure this database is also highly available and backed up.
  • External Artifact Storage: Store build artifacts in external, highly durable services like AWS S3 instead of on the Jenkins controller’s local storage. This offloads storage, improves performance, and ensures artifacts are resilient even if Jenkins is lost.
  • External Log Aggregation: Send all Jenkins logs to a centralized, highly available logging solution e.g., CloudWatch Logs, Splunk, ELK stack. This ensures logs are accessible even if the Jenkins instance is down.

By meticulously planning and implementing these HA and DR strategies, you can transform your Jenkins on EKS into a truly resilient and dependable CI/CD platform, capable of weathering various types of outages with minimal impact to your development workflow.

Continuous Improvement and Advanced Topics

Successfully migrating Jenkins to EKS is a significant achievement, but the journey doesn’t end there.

This final section explores advanced topics and strategies to further optimize, secure, and extend your Jenkins on EKS setup, ensuring it remains a performant, cost-effective, and future-proof platform.

Automating Jenkins Upgrades and Patching

Manual upgrades for Jenkins controller, plugins, and worker nodes are time-consuming and prone to errors.

Automating these processes is crucial for maintaining security, stability, and access to new features.

*   Helm for Upgrades: Use Helm to upgrade the Jenkins controller. Update the `tag` in your `jenkins-values.yaml` to a newer LTS version and run `helm upgrade jenkins -n jenkins -f jenkins-values.yaml jenkins/jenkins`.
*   Canary Deployments/Blue/Green: For critical production Jenkins instances, consider using more advanced deployment strategies. Deploy a new version alongside the old one canary or completely replace it blue/green after thorough testing. This minimizes downtime and risk.
*   Automated Testing: Implement automated tests for your Jenkins instance itself e.g., testing plugin functionality, JCasC application within a dedicated CI/CD pipeline that runs before deploying new Jenkins versions.
  • Plugins:
    • JCasC for Plugin Versions: As highlighted before, manage plugin versions in JCasC. This ensures reproducibility and allows you to update plugin versions in your version control, then apply the change.
    • Automated Plugin Updates Cautiously: While Jenkins has features for automated plugin updates, use them with caution in production. It’s often safer to pin versions in JCasC and update them after testing.
  • EKS Worker Nodes:
    • EKS Managed Node Groups: If you used Managed Node Groups, AWS handles patching and updating the underlying EC2 instances. You can configure rollouts for updates.
    • Karpenter: Karpenter’s ttlSecondsAfterAllocation and ttlSecondsAfterEmpty allow it to automatically terminate old nodes and provision new ones with the latest AMIs, effectively automating node refreshing and patching.
    • Automated AMI Updates: Implement a process to regularly update the AMI used by your node groups or Karpenter AWSNodeTemplate to pick up the latest security patches.

Implementing GitOps for Jenkins Management

GitOps is an operational framework that takes DevOps best practices like version control, collaboration, compliance, and CI/CD and applies them to infrastructure automation. For Jenkins on EKS, it means managing your cluster configuration, Jenkins deployment, and JCasC files entirely through Git.

  • Core Principle: Git is the single source of truth for your desired state.
  • Workflow:
    1. Developer makes a change: e.g., updates a Jenkins plugin version in jenkins-values.yaml or a JCasC configuration.
    2. Pushes to Git: The change is committed and pushed to a Git repository.
    3. CI/CD Pipeline: A pipeline e.g., Jenkins itself, GitHub Actions, GitLab CI builds/validates the change.
    4. Argo CD/Flux CD Reconciliation: A specialized GitOps operator like Argo CD or Flux CD deployed in your EKS cluster continuously monitors the Git repository. When it detects a change, it automatically applies the desired state to the cluster.
  • Benefits:
    • Automated Deployments: Changes are deployed automatically upon Git commit.
    • Auditable Changes: Every change to your infrastructure is version-controlled and auditable.
    • Rollbacks: Reverting to a previous state is as simple as reverting a Git commit.
    • Faster Disaster Recovery: The entire cluster state is defined in Git, enabling rapid recovery.

Implementation with Argo CD Example:

  1. Install Argo CD:
    kubectl create namespace argocd

    Kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

  2. Create Argo CD Application: Define an Application resource in Argo CD that points to your Git repository containing Jenkins Helm chart values and JCasC configs.

    argocd-jenkins-app.yaml

    apiVersion: argoproj.io/v1alpha1
    kind: Application
    name: jenkins
    namespace: argocd
    destination:
    namespace: jenkins
    server: https://kubernetes.default.svc
    project: default
    source:
    repoURL: https://github.com/your-org/your-jenkins-config-repo.git # Your Git repo
    targetRevision: HEAD
    path: jenkins-helm-chart/ # Path to your Helm chart values and JCasC
    helm:
    valueFiles:
    – values.yaml
    – jenkins-casc-configmap.yaml # If using configmap for JCasC
    syncPolicy:
    automated:
    prune: true
    selfHeal: true # Automatically sync if drift is detected
    syncOptions:
    – CreateNamespace=true
    Apply: kubectl apply -f argocd-jenkins-app.yaml

    Now, any changes pushed to your Git repository for Jenkins will be automatically picked up and deployed by Argo CD.

Extending Jenkins with Serverless and Other AWS Services

Jenkins on EKS is powerful, but you can augment its capabilities and further optimize costs by integrating with other AWS services, particularly serverless options.

  • AWS Lambda for Triggering:
    • Use AWS Lambda functions to trigger Jenkins jobs in response to events e.g., S3 object creation, DynamoDB stream updates, CloudWatch alarms. This eliminates the need for Jenkins to constantly poll for changes, saving resources.
    • Example: A Lambda function triggered by a new file in an S3 bucket could call the Jenkins API to start a pipeline for processing that file.
  • AWS Step Functions for Complex Workflows:
    • For very complex, multi-stage CI/CD workflows that might span multiple Jenkins jobs, external systems, or human approvals, consider orchestrating them with AWS Step Functions.
    • Jenkins jobs can be steps within a larger Step Functions state machine, allowing for resilient, retryable, and observable workflows.
  • AWS CodeBuild/CodePipeline for Specialized Tasks:
    • While Jenkins is your primary orchestrator, leverage AWS CodeBuild for highly parallelized or specialized build tasks e.g., Docker image builds, serverless deployments. CodeBuild is serverless and highly scalable.
    • Use AWS CodePipeline to orchestrate the end-to-end delivery process, potentially calling Jenkins as one of its stages, alongside other Code services CodeCommit, CodeDeploy. This provides a more managed, integrated AWS native CI/CD experience.
  • AWS Security Hub / GuardDuty Integration:
    • Feed EKS logs and metrics into Security Hub or GuardDuty for threat detection and compliance monitoring.
    • Use AWS Config to track changes to your EKS cluster and ensure compliance with security baselines.

By continuously evaluating and adopting these advanced practices, your Jenkins on AWS EKS environment will remain at the forefront of CI/CD efficiency, security, and scalability, supporting your development teams for years to come.

Frequently Asked Questions

What are the main benefits of running Jenkins on AWS EKS?

The main benefits of running Jenkins on AWS EKS include enhanced scalability, as build agents can dynamically scale as Kubernetes pods based on demand. improved resilience, with Kubernetes’s self-healing capabilities ensuring high availability. better resource utilization and cost optimization through features like the Cluster Autoscaler or Karpenter and Spot Instances. and consistent, isolated build environments provided by containerization.

Is EKS a good choice for small Jenkins instances or only large ones?

EKS offers robust features beneficial to both small and large Jenkins instances.

For small instances, it provides inherent resilience and easy scaling when growth occurs.

For large instances, it’s essential for handling high concurrency, managing diverse agent types, and maintaining performance.

While there’s an initial setup overhead, the long-term operational benefits often outweigh it, even for moderately sized setups.

How does Jenkins handle persistent data on EKS?

Jenkins handles persistent data on EKS by utilizing PersistentVolumes PVs and PersistentVolumeClaims PVCs. The recommended storage solution for Jenkins on EKS is Amazon Elastic File System EFS, mounted using the EFS CSI driver. EFS provides shared, highly available, and scalable storage for the Jenkins home directory, ensuring data durability across pod restarts and node failures.

Amazon

What is Jenkins Configuration as Code JCasC and why is it important for EKS?

Jenkins Configuration as Code JCasC allows you to define your Jenkins setup, including plugins, security, credentials, and agent configurations, in declarative YAML files. It’s important for EKS because it enables reproducibility, version control, and automated deployment of your Jenkins instance. This aligns with the Infrastructure-as-Code IaC principles common in Kubernetes, simplifying management, auditing, and disaster recovery.

How do I expose Jenkins running on EKS to external users?

You can expose Jenkins running on EKS to external users primarily through a Kubernetes Service of type LoadBalancer which provisions an AWS Elastic Load Balancer or, more robustly, by using a ClusterIP Service combined with an Ingress Controller like NGINX Ingress Controller or AWS ALB Ingress Controller. The Ingress controller allows for custom domains, SSL termination, and more advanced routing rules.

How do I manage Jenkins agent resources on EKS?

Jenkins agent resources on EKS are managed through Kubernetes Pod Templates configured within the Jenkins Kubernetes Plugin or via JCasC. For each Pod Template, you specify the Docker image for the agent, and crucially, define CPU and memory requests and limits for each container within the agent pod. This ensures agents get the necessary resources for builds and helps with cost optimization.

What is the role of the Kubernetes Plugin in Jenkins on EKS?

The Kubernetes Plugin is essential for Jenkins on EKS as it allows the Jenkins controller to dynamically provision and de-provision build agents as Kubernetes pods. When a build is queued, the plugin communicates with the Kubernetes API to launch a new agent pod based on the defined Pod Templates, and once the build is complete, the pod is automatically terminated.

How can I monitor Jenkins and my EKS cluster for performance and issues?

You can monitor Jenkins and your EKS cluster using AWS CloudWatch Container Insights for aggregated metrics and logs, and by deploying an open-source observability stack like Prometheus for metrics collection and alerting and Grafana for dashboarding. Fluent Bit can be used to forward container logs to CloudWatch Logs for centralized logging.

What are IAM Roles for Service Accounts IRSA and why are they important for EKS?

IAM Roles for Service Accounts IRSA allow you to assign fine-grained AWS IAM permissions directly to Kubernetes Service Accounts and thus to specific pods. This is crucial for EKS security because it enables the principle of least privilege, allowing Jenkins pods to interact with AWS services like S3 or ECR using specific roles rather than inheriting broad permissions from the underlying EC2 instance roles.

How can I optimize costs for Jenkins on EKS?

Cost optimization for Jenkins on EKS involves several strategies:

  • Using Cluster Autoscaler or Karpenter for dynamic node scaling.
  • Right-sizing Jenkins controller and agent pod CPU/memory requests and limits.
  • Leveraging Spot Instances for Jenkins build agents.
  • Implementing EFS lifecycle policies to move cold data to Infrequent Access storage.
  • Considering AWS Graviton instances for better price-performance of worker nodes.

What is the best practice for high availability HA of Jenkins on EKS?

The best practice for high availability of Jenkins on EKS is to deploy the Jenkins controller as a single instance but ensure its underlying infrastructure EKS cluster, EFS is highly available across multiple Availability Zones.

Kubernetes will automatically reschedule the Jenkins pod if a node fails, and EFS ensures data durability.

For disaster recovery, implement cross-region backups and a clear recovery plan.

Should I use AWS CodeBuild/CodePipeline instead of Jenkins on EKS?

Whether to use AWS CodeBuild/CodePipeline or Jenkins on EKS depends on your needs.

CodeBuild/CodePipeline offers a fully managed, serverless CI/CD solution deeply integrated with AWS services, which is great for simpler, AWS-native pipelines.

Jenkins on EKS provides more flexibility, extensibility plugins, and control, especially for complex, legacy, or hybrid cloud environments.

Many organizations use a hybrid approach, leveraging Jenkins for orchestration while offloading specific build tasks to CodeBuild.

How do I handle secrets and credentials in Jenkins on EKS?

Secrets and credentials in Jenkins on EKS should be handled securely. Use the Jenkins Credentials Plugin for general credentials. For highly sensitive secrets, integrate Jenkins with AWS Secrets Manager or AWS Parameter Store, possibly using the Kubernetes Secrets Store CSI driver to inject secrets directly into pods. Avoid hardcoding secrets in configuration files or Docker images.

Can I run multiple Jenkins controllers on the same EKS cluster?

Yes, you can run multiple Jenkins controllers on the same EKS cluster, typically in separate namespaces, or with distinct Helm releases.

This is useful for isolating different teams or projects, or for maintaining separate Jenkins instances for different environments e.g., dev, staging. Each would require its own EFS volume and configuration.

How do I ensure reproducible builds with Jenkins on EKS?

Reproducible builds on Jenkins on EKS are achieved by:

  • Using Docker images: Ensuring build tools and dependencies are packaged consistently.
  • Pinning Docker image versions: Avoid latest tags.
  • Using JCasC: To define Jenkins configurations plugins, agent templates declaratively and version-controlled.
  • Externalizing dependencies: Fetching dependencies from consistent sources e.g., artifactory, private Git repos.

What is the typical network traffic pattern for Jenkins on EKS?

Jenkins on EKS typically has the following network traffic patterns:

  • Jenkins controller to Kubernetes API: For dynamic agent provisioning.
  • Jenkins controller to EFS: For persistent storage.
  • Jenkins controller/agents to Git repositories: For source code.
  • Jenkins controller/agents to artifact repositories: For pulling/pushing build artifacts.
  • Jenkins controller/agents to external services/APIs: For integrations.
  • External users to Jenkins UI: Via LoadBalancer or Ingress.

How do I update the Kubernetes version of my EKS cluster running Jenkins?

Updating the Kubernetes version of your EKS cluster involves two main steps:

  1. Updating the EKS Control Plane: This is done via the AWS console or eksctl upgrade cluster.
  2. Updating Worker Nodes: For EKS Managed Node Groups, AWS provides an update mechanism. For self-managed nodes, you’ll need to update their AMIs and perform a rolling update. Karpenter automatically handles node updates as part of its lifecycle management if configured. Always check the Jenkins and Kubernetes plugin compatibility matrix before upgrading.

What are the considerations for pipeline security in Jenkins on EKS?

Pipeline security in Jenkins on EKS includes:

  • Using secrets management for sensitive data within pipelines.
  • Implementing RBAC to restrict who can define or run pipelines.
  • Scanning Docker images used by agents for vulnerabilities.
  • Restricting network access of agent pods using Network Policies.
  • Sanitizing user input in pipelines to prevent injection attacks.
  • Auditing pipeline execution and access.

Can Jenkins on EKS integrate with other AWS services besides EFS and ECR?

Yes, Jenkins on EKS can integrate with many other AWS services:

  • S3: For artifact storage, build logs, and backup.
  • CloudWatch: For monitoring, logging, and alarms.
  • SNS/SQS: For event-driven triggers and notifications.
  • VPC Endpoints: For secure, private connectivity to AWS services without traversing the public internet.
  • Secrets Manager/Parameter Store: For managing sensitive credentials.
  • RDS: If using an external database for Jenkins.
  • CodeCommit: For source code management.

How do I handle Jenkins plugins in a reproducible manner on EKS?

To handle Jenkins plugins reproducibly on EKS, list all required plugins and their specific versions in your Helm values.yaml file under controller.installPlugins. This ensures that every time the Jenkins controller pod is deployed, it starts with the exact set of plugins.

For further configuration of these plugins, use Jenkins Configuration as Code JCasC to define their settings in version-controlled YAML files.

How useful was this post?

Click on a star to rate it!

Average rating 0 / 5. Vote count: 0

No votes so far! Be the first to rate this post.

•

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *