GKE Ingress with Shared VPC, ILB, and NEG

I recently configured container-native load balancing using the GKE Ingress in a Shared VPC and wanted to share the steps for others to use.

Here are the steps:

1. Create two projects: test-project-00 and test-project-01.

2. Enable Shared VPCs between the two projects.

   gcloud beta compute shared-vpc enable test-project-00
   gcloud beta compute shared-vpc associated-projects add test-project-01 \
     --host-project test-project-00
   

3. In test-project-00 run the following commands:

   gcloud compute networks create lb-network --subnet-mode=custom
   gcloud compute networks subnets create backend-subnet \
       --network=lb-network \
       --range=10.1.0.0/16 \
       --region=us-central1
   gcloud compute networks subnets create proxy-only-subnet \
       --purpose=INTERNAL_HTTPS_LOAD_BALANCER \
       --role=ACTIVE \
       --region=us-central1 \
       --network=lb-network \
       --range=10.129.0.0/23
   gcloud compute firewall-rules create fw-allow-ssh \
       --network=lb-network \
       --action=allow \
       --direction=ingress \
       --target-tags=allow-ssh \
       --rules=tcp:22
   gcloud compute firewall-rules create fw-allow-health-check \
       --network=lb-network \
       --action=allow \
       --direction=ingress \
       --source-ranges=130.211.0.0/22,35.191.0.0/16 \
       --target-tags=load-balanced-backend \
       --rules=tcp
   gcloud compute firewall-rules create fw-allow-proxies \
       --network=lb-network \
       --action=allow \
       --direction=ingress \
       --source-ranges=10.129.0.0/23 \
       --target-tags=load-balanced-backend \
       --rules=tcp:80,tcp:443,tcp:8080
   

4. In the Google Cloud console, edit the backend-subnet and add the following secondary IP ranges:

• cluster = 10.2.0.0/16
• services = 10.3.0.0/16

5. In test-project-01 run the following commands (YAML files are at the end):

   gcloud container clusters create neg-demo-cluster-00 \
       --enable-ip-alias \
       --region us-central1 \
       --network projects/test-project-00/global/networks/lb-network \
       --subnetwork projects/test-project-00/regions/us-central1/subnetworks/backend-subnet \
       --cluster-secondary-range-name cluster \
       --services-secondary-range-name services
   kubectl apply -f neg-demo-app.yaml
   kubectl apply -f neg-demo-svc.yaml
   kubectl apply -f neg-demo-ing.yaml
   watch kubectl describe ingress neg-demo-ing
   

• Note: record the IP in the Address field from the above watch command as INGRESS_IP
• Note: if you need to use a specific HTTP path for the NEG healthcheck (e.g. springboot actuator’s /actuator/health), you’ll need to do the following:
  • Create a backendconfig:

    apiVersion: cloud.google.com/v1beta1
    kind: BackendConfig
    metadata:
      name: healthcheck-path
      namespace: default
    spec:
      healthCheck:
        checkIntervalSec: 30
        port: 8080
        type: HTTP
        requestPath: /actuator/health
    

  • Add the following in neg-demo-svc.yaml under metadata.annotations:

    cloud.google.com/backend-config: '{"default": "healthcheck-path"}'
    

6. The logs from the watch command will tell you to create a firewall rule. Copy the output and paste it into the terminal and run it.
7. In the Google Cloud console, go to the Firewall page and edit the rule that was automatically created that ends with -all. Change the source to 0.0.0.0/0.
8. SSH to one of the GKE nodes and run curl INGRESS_IP to validate its working.

YAML Files

neg-demo-app.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
 labels:
   run: neg-demo-app # Label for the Deployment
 name: neg-demo-app # Name of Deployment
spec:
 selector:
   matchLabels:
     run: neg-demo-app
 template: # Pod template
   metadata:
     labels:
       run: neg-demo-app # Labels Pods from this Deployment
   spec: # Pod specification; each Pod created by this Deployment has this specification
     containers:
     - image: k8s.gcr.io/serve_hostname:v1.4 # Application to run in Deployment's Pods
       name: hostname # Container name
       ports:
       - containerPort: 9376
         protocol: TCP

neg-demo-svc.yaml

apiVersion: v1
kind: Service
metadata:
 name: neg-demo-svc # Name of Service
 annotations:
   cloud.google.com/neg: '{"ingress": true}' # Creates a NEG after an Ingress is created
spec: # Service's specification
 type: ClusterIP
 selector:
   run: neg-demo-app # Selects Pods labelled run: neg-demo-app
 ports:
 - name: http
   port: 80 # Service's port
   protocol: TCP
   targetPort: 9376

neg-demo-ing.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: neg-demo-ing
 annotations:
   kubernetes.io/ingress.class: "gce-internal"
spec:
 defaultBackend:
   service:
     name: neg-demo-svc # Name of the Service targeted by the Ingress
     port:
       number: 80 # Should match the port used by the Service