Troubleshooting and Updating Nginx Ingress for Improved Stability

In this blog post, we will explore a critical issue encountered at Safepay when both the sandbox and production URLs/APIs unexpectedly stopped functioning. Through careful investigation and troubleshooting, we identified the root cause to be an outdated version of the Nginx Ingress Controller. This post will walk you through the steps we took to update the Nginx Ingress definitions and resolve the downtime issue.

Identifying the Issue:Upon discovering the malfunctioning URLs/APIs, our initial troubleshooting efforts led us to the Nginx Ingress Controller. Log analysis and thorough investigation revealed that an outdated version of the controller was causing the problem.

Updating Nginx Ingress Definitions:To resolve the issue, we needed to update the Nginx Ingress definitions from the deprecated extensions/v1beta API version to the new apps/v1 version. Here are the steps we followed:

1. Ensure Permissions: We made sure we had the necessary permissions to modify Kubernetes resources.
2. Identify Ingress Resources: Using the command kubectl get ingress -o yaml, we identified the Ingress resources that were using the extensions/v1beta API version.
3. Create Backup: As a precautionary measure, we created a backup of the existing Ingress definitions to have a fallback option if any issues arose during the update.
4. Modify API Version: We opened each Ingress definition file and updated the apiVersion field from extensions/v1beta to networking.k8s.io/v1. Additionally, we adjusted the kind field to IngressV1 or Ingress.
5. Apply Changes: We saved the modified Ingress definition files and applied the changes using the kubectl apply -f <filename> command for each file, effectively updating the Ingress resources to use the apps/v1 API version.
6. Verification: After applying the changes, we ran kubectl get ingress to verify that the updated Ingress resources displayed the networking.k8s.io/v1 API version. We also tested our applications to ensure they were functioning as expected with the updated Ingress definitions.

Updating Nginx Ingress Controller:To further enhance stability, we decided to replace the existing Nginx controller with an updated version. Here's what we did:

1. Installation: We installed the latest version of the Nginx Ingress Controller, which generated a fresh load balancer along with a new elastic IP address.
2. DNS Records: We updated the DNS records in Google Cloud DNS to ensure they pointed to the IP address of the newly created load balancer.

Addressing Routing Errors:Despite the DNS adjustments, we encountered routing errors. To tackle this issue, we made specific modifications within our Ingress file:

1. Annotation:
   We included the following annotation within the Ingress file: nginx.ingress.kubernetes.io/rewrite-target: '/$2'
   This annotation enabled us to appropriately rewrite the target path.
2. Path Adjustments:
   We modified the path itself using the format: path: "/auth(/|$)(.*)”
   

These adjustments helped to resolve the routing errors and ensure proper functioning of the system.

Monitoring and Recommendations:Following the update, we closely monitored the system for any potential issues arising from the API version change. It is always advisable to perform such updates in a test or staging environment before applying them to production.

Conclusion:By applying the necessary updates to the Nginx Ingress definitions and the Nginx Ingress Controller, we successfully resolved the critical issue that caused downtime at Safepay.

Our Uptime notifications played a vital role in promptly identifying the problem and minimizing the impact on our users. To stay up to date with our system's availability and receive real-time updates, we encourage you to visit our dedicated status page at https://safepay.betteruptime.com.

By implementing these measures and ensuring regular updates, we strive to provide a stable and reliable platform for our users.

‍