Microsoft Azure Cloud Services

AKS Cluster Not Starting After Upgrade: Our Azure Experience

We often encounter a situation: after a nightly AKS cluster upgrade in Azure cloud services, the cluster stops responding — FailedNodeOSUpgrade, nodes in NotReady status, pods evicted. The cause is automatic node image upgrade without considering PodDisruptionBudgets. Over 7 years of working with Azure, we have developed a proven scheme: bind autoUpgradeProfile.upgradeChannel and nodeOSUpgradeChannel to maintenanceConfigurationId with a dedicated window during off-hours. This eliminates breakage during peak load. One client after implementation reduced MTTR from 4 hours to 15 minutes — now their engineers sleep peacefully.

Why Use Key Vault CSI Instead of Kubernetes Secrets?

Storing credentials in Kubernetes Secrets is a dangerous practice. The Secret lies in etcd in base64 (not encrypted by default) and is accessible to anyone with get/list on Secrets in the namespace. Azure Key Vault CSI Driver fundamentally solves this: secrets live in Key Vault, are mounted into Pods as files or env variables, access is controlled via Azure RBAC. The workflow:

1. AKS obtains a Managed Identity or Workload Identity.
2. The Identity is assigned the Key Vault Secrets User role on the target Key Vault.
3. The CSI provider mounts secrets at pod startup via SecretProviderClass.

apiVersion: secrets-store.csi.x-k8s.io/v1 kind: SecretProviderClass metadata: name: azure-kvname spec: provider: azure parameters: usePodIdentity: "false" useVMManagedIdentity: "true" userAssignedIdentityID: "" keyvaultName: "company-prod-kv" objects: | array: - | objectName: db-connection-string objectType: secret objectVersion: "" tenantId: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" 

The Pod mounts the SecretProviderClass as a CSI volume — files appear in /mnt/secrets-store/. Additionally, the CSI can sync them into a Kubernetes Secret for env variables, but this is optional. Azure Workload Identity (replacement for Pod Identity v1) provides even more granular control: different ServiceAccounts get different Azure AD tokens with different permissions in Key Vault — no managed identity on nodes. In one project, we migrated 60 microservices from plain Secrets to Key Vault CSI in a week: data leaks stopped, and the Key Vault cost increased by only a minimal amount.

How to Set Up Azure DevOps Pipelines for AKS?

Azure DevOps Pipelines are deeply integrated with AKS, ACR, and Azure App Service. The YAML pipeline below is a base for CI/CD with manual deployment approval.

trigger: branches: include: - main pool: vmImage: 'ubuntu-latest' stages: - stage: Build jobs: - job: BuildAndPush steps: - task: Docker@2 inputs: containerRegistry: 'acr-service-connection' repository: 'company/api' command: 'buildAndPush' tags: | $(Build.BuildId) latest - stage: Deploy dependsOn: Build environment: 'production' jobs: - deployment: DeployToAKS strategy: runOnce: deploy: steps: - task: KubernetesManifest@1 inputs: action: 'deploy' kubernetesServiceConnection: 'aks-prod-connection' manifests: 'k8s/production/*.yaml' containers: 'company.azurecr.io/company/api:$(Build.BuildId)' 

environment: 'production' with approvals — deployment requires manual confirmation from an approver group. Deployment history and rollbacks are available via the Azure DevOps Environments UI. If the cluster is in a private VNet, use self-hosted agents: run them as a Pod in the same AKS or as a VM in the VNet — this gives access to Azure SQL without a public endpoint. This approach reduces deployment time by 2 times compared to external agents.

How to Leverage Azure Functions for .NET Serverless?

Azure Functions is serverless computing with deep integration into Azure services. Triggers: HTTP, Service Bus, Event Hub, Blob Storage, Timer (cron), Cosmos DB change feed. Durable Functions extend capabilities to stateful orchestrations. The Fan-out/Fan-in pattern: the orchestrator launches N parallel tasks and collects results.

[FunctionName("Orchestrator")] public static async Task<List<string>> RunOrchestrator( [OrchestrationTrigger] IDurableOrchestrationContext context) { var tasks = new List<Task<string>>(); foreach (var item in items) tasks.Add(context.CallActivityAsync<string>("ProcessItem", item)); return new List<string>(await Task.WhenAll(tasks)); } 

Cold start in .NET is an issue for HTTP triggers with SLA. Premium Plan eliminates cold start via pre-warmed instances, but costs a fixed amount regardless of load. For production HTTP APIs with constant traffic, Container Apps or App Service are often more cost-effective. For example, one client with 50,000 requests/hour switched from Premium Plan to Container Apps and saved 35% of budget. We perform a comparative analysis and provide recommendations tailored to your profile.

How to Design Partition Key and Monitor RU in Cosmos DB?

Cosmos DB is a multi-model NoSQL database with guarantees: latency <10 ms for writes and reads at the 99th percentile, SLA 99.999% for multi-region writes (Azure Cosmos DB documentation). API compatibility: NoSQL (native), MongoDB, Cassandra, Gremlin (graphs), Table Storage. Migration from MongoDB requires minimal code changes, but the billing model and consistency guarantees differ.

Partition key is the most important design decision. A poor partition key (e.g., country with 5 values for millions of documents) creates 'hot' partitions — all requests hit one physical partition, RU limit exhausted quickly. A good partition key is userId or tenantId with high cardinality and even request distribution. Request Units (RU) is an abstract unit. One read of a 1KB document = 1 RU. One write = 5 RU. SELECT * over a large partition = hundreds of RU. Monitor via x-ms-request-charge in API responses or Azure Monitor metrics. Without RU monitoring, you unexpectedly get 429 Too Many Requests in production.

Why Does 429 Too Many Requests Occur in Cosmos DB?

Main causes: Uneven request distribution due to poor partition key. Exceeding provisioned RU/s during peak load. Lack of autoscaling or incorrect threshold.

We recommend using autoscale with a 30% buffer above average consumption. In one case, we increased throughput from 400 to 80,000 RU/s by changing the partition key from deviceType to deviceId — 429 errors disappeared, and cost increased by only 12% due to even load.

What Is Included in Azure Optimization Work?

Company Experience

• Over 7 years working with Microsoft Azure.
• More than 50 migration and optimization projects for Azure infrastructure.
• Certifications: Azure Solutions Architect Expert, DevOps Engineer Expert.

Typical Timelines and Cost

Cost is calculated individually after an audit of your infrastructure. For an accurate estimate, contact us — we will send a data checklist for the initial assessment.

Order an Azure Cloud Services Audit

Get a consultation from an engineer — we will review your current architecture, find bottlenecks, and propose optimization with a forecast of up to 30% cost savings on Azure. Contact us via Telegram or the feedback form. If your cluster is already experiencing autoscaling issues or 429 errors, don't delay — reach out now.