Workflow Blue/Green vs Canary Deploy Enterprise

TL;DR

• Blue/Green: MTTR rollback 15-30 detik, capacity 2x, ideal untuk service payment / auth dengan SLO ketat.
• Canary: gradual traffic shift dengan auto-rollback metric-based, capacity 1,1-1,3x, ideal untuk feature validation.
• Tools: Argo Rollouts (default K8s 2026), Flagger (FluxCD), Spinnaker (multi-cloud enterprise).
• Verdict: Conditional — Blue/Green untuk service kritis, Canary untuk feature service. Pattern hybrid di enterprise modern.

Konteks

Saya jalankan kedua pattern paralel 24 bulan (Juni 2024 - Mei 2026) di:

• Fintech series-B Jakarta: 18 microservice di GKE Jakarta dengan Argo Rollouts
  • Service payment + auth: Blue/Green
  • Service product + dashboard: Canary
  • Service internal admin: Rolling update sederhana
• BUMN energy Jakarta: 14 service di Rancher on-prem dengan Argo Rollouts + Istio
  • Service customer-facing: Canary
  • Service core billing: Blue/Green

Pengalaman saya sebelumnya: Spinnaker 3 tahun di SaaS klien (2022-2024), basic blue/green via NGINX + manual flip 5 tahun (2019-2024).

Pricing + tooling cost

Argo Rollouts

• Software gratis (Apache 2.0)
• Compute: 2 controller pod e2-small di GKE Jakarta: Rp 460 ribu/bulan
• Tidak butuh service mesh wajib (basic deployment cukup), tapi integrasi Istio/Linkerd add capability

Flagger

• Software gratis (Apache 2.0)
• Compute: similar Argo Rollouts
• Butuh service mesh (Istio/Linkerd/Contour) untuk traffic shifting — add ops complexity

Spinnaker

• Software gratis (Apache 2.0)
• Self-host complex: 6-10 microservice (Orca, Igor, Echo, Front50, Gate, dll)
• Compute: Rp 4-6 juta/bulan
• Plus ops 30-50 jam/bulan
• Managed version (Armory): USD 12k-50k+/tahun

Total tooling cost

Item	Cost/bulan
Argo Rollouts controller + monitoring	Rp 460 ribu
Istio service mesh (untuk canary traffic split)	Rp 1,8 juta
Prometheus + custom metric (untuk analysis)	already in observability budget
Grafana dashboard rollout	already in observability budget
Total	Rp 2,26 juta/bulan

Bandingkan kalau Spinnaker: Rp 5-8 juta/bulan infra + ops time signifikan. Argo Rollouts win-win untuk K8s-native enterprise modern.

SLO + performance (24 bulan)

Blue/Green metric

Metrik	Target	Realisasi service payment
Cutover time (DNS / Ingress alias flip)	< 30 detik	12-18 detik
MTTR rollback (re-flip ke blue)	< 60 detik	38 detik
Capacity overhead saat deploy	2x	2x (manageable cost-wise)
Deploy success rate	> 98%	98,9%
Customer-facing impact saat cutover	< 1% req	0,3% (connection draining gap)

Canary metric

Metrik	Target	Realisasi service product
Total rollout time (1% → 100%)	30-90 menit	45 menit typical
Auto-rollback trigger latency	< 60 detik	18-42 detik
Auto-rollback success rate	> 99%	99,8%
Capacity overhead saat rollout	1,1-1,3x	1,2x avg
Bug caught at canary 5% stage	rate	14 bug dalam 24 bulan caught at < 25% canary

Architecture pattern

Blue/Green dengan Argo Rollouts

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: payment-service
spec:
  replicas: 6
  strategy:
    blueGreen:
      activeService: payment-service-active
      previewService: payment-service-preview
      autoPromotionEnabled: false  # manual promote, OJK compliance
      scaleDownDelaySeconds: 600    # keep blue 10 menit setelah cutover
      prePromotionAnalysis:
        templates:
        - templateName: payment-smoke-test
      postPromotionAnalysis:
        templates:
        - templateName: payment-error-rate
        - templateName: payment-business-metric

Flow:

1. Deploy new version ke preview service (green)
2. Pre-promotion analysis: smoke test 5 menit
3. Manual approval gate (compliance OJK)
4. Promote: switch active service ke green
5. Post-promotion analysis: monitor 30 menit
6. Kalau analysis fail dalam 30 menit: auto-rollback ke blue (re-switch)
7. Setelah 10 menit stable, scale down blue

Canary dengan Argo Rollouts + Istio

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: product-service
spec:
  replicas: 6
  strategy:
    canary:
      canaryService: product-canary
      stableService: product-stable
      trafficRouting:
        istio:
          virtualService:
            name: product-vs
      steps:
      - setWeight: 5
      - pause: { duration: 5m }
      - analysis:
          templates: [error-rate, latency-p99]
      - setWeight: 25
      - pause: { duration: 10m }
      - analysis: ...
      - setWeight: 50
      - pause: { duration: 15m }
      - analysis: ...
      - setWeight: 100

Flow:

1. Deploy canary pod (10% replica baru)
2. Istio VirtualService route 5% traffic ke canary
3. Wait 5 menit + analysis (error rate < baseline + 1%, p99 latency < baseline × 1,2)
4. Increment ke 25%, 50%, 100% dengan analysis tiap step
5. Kalau analysis fail kapanpun: auto-rollback (set weight back to 0, scale down canary)

Analysis template

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: error-rate
spec:
  metrics:
  - name: error-rate
    interval: 30s
    successCondition: result[0] <= 0.01
    failureLimit: 3
    provider:
      prometheus:
        address: http://prometheus.monitoring.svc:9090
        query: |
          sum(rate(http_requests_total{service="product-canary",status=~"5.."}[1m]))
          /
          sum(rate(http_requests_total{service="product-canary"}[1m]))

3 consecutive fail (90 detik) → rollback trigger.

Capability comparison

Capability	Blue/Green	Canary
Cutover speed	instant (alias flip)	gradual (minutes-hours)
MTTR rollback	15-60 detik	depends on % shifted
Capacity overhead	2x	1,1-1,3x
Cost during deploy	2x normal	10-30% extra
Bug detection blast radius	100% atau 0%	percentage-based gradual
Database migration compat	wajib backward compat	wajib backward compat
Session preservation	tricky (connection drain)	smooth (gradual)
Cache warming	wajib (full new fleet)	gradual (low overhead)
Business metric validation	post-deploy only	inline during rollout
Ops complexity	rendah	sedang
Best for	service kritis SLO ketat	feature service validation

Trade-off arsitektural

Pilih Blue/Green kalau:

• Service SLO 99,95%+ (payment, auth, core API)
• MTTR rollback < 60 detik wajib
• Stateless service (atau session externalized)
• Capacity 2x acceptable cost-wise
• Compliance audit-friendly (atomic cutover, clear before/after)

Pilih Canary kalau:

• Feature validation butuh real traffic + real user
• Business metric (conversion, completion rate) jadi rollback trigger
• Cost capacity 2x tidak acceptable
• Bug yang muncul gradual / pattern-dependent
• Service tolerates gradual exposure

Pilih Rolling update sederhana (default K8s) kalau:

• Service internal admin / batch
• SLO longgar (99,5% atau lower)
• Cost optimization prioritas
• Tim familiar dengan K8s standard

Database migration compatibility

Both Blue/Green dan Canary butuh database backward compat selama rollout:

Pattern: expand-contract

1. Expand: deploy schema change yang backward compatible (add column nullable, new table, dll). Tidak break existing code.
2. Migrate code: deploy new code yang pakai new schema, tapi tetap support old schema.
3. Contract: setelah confident new code stable, drop old schema artifact (column unused, dll).

Setiap phase = deploy terpisah. Untuk service payment: rule “no breaking schema change dalam 1 deploy” wajib di-enforce via PR review.

Workflow integration

CI/CD pipeline pattern

git push ─→ GitHub Actions
              │
              └─→ Build + Test + Image push to GHCR
                    │
                    └─→ ArgoCD detect new image
                          │
                          └─→ Argo Rollouts trigger
                                │
                                ├─→ Blue/Green: cutover dengan approval gate
                                │   └─→ Smoke test → Analysis → Manual approve → Switch
                                │
                                └─→ Canary: gradual 5%, 25%, 50%, 100%
                                    └─→ Analysis per step → Auto-promote atau Auto-rollback

Approval gate untuk fintech compliance

OJK mensyaratkan change control untuk service payment. Setup di Argo Rollouts:

• Approval gate via Slack bot (/approve payment-service v1.2.4)
• 2 approver mandatory: 1 engineering lead + 1 SRE on-call
• Audit log siapa approve kapan di Splunk

Common pitfall

1. Health check kasar untuk auto-rollback. httpGet /healthz return 200 = service hidup, bukan service correct. Implement custom analysis: error rate, business metric, downstream dependency latency.
2. Capacity buffer kurang untuk Blue/Green. 2x capacity selama deploy = pod request 2x. Setup cluster autoscaler + buffer 30%, atau pakai pre-emptible node untuk preview environment.
3. Database schema breaking change deployed dengan new code. Rollback impossible. Wajib expand-contract pattern dengan multiple deploy.
4. Connection draining tidak setup. Cutover tanpa graceful shutdown = active connection drop. Setup terminationGracePeriodSeconds: 30 + readiness probe drain.
5. Canary analysis template tidak tuned. Default threshold (5xx rate > 1%) bisa terlalu loose untuk payment, terlalu strict untuk batch. Tune per service SLO.

Multi-region deployment

Untuk fintech dengan DR Jakarta + Singapore:

Jakarta (primary) ──→ Canary deploy 5% → 100%
                       │
                       └─→ Setelah stable 24 jam ──→ Singapore (DR) canary 5% → 100%

Pattern: primary first, DR follow setelah confident. Tidak deploy bersamaan untuk minimize multi-region failure scenario.

Untuk service BUMN dengan dual data center (Cibitung + Tangerang): same pattern, manual coordination tambah karena on-prem network constraint.

Cost of ownership 36 bulan

Skenario: 18 microservice fintech, 6 service kritis (Blue/Green), 12 service standard (Canary).

Item	Cost/bulan	Cost 3 tahun
Tooling (Argo Rollouts + Istio)	Rp 2,26 juta	Rp 81 juta
Capacity overhead Blue/Green (6 service)	Rp 3,5 juta	Rp 126 juta
Capacity overhead Canary (12 service, 20% avg)	Rp 1,4 juta	Rp 50 juta
Ops time engineer (8 jam/bulan)	Rp 2,1 juta	Rp 76 juta
Total	Rp 9,26 juta/bulan	Rp 333 juta

Bandingkan dengan Spinnaker setup: setup awal Rp 200-400 juta, run-rate Rp 8-12 juta/bulan + 30-50 jam ops. 3 tahun cost Rp 480-720 juta. Argo Rollouts saves 30-50%.

Migration risk

Dari rolling update standard ke Argo Rollouts

Smooth, additive:

1. Install Argo Rollouts controller
2. Convert Deployment → Rollout untuk 1 service pilot
3. Setup analysis template per service
4. Rollout gradual ke service lain

Effort: 2-3 minggu per service untuk tuning analysis. Total 12-18 minggu untuk 18 service.

Dari Spinnaker ke Argo Rollouts

Lebih kompleks. Pipeline Spinnaker punya banyak feature (manual stage, parallel branch, dll) yang harus dimap ke Argo Workflow + Rollouts.

Effort: 4-8 minggu untuk 10-15 service. Trade-off: Argo ekosistem lebih simple long-term.

Indonesia specific

Compliance OJK

OJK Cyber Resilience Guideline + ITRG mensyaratkan change control + audit trail. Argo Rollouts approval gate + audit log via Splunk = compliant. Saya pakai pattern: approval mandatory untuk service classified P0 (payment, auth, core ledger).

Bandwidth + cost

Canary cross-region (Jakarta canary + Singapore stable) butuh inter-region network. Cost egress GCP intra-region Asia: low (~Rp 50/GB). Tetap implement caching agresif di edge untuk reduce cross-region traffic.

Hiring

Engineer berpengalaman Argo Rollouts di Jakarta: pool kecil. Senior SRE familiar dengan progressive delivery: kompensasi Rp 35-50 juta/bulan. Training internal: 2-3 hari workshop + pair programming first month.

Yang surprising

Setelah 24 bulan: 14 bug caught at canary stage < 25% (sebelum reach 100% user). Tanpa canary, 14 bug akan jadi P1/P2 incident dengan customer-facing impact. ROI canary jelas untuk feature service.

Surprise lain: Blue/Green approval gate ternyata jadi quality bar yang baik. Tim cenderung lebih disiplin sebelum trigger deploy karena tahu 2 approver mandatory. Deploy frequency turun sedikit (12 → 8 deploy/minggu untuk service payment), tapi quality naik (P0 incident dari 4 → 1 per tahun).

Verdict

Conditional dengan rule konkret:

• Service payment / auth / core ledger SLO 99,95%+: Blue/Green dengan approval gate.
• Feature service product / dashboard: Canary dengan auto-rollback metric-based.
• Service internal / batch SLO 99,5%: Rolling update K8s default.
• Tools default 2026: Argo Rollouts. Default untuk K8s-native enterprise modern.
• Skip Spinnaker untuk greenfield kecuali multi-cloud kompleks. Ops overhead tidak justify.

Threshold konkret untuk adopt progressive delivery: 5+ deploy/minggu + service customer-facing + SRE bandwidth 8+ jam/bulan untuk maintain analysis template. Di bawah ini, rolling update standar cukup.