ksqlDB vs Flink vs Spark Streaming

TL;DR

• Apache Flink: stream processing latency-critical (< 300ms p99), stateful processing matang, exactly-once. Default untuk fraud detection enterprise.
• ksqlDB: SQL streaming sederhana, learning curve rendah. Development melambat di 2025 — evaluate sebelum heavy bet.
• Spark Streaming: micro-batch, latency 1-10 detik, cocok untuk near-real-time dengan batch-like semantics.
• Kafka Streams: Java library embedded, simpler ops, cocok untuk single-team mid-scale.
• Verdict: Watch — ekosistem masih shift. Flink default untuk new project, hindari ksqlDB heavy investment.

Konteks

Saya jalankan 3 engine stream processing paralel 18 bulan (Desember 2024 - Mei 2026) di fintech series-B Jakarta:

• Apache Flink 1.20: fraud detection real-time + risk scoring + payment anomaly detection (workload primary, mission-critical)
• ksqlDB: SQL aggregation untuk dashboard near-real-time + transformation ETL (low-criticality, evaluation)
• Spark Streaming: legacy data pipeline batch-like, 1 use case migrate dari standalone batch ke streaming

Workload skala:

• Total Kafka topic input: 18 topic, 280k events/sec peak campaign payday
• Fraud detection: 80k events/sec primary stream
• Risk scoring: 45k events/sec primary stream
• ETL streaming: 25k events/sec
• Dashboard aggregation: 12k events/sec

Pengalaman saya sebelumnya: Apache Storm 2 tahun (legacy, deprecated), Spark Streaming 4 tahun (batch-heavy use case), Flink 3 tahun (sejak 1.13).

Pricing + cost (Juni 2026)

Apache Flink self-host

• Software gratis (Apache 2.0)
• Cluster 3 JobManager + 6 TaskManager (n2-standard-4, 4 vCPU 16GB):
  • Compute: 9 × Rp 1,6 juta = Rp 14,4 juta/bulan
  • State backend (RocksDB local + S3 checkpoint): Rp 1,2 juta/bulan
  • Network egress (Kafka ↔ Flink): Rp 800 ribu/bulan
• Total Flink: Rp 16,4 juta/bulan + ops 25-35 jam/bulan

ksqlDB self-host (via Confluent Platform CE)

• Software: Confluent Community Edition gratis untuk ksqlDB
• Cluster 3 ksqlDB Server: Rp 4,2 juta/bulan
• Total: Rp 4,2 juta/bulan + ops 12-18 jam/bulan

Spark Streaming self-host

• Software gratis (Apache 2.0)
• Spark cluster 4 worker via Spark on K8s: Rp 6,8 juta/bulan
• Total: Rp 6,8 juta/bulan + ops 18-25 jam/bulan

Confluent Cloud managed (sebagai pembanding)

• ksqlDB on Confluent Cloud: USD 0,30-1,00 per CSU-hour
• Untuk workload 100k events/sec: ~USD 4.000-6.000/bulan = Rp 64-96 juta/bulan
• Flink on Confluent Cloud (managed Apache Flink): USD 0,40-1,20 per CFU-hour = USD 5.000-12.000/bulan

AWS Kinesis Data Analytics (Flink managed)

• USD 0,11 per KPU-hour + storage
• Untuk equivalent workload: ~USD 4.500-9.000/bulan = Rp 72-144 juta/bulan

Total stream processing cost

Komponen	Cost/bulan
Flink self-host (3 production job)	Rp 16,4 juta
ksqlDB self-host (4 transform query)	Rp 4,2 juta
Spark Streaming self-host (1 ETL)	Rp 6,8 juta
Monitoring (Prometheus Flink exporter, Grafana dashboard)	Rp 600 ribu
Total	Rp 28 juta/bulan

Bandingkan kalau full managed Confluent Cloud / Kinesis equivalent: ~Rp 140-220 juta/bulan. Self-host saves 80%, trade-off ops 55-75 jam/bulan total.

SLO + performance (18 bulan)

Flink

Metrik	Target	Realisasi
Event-to-decision latency p99	< 300ms	185ms
Throughput sustained	> 100k events/s	145k peak
Exactly-once semantics	wajib	ya (verified)
Checkpoint interval	< 60 detik	30 detik
Recovery time setelah crash	< 3 menit	1,5-2,5 menit
Backpressure event	< 5/bulan	8/bulan rata-rata
Job availability	> 99,9%	99,92%

ksqlDB

Metrik	Realisasi
Query execution latency p99	380ms
Throughput	25k events/s
Pattern complexity	terbatas (SQL DSL)
Operational stability 18 bulan	99,76% (lebih sering restart manual)

Spark Streaming

Metrik	Realisasi
Micro-batch latency p99	4,2 detik
Throughput	35k events/s
Operational stability	99,82%

Capability comparison

Capability	Flink	ksqlDB	Spark Streaming	Kafka Streams
Latency p99	100-300ms	300-800ms	1-10 detik	50-200ms
Stateful processing	matang (RocksDB + checkpoint)	terbatas	medium (state store)	matang (RocksDB)
Exactly-once semantics	ya (best-in-class)	at-least-once + dedup	at-least-once	ya
Windowing (tumbling, sliding, session)	ya (semua)	ya (basic)	ya	ya
Watermark / event-time handling	matang	basic	basic	matang
SQL DSL	Flink SQL (mature)	native SQL	Spark SQL	tidak (Java DSL only)
Java/Scala API	ya	tidak	ya	ya
Python API	ya (PyFlink)	tidak	ya (PySpark)	tidak
State recovery time	1-3 menit	30 detik - 2 menit	1-5 menit	seamless (rebalance)
Operational complexity	tinggi	sedang	tinggi	rendah
Ekosistem connector	besar (200+)	Kafka-only	besar	Kafka-only
Multi-source join	ya	terbatas	ya	terbatas

Use case mapping

Use case: Fraud detection real-time

Requirement: setiap transaksi cek pattern 30-detik window + cek customer history + machine learning score → decide approve/reject dalam < 300ms.

Pilihan: Apache Flink.

Implementation: Flink job consume topic payment-events, join dengan state customer history (RocksDB), call ML inference service, output decision ke topic payment-decisions. Latency budget 300ms; realisasi 185ms p99.

Use case: Real-time dashboard aggregation

Requirement: aggregate per-minute revenue, count active user, dll. SLA refresh 30-60 detik.

Pilihan: ksqlDB (atau Flink SQL).

Implementation: ksqlDB query CREATE TABLE revenue_per_minute AS SELECT TUMBLINGWINDOW... FROM payments. Output table consumed via Materialized View. Latency 30-60 detik OK.

Alternative: Flink SQL juga bisa, tapi overhead operasional lebih besar untuk job simple. Saya pakai ksqlDB sebelum 2025; evaluasi migration ke Flink SQL karena ksqlDB development melambat.

Use case: ETL streaming (Kafka → BigQuery)

Requirement: transform + enrich + push ke BigQuery untuk BI. Latency 5-30 detik OK.

Pilihan: Spark Streaming (legacy) atau Flink (preferred).

Spark Streaming legacy yang saya migrate dari batch — work, tapi overhead operasional Spark on K8s tinggi. Migration plan ke Flink.

Use case: Per-user behavior tracking

Requirement: track event sequence per user, detect funnel completion, trigger notification.

Pilihan: Kafka Streams atau Flink.

Untuk single-team domain ownership: Kafka Streams embedded di service. Simpler ops, scale terikat partition. Untuk cross-team shared platform: Flink.

Trade-off arsitektural

Pilih Flink kalau:

• Latency p99 < 300ms wajib
• Exactly-once semantics kritis (payment, fraud, ledger)
• Stateful processing kompleks (windowing, join, CEP pattern matching)
• Skala > 50k events/sec sustained
• Multi-team platform shared

Pilih ksqlDB kalau:

• SQL DSL preference (lower learning curve)
• Aggregation + transformation sederhana
• Skala < 30k events/sec
• Tim familiar SQL, tidak Java/Scala

Caveat 2026: ksqlDB development di Confluent melambat sejak 2024-2025. Roadmap unclear. Hindari heavy bet untuk new project. Pertimbangkan Flink SQL sebagai alternative.

Pilih Spark Streaming kalau:

• Micro-batch acceptable (latency 1-10 detik)
• Heavy investment Spark batch existing
• Tim familiar PySpark / Scala Spark
• Workload mixed batch + streaming

Pilih Kafka Streams kalau:

• Single-team microservice scope
• Skala < 30k events/sec
• Operational simplicity prioritas (no separate cluster)
• Java/Spring Boot stack existing

HA + DR

Flink HA

• 3 JobManager dengan ZooKeeper coordination
• TaskManager auto-restart via K8s
• RPO: 30 detik (checkpoint interval)
• RTO: 1,5-2,5 menit (job restart + state restore)
• Checkpoint state ke S3 / GCS untuk recovery cross-region
• Savepoint untuk planned downtime (upgrade) — manual trigger

ksqlDB HA

• 3-node ksqlDB Server cluster (active-active)
• State store di local + replicate via Kafka topic
• RPO: 0 (Kafka commit-log replication)
• RTO: 30 detik - 2 menit

Spark Streaming HA

• Spark on K8s Operator manage lifecycle
• Checkpoint to S3
• RPO: 30 detik - 5 menit
• RTO: 2-5 menit

Operational complexity

Pengalaman 18 bulan ops effort per platform:

Engine	Avg ops time/bulan	Skill ramp-up baru
Flink	25-35 jam	8-12 minggu untuk produktif
ksqlDB	12-18 jam	2-4 minggu (kalau familiar SQL)
Spark Streaming	18-25 jam	4-6 minggu
Kafka Streams (embedded in service)	4-8 jam	4-6 minggu

Flink butuh skill khusus: JVM tuning, state backend (RocksDB), checkpoint optimization, savepoint management. Senior Flink engineer di Jakarta sangat scarce — kompensasi premium (Rp 40-60 juta/bulan).

Common pitfall

1. Flink checkpoint terlalu sering. Default 60 detik OK, tapi state besar = checkpoint expensive. Tune interval + incremental checkpoint untuk state > 5GB.
2. Backpressure tidak monitor. Flink job slow downstream = backpressure propagate upstream = lag accumulate. Monitor flink_taskmanager_job_task_backPressuredTimeMsPerSecond dengan alert > 50ms/s.
3. State backend salah pilih. RocksDB untuk state besar (> 1GB), Heap state untuk state kecil dengan latency critical. Default RocksDB OK untuk most case.
4. Watermark misconfigured. Out-of-order event = window result salah. Tune watermark strategy sesuai event-time delay realistic (5-60 detik typical).
5. ksqlDB persistent query terlalu banyak. 50+ query di 1 cluster = resource contention. Split per domain ke cluster terpisah, atau migrate ke Flink SQL untuk scale lebih baik.

Migration risk

ksqlDB → Flink SQL

Pattern saya plan 2026: ksqlDB development uncertain, migrate transform + aggregation ke Flink SQL.

• Effort: 6-10 minggu untuk 20 query
• Pain point: SQL semantic beda subtle (windowing syntax, time function)
• Trade-off: ops complexity naik (Flink lebih kompleks), tapi vendor risk turun

Spark Streaming → Flink

Saya jalankan 1 use case 2025:

• Effort: 4-6 minggu untuk 1 job medium complexity
• Pain point: rewrite logic dari DataFrame API ke DataStream API
• Outcome: latency turun dari 4 detik ke 250ms, throughput naik 2x

Kafka Streams ↔ Flink

Lebih mudah translate (similar concept), tapi rewrite tetap perlu. Worth migrate Kafka Streams → Flink kalau scale outgrow partition count.

Cost of ownership 36 bulan

Skenario: fintech fraud detection + real-time analytics, 100k events/s sustained.

Item	Flink self-host	Confluent Cloud Flink
Software/SaaS 3 tahun	Rp 0	Rp 2,9 miliar
Infra	Rp 590 juta	Rp 0
Ops engineer	Rp 720 juta	Rp 130 juta
Migration	Rp 80 juta	Rp 35 juta
Incident	Rp 24 juta	Rp 0
Total 3 tahun	Rp 1,41 miliar	Rp 3,07 miliar

Self-host saves Rp 1,66 miliar dengan trade-off ops 720 juta = ~2 FTE 3 tahun. Untuk fintech series-B dengan SRE dedicated streaming: worth. Untuk SMB tanpa SRE: managed.

Indonesia specific

Data residency

Flink self-host di GKE Jakarta + state backend ke R2 Jakarta = data tinggal di Indonesia. Untuk fintech regulated: wajib. Confluent Cloud Standard tier Singapore: data keluar Indonesia (perlu disclosure customer).

Hiring

Senior Flink engineer di Jakarta: pool sangat kecil (estimasi < 50 senior se-Jakarta dengan 3+ tahun experience). Kompensasi Rp 40-60 juta/bulan + signing bonus tipikal. Strategy saya: train Spring Boot engineer existing ke Flink (12 minggu ramp-up dengan pair programming + 1 senior consultant 3 bulan).

Compliance audit

Flink job audit log: setup file appender log4j + ship ke Splunk via Fluentbit. Audit retention 7 tahun (OJK). Custom audit event: job start/stop, savepoint, checkpoint trigger.

Yang surprising

Setelah 18 bulan: ksqlDB ternyata jadi pain point operasional lebih besar dari ekspektasi. Development cycle melambat di Confluent sejak 2024 (fokus pivot ke Flink-based product). Bug fix lambat masuk ke OSS release. Saya plan migrate semua ksqlDB query ke Flink SQL dalam 12 bulan.

Surprise lain: Flink performance default sangat baik. Setelah initial tuning (heap, checkpoint interval, parallelism), p99 latency stable di 150-220ms tanpa banyak ad-hoc tuning. JVM tuning Flink lebih predictable dari Spark.

Verdict

Watch dengan rule konkret:

• Default new project stream processing: Apache Flink. Untuk semua use case latency-critical.
• Avoid heavy investment di ksqlDB untuk new project — development melambat, vendor strategic shift.
• Spark Streaming: maintain only kalau legacy investment, plan migrate ke Flink dalam 12-18 bulan.
• Kafka Streams: cocok untuk single-team microservice scope, tidak compete dengan Flink di multi-team platform.
• Managed (Confluent Cloud Flink, AWS Kinesis): kalau SMB tanpa SRE dedicated. Trade-off 4-6x cost.

Threshold konkret untuk adopt platform stream processing: > 10k events/sec sustained + use case latency-critical (< 1 detik). Di bawah ini, batch processing per-minute (Cron + script) cukup dan jauh lebih simple.