Context & Why: Breaking Through the Single-Node MySQL Ceiling
Pushing a monolithic MySQL database past 80,000 queries per second (QPS) is where vertical scaling hits a wall. In our production cluster, a 4.2 TB orders table was crushing an AWS r6i.16xlarge instance (64 vCPUs, 512 GB RAM). Even with provisioned IOPS cranked up to 40,000, disk queue depths spiked during traffic surges. Write replication lag regularly drifted beyond 15 seconds, risking stale reads across secondary replicas.
Application-level sharding is the typical workaround. But baking shard keys into application code is a maintenance trap. Backend services quickly turn into a tangle of custom connection pools, brittle distributed transactions, and hardcoded shard maps. Vitess takes a cleaner approach.
Originally built at YouTube to handle tens of billions of daily video views across thousands of nodes, Vitess acts as an abstraction proxy. Microservices talk to it through standard MySQL protocol drivers as if connecting to a single monolithic database. Under the hood, Vitess divides operational responsibilities into dedicated tiers:
- VTGate: Stateless proxies serving as the public frontend. They parse incoming SQL syntax, look up partition targets in cluster metadata, and route queries concurrently to the right database shards.
- VTTablet: A sidecar daemon running alongside each
mysqldprocess. It manages local connection pooling, kills runaway queries exceeding execution limits (such as queries holding locks past 10 seconds), and tracks replication state. - Topology Service: A consensus cluster (typically etcd or Consul) storing tablet addresses, shard ranges, and failover states.
- VSchema: The routing configuration that maps table structures, primary VIndexes (hash or lookup keys), and partition boundaries.
Installation: Deploying a Vitess Cluster on Kubernetes
The PlanetScale Vitess Operator handles cluster lifecycle tasks on Kubernetes, including automated failovers, backup jobs, and rolling configuration rollouts.
Start by adding the Helm repository and deploying the operator controller:
helm repo add vitess https://vitess.io/helm-charts/
helm repo update
kubectl create namespace vitess
helm install vitess-operator vitess/vitess-operator --namespace vitessNext, define the cluster specification in cluster.yaml. This manifest provisions an etcd topology backer, a cell named zone1, two VTGate proxies, and an initial unsharded keyspace named commerce:
apiVersion: planetscale.com/v2
kind: VitessCluster
metadata:
name: production-db
namespace: vitess
spec:
cells:
- name: zone1
gateway:
replicas: 2
resources:
limits:
cpu: "2"
memory: 4Gi
requests:
cpu: "500m"
memory: 1Gi
topology:
etcd:
replicas: 3
keyspaces:
- name: commerce
turndownStrategy: Immediate
partitionings:
- equal:
parts: 1
shardTemplate:
tabletControl:
- tabletType: replica
cells: [zone1]
tablets:
- type: replica
replicas: 2
resources:
limits:
cpu: "4"
memory: 8GiApply the manifest and wait for the pods to pass their health checks:
kubectl apply -f cluster.yaml
kubectl wait --for=condition=Ready vitesscluster/production-db -n vitess --timeout=300sConfiguration: Designing VSchema and Splitting into Shards
With the cluster alive, the next milestone is defining a sharded keyspace and partitioning high-volume tables across independent MySQL instances.
1. Define the VSchema
Vitess uses 64-bit keyspace IDs to distribute rows across physical tablets. Save this schema as vschema_orders.json. It uses the built-in hash VIndex to distribute rows evenly across shard boundaries based on customer_id:
{
"sharded": true,
"vindexes": {
"hash_index": {
"type": "hash"
}
},
"tables": {
"customers": {
"column_vindexes": [
{
"column": "customer_id",
"name": "hash_index"
}
]
},
"orders": {
"column_vindexes": [
{
"column": "customer_id",
"name": "hash_index"
}
]
}
}
}Upload the VSchema to the target keyspace (customer_db) using vtctlclient:
vtctlclient --server vtctld.vitess:15999 ApplyVSchema \
--vschema-file vschema_orders.json customer_db2. Split the Keyspace into Shards
Vitess describes shard intervals using hexadecimal ranges. A two-shard topology splits the keyspace hash range 00 through ff into two halves: -80 (0x00 to 0x7f) and 80- (0x80 to 0xff):
# Provision primary and replica tablets for Shard 1 (-80)
vtctlclient --server vtctld.vitess:15999 CreateShard customer_db/-80
# Provision primary and replica tablets for Shard 2 (80-)
vtctlclient --server vtctld.vitess:15999 CreateShard customer_db/80-3. Move Tables and Cut Over Routing
Online data splitting runs through the MoveTables workflow without downtime. VTGate handles dual-writes, syncs historical records, and buffers in-flight transactions during the final transition:
# Start online background data copy
vtctlclient --server vtctld.vitess:15999 MoveTables \
--workflow=order_migration customer_db/-80,customer_db/80- Create
# Inspect replication lag between source and destination shards
vtctlclient --server vtctld.vitess:15999 MoveTables \
--workflow=order_migration Status
# Cut over read and write traffic with zero dropped connections
vtctlclient --server vtctld.vitess:15999 MoveTables \
--workflow=order_migration SwitchTrafficDuring a migration of 120 million rows, replication lag consistently stayed below 150ms. The final write cutover buffered client requests for only 85ms before completing.
Verification & Monitoring: Testing Query Routing and Health
After switching traffic, you must confirm that VTGate routes client queries to the correct shards and avoids scatter-gather penalties where possible.
1. Test VTGate Routing via MySQL CLI
VTGate exposes standard MySQL port 15306. Connect directly with standard database client utilities:
mysql -h vtgate.vitess.svc.cluster.local -P 15306 -u vtgate_userPrefix target SQL queries with EXPLAIN FORMAT = VITESS to inspect execution plans:
USE customer_db;
-- Direct single-shard target via hashed customer_id
EXPLAIN FORMAT = VITESS
SELECT * FROM orders WHERE customer_id = 1045;
-- Scatter-gather query across all shards
EXPLAIN FORMAT = VITESS
SELECT COUNT(*) FROM orders WHERE created_at > '2026-01-01';The first query inspects the MD5 hash of customer_id = 1045. The hash falls into range -80, so VTGate routes the query straight to tablet customer_db/-80 with sub-3ms latency. The second query lacks a shard key. VTGate must broadcast the statement across both shards, collect partial sums, and return the aggregated total, adding a modest latency cost (around 11ms).
2. Core Prometheus Metrics for Vitess
All Vitess daemons expose Prometheus metrics on port 15000. Watch these four production indicators closely:
vtgate_queries_total{keyspace="customer_db"}: Tracks query rates bykeyspace and plan type. Spike warnings here reveal unexpected scatter-gather query bursts.vttablet_query_latency_seconds_bucket: Monitors execution duration on individual shard nodes. Set a P99 alert if latency climbs past 25ms.vttablet_pool_wait_count: Indicates connection saturation. A rate greater than 5 waits/sec means the VTTablet connection pool needs expansion.vitess_replication_lag_seconds: Measures read replica delay. Keep an alert threshold at 5 seconds to prevent stale data reads on read-only replicas.
Combining VTGate proxy routing with Kubernetes automation eliminates the single-node bottleneck. Your MySQL cluster gains linear write scaling and can comfortably handle over 200,000 QPS without application-layer complexity.
