Using ClickHouse to Power High-Throughput Quantum Experiment Analytics

Stop hunting reproducible qubit data across notebooks and cloud consoles — get analytics that scales

Quantum labs in 2026 run hundreds to thousands of experiments per week. The friction points are familiar: scattered Qiskit and Cirq logs, terabytes of shot-by-shot telemetry, and slow, expensive OLAP cycles that block hypothesis testing. This guide shows how to use ClickHouse as a low-cost, high-throughput engine to collect, store, and analyze time-series qubit telemetry. You'll get practical DDL, ingestion code for Qiskit and Cirq, cloud-run examples, CI/CD suggestions, and proven OLAP queries that let you iterate on calibration hypotheses in minutes — not days.

Why ClickHouse for quantum experiment analytics in 2026

ClickHouse has matured into a mainstream OLAP engine — fast, columnar, and cost-effective. By late 2025 and into 2026 the project and vendor landscape accelerated (notably a major funding round that expanded enterprise adoption). Labs choosing ClickHouse get:

• High ingest throughput for shot-level telemetry (millions of rows/minute) via HTTP, native clients, or Kafka pipelines.
• Low storage cost because of columnar compression and MergeTree storage engines compared with commodity cloud warehouses.
• Fast OLAP for aggregate and windowed queries: run drift analysis, per-qubit error heatmaps, and hypothesis tests interactively.
• Operational controls (TTL, data tiering, materialized views) that let labs implement lifecycle and retention policies.

Tip: In 2026, several medium labs have cut their analytics bills by 40–70% after migrating QPU telemetry from cloud warehouses to ClickHouse clusters tuned for time-series.

High-level architecture

Keep responsibilities separate and auditable. A typical real-time analytics stack looks like:

1. Experiment producers: Qiskit/Cirq jobs run on cloud or local hardware; SDKs emit JSON/NDJSON logs and parity files.
2. Ingestion layer: lightweight microservices (Cloud Run) or Kafka topics buffer and transform logs into ClickHouse-friendly NDJSON/CSV/Parquet.
3. ClickHouse cluster: MergeTree tables for raw telemetry, materialized views for rollups and aggregations, TTL rules for retention.
4. Object storage (S3/MinIO): canonical experiment artifacts (raw job blobs, circuits, seeds) for reproducibility; ClickHouse stores pointers and hashes — important for provenance and compliance.
5. Notebooks & dashboards: Jupyter/VS Code, Superset, or Grafana for fast exploration and hypothesis testing.

Schema design: time-series qubit telemetry

Design goals:

• High-cardinality support: handle many qubits, experiments, and timestamps efficiently.
• Fast selective reads: slice by experiment, qubit, time window, or calibration batch.
• Retention and tiering: keep raw shots for a window, roll up older data into aggregates. See guidance on data rules and retention in regulation & compliance.

Recommended schema

Core tables: telemetry_raw, experiments, and telemetry_rollup.

-- telemetry_raw: shot-by-shot time-series telemetry
CREATE TABLE telemetry_raw (
  experiment_id UUID,
  run_id String,
  qubit_id UInt16,
  ts DateTime64(6),
  shot_index UInt32,
  measured_bits String, -- e.g. "0101"
  outcome UInt8,        -- 0/1 mapped single-qubit or bitmask
  analog_samples Array(Float32), -- optional waveform samples
  backend String,
  sdk_version String,
  tags Nested(key String, value String)
) 
ENGINE = MergeTree()
PARTITION BY toYYYYMM(ts)
ORDER BY (experiment_id, qubit_id, ts, shot_index)
TTL ts + INTERVAL 90 DAY TO VOLUME 'long_term' -- move or expire
SETTINGS index_granularity = 8192;

-- experiments: metadata and pointers to raw artifacts in object store
CREATE TABLE experiments (
  experiment_id UUID,
  created_at DateTime64(6),
  name String,
  circuit_hash String,
  seed UInt64,
  backend String,
  backend_version String,
  sdk String,
  sdk_version String,
  s3_path String, -- canonical storage pointer for raw job
  primary_key String,
  notes String
) ENGINE = MergeTree()
PARTITION BY toYYYYMM(created_at)
ORDER BY (experiment_id);

-- telemetry_rollup: hourly aggregates used for fast explorations
CREATE MATERIALIZED VIEW telemetry_rollup
ENGINE = AggregatingMergeTree()
PARTITION BY toYYYYMM(ts_hour)
ORDER BY (experiment_id, qubit_id, ts_hour)
AS SELECT
  experiment_id,
  qubit_id,
  toStartOfHour(ts) AS ts_hour,
  countState() AS shots_state,
  sumState(outcome) AS ones_state,
  quantilesState(0.5, 0.9)(arrayMap(x->x, analog_samples)) AS waveform_quantiles_state
FROM telemetry_raw
GROUP BY experiment_id, qubit_id, ts_hour;

Rationale:

• Partition by month keeps partition counts modest and allows monthly maintenance.
• ORDER BY prioritizes queries that filter by experiment_id and qubit and then by time.
• TTL rules move old raw rows to cheaper volumes or expire them after your retention window — a pattern that must be aligned with your legal and audit obligations (regulation & compliance).
• Materialized views with AggregatingMergeTree produce compact hourly aggregates for interactive queries.

Ingesting Qiskit experiment logs into ClickHouse

Qiskit results are JSON-rich. Best practice: persist the canonical job artifact to object storage (S3), then stream shot-level telemetry as NDJSON into ClickHouse with a skinny schema that references the artifact. This keeps ClickHouse storage efficient while maintaining reproducibility — store canonical artifacts with provenance pointers (see provenance guidance).

Python example: direct HTTP insert (synchronous)

from qiskit import IBMQ
import requests, json, uuid, time

# Assume job is done and result is available
job_result = job.result()
experiment_id = str(uuid.uuid4())

# Save canonical artifact to S3 separately and get s3_path
s3_path = upload_to_s3(job_result.to_dict(), key=f"jobs/{experiment_id}.json")

# Stream shots as JSONEachRow to ClickHouse
url = 'http://clickhouse:8123/?query=INSERT+INTO+telemetry_raw+FORMAT+JSONEachRow'

for idx, (ts, meas) in enumerate(extract_shots(job_result)):
    row = {
      'experiment_id': experiment_id,
      'run_id': job.job_id(),
      'qubit_id': meas['qubit'],
      'ts': ts.isoformat(),
      'shot_index': idx,
      'measured_bits': meas['bits'],
      'outcome': meas['outcome'],
      'backend': job.backend().name(),
      'sdk_version': qiskit.__version__
    }
    requests.post(url, data=json.dumps(row))
    time.sleep(0.001)  # small backpressure if needed

Notes:

• For high throughput, batch rows into arrays and insert with a single POST containing many lines of JSONEachRow.
• Use ClickHouse's native Python clients or the HTTP compression (gzip) to reduce bandwidth.
• For mission-critical ingestion, put an intermediate Kafka buffer between your SDK and ClickHouse.

Ingesting Cirq experiment logs

Cirq often returns Pandas-compatible tables. Convert to NDJSON or CSV and use ClickHouse's HTTP insert or clickhouse-client.

import cirq, pandas as pd, requests

result = sampler.run(program, repetitions=1024)
df = pd.DataFrame(result.data)
# normalize columns to (qubit, shot_index, bits, ts)
rows = []
for shot_index, row in df.iterrows():
    for q in row.index:
        rows.append({
            'experiment_id': exp_id,
            'qubit_id': int(q.name.split('_')[-1]),
            'ts': pd.Timestamp.now().isoformat(),
            'shot_index': int(shot_index),
            'measured_bits': str(row[q.name]),
            'outcome': int(row[q.name])
        })

payload = '\n'.join(json.dumps(r) for r in rows)
requests.post(clickhouse_url, data=payload)

Tip: When your payloads include waveform arrays, prefer Parquet for transport and use clickhouse-local or the client bulk loader that supports Parquet.

OLAP queries for rapid hypothesis testing

Once your data is in ClickHouse and you have materialized rollups, you can iterate on hypotheses interactively. Below are common patterns for quantum labs.

1) Per-qubit error rate time-series (using rollups)

SELECT
  experiment_id,
  qubit_id,
  ts_hour,
  (sumMerge(ones_state) / countMerge(shots_state)) AS error_rate
FROM telemetry_rollup
WHERE experiment_id = '...' AND ts_hour BETWEEN now() - INTERVAL 7 DAY AND now()
GROUP BY experiment_id, qubit_id, ts_hour
ORDER BY qubit_id, ts_hour

2) Drift detection: identify qubits with rising error trend

SELECT qubit_id,
  (arrayMax(diff_rates) > 0.02) AS has_spike
FROM (
  SELECT qubit_id,
    arrayMap((x,y)->y-x, arraySlice(rates,1,length(rates)-1), arraySlice(rates,2,length(rates))) AS diff_rates
  FROM (
    SELECT qubit_id, groupArray(error_rate ORDER BY ts_hour) AS rates
    FROM (
      -- subquery using telemetry_rollup as above
    )
    GROUP BY qubit_id
  )
)
WHERE has_spike = 1;

3) Quick A/B calibration test across two backend versions

SELECT backend_version, qubit_id,
  sumMerge(ones_state) / countMerge(shots_state) AS error_rate
FROM telemetry_rollup
WHERE experiment_id IN ('exp_a','exp_b')
GROUP BY backend_version, qubit_id
ORDER BY backend_version, qubit_id;

Advanced pattern: use approximate functions (uniqCombined64, quantiles) for multi-million-row exploratory queries. Use SAMPLE if you need microsecond query response for dashboards.

Cloud Run example: containerized ingestion microservice

Cloud Run lets you host a stateless ingestion endpoint that accepts job artifacts and streams them into ClickHouse. Keep the service minimal — hand off longer processing to worker queues.

# Dockerfile (outline)
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY app.py .
CMD ["gunicorn","-b","0.0.0.0:8080","app:app"]

# app.py (FastAPI simplified)
from fastapi import FastAPI, UploadFile
import requests
app = FastAPI()

@app.post('/ingest')
async def ingest(file: UploadFile):
    data = await file.read()
    # validate and transform into JSONEachRow
    requests.post(CLICKHOUSE_URL, data=data)
    return {"status": "ok"}

Deploy with Cloud Build/Cloud Run and configure IAM to allow only your lab services to POST. Use a signed URL pattern for S3 uploads to decouple large artifact storage.

CI/CD for quantum workflows and data pipelines

Reliable analytics require schema migrations, tests, and reproducible artifacts. Example GitHub Actions flow:

1. Pre-commit: lint DDL, run unit tests on parsers (pytest).
2. PR checks: spin up a ClickHouse test container (ClickHouse server Docker image), run schema migrations, and execute integration inserts with sample Qiskit/Cirq artifacts.
3. Merge: apply migrations to staging ClickHouse, run smoke queries, and deploy ingestion service to Cloud Run.
4. Nightly: archive experiments to S3 and run aggregate jobs to populate telemetry_rollup.

Use dbt-clickhouse or a lightweight migration tool for versioned DDL. Store canonical job artifacts in a content-addressed S3 repo and reference SHA256 hashes from ClickHouse rows to guarantee reproducibility.

Cost, scaling and operational tips

• Compression: columnar compression and skipping indexes make ClickHouse storage competitive with cold object stores for aggregated telemetry.
• Partitioning: monthly partitions are a good default; daily partitions can explode for many experiments.
• Order key: choose ORDER BY to match your common query patterns (experiment_id + qubit_id + ts is a common sweet spot).
• Backups: keep canonical artifacts in object storage and store lightweight pointers in ClickHouse to minimize backup size — combine this with provenance practices for auditability.
• Scale: use sharded ClickHouse clusters for multi-lab, multi-PB setups; single-node installations work well for small-to-medium labs in 2026.

Advanced strategies and 2026 trends

Expect these trajectories in 2026 and beyond:

• Standardized telemetry schemas for qubit experiments, making cross-lab comparisons easier. Adopt a shared schema early to facilitate federated queries.
• Real-time drift detection pipelines that combine ClickHouse SQL alerts with ML models to trigger recalibration automatically — heavy model fitting and parameter estimation may run on GPUs or hybrid stacks described by platforms like Edge AI.
• Hybrid compute: heavy model fitting and T1/T2 parameter estimation run on GPUs, orchestrated by the analytics stack and using ClickHouse as the fast feature store.
• Interoperability with quantum data registries: record provenance (SDK versions, seed, hardware revision) and surface it in analytics for reproducibility audits.

Actionable checklist: get started in one week

1. Design your core schema (use the DDL above as a starting point).
2. Implement an ingestion microservice (Cloud Run) that saves raw job artifacts to S3 and streams NDJSON to ClickHouse.
3. Build a materialized view for hourly rollups and test the rollup queries in notebooks.
4. Create a CI workflow that runs schema migrations and sanity checks against a test ClickHouse container.
5. Run a cost and retention policy review: set TTLs for raw and aggregated tables.

Case study: small lab migration (brief)

In late 2025 a 12-person experimental group migrated their Qiskit shot-level telemetry off a cloud warehouse into a single ClickHouse node with hourly rollups. Results in the first quarter:

• Query latency for typical rollup queries dropped from ~10s to <200ms.
• Monthly analytics spend dropped ~55% thanks to compression and self-hosted infrastructure.
• Faster hypothesis cycles: calibration A/B tests completed in hours instead of days.

Final recommendations

ClickHouse is an excellent fit for labs that need interactive OLAP on shot-level telemetry without the long tail cost of cloud data warehouses. The pattern that works well in 2026: persist canonical artifacts to object storage, keep time-series telemetry in ClickHouse with clear partitioning and rollups, deploy a lightweight ingestion service on Cloud Run, and automate migrations and tests in CI. This approach yields reproducible analytics, fast hypothesis feedback loops, and predictable costs.

Key takeaways

• Architecture: separate artifact storage (S3) from telemetry (ClickHouse).
• Schema: MergeTree + monthly partitions + ORDER BY (experiment_id, qubit, ts).
• Ingestion: stream NDJSON (JSONEachRow) from Qiskit/Cirq; buffer with Kafka for resilience.
• OLAP: use materialized views for rollups and ClickHouse approximations for quick exploration.
• Ops: automate schema migrations, CI integration tests, and TTL-based lifecycle policies.

Want a ready-to-deploy starter repo with ClickHouse schema, Cloud Run ingestion service, and GitHub Actions CI for Qiskit and Cirq? Visit qbitshare to clone the example, or contact our team for a hands-on migration workshop tailored to your lab's scale.

Call to action

If you run experiments with Qiskit or Cirq and are tired of slow analytics cycles or ballooning cloud costs, try the starter kit on qbitshare: a prebuilt ClickHouse schema, Cloud Run ingestion template, and CI pipeline. Start a migration pilot this week and cut inquiry-to-insight times from days to minutes.

Related Reading

• Feature Deep Dive: Live Schema Updates and Zero-Downtime Migrations
• Edge AI at the Platform Level: On‑Device Models, Cold Starts and Developer Workflows (2026)
• Hybrid Edge–Regional Hosting Strategies for 2026: Balancing Latency, Cost, and Sustainability
• Provenance, Compliance, and Immutability: How Estate Documents Are Reshaping Appraisals in 2026
• Short-Term Fixes When Your Ordered Toilet Is Delayed: DIY Plumbing Options
• A Caregiver’s Guide to Media Overload During Big Events: Staying Present When Everyone’s Bingeing
• Common Data Mistakes Blocking AI in Sports—and How to Fix Them
• Quote-Driven Pitch Templates for Selling Unscripted Ideas to Networks (Lessons From BBC-YouTube Talks)
• Tariff Winners and Losers: Scan Your Portfolio for Hidden Trade Risks in 2026