Why ClickHouse® fills the disk in self-hosted Langfuse and SigNoz, and how to fix it

Also in Spanish, Portuguese, Russian, Japanese, Korean and Chinese.

Your Langfuse or SigNoz server is running out of disk, but your traces are small. In most public reports the space goes to ClickHouse® itself: to its own log tables, such as system.trace_log and system.text_log, which have no size limit by default. ClickStack has the same problem.

This guide shows how to confirm it with one read-only query, free the space now, and stop it from coming back without hitting the known traps. You don't need any extra tools.

Tested on stock clickhouse/clickhouse-server images 24.8, 25.12 and 26.9 (the SigNoz commands on 25.5 and 25.12), with the ClickHouse service and config taken from Langfuse's and SigNoz's compose files.

Looking up one error message, such as Code 243 or Code 359? See common errors and fixes.

TL;DR

  1. Check. Run one read-only query on system.parts. If system.*_log tables are bigger than your data, ClickHouse's own logs are the problem.
  2. Free space now. TRUNCATE the big logs. For a table over 50 GB, add SETTINGS max_table_size_to_drop = 0.
  3. Stop it coming back. Add a config.d file with a TTL for each log (for opentelemetry_span_log the TTL goes inside <engine>), then restart ClickHouse.
  4. Clean up. Drop the *_log_0, *_log_1 copies that the restart leaves behind.
  5. Still full? Check Docker's container logs, deleted rows and stuck parts. On Langfuse, update Langfuse before you move ClickHouse to 26.8+.

1. What is using the disk? Check ClickHouse disk usage by table

Save this query as size.sql. It lists the biggest tables by size on disk:

SELECT database, table,
       formatReadableSize(sum(bytes_on_disk)) AS size,
       sum(rows) AS rows,
       count() AS parts
FROM system.parts
WHERE active
GROUP BY database, table
ORDER BY sum(bytes_on_disk) DESC
LIMIT 15;

Langfuse (docker compose). Run this in the folder with Langfuse's docker-compose.yml. Its compose file already sets CLICKHOUSE_USER and CLICKHOUSE_PASSWORD inside the container:

docker compose exec -T clickhouse sh -c \
  'clickhouse-client --user "$CLICKHOUSE_USER" --password "$CLICKHOUSE_PASSWORD" --readonly=1 --format PrettyCompact' < size.sql

SigNoz (Docker). The container is signoz-clickhouse if you installed with the compose files from the SigNoz repo (up to v0.129.0), and signoz-telemetrystore-clickhouse-0-0 with Foundry, SigNoz's Docker install since v0.130.0 (check with docker ps). Both leave the default user without a password. The commands below use signoz-clickhouse; with Foundry, put the other name in its place:

docker exec -i signoz-clickhouse clickhouse-client --readonly=1 --format PrettyCompact < size.sql

--readonly=1 makes the session read-only: ClickHouse refuses any change.

How to read it. Compare the system rows with your own data (default for Langfuse, signoz_* for SigNoz). If system.trace_log or system.text_log is at the top, go on to section 2.

2. How do I free the disk space right now?

Open a client with write access:

# Langfuse
docker compose exec clickhouse sh -c 'clickhouse-client --user "$CLICKHOUSE_USER" --password "$CLICKHOUSE_PASSWORD"'
# SigNoz (Foundry: signoz-telemetrystore-clickhouse-0-0)
docker exec -it signoz-clickhouse clickhouse-client

Empty the big logs:

TRUNCATE TABLE system.trace_log;
TRUNCATE TABLE system.text_log;
TRUNCATE TABLE system.metric_log;
TRUNCATE TABLE system.query_log;

This deletes only ClickHouse's own diagnostics, not your traces, and needs no restart.

Langfuse reads system.query_log to follow some of its own queries (#13123). So empty it, but don't switch it off.

TRUNCATE fails with Code 359? Tables over 50 GB

ClickHouse won't drop or truncate a table bigger than max_table_size_to_drop (default 50,000,000,000 bytes, about 46.6 GiB). The limit applies to TRUNCATE too, and a bigger table fails with Code: 359 ... (TABLE_SIZE_EXCEEDS_MAX_DROP_SIZE_LIMIT).

Big logs do hit it: the text_log in Langfuse discussion #15024 was 59.20 GiB. There are two ways around it.

Option A. Lift the limit for one statement:

-- ClickHouse 23.12+: lift the limit for this one statement
TRUNCATE TABLE system.text_log SETTINGS max_table_size_to_drop = 0;

Option B. Create the one-time flag file, then run the plain TRUNCATE:

docker compose exec clickhouse sh -c 'touch /var/lib/clickhouse/flags/force_drop_table && chmod 666 /var/lib/clickhouse/flags/force_drop_table'

(SigNoz: docker exec signoz-clickhouse sh -c '...' with the same command.) ClickHouse deletes the flag after the first DROP or TRUNCATE that needs it, so create it again before the next big table.

The space is back now, but the logs start growing again right away. Section 4 stops that. Section 3 explains why it happens.

3. Why are trace_log and text_log so big?

ClickHouse writes its own diagnostics into tables in the system database: query_log, trace_log, text_log, metric_log and more. The documentation says it plainly: "By default, table growth is unlimited."

Recent default configs set a TTL only for a few small logs (since 25.2, processors_profile_log keeps 30 days), not for the big ones. Two defaults make the big ones grow fast:

  • the query profiler is on and writes stack samples of running queries into trace_log;
  • text_log stores the server log at trace level.

Public examples:

  • Langfuse #13123: system.trace_log was 66.86 GiB, while the Langfuse tables listed there (traces, observations, scores) took about 51 MiB together. The maintainers decided not to override ClickHouse's defaults and added a FAQ page instead.
  • SigNoz #12050: more than 80 GB of system logs (including a trace_log_0 copy) next to less than 500 MB of telemetry. Truncating the system.* tables freed about 80 GB. SigNoz's new Foundry installer sets TTLs; older docker compose installs don't have them.
  • ClickStack Helm chart PR #275: a 10 Gi volume hit 100% after 10 days, with 5.2 GB of trace-level server logs and a 3.7 GB system.text_log against ~70 MB of telemetry. The PR, merged in September 2026, adds a 7-day TTL.

4. How do I set a TTL on ClickHouse system logs?

A TTL makes ClickHouse delete old log rows by itself. Set it after the TRUNCATE from section 2, because of how ClickHouse applies it.

When you add a TTL and restart, ClickHouse doesn't shrink the old table. It renames it to trace_log_0 with all its rows and creates a new one with the TTL. If you truncated first, these copies are tiny, and step 4 drops them.

Run the SQL below in the same write-access client as in section 2.

Step 1: which logs have no TTL?

SELECT name,
       formatReadableSize(total_bytes) AS size,
       extract(engine_full, 'TTL [^S]*') AS ttl
FROM system.tables
WHERE database = 'system' AND engine LIKE '%MergeTree' AND match(name, '_log(_[0-9]+)?$')
ORDER BY total_bytes DESC;

An empty ttl means the log is kept forever. Names like trace_log_0 are old copies; step 4 removes them.

Step 2: add a TTL file in config.d

Save this as clickhouse-ttl.xml next to docker-compose.yml:

<clickhouse>
    <!-- Keep 7 days of ClickHouse's own logs. List only logs your server has. -->
    <query_log><ttl>event_date + INTERVAL 7 DAY DELETE</ttl></query_log>
    <trace_log><ttl>event_date + INTERVAL 7 DAY DELETE</ttl></trace_log>
    <text_log><ttl>event_date + INTERVAL 7 DAY DELETE</ttl></text_log>
    <metric_log><ttl>event_date + INTERVAL 7 DAY DELETE</ttl></metric_log>
    <asynchronous_metric_log><ttl>event_date + INTERVAL 7 DAY DELETE</ttl></asynchronous_metric_log>
    <part_log><ttl>event_date + INTERVAL 7 DAY DELETE</ttl></part_log>

    <!-- The stock config defines this log with <engine>, so a separate <ttl>
         stops the server from starting (ClickHouse#88366). TTL goes inside. -->
    <opentelemetry_span_log>
        <engine>ENGINE = MergeTree PARTITION BY toYYYYMM(finish_date) ORDER BY (finish_date, finish_time_us) TTL finish_date + INTERVAL 7 DAY DELETE</engine>
    </opentelemetry_span_log>
</clickhouse>

Mount it into the clickhouse service (keep your existing volume lines):

  clickhouse:
    volumes:
      - ./clickhouse-ttl.xml:/etc/clickhouse-server/config.d/clickhouse-ttl.xml:ro

Three traps in this step:

  • opentelemetry_span_log takes its TTL inside <engine>. The stock config defines this log with <engine>, and per the docs that conflicts with <ttl>. The server then exits with If 'engine' is specified for system table, TTL parameters should be specified directly inside 'engine' (ClickHouse #88366). Copy PARTITION BY and ORDER BY from SELECT engine_full FROM system.tables WHERE database = 'system' AND name = 'opentelemetry_span_log' (SigNoz's config also sorts by trace_id). Or switch the log off with <opentelemetry_span_log remove="1"/> (Langfuse's scaling docs). remove doesn't delete the existing table, so drop it yourself.
  • List only the logs your server has. A section in the config is what turns a log on (the stock config.xml disables session_log by commenting it out). SigNoz's own config.xml doesn't enable text_log, so delete that line there. Also add processors_profile_log, which has no TTL in that file.
  • Profile settings go into users.d, not config.d. If you also turn off the query profiler (query_profiler_real_time_period_ns and query_profiler_cpu_time_period_ns = 0, as in Langfuse's scaling docs), put that <profiles> block into users.d/. In config.d/ it is silently ignored (trigger.dev #4343).

Step 3: restart ClickHouse

The TTL from config.d takes effect only when the server restarts (Altinity KB). A config reload is not enough. With compose:

docker compose up -d clickhouse     # recreates the container with the new mount
docker compose ps clickhouse

If ClickHouse keeps restarting, the reason is in its error log inside the container. docker compose logs may not show it:

docker cp "$(docker compose ps -aq clickhouse)":/var/log/clickhouse-server/clickhouse-server.err.log .
tail -n 20 clickhouse-server.err.log

Step 4: drop the old trace_log_0 copies

After the restart, every changed log has a copy like trace_log_0 (then _1, _2 after later changes). The copy keeps all the old rows and has no TTL (ClickStack PR #275 upgrade note, Sentry snuba #7311).

This query writes the DROP statements for you. Read them, then paste them into the client:

SELECT 'DROP TABLE system.' || name || ' SYNC SETTINGS max_table_size_to_drop = 0;'
FROM system.tables
WHERE database = 'system' AND match(name, '_log_[0-9]+$');

ClickHouse no longer writes to these tables. DROP can't be undone.

SYNC gives the space back at once. Without it, the system database (Atomic) keeps the data on disk for 8 minutes (database_atomic_delay_before_drop_table_sec = 480 s). SYNC goes before SETTINGS; after it, the statement fails with Code 62 (SYNTAX_ERROR).

Step 5: check the result

Run the step 1 query again. Every big log should show a TTL, and no _N names should be left.

ClickHouse removes expired rows during merges, by default at most every 4 hours (merge_with_ttl_timeout), so the size shrinks over the next hours.

A script can do the checking for you: diskvet runs the step 1 check read-only and prints the TRUNCATE commands, the TTL file and the DROP list for your tables, with the flag for tables over the drop limit (see section 7).

5. Disk still full? Where else the space goes

Compare df -h with the total size of all parts (SELECT formatReadableSize(sum(bytes_on_disk)) FROM system.parts). If the disk holds much more than that, the space is outside ClickHouse's tables.

Are Docker container logs filling the disk?

Docker's default json-file log driver has no size limit (max-size defaults to -1, see the Docker docs). In Langfuse #16339 the ClickHouse container's log reached 89.1 GiB and filled a 244 GiB root disk.

Find the biggest ones:

sudo sh -c 'du -h /var/lib/docker/containers/*/*-json.log | sort -h | tail -5'
docker ps -a --no-trunc --format '{{.ID}} {{.Names}}'   # the folder name is the container ID

To fix it, add a logging: block to every service in docker-compose.yml. In #16339 the Langfuse maintainers recommended doing it for all containers, not only ClickHouse. SigNoz's compose file already sets 50m × 3.

    logging:
      driver: json-file
      options:
        max-size: "50m"
        max-file: "3"

Then run docker compose up -d --force-recreate: log options apply only to newly created containers. Recreating also removes the old log file; in #16339 that freed 81 GiB.

Two notes:

  • If your Docker daemon uses another logging driver, set retention there instead, because driver: json-file overrides it.
  • Langfuse's README (PR #16363) describes the daemon-wide way: set max-size and max-file as daemon defaults, then restart Docker and recreate the containers. It also advises against truncating or rotating these files with external tools.

Are ClickHouse's own log files big?

ClickHouse also writes plain log files to /var/log/clickhouse-server (a volume in Langfuse's compose). By default they are at trace level, rotated at 1000M, with up to 10 old files kept (logger docs).

ClickStack PR #275 uses the values below. Save them as another config.d file, mount it like the TTL file and restart ClickHouse (section 4, step 3):

<clickhouse>
    <logger><level>information</level><size>100M</size><count>10</count></logger>
</clickhouse>

Why do deleted rows still take space?

Langfuse's data retention removes old rows with a lightweight DELETE. ClickHouse only marks such rows; the space comes back when the part is merged.

Background merges skip parts over max_bytes_to_merge_at_max_space_in_pool (150 GiB), and old monthly partitions are rarely merged. In Langfuse discussion #13969 a May partition of ~802 GiB was ~85% deleted rows.

Find the partitions with deleted rows:

SELECT database, table, partition_id, count() AS parts,
       formatReadableSize(sum(bytes_on_disk)) AS size
FROM system.parts
WHERE active AND has_lightweight_delete
GROUP BY database, table, partition_id
ORDER BY sum(bytes_on_disk) DESC;

Then apply the delete mask, one partition at a time (APPLY DELETED MASK):

ALTER TABLE default.observations APPLY DELETED MASK IN PARTITION ID '202605';

Careful: this is a heavyweight mutation. It rewrites the affected parts, so it needs free space for the new copies and loads the disk. Run it off-peak and watch it with SELECT * FROM system.mutations WHERE NOT is_done. In #13969 it freed ~360 GiB per replica in ~17 minutes.

If the query shows ids like patch-f18f7271629a324b0d26b6ad0b83a6c2-202605, the DELETE ran with lightweight_delete_mode = 'lightweight_update'. The mask is then in these small patch parts, and the big parts it applies to keep has_lightweight_delete = 0, so the query doesn't show them. With the full patch- id the command runs without an error and removes nothing. Use the plain id, everything after patch-<hash>- (202605 here, 202605-1 with a tuple partition key), and run the command twice: the first run only writes the mask into the parts, the second removes the rows. Pasting both at once works:

ALTER TABLE default.observations APPLY DELETED MASK IN PARTITION ID '202605';
ALTER TABLE default.observations APPLY DELETED MASK IN PARTITION ID '202605';

The patch parts can stay in the query for a few minutes after that (test details).

Since v3.179.0 the Langfuse worker has a cleaner, off by default: LANGFUSE_CLICKHOUSE_DELETED_MASK_CLEANER_ENABLED=true (PR #14035). But it picks only patch- partitions and skips the current month (helpers.ts), and a DELETE writes those only with CLICKHOUSE_LIGHTWEIGHT_DELETE_MODE=lightweight_update on a table with the _block_number and _block_offset columns (requirements). With the default alter_update (env.ts), and on v3's tables, which have neither column, it finds nothing: run APPLY DELETED MASK yourself.

Are old parts stuck on disk? (inactive and detached parts)

SELECT database, table, count() AS parts,
       formatReadableSize(sum(bytes_on_disk)) AS size,
       min(remove_time) AS oldest, max(refcount) AS max_refcount
FROM system.parts
WHERE NOT active
GROUP BY database, table
ORDER BY sum(bytes_on_disk) DESC;

Inactive parts are left over from merges. ClickHouse deletes them after old_parts_lifetime (480 s), once no query uses them (refcount = 1).

If they pile up (112.55 GiB of them in Langfuse discussion #15024) and oldest is hours ago, look for long queries in system.processes and stop them with KILL QUERY WHERE query_id = '...'.

Detached parts stay on disk until you remove them:

SELECT database, table, reason, count() AS parts,
       formatReadableSize(sum(bytes_on_disk)) AS size
FROM system.detached_parts
GROUP BY database, table, reason;

Check reason first. Remove a part with ALTER TABLE <db>.<table> DROP DETACHED PART '<name>' SETTINGS allow_drop_detached = 1, not by deleting folders.

6. Upgrading ClickHouse? Check this first

  • Langfuse + ClickHouse 26.8 or newer: update Langfuse first. ClickHouse 26.8 changed the default of input_format_read_datetime_number_as_raw_value from 1 to 0. Older Langfuse sends DateTime64 values as millisecond numbers, and on 26.8+ they end up stored as 9999-12-31 23:59:59 (Langfuse #16858: 0.59% to 21.43% of writes per project). The fix is PR #16892 (backported to v3 in PR #16957), released in Langfuse v4.28.0 and v3.225.7. Langfuse partitions by month, so such rows land in partition 999912. Check your version and look for them:

    SELECT version();
SELECT database, table, partition_id, count() AS parts
FROM system.parts
WHERE active AND startsWith(partition_id, '9999')
GROUP BY database, table, partition_id;

  • After any upgrade, run the step 1 query from section 4 again. When a release changes a log's schema, ClickHouse renames the current table and creates a new one, so new _N copies appear. Drop them as in step 4.

7. Check everything at once

diskvet is an open-source (Apache-2.0) read-only script that runs most of the checks above and prints a report with the fix commands. It can't see Docker's log files, but it shows how much of the disk is outside ClickHouse's parts.

On the machine with Langfuse's docker-compose.yml (read checks.sql before you run it):

curl -fsSLO https://github.com/Protemir/diskvet/releases/latest/download/diskvet.sh
curl -fsSLO https://github.com/Protemir/diskvet/releases/latest/download/checks.sql
sh diskvet.sh report --docker auto > report.md

For SigNoz, run the same command: --docker auto finds its ClickHouse by the image (clickhouse/clickhouse-server), or by the service clickhouse in the folder of SigNoz's old docker-compose.yaml. With other ClickHouse containers on the machine, pass the name from docker ps: --docker signoz-telemetrystore-clickhouse-0-0 (Foundry) or --docker signoz-clickhouse (old compose files).

ClickHouse in Kubernetes works too, through kubectl exec into its pod:

sh diskvet.sh report --k8s auto > report.md

--k8s NAMESPACE/POD picks the pod. The password stays inside the pod, and diskvet needs only get and list on pods and pods/exec. Which pod each Helm chart runs and where its fixes go: Kubernetes recipe.

It reads only metadata from system.* tables (system.tables, system.parts, system.disks, system.detached_parts, system.mutations, system.part_log and a few more), with readonly=2 and resource limits. It never reads rows of your tables, system.query_log or query texts, and sends nothing anywhere. Nothing runs by itself: you read each fix and run it.

An hourly version that emails you before the disk fills is planned. If you'd use it, say so in Early access: it gets built if enough people ask, and it will be free.

Sources

ClickHouse documentation and source:

Docker:

Langfuse:

SigNoz, ClickStack and others:

ClickHouse is a registered trademark of ClickHouse, Inc. diskvet is not affiliated with ClickHouse, Inc. Langfuse, SigNoz and ClickStack are trademarks of their respective owners.

Source on GitHub