pg_diskann PQ mode causes TOAST corruption during vector insert (possibly after deleting items and index rebuild doesnt work correctly)

Hello Geoff Fletcher,
It sounds like you’re running into TOAST-table corruption when using DiskANN’s Product Quantization (PQ) mode for vector inserts. In PQ mode your vectors get chunked and stored in a pg_toast table, and it looks like some chunks are going missing or getting orphaned—hence the “missing chunk number 0” error in pg_toast_27911. You said disabling PQ makes ingestion succeed, and rebuilding the index temporarily fixes it, but it recurs after deletes + index rebuild.

Here are a few things to try and investigate:

Vacuum / bloat on the TOAST table

• After big deletes, pg_toast_xxxx can accumulate dead chunks. If autovacuum isn’t keeping up, you’ll see orphaned TOAST entries.
  • Run a manual VACUUM (FULL, ANALYZE) on the base table (or directly on pg_toast_27911) to compact and reclaim space.
    • Consider using pg_repack to defragment without downtime.
    Check autovacuum settings

    - Verify autovacuum is active on your database and that there aren’t long-running transactions holding old snapshots (which block TOAST cleanup).
    
       - Monitor `pg_stat_progress_vacuum` for the toast table. If it never fires, you may need to tune `autovacuum_vacuum_cost_delay`, `autovacuum_work_mem`, or increase `autovacuum_max_workers`.
    
       Confirm index rebuild actually cleans up TOAST
    
          - A plain `REINDEX` may not reclaim all orphaned toast chunks. Pair it with `VACUUM FULL` or drop & recreate the table to be sure.
    
          Check DiskANN/PQ versions and known bugs
    
             - There have been reports of missing‐chunk issues in earlier PQ implementations. If you’re not on the latest DiskANN build, try upgrading.
    
                - As a temporary workaround, use a flat index or quantizedFlat mode to see if the behavior changes.
    
                Inspect server logs around inserts
    
                   - Look for any autovacuum errors or disk-IO warnings.
    
                      - Verify there’s no out-of-disk or resource exhaustion happening right before your final batch fails.
    

Hopefully one of those steps uncovers the culprit. If the issue persists, we can dig deeper once we have more details.

The described TOAST corruption during inserts with pg_diskann Product Quantization (PQ) enabled is not covered in the provided documentation, and no specific fix or known issue is documented in the context.

The available guidance for pg_diskann with PQ is limited to:

1. PQ must be enabled explicitly on the index:

      CREATE INDEX demo_embedding_diskann_idx ON demo USING diskann(embedding vector_cosine_ops) 
      WITH(
          product_quantized=true
      );
   

2. PQ is supported starting from pg_diskann v0.6 and above.
3. For high-dimensional embeddings (up to 16,000 dimensions), PQ must be turned on and recommended settings are:
   • product_quantized = true
   • pq_param_num_chunks = embedding_dimension / 3
   • pq_param_training_samples is auto-determined unless explicitly set.
4. To improve accuracy with PQ, use a two-step query with approximate search followed by full-vector reranking.
5. For certain DiskANN index metadata/version errors, the documented recovery is to run REINDEX / REINDEX CONCURRENTLY or rebuild/upgrade the index, but there is no mention of TOAST corruption or a related bug.

Given the context, only the following supported actions can be recommended:

1. Ensure pg_diskann version
   • Confirm that pg_diskann is v0.6 or higher, as PQ support is only documented from v0.6 onward.
2. Rebuild or reindex when index metadata issues occur
   • For index metadata or version issues, the documented recovery pattern is:

          REINDEX INDEX your_index_name;          -- or REINDEX CONCURRENTLY
          -- or
          DROP INDEX your_index_name;
          CREATE INDEX your_index_name ON your_table USING diskann(your_vector_column vector_cosine_ops);
     

   • And, when prompted by a version-upgrade message, using upgrade_diskann_index() to upgrade all DiskANN indexes.
   • Although this pattern is documented for specific DiskANN errors (not TOAST corruption), it is the only supported remediation pattern in the context for index-related issues.
3. Use PQ with recommended parameters for large/high-dimensional workloads
   • When PQ is enabled, follow the documented recommendations for pq_param_num_chunks and allow pq_param_training_samples to be auto-determined unless there is a specific need to override it.

Because the context does not document TOAST corruption, a bug, or a configuration workaround for this specific behavior, no further product-specific remediation steps can be provided from the supplied material.

References:

• Enable and use DiskANN extension