Currently, there is just 1 index and it is fully in-memory (RAM).
ChunkIndex: object_id -> flags, size, pack_id, obj_offset, obj_size # 32 + 4 + 4 + 32 + 4 + 4 = 80B
borg create: for deduplication against already existing objects, we only need to know which object_ids we already have in the repository (and maybe size and flags could be also useful):
DedupIndex: object_id -> flags, size # 32 + 4 + 4 = 40B
We would also have a ChunkIndex, but only for the new objects of that session. This would roughly reduce the RAM usage by 50% for a usual daily backup.
borg extract: to restore a backup, we just need the pack infos:
PackIndex: object_id -> pack_id, obj_offset, obj_size # 32 + 32 + 4 + 4 = 72B
Maybe not worth it, the ChunkIndex is only 10% more.
Currently, there is just 1 index and it is fully in-memory (RAM).
borg create: for deduplication against already existing objects, we only need to know which object_ids we already have in the repository (and maybe size and flags could be also useful):
We would also have a ChunkIndex, but only for the new objects of that session. This would roughly reduce the RAM usage by 50% for a usual daily backup.
borg extract: to restore a backup, we just need the pack infos:
Maybe not worth it, the ChunkIndex is only 10% more.