Hytale World Storage: Persistence, Chunks, and Serialization

When a Hytale world is saved, every modified chunk takes a deterministic trip: its entity components are serialized to a BSON document, the document bytes are zstd-compressed, and the compressed blob is written into a region file keyed by the chunk's local coordinate. This reference walks that path end to end against Hytale server version 2026.03.26, using the real classes in com.hypixel.hytale.storage and com.hypixel.hytale.server.core.universe.world.storage.

The storage layer separates three concerns: a low-level on-disk container (IndexedStorageFile) that knows nothing about chunks, a serialization bridge (BufferChunkLoader / BufferChunkSaver) that turns chunk Holders into byte buffers, and a pluggable provider interface (IChunkStorageProvider) that lets a world pick its backend — flat region files, RocksDB, or a migration chain. These three layers are the key to writing mods that read, transform, or migrate world data safely.

#The IndexedStorageFile Blob Container

IndexedStorageFile (in com.hypixel.hytale.storage) is the foundation. It is a single-file, fixed-slot blob store: a header, an array of blob indexes, then a body of fixed-size segments. Each region file begins with the magic string HytaleIndexedStorage and a version int.

public class IndexedStorageFile implements Closeable {
    public static final String MAGIC_STRING = "HytaleIndexedStorage";
    public static final int VERSION = 1;
    public static final int DEFAULT_BLOB_COUNT = 1024;     // slots per file
    public static final int DEFAULT_SEGMENT_SIZE = 4096;   // bytes per segment
    public static final int DEFAULT_COMPRESSION_LEVEL = 3; // zstd level

    public static IndexedStorageFile open(Path path, OpenOption... options) throws IOException;

    public void writeBlob(int blobIndex, ByteBuffer src) throws IOException;
    public ByteBuffer readBlob(int blobIndex) throws IOException;  // null if empty
    public void removeBlob(int blobIndex) throws IOException;
    public IntList keys() throws IOException;                      // occupied slots
}

A blob is addressed by an integer blobIndex in [0, blobCount). The memory-mapped index region (memoryMapBlobIndexes()) stores, per blob, the first segment index of its data. writeBlob compresses the source with Zstd.compress, allocates a contiguous run of free segments through findFreeSegment, writes the blob header (SRC_LENGTH_OFFSET, COMPRESSED_LENGTH_OFFSET) plus the payload, then atomically points the blob's index at the new segment. readBlob reverses this: read header, pull compressed bytes via readSegments, return Zstd.decompress(src, srcLength).

Concurrency is fine-grained: every blob slot has its own StampedLock (indexLocks[]), free-segment tracking uses a BitSet (usedSegments) guarded by usedSegmentsLock, and segment writes take per-segment locks — so two threads can write different chunks in the same region file without serializing.

#Format versioning and migration

The class carries a deprecated predecessor, IndexedStorageFile_v0, which used a linked-list segment layout (each segment stored a NEXT_SEGMENT_OFFSET) and a temp-index crash-recovery scheme (processTempIndexes reclaims orphaned segments on open). When open() reads a version == 0 file, it calls migrateV0 — move the old file aside, create a fresh v1 file, copy every blob across via readBlob/writeBlob. This is why old worlds upgrade transparently.

#From Chunk to Bytes: BSON Serialization

The container stores opaque byte buffers; the chunk-to-bytes translation lives in two abstract classes. BufferChunkSaver.saveHolder serializes a chunk's entity Holder to BSON and hands the bytes down to a saveBuffer implementation:

public abstract class BufferChunkSaver implements IChunkSaver {
    public CompletableFuture<Void> saveHolder(int x, int z, Holder<ChunkStore> holder) {
        BsonDocument document = ChunkStore.REGISTRY.serialize(holder);
        ByteBuffer buffer = ByteBuffer.wrap(BsonUtil.writeToBytes(document));
        return this.saveBuffer(x, z, buffer);
    }
    public abstract CompletableFuture<Void> saveBuffer(int x, int z, ByteBuffer buffer);
}

BufferChunkLoader.loadHolder is the mirror image — BsonUtil.readFromBuffer rebuilds the BsonDocument, ChunkStore.REGISTRY.deserialize reconstructs the Holder<ChunkStore>, and the chunk's WorldChunk component is re-bound to the world via worldChunkComponent.loadFromHolder(world, x, z, holder).

Serialization is driven by Hytale's component registry. ChunkStore.REGISTRY is a ComponentRegistry<ChunkStore>; its serialize(Holder) / deserialize(BsonDocument) methods walk the chunk entity's components against a codec-defined BSON shape. Crucially, non-serialized components are excluded — the save query is

public static final Query<ChunkStore> QUERY = Query.and(
    WorldChunk.getComponentType(),
    Query.not(ChunkStore.REGISTRY.getNonSerializedComponentType()));

So a chunk is only persisted if it carries a WorldChunk component and is not marked non-serialized (NonSerialized). This is the hook modders care about: attach your custom component with a registered codec and it rides along in the chunk blob automatically; mark it transient and it is dropped on save.

#Region Files: Coordinate Math

IndexedStorageChunkStorageProvider is the default backend. It maps the world's chunk grid onto region files of 32×32 chunks. The arithmetic is worth memorizing because it is the same in the loader, saver, and remover:

int regionX = x >> 5;          // 32 chunks per region
int regionZ = z >> 5;
int localX  = x & 31;          // position within the region
int localZ  = z & 31;
int index   = ChunkUtil.indexColumn(localX, localZ);  // blobIndex 0..1023

The region file name is built by toFileName(regionX, regionZ) as "<x>.<z>.region.bin" (e.g. 3.-2.region.bin); fromFileName parses it back. Files are cached per region in an IndexedStorageCache (a Long2ObjectConcurrentHashMap<IndexedStorageFile>), opened lazily via getOrTryOpen (reads) or getOrCreate (writes), all under world.getSavePath().resolve("chunks").

getIndexes() reconstructs the full set of stored chunks by listing the chunks directory, opening each region, and expanding every occupied blob slot back into a global chunk index via xFromColumn / zFromColumn and the regionX << 5 | localX reverse mapping. This is what feeds world-wide migration and backup tooling.

#Pluggable Backends: IChunkStorageProvider

Every world chooses a backend through world.getWorldConfig().getChunkStorageProvider(). The interface is small and codec-registered, so providers are selectable from world config by a "Type" discriminator:

public interface IChunkStorageProvider<Data> {
    BuilderCodecMapCodec<IChunkStorageProvider<?>> CODEC =
        new BuilderCodecMapCodec<>("Type", true);

    Data initialize(Store<ChunkStore> store) throws IOException;
    IChunkLoader getLoader(Data data, Store<ChunkStore> store) throws IOException;
    IChunkSaver  getSaver(Data data, Store<ChunkStore> store) throws IOException;
    void close(Data data, Store<ChunkStore> store) throws IOException;

    default <O> Data migrateFrom(Store<ChunkStore> store,
                                 IChunkStorageProvider<O> other) throws IOException;
}

The shipped implementations each register a string ID:

DefaultChunkStorageProvider is just a thin delegate whose internal provider is a new IndexedStorageChunkStorageProvider() — so "default" means flat region files today, and its isSame check treats the indexed provider as equivalent to avoid spurious migrations.

The RocksDB backend is the interesting alternative. It opens a db directory with a dedicated chunks column family tuned for spatial data: useFixedLengthPrefixExtractor(8) over an 8-byte big-endian (x, z) key (toKey), IndexType.kHashSearch, a BloomFilter, LZ4 for hot levels with ZSTD at the bottommost level, and blob files with garbage collection enabled. Its getIndexes() iterates the column family with a RocksIterator. Same BSON-over-buffer contract, completely different engine — the point of the abstraction.

#The Save/Load Pipeline at Runtime

Loading and saving are asynchronous and tick-aware. ChunkStore.getChunkReferenceAsync(index, flags) is the single entry point for getting a chunk in memory; the flags come from GetChunkFlags:

public class GetChunkFlags {
    public static final int NONE = 0;
    public static final int NO_LOAD = 1;        // skip disk load
    public static final int NO_GENERATE = 2;    // skip worldgen fallback
    public static final int SET_TICKING = 4;
    public static final int BYPASS_LOADED = 8;
    public static final int POLL_STILL_NEEDED = 16;
}

The default behavior is load-then-generate: getChunkReferenceAsync first calls loader.loadHolder(x, z), and only if disk returns nothing does it fall back to the generator. Per-chunk ChunkLoadState objects (a StampedLock, in-flight future, and exponential failure backoff capped at MAX_FAILURE_BACKOFF_NANOS = 10s) prevent duplicate loads and stop a corrupt chunk being retried in a hot loop.

Saving is driven by ChunkSavingSystems. A WorldChunk is only written when getNeedsSaving() is true and it is not already isSaving(). The ticking system (ChunkSavingSystems.Ticking) polls dirty chunks on a 0.5s timer, fires a cancellable ChunkSaveEvent, copies a serializable snapshot with store.copySerializableEntity(reference), and dispatches up to ForkJoinPool.commonPool().getParallelism() saves per tick. On success it sets ChunkFlag.ON_DISK and clears the dirty flag via consumeNeedsSaving().

// ChunkSavingSystems.saveChunk — the core of one save (condensed)
Holder<ChunkStore> holder = worldChunkComponent.toHolder();
IChunkSaver saver = chunkStore.getSaver();
saver.saveHolder(worldChunkComponent.getX(), worldChunkComponent.getZ(), holder)
     .whenComplete((v, throwable) -> {
         if (throwable == null) worldChunkComponent.setFlag(ChunkFlag.ON_DISK, true);
     });
worldChunkComponent.consumeNeedsSaving();

On world shutdown, ChunkSavingSystems.WorldRemoved runs a blocking full flush — saveChunksInWorld(store).join() — unless the world config has disabled saving (canSaveChunks()), in which case it logs that the world "will not be saved on shutdown." Newly generated chunks are only queued for saving if worldConfig.shouldSaveNewChunks() is set, which is checked in ChunkStore.preLoadChunkAsync before markNeedsSaving().

#Putting It Together

A mod that touches world storage almost always does one of three things: read existing chunks (use getIndexes() + a loader), add persistent data (register a serializable component on ChunkStore.REGISTRY), or migrate formats (configure MigrationChunkStorageProvider). All three ride on the same BSON-over-blob contract, so you rarely touch IndexedStorageFile directly — you let the provider and the component registry do the work, and you mark anything transient as NonSerialized so it never reaches disk.

// Conceptual: enumerate every stored chunk in a world (not literal source)
IChunkStorageProvider<?> provider = world.getWorldConfig().getChunkStorageProvider();
Object data = provider.initialize(store);
try (IChunkLoader loader = provider.getLoader(data, store)) {
    LongSet stored = loader.getIndexes();
    for (long index : stored) {
        int cx = ChunkUtil.xOfChunkIndex(index);
        int cz = ChunkUtil.zOfChunkIndex(index);
        Holder<ChunkStore> holder = loader.loadHolder(cx, cz).join();
        // inspect or transform the chunk's components here
    }
}