Structure of Journal Data File
Each batch record starts with a batch header. Batch header consists of the batch header size, record type, magic string, size of the batch and checksum. Each component in the header has its own significance.
- Batch header size – this lets us know how many bytes to skip to reach to the first journal record.
- Record type – 2
- Magic string – ‘WRITE_BATCH’
- Size of the batch – Size of the journal records in the batch (in bytes).
- Checksum – Journal records data in the batch must correspond to this checksum else the data will be considered corrupted.
A separate thread is started to write the batch, it will be in waiting state till a new batch kicks in.
On notifying the thread of a new batch, it wakes up to writing the journal records. First, the batch header is written followed each journal record till it reaches the last journal record in the batch.
A new batch may get started during the time the thread is processing the batch. Every write that is issued during this time will get appended to the batch. Only when the batch thread is done writing the batch, will it take up the new batch.
If the maximum batch size is reached or the journal file has reached its maximum length, the write will not be written to the current batch and will wait for the batch to be processed. If the journal file has reached its maximum length, a new file will be created.
Journal data is stored in one or more data files. A data file consists of batches. Each batch in turn consists of journal records. Each data file is validated to make sure there are no corrupted blocks. In order to validate the data file, following two things are checked:
- A batch record should start with the batch header. Except the last two components – the batch size and checksum, each byte is checked to make sure it is a batch record.
- The batch size lets us read the journal data which will be used to calculate the checksum. The calculated checksum must match with the header checksum. If it doesn’t match then it means the batch is somehow corrupted.
Once the entire journal data file is validated, each data file under it will be aware of the corrupted blocks.
Since a corrupted block is nothing but the range of corrupt bytes, it knows the start (x) to end (y) offset in the data file.