MultiThreadedFileReader

MultiThreadedFileReader reads one chunk per worker on each call. Each worker owns an independent aare::File, while all workers write into non-overlapping regions of the destination buffer. The output is ordered by frame index and successive calls advance through the file.

#include "aare/MultiThreadedFileReader.hpp"

// Four workers, chunks of 128 frames, and at most 10,000 frames.
aare::experimental::MultiThreadedFileReader reader(path, 4, 128, 10'000);
while (reader.remaining_frames() != 0) {
    // Contains at most 4 * 128 frames.
    auto batch = reader.read();
    process(batch);
}

Omit the final argument to read every frame in the source. An explicit value of zero requests an empty result. The low-level read_into overload avoids an allocation when the caller already owns a buffer of at least reader.next_read_bytes() bytes. Use read_all() to read every frame remaining from the current position, and seek() to reposition the reader. Call close() to release all worker file handles early.

Note

Multiple workers do not guarantee faster reads. Performance depends on the storage device and file format, so the thread count and chunk size should be benchmark-driven.

class MultiThreadedFileReader

Read independent chunks of a file in parallel.

Each worker opens its own File instance, so seeking and reading do not share mutable file state. Chunks are written directly to their position in the destination buffer and the resulting frame order is the same as in the file.

Public Functions

MultiThreadedFileReader(std::filesystem::path fname, size_t n_threads, size_t chunk_size, std::optional<size_t> total_frames = std::nullopt)

Parameters:

• fname – path accepted by File

• n_threads – maximum number of worker threads

• chunk_size – number of frames claimed by a worker at a time

• total_frames – number of frames to read, or all frames when omitted

MultiThreadedFileReader(const MultiThreadedFileReader&) = delete

MultiThreadedFileReader &operator=(const MultiThreadedFileReader&) = delete

MultiThreadedFileReader(MultiThreadedFileReader&&) noexcept = default

MultiThreadedFileReader &operator=(MultiThreadedFileReader&&) noexcept = default

size_t read_into(std::byte *destination)

Read one chunk per active worker into a caller-owned buffer.

The buffer must hold at least next_read_bytes() bytes. The reader’s position advances by the returned number of frames. At the end of the configured range this function returns zero and does not access the destination.

std::vector<std::byte> read()

Read the next wave of chunks into an owned byte buffer.

std::vector<std::byte> read_all()

Read every frame remaining from the current position.

void seek(size_t frame_index)

Set the next frame index to read. The end position is valid.

inline size_t tell() const noexcept

Return the next frame index to read.

inline void close() noexcept

Close all worker files. Safe to call more than once.

inline bool is_open() const noexcept

Return whether the worker files are open.

inline size_t n_threads() const noexcept

inline size_t chunk_size() const noexcept

inline size_t total_frames() const noexcept

inline size_t source_total_frames() const noexcept

inline size_t rows() const noexcept

inline size_t cols() const noexcept

inline size_t bitdepth() const noexcept

inline Dtype dtype() const noexcept

inline size_t bytes_per_frame() const noexcept

inline size_t total_bytes() const noexcept

size_t remaining_frames() const noexcept

size_t next_read_frames() const noexcept

inline size_t next_read_bytes() const noexcept

Private Functions

void ensure_open() const

Private Members

std::filesystem::path m_fname

size_t m_n_threads

size_t m_chunk_size

size_t m_total_frames

size_t m_source_total_frames

size_t m_rows

size_t m_cols

size_t m_bitdepth

Dtype m_dtype

size_t m_bytes_per_frame

size_t m_total_bytes

size_t m_current_frame

std::vector<File> m_files