How to provide ticks and candles

Collections of ticks, candlesticks, and derived market data often accumulate to sizes on the order of gigabytes and beyond. Consequently, it becomes essential to store this data in databases or plain files.

To prevent excessive memory allocation and slow loading times, the playground engine incorporates a generator mechanism. This mechanism retrieves data only when necessary, ensuring that only the data directly utilized is stored in memory.

For this particular case, we are using function* generators (opens new window).

Ticks generators

The following is a simple implementation of a tick generator from a file, it reads a file line by line and generates the respective tick.

import * as fs from "node:fs";
import * as readline from "node:readline";

// The CSV format for every line must be: TIMESTAMP,BID,ASK
export async function* readTicksFromFile (file: string, symbol: string): AsyncGenerator<MidaTick | undefined> {
    const reader: readline.Interface = readline.createInterface({
        input: fs.createReadStream(file),
        crlfDelay: Infinity,
    });
    
    for await (const line of reader) {
        if (line.length === 0) {
            continue;
        }

        const csvTick = line.split(",");

        try {
            yield new MidaTick({
                symbol,
                bid: decimal(csvTick[1]),
                ask: decimal(csvTick[2]),
                date: date(csvTick[0]),
            });
        }
        catch (e) {
            console.log(e);
        }
    }

    reader.close();

    return undefined;
};