Feed

A feed is responsible for delivering the data that is required to test and run strategies. There are no restrictions on the type of data that a Feed can emit. It can be a simple as a stock prices or as complex as a satellite image.

A Feed can be used to deliver historic and live data. A feed wraps its information in an event and put that event on a EventChannel. This is all done asynchronously from rest of the trading logic. And although an EventChannel is used to deliver the events, this channel is normally only directly accessed by the Roboquant instance. A strategy will be passed just the plain Event object.

To read more why roboquant has selected this event-driven approach, read this introduction on event-driven software.

Event

An Event instance contains all the data that happens at a certain point in time. It contains the following two attributes

  1. The time attribute when the information within the event became available.

  2. The list of all the actions at that time

Events are always delivered in chronological order.

Time

The time in an event is of the type Instant. It is very important that the time in the event reflects when the actions in that same event became available, not when they happened. Typically, there is some delay between when something occurred and when it is published.

For example when you want to use unemployment numbers as part of your feed, you should use the time when they are published, and you have access to them, not the last day of the period they cover. Otherwise, your strategy is looking into the future during back testing and can make decisions based on information it will never have in live trading.

Action

Action is an abstraction for any type of information that can be made available in an event.

An action can range from price actions, like price-bars and order-book snapshots, to social media content or even satellite images. There is no commonality between actions, and you can add any type of actions you like to the platform. The only requirements is that they implement the empty interface Action

It is up to the strategy to filter for the action types it is interested in, for example:

for (priceBar in event.actions.filterIsInstance<PriceBar>()) {
    // your code
}

There are however a few helper methods in Event to access PriceActions that are of interest of many strategies:

// iterate over all price actions in an event
for (priceAction in event.prices) {
    println(priceAction)
}

// get the first price for a particular asset
val price:Double? = event.getPrice(Asset("XYZ"))

Included Feeds

Roboquant includes several Feed implementations out of the box:

  • CSVFeed that can process CSV files stored somewhere on your local machine.

  • AvroFeed that can process an Avro file stored somewhere on your local machine.

  • RandomWalk that (as the name suggests) generates random price actions for random assets.

  • Integration with third party data providers that make their data available through an API. See also the integration documentation

Custom Feeds

If you have a data source that is not yet covered by roboquant, you can implement your own Feed.The main method to implement is the play method that will put the available events on the channel.

interface Feed {

    val timeFrame: TimeFrame
        get() = TimeFrame.FULL

    suspend fun play(channel: EventChannel)
}

The following implementation of the play method sends out 100 empty events, empty meaning an event without any actions included. In case the EventChannel is full, the channel.send method will be blocking.

suspend fun play(channel: EventChannel) {
    for (i in 1..100) {
        val actions = listOf<Action>() // replace with real actions
        val now = Instant.now()
        val event = Event(actions, now)
        channel.send(event)
    }
}

CSV Feed

CSV files are one of the most common used file formats for storing historic price data. However, since there is no standard defined, they differ a lot depending on where you downloaded them from. Additionally, they often lack some required meta-data, like the currency of the price or exchange they ar traded.

The CSVFeed class in roboquant offers several configuration options to cater to these differences, allowing you to parse almost all CSV files found on the internet.

Basic usage

The simplest usage is to provide the directory that contains the CSV files and the CSVFeed will try to parse them based on defaults and the content it finds.

val feed = CSVFeed("data/test")
roboquant.run(feed)
feed.assets.summary().log()

Configuration

Since the format of CSV files differs a lot, the CSVFeed allows for a lot of configuration to cather to this.

The following properties decide on how the CSV files are parsed

asset.type = STOCK
asset.exchange = AEB
asset.currency = EUR

file.extension = csv
file.skip = ABN.csv ASML.csv

Memory Constraints

The default CSVFeed loads all the data found in all the files into memory during initializing. The benefit is that after initialization the CSVFeed is very fast when used in back tests. And since you reuse the feed in concurrent back-tests, this is becomes even more important when doing extensive back testing.

However, the downside are possible memory constraints. So there is also a CSV Feed implementation that only reads the data when required. This is substantial slower, but consumes a lot less memory. It supports the same configuration, the only difference is that you need to LazyCSVFeed instead of the regular CSVFeed

val feed = LazyCSVFeed("data/test")
roboquant.run(feed)
Tip
Convert the CSV files to an AvroFeed, that way you have the best of both worlds: low memory consumption and high performance. See Convert to an AvroFeed on how to do that.

Avro Feed

Apache Avro™ is a data serialization system. Avro provides:

  • Rich data structures.

  • A compact, fast, binary data format.

  • A container file, to store persistent data.

Avro files make it easy to store large amount of data in a single file and efficiently use it during back testing. The AvroFeed implementation of roboquant only loads data in memory when it is required, so very large datasets can be used without having to worry about memory issues.

Also since the Avro Feed uses a single file, it is easier to distribute than for example directories full of CSV files.

Using an AvroFeed

To use any compatible Avro file as a feed is straight forward.You just provide the file location to the constructor as a parameter and the feed will be ready to use.

val feed = AvroFeed("myfile.avro")
roboquant.run(feed)

A single feed can contain mixed price actions if required.So you can have Trade Prices and Price Bars in the same feed.

Convert to an AvroFeed

You can also convert other feeds into Avro files that you can then later use for back-testing.This works with both historic data feeds as wel as live data feeds.The current limitation is that it only works with price actions and not other type of actions like image data or social media post.

val feed = CSVFeed("some/path/")
AvroUtil.record(feed, "myfile.avro")

// Later you can use the same file in a AvroFeed
val feed2 = AvroFeed("myfile.avro")
Tip
If you work a lot with directories full of CSV files, you can convert them to a single Avro file and use that from then on.This is both faster and has lower memory consumption.

The nice thing is that the recording also works with live feeds.So you can record a live feed for a specified timeframe and then use it from then on in your own back test.

The following example shows how you can record 4 hours of Bitcoin price-bars from Binance.

val feed = BinanceLiveFeed()
feed.subscribePriceBar("BTCUSDT")
val timeframe = Timeframe.next(4.hours)
AvroUtil.record(feed, "bitcoin.avro", timeframe)

Default AvroFeeds

Roboquant comes with a few default AvroFeeds. The main benefit is that you can quickly test if your code is working without having to download and set up your own feeds.

val feed1 = AvroFeed.sp500() // S&P500 for 5 years
val feed2 = AvroFeed.usTest() // 6 popular US stocks for last 60 years

In the future there will be more default AvroFeeds that you can use, also depending on the feedback from the community. Likely candidates that are currently missing are FOREX feeds and cryptocurrency feeds.

Warning
these default feeds should not be relied on for serious back testing.