Feed

What is a feed

A feed is responsible for delivering the data that is needed to test and run strategies. There are no restrictions on the type of data that a Feed can emit. It can be as simple as stock prices or as complex as satellite images. A Feed is used to deliver historic and live data.

A feed wraps its information in an Event and puts that Event on an EventChannel. This is all done asynchronously from rest of the trading logic. And although an EventChannel is used to deliver the events, this channel is normally only accessed by internal roboquant logic. A strategy will be passed just the Event object.

To read more about why roboquant has selected this event-driven approach, read this introduction to event-driven software.

Event

An Event instance contains all the data that happens at a certain point in Time. It contains the following two attributes

  1. The time attribute when the information within the event became available.

  2. The list of all the Action instances at that time. This can be a mixed list of classes that implement the Action interface. A single Event can contain prices, satellite images and social media news at the same time.

Events are always expected to be delivered in chronological order.

Time

The time in an event is of the Java type java.time.Instant. So they can be easily processed without having to worry about timezones. When a feed delivers events, they are always in chronological order.

It is very important that the time in the event reflects when the actions in that same event became available, not when they happened. Typically, there is some delay between when something occurred and when it is published. An event should reflect the published time.

For example when you want to use unemployment numbers as part of your feed, you should use the time when they are published, and you have access to them, not the last day of the period they cover. Otherwise, your strategy is looking into the future during back testing and can make decisions based on information it will never have in live trading.

Action

Action is an abstraction for any type of information that can be made available in an Event. An action ranges from price actions, like price-bars and order-book snapshots, to social media content or images.

There is no commonality between actions, and you can add any type of action you like to the platform. The only requirements is that they implement the empty interface Action. It is up to the strategy to filter for the action types it is interested in.

For example if your strategy would require PriceBar (candlestick) data, you could use the following pattern to filter this type:

for (priceBar in event.actions.filterIsInstance<PriceBar>()) {
    // your code
}

There are also a few helper methods in the Event class to make access to prices more convenient, since these are of interest to many strategies:

// iterate over all price actions in an event
for (priceAction in event.prices) {
    println(priceAction)
}

// get the first price for a particular asset
val price: Double? = event.getPrice(Asset("XYZ"))

Included Feeds

Roboquant includes several Feed implementations out of the box:

  • CSVFeed that can process CSV files stored somewhere on your local machine.

  • AvroFeed that can process an Avro files stored somewhere on your local machine.

  • RandomWalkFeed that (as the name suggests) generates random price actions for random assets.

  • Integration with third party data providers that make their data available through an API. See also the integration documentation.

Custom Feeds

If you have a data source that is not yet covered by roboquant, you can implement your own Feed. The main method to implement is the play method that will put the available events on the EventChannel.

class MYFeed : Feed {

    val timeFrame: Timeframe
        get() = Timeframe.INFINITE

    override suspend fun play(channel: EventChannel) {
        TODO() // Your code goes here
    }
}

The following dummy implementation of the play method sends out 100 empty events, empty meaning an event without any actions included. When the EventChannel is full, the channel.send method will be blocking.

suspend fun play(channel: EventChannel) {
    for (i in 1..100) {
        val actions = emptyList<Action>() // replace with real list of actions
        val now = Instant.now()
        val event = Event(actions, now)
        channel.send(event)
    }
}

CSV Feed

CSV files are one of the most common used file formats for storing historic price data. However, since there is no standard defined, they differ a lot depending on where you downloaded them from. Additionally, they often lack some required meta-data, like the currency of the price or the exchange where they are traded.

The CSVFeed class in roboquant offers several configuration options to cater to these differences, allowing you to parse almost any CSV files found on the internet.

Basic usage

The simplest usage is to provide the directory that contains the CSV files and the CSVFeed will try to parse them based on defaults and the content it finds.

val feed = CSVFeed("data/test")
roboquant.run(feed)
println(feed.assets.summary())

Configuration

Since the format of CSV files differs a lot, the CSVFeed allows for a lot of configuration to cater to this.

The following properties determine how the CSV files are parsed

asset.type = STOCK
asset.exchange = AEB
asset.currency = EUR

file.extension = csv
file.skip = ABC.csv XYZ.csv

In order to make it easier to use CSVFeed, there are several popular market data providers of CSV feed pre-configured.

// CSV files downloaded from stooq.pl
val feed1 = CSVFeed.fromStooq("somepath")

// CSV files downloaded from histdata.com
val feed2 = CSVFeed.fromHistData("somepath")

Memory Constraints

The default CSVFeed loads all the data found in all the files into memory during initialization. The benefit is that after initialization the CSVFeed is very fast when used in back tests. And since you can reuse the feed in parallel back-tests, this becomes even more important when doing extensive back testing.

However, a downside is memory usage when working in a memory-constrained environment. So there is also a CSV Feed implementation that only reads the data when required. This is substantial slower, but consumes a lot less memory. It supports the same configuration, the only difference is that you need to use LazyCSVFeed instead of the regular CSVFeed

val feed = LazyCSVFeed("data/test")
roboquant.run(feed)
Convert the CSV files to an AvroFeed, that way you have the best of both worlds: low memory consumption and high performance. See Convert to an AvroFeed on how to do that.

AvroFeed

Apache Avro™ is a data serialization system. Avro provides a compact, fast, binary data format. Avro file format is a row-based repository, making a good match for event-driven architectures like roboquant.

Avro files make it easy to store large amount of data in a single file and efficiently use it during back testing. The AvroFeed implementation of roboquant only loads prices in memory when it is required, so very large datasets can be used without having to worry about memory issues. Also since an AvroFeed uses a single file to store its prices, it is easier to distribute it than for example many directories full of CSV files.

As you might have guessed, the use of the AvroFeed comes highly recommended. It is fast and scales well with large sets of data.

Using an AvroFeed

Using any compatible Avro file as a feed is straightforward. You just provide the file location to the constructor as a parameter and the feed will be ready to use.

val feed = AvroFeed("my_file.avro")
roboquant.run(feed)

A single feed can contain mixed price actions if required, so you can have Trade Prices and Price Bars in the same feed.

Convert to an AvroFeed

You can also convert other feeds into Avro files that you can later use for back-testing. This works with both historic data feeds and live data feeds. The current limitation is that it only works with price actions and not other type of actions like image data or social media posts.

val fileName = "my_file.avro"

val feed = CSVFeed("some/path/")
AvroFeed.record(feed, fileName)

// Append to existing avro file
val feed2 = CSVFeed("some/path2/")
AvroFeed.record(feed2, fileName, append = true)

// Later you can use the same file in a AvroFeed
val feed3 = AvroFeed(fileName)
If you work a lot with directories full of CSV files, you can convert them to a single Avro file and use that from then on (using the above code snippet as an example). This is both faster and has lower memory consumption.

The nice thing is that the recording also works with live feeds. So you can record a live feed for a specified timeframe and then use it from then on in your own back test.

The following example shows how you can record 4 hours worth of Bitcoin price-bars from the Binance exchange.

val feed = BinanceLiveFeed()
feed.subscribePriceBar("BTCUSDT")
var timeframe = Timeframe.next(4.hours)
AvroFeed.record(feed, "bitcoin.avro", timeframe = timeframe)

// You can also append to an existing Avro file
timeframe = Timeframe.next(8.hours)
AvroFeed.record(feed, "bitcoin.avro", timeframe = timeframe, append = true)

Default AvroFeeds

Roboquant comes with a few default AvroFeeds. The main benefit is that you can quickly test if your code is working without having to download or set up your own feeds. There are two feeds available:

  1. A feed with daily PriceBar market data for the S&P 500 stocks for the last few years

  2. A feed with a few minutes of price quotes, again for the S&P 500 stocks

val feed1 = AvroFeed.sp500() // S&P500 with daily PriceBar data
val feed2 = AvroFeed.sp500Quotes() // S&P500 with PriceQuote data
val feed3 = AvroFeed.forex() // EUR/USD forex data

In the future there will be more default AvroFeeds that you can use, depending on the feedback from the community. Likely candidates that are currently missing are FOREX feeds and cryptocurrency feeds.

These default feeds should not be relied on for serious back testing. They just serve to quickly run a back test and see if your strategy and policy are behaving as expected.