Feed

What is a feed

A feed is responsible for delivering the data that is needed to test and run strategies. There are no restrictions on the type of data that a Feed can emit. It can be as simple as stock prices or as complex as a satellite images. A Feed is used to deliver historic and live data.

A feed wraps its information in an Event and puts that Event on an EventChannel. This is all done asynchronously from rest of the trading logic. And although an EventChannel is used to deliver the events, this channel is normally only accessed by internal roboquant logic. A strategy will be passed just the Event object.

To read more why roboquant has selected this event-driven approach, read this introduction on event-driven software.

Event

An Event instance contains all the data that happens at a certain point in Time. It contains the following two attributes

  1. The time attribute when the information within the event became available.

  2. The list of all the Action instances at that time. This can be a mixed list of classes that implement the Action interface. A single Event can contain prices, satellite images and social media news at the same time.

Events are always expected to be delivered in chronological order.

Time

The time in an event is of the Java type java.time.Instant. So they can be easily processed without having to worry about timezones. When a feed delivers events, they are always in chronological order.

It is very important that the time in the event reflects when the actions in that same event became available, not when they happened. Typically, there is some delay between when something occurred and when it is published. An event should reflect the published time.

For example when you want to use unemployment numbers as part of your feed, you should use the time when they are published, and you have access to them, not the last day of the period they cover. Otherwise, your strategy is looking into the future during back testing and can make decisions based on information it will never have in live trading.

Action

Action is an abstraction for any type of information that can be made available in an Event. An action ranges from price actions, like price-bars and order-book snapshots, to social media content or images.

There is no commonality between actions, and you can add any type of action to the platform. The only requirements is that they implement the empty interface Action. It is up to the strategy to filter for the action types it is interested in.

For example if your strategy would require PriceBar (candlestick) data, you could use the following pattern to filter this type:

for (priceBar in event.actions.filterIsInstance<PriceBar>()) {
    // your code
}

There are also a few helper methods in the Event class to make access to prices more convenient, since these are of interest of many strategies:

// iterate over all price actions in an event
for (priceAction in event.prices) {
    println(priceAction)
}

// get the first price for a particular asset
val price: Double? = event.getPrice(Asset("XYZ"))

Included Feeds

Roboquant includes several Feed implementations out of the box:

  • CSVFeed that can process CSV files stored somewhere on your local machine.

  • AvroFeed that can process an Avro file stored somewhere on your local machine.

  • RandomWalkFeed that (as the name suggests) generates random price actions for random assets.

  • Integration with third party data providers that make their data available through an API. See also the integration documentation

Custom Feeds

If you have a data source that is not yet covered by roboquant, you can implement your own Feed. The main method to implement is the play method that will put the available events on the EventChannel.

class MYFeed : Feed {

    val timeFrame: Timeframe
        get() = Timeframe.INFINITE

    override suspend fun play(channel: EventChannel) {
        TODO() // Your code goes here
    }
}

The following dummy implementation of the play method sends out 100 empty events, empty meaning an event without any actions included. In case the EventChannel is full, the channel.send method will be blocking.

suspend fun play(channel: EventChannel) {
    for (i in 1..100) {
        val actions = emptyList<Action>() // replace with real actions
        val now = Instant.now()
        val event = Event(actions, now)
        channel.send(event)
    }
}

CSV Feed

CSV files are one of the most common used file formats for storing historic price data. However, since there is no standard defined, they differ a lot depending on where you downloaded them from. Additionally, they often lack some required meta-data, like the currency of the price or exchange they ar traded.

The CSVFeed class in roboquant offers several configuration options to cater to these differences, allowing you to parse almost all CSV files found on the internet.

Basic usage

The simplest usage is to provide the directory that contains the CSV files and the CSVFeed will try to parse them based on defaults and the content it finds.

val feed = CSVFeed("data/test")
roboquant.run(feed)
println(feed.assets.summary())

Configuration

Since the format of CSV files differs a lot, the CSVFeed allows for a lot of configuration to cather to this.

The following properties decide on how the CSV files are parsed

asset.type = STOCK
asset.exchange = AEB
asset.currency = EUR

file.extension = csv
file.skip = ABC.csv XYZ.csv

Memory Constraints

The default CSVFeed loads all the data found in all the files into memory during initializing. The benefit is that after initialization the CSVFeed is very fast when used in back tests. And since you reuse the feed in concurrent back-tests, this is becomes even more important when doing extensive back testing.

However, the downside are possible memory constraints. So there is also a CSV Feed implementation that only reads the data when required. This is substantial slower, but consumes a lot less memory. It supports the same configuration, the only difference is that you need to LazyCSVFeed instead of the regular CSVFeed

val feed = LazyCSVFeed("data/test")
roboquant.run(feed)
Tip
Convert the CSV files to an AvroFeed, that way you have the best of both worlds: low memory consumption and high performance. See Convert to an AvroFeed on how to do that.

Avro Feed

Apache Avro™ is a data serialization system. Avro provides:

  • Rich data structures.

  • A compact, fast, binary data format.

  • A container file, to store persistent data.

Avro files make it easy to store large amount of data in a single file and efficiently use it during back testing. The AvroFeed implementation of roboquant only loads data in memory when it is required, so very large datasets can be used without having to worry about memory issues.

Also since the Avro Feed uses a single file, it is easier to distribute it than for example directories full of CSV files.

Using an AvroFeed

To use any compatible Avro file as a feed is straight forward.You just provide the file location to the constructor as a parameter and the feed will be ready to use.

val feed = AvroFeed("myfile.avro")
roboquant.run(feed)

A single feed can contain mixed price actions if required.So you can have Trade Prices and Price Bars in the same feed.

Convert to an AvroFeed

You can also convert other feeds into Avro files that you can then later use for back-testing. This works with both historic data feeds as wel as live data feeds. The current limitation is that it only works with price actions and not other type of actions like image data or social media post.

val feed = CSVFeed("some/path/")
AvroFeed.record(feed, "myfile.avro")

// Later you can use the same file in a AvroFeed
val feed2 = AvroFeed("myfile.avro")
Tip
If you work a lot with directories full of CSV files, you can convert them to a single Avro file and use that from then on (using the above code snippet as an example). This is both faster and has lower memory consumption.

The nice thing is that the recording also works with live feeds. So you can record a live feed for a specified timeframe and then use it from then on in your own back test.

The following example shows how you can record 4 hours worth of Bitcoin price-bars from the Binance exchange.

val feed = BinanceLiveFeed()
feed.subscribePriceBar("BTCUSDT")
val timeframe = Timeframe.next(4.hours)
AvroFeed.record(feed, "bitcoin.avro", timeframe = timeframe)

Default AvroFeeds

Roboquant comes with a few default AvroFeeds. The main benefit is that you can quickly test if your code is working without having to download or set up your own feeds. There are two feeds available:

  1. A feed with daily PriceBar market data for the S&P 500 stocks for the last 2.5 years

  2. A feed with 5 minutes of price quotes, again for the S&P 500 stocks

val feed1 = AvroFeed.sp500() // S&P500 with daily PriceBar data
val feed2 = AvroFeed.sp500Quotes() // S&P500 with PriceQuote data

In the future there will be more default AvroFeeds that you can use, also depending on the feedback from the community. Likely candidates that are currently missing are FOREX feeds and cryptocurrency feeds.

Warning
these default feeds should not be relied on for serious back testing.