Integration

There are several categories of functionality available from 3rd party providers:

Historic data feeds
Live data feeds
Broker functionality
Exchange rates

In some cases, a single 3rd party provides all types of functionality, or sometimes they are just offer a subset. You can combine several of these 3rd parties together in a single run, for example, using a data feed from one provider and the broker functionality from another one.

One of the design principles is that roboquant tries to enforce is avoiding exposing 3rd party APIs directly. Typically, these APIs are wrapped in roboquant specific APIs and types. This makes it easier to switch between different providers and also reduces the learning curve.

Modules & Packages

The third party providers have their own modules and are not part of the core roboquant module:

roboquant-extra for the "traditional" broker and data feed providers
roboquant-crypto for cryptocurrency related 3rd parties
roboquant-ibkr for Interactive Brokers

You can include these modules in your Maven or Gradle build. In order to use the 3rd party integration, you’ll have to import the package. All third party integrations have their own package directly under org.roboquant.

So for example, you can import all the required classes for Alpaca as follows:

import org.roboquant.alpaca.*

This wildcard import is especially useful if you use roboquant in a Jupyter Notebook. If you use roboquant in an IDE like IntelliJ IDEA, the IDE will automatically suggest which packages and classes to import for you.

Configuration

When you want to integrate with a 3rd party, you most likely require credentials to gain access to their APIs. Often this is in the form of an API key and/or secret. You can include these credentials directly in your source code. However, this is not considered good practice since it is easy by mistake to version these files and make the credentials available to a wider audience.

So roboquant offers better and more secure alternatives. The following list shows the order of steps roboquant uses to find these configuration parameters:

If the credentials are provided directly in the code when calling the API, use these, otherwise go to the next step.

// instantiation with hard-coded configuration
val feed = AlpacaHistoricFeed {
    publicKey = "123"
    secretKey = "456"
    dataType = DataAPIType.IEX
}

If the credentials are provided as startup parameter to the JVM with the -D syntax, use these, otherwise go to next step.
```
java myapp.jar -Dalpaca.secret.key=... -Dalpaca.pubic.key=...
```
If the credentials are set as system environment variable (export my_key="some value"), use these, otherwise go to the next step. In this case use _ and not . in the key names.
```
export alpaca_secret_key=...
export alpaca_public_key=...
java myapp.jar
```
If the credentials are set in a property file called .env in the working directory, use these, otherwise go to next step.
If the credentials are set in a property file called dotenv in the working directory, use these, otherwise go to next step. The reason to support this variation is that .env files are hidden in Jupyter notebook directories and then dotenv is a good alternative.
If the credentials are set in a property file called .env in the $HOME/.roboquant directory, use these, otherwise not found.

The following provides an overview of the used properties in roboquant:

# This property file contains configuration settings that will be made available to roboquant
# Many data feed providers and brokers require one or more keys to be able to access their APIs,
# and this is one of the ways to provide those credentials.
#
# If you store credentials in here, make sure other people don't have access to it and
# don't version control it to a public repo by mistake.
# One solution is to place this file at ~/.roboquant/.env

# Sample entry
# sample.public.key=some_key_value
# sample.secret.key=another_key_value

# Uncomment the following two lines and change the values for Alpaca API access
# alpaca.public.key=your_api_key_id
# alpaca.secret.key=your_api_secret_key

# Uncomment the following line and change the value for Alpha Vantage access
# alphavantage.key=your_api_key

# Uncomment the following two lines and change the values for Binance access
# binance.public.key=your_key
# binance.secret.key=you_secret

# Uncomment the following line and change the value for Polygon.io access
# polygon.key=your_key

Supported 3rd parties

The following table shows which 3rd parties are supported out of the box and what functionality the integration offers:

3rd party	Type	Historic feed	Live feed	Broker	Exchange rates
Interactive Brokers	API	✓	✓	✓	✓
Alpaca	API	✓	✓	✓
Polygon.io	API	✓	✓
Alpha Vantage	API	✓
Binance	API	✓	✓	✓
XChange (60+ crypto exchanges)	API	✓	✓	✓
ECB	CSV File				✓
Stooq	CSV File	✓
MT5	CSV File	✓
HistData	CSV File	✓
Kraken	CSV File	✓
Yahoo Finance	CSV File	✓

3rd party

Type

Historic feed

Live feed

Broker

Exchange rates

Interactive Brokers

API

✓

Alpaca

API

✓

Polygon.io

API

✓

Alpha Vantage

API

✓

Binance

API

✓

XChange (60+ crypto exchanges)

API

✓

ECB

CSV File

✓

Stooq

CSV File

✓

MT5

CSV File

✓

HistData

CSV File

✓

Kraken

CSV File

✓

Yahoo Finance

CSV File

✓

Roboquant is open source software that is developed independently of these 3rd party companies and has no associations with them. There is also no direct on indirect recommendation using them.

Alpaca

The integration with Alpaca covers their broker API as well as their historic and live market data APIs. In order to use these, you’ll need at least register for a paper-trading account and create an API-key.

val feed = AlpacaHistoricFeed()
val tf = Timeframe.past(100.days)

// You can retrieve historic price-bars, quotes or trades
feed.retrieveStockPriceBars("AAPL", "JPM", "TSLA", timeframe = tf)
feed.retrieveStockQuotes("AAPL", "JPM", "TSLA", timeframe = tf)
feed.retrieveStockTrades("AAPL", "JPM", "TSLA", timeframe = tf)

val feed = AlpacaLiveFeed()
feed.subscribeStocks("AAPL", "IBM")

val tf = Timeframe.next(120.minutes)
roboquant.run(feed, tf)

// instantiation with hard-coded configuration, rather than using dotenv property file
val broker = AlpacaBroker {
    publicKey = "123"
    secretKey = "456"
    accountType = AccountType.PAPER
    dataType = DataAPIType.IEX
}
println(broker.account.summary())
println(broker.availableAssets)

// place a market order to buy 100 stocks Apple
val order = MarketOrder(Asset("AAPL"), 100)
broker.place(listOf(order))

Polygon

Polygon has several type of subscriptions available, depending on your needs. The free version only provides access to historic data, while the paid versions also provide live data.

val feed = PolygonHistoricFeed()
val tf = Timeframe.past(100.days)
feed.retrieve("AAPL", "JPM", "TSLA", timeframe = tf)

val feed = PolygonLiveFeed()
feed.subscribe("AAPL", "JPM", "TSLA")

val tf = Timeframe.next(120.minutes)
roboquant.run(feed, tf)

ECB

The ECB (European Central Bank) publishes since the start of the Euro the daily historic exchange rates with almost all other fiat currencies in the world. ECBExchangeRates will automatically download the exchange rates from their website and use them for currency conversions during trading. There is no need for registration or an API key, since this data is available on their public website.

// Download the latest rates from the ECB website
// The results are cached by default
Config.exchangeRates = ECBExchangeRates.fromWeb()

// Create a wallet holding different currencies
val wallet = 100.EUR + 20.USD + 1000.JPY

// convert the wallet to GBP at today's exchange rates
wallet.convert(Currency.GBP)

//  convert the wallet to GBP using 5 years ago exchange rates
wallet.convert(Currency.GBP, Instant.now() - 5.years)

Binance

The integration with Binance covers their broker API as well as their historic and live market data APIs. From an algo-trading perspective, it is also nice to see that the access to (high-frequency) market data is free. Even without a trading account with them, you can still access the data.

val feed = BinanceHistoricFeed()
println(feed.availableAssets.summary())

// Retrieve 1-minute candlesticks for two currency pairs for the last day
val timeframe = Timeframe.past(1.days)
feed.retrieve("BTCBUSD", "ETHBUSD", timeframe = timeframe, interval = Interval.ONE_MINUTE)

To subscribe to live quotes for one or more currency pairs is just as easy:

val feed = BinanceLiveFeed()
println(feed.availableAssets.summary())

// subscribe to live quotes for two currency pairs
feed.subscribePriceQuote("BTCBUSD", "ETHBUSD")

Currently, the broker integration with Binance is largely untested and core features are disabled to avoid costly bugs. People have been using it as a starting point for their own implementation, but the included implementation should not be relied upon yet.

val broker = BinanceBroker {
    publicKey = "123"
    secretKey = "456"
}
println(broker.account.fullSummary())

XChange

XChange is a library providing a consistent API for interacting with 60+ Bitcoin and other cryptocurrency exchanges, providing an interface for trading and accessing market data.

Roboquant integrates with XChange for retrieving historic data, live data and accessing trading functionality. Because there is support for so many crypto exchanges, the initial setup is a bit more elaborate than with other 3rd party providers.

You’ll always have to add the additional exchange-specific libraries to your project. We’ll use Maven and Bitstamp as an example:

<dependencies>
    <dependency>
        <groupId>org.knowm.xchange</groupId>
        <artifactId>xchange-bitstamp</artifactId>
        <version>5.1.0</version>
    </dependency>
    <dependency>
        <groupId>org.knowm.xchange</groupId>
        <artifactId>xchange-stream-bitstamp</artifactId>
        <version>5.1.0</version>
    </dependency>
</dependencies>

To get live data, you’ll first need to acquire an instance of a StreamingExchange. Using Bitstamp again as an example, you can use the following code snippet:

val exchange = StreamingExchangeFactory.INSTANCE.createExchange(BitstampStreamingExchange::class.java)

// Connect to the Exchange WebSocket API. Here we use a blocking wait.
exchange.connect().blockingAwait()

Now you are ready to create an instance of XChangeLiveFeed and subscribe to one or more currency pairs.

val feed = XChangeLiveFeed(exchange)

// subscribe to live quotes for two currency pairs
feed.subscribeTicker("BTC_USD", "ETH_USD")

Alpha Vantage

Alpha Vantage provides a rich set of historic stock prices, not only for stocks listed on US exchanges. Although Alpha Vantage also provides other information besides stock market data, that is not yet supported by the AlphaVantageHistoricFeed.

val feed = AlphaVantageHistoricFeed()

// You can retrieve the end-of-day prices
val assets = listOf(
    // regular US stock
    Asset("AAPL"),

    // stock listed on a non-US exchange
    Asset("DAI.DEX", currency = Currency.EUR, exchange = Exchange.DEX)
)
feed.retrieveDaily(*assets.toTypedArray())

// You can retrieve intra-day prices
feed.retrieveIntraday(Asset("TSLA"), interval = Interval.ONE_MIN)

Interactive Brokers

Interactive Brokers integration requires running either TWS or IB Gateway on a local machine. So you cannot directly connect from roboquant to the IBKR servers. Other than that, the integration is the same as other brokers and data feeds.

IB Gateway provides some benefits over TWS, in that you can more easily monitor the messages being exchanged, and it is a lighter on resources.

val feed = IBKRHistoricFeed()
val asset = Asset("TSLA")
feed.retrieve(asset)
roboquant.run(feed)

If you run your software in a Docker container and the IB Gateway is running on the host, you still need to be able to connect from your software to the IB Gateway.

One way to achieve this, is to set the IBKR host property to "host.docker.internal" in your application.

Config.setProperty("ibkr.host", "host.docker.internal")

Adding new 3rd party integrations

Adding integration with a 3rd party is not difficult. Integration with 3rd party data providers is straight forward and requires implementing a single interface. Depending on the client library that is available, this could take as little as a few hours to a few days.

For both custom Feed and Broker integrations, it is recommended to first look at some existing integrations and use them as a starting point. And if you have questions, you can always reach out via one of the channels mentioned on the community page.

Feed integration

You can read more about the Feed interface that you need to implement it right here.

The following code-snippet shows the template of what you would normally implement for a historic data feed. By convention, roboquant uses retrieve for historic feeds and subscribe for live feeds.

class MyHistoricFeed : HistoricPriceFeed() {

    // The api of your provider
    private val api = FeedAPI()

    fun retrieve(symbol: String, timeframe: Timeframe) {

        // Make the API call
        val trades = api.getTrades(symbol, timeframe.start, timeframe.end)

        // The API returns American stocks
        val asset = Asset(symbol, AssetType.STOCK, Currency.USD, Exchange.US)

        // Loop over trades
        for ((date, price, volume) in trades) {

            // Convert to a TradePrice
            val tradePrice = TradePrice(asset, price, volume.toDouble())

            // Convert to an Instant
            val time = asset.exchange.getClosingTime(date)

            // Add it
            add(time, tradePrice)
        }
    }

}

Broker integration

Integrating with a new broker is not that difficult either, but requires more effort. Most of the work will be the translation of object exposed by the broker to those used by roboquant. For example, an Order in roboquant will be different from an Order expected by the client library of the broker, and a conversion needs to happen in both directions.

To get started, it is most helpful to look at one of the existing implementations, like the BinanceBroker or IBKRBroker. This will give you a good idea of how to approach this.

A few things that might make the task easier:

You can use the InternalAccount class to store the state in the broker implementation. It makes it easy to create an Account instance.
BuyingPower tells you how much money there is to place new orders. You have to make sure that this value is correct.
Open Orders and Positions are also important to decide what to do in your logic and should reflect the latest status.
Closed orders, Cash and Trades are typically not required when making decisions for new orders. So if you don’t map these, typically a few metrics might be wrong, but it should not impact your overall trading logic.

The following snippet can serve as a template:

class MyBroker : Broker {

    private val api = BrokerApi()

    // Internal Account is a mutable version of Account
    private val iAccount = InternalAccount(baseCurrency = Currency.USD)

    override val account: Account
        get() = iAccount.toAccount()

    override fun sync(event: Event) {
        // All the hard work goes here where you sync the InternalAccount object with your broker
        // The api calls are all fictitious examples
        val now = Instant.now()

        // Optional check to avoid to many calls to the broker
        if (iAccount.orders.isEmpty() && (now < account.lastUpdate + 1.minutes)) return

        // Sync the positions
        iAccount.portfolio.clear()
        for (position in api.getPositions()) {
            TODO()
            // iAccount.setPosition(...)
        }

        // Sync the open orders
        for (order in iAccount.orders) {
            val brokerOrder = api.getOrder(order.orderId)

            // Fictitious implementation
            when (brokerOrder.status) {
                "RECEIVED" -> iAccount.updateOrder(order.order, now, OrderStatus.ACCEPTED)
                "DONE" -> iAccount.updateOrder(order.order, now, OrderStatus.COMPLETED)
            }
        }

        // Sync buying-power
        val buyingPower = api.getBuyingPower()
        iAccount.buyingPower = buyingPower.USD

        // Set the lastUpdate time
        iAccount.lastUpdate = now
    }


    override fun place(orders: List<Order>, time: Instant) {

        // Optional sanity check that we don't use this broker in historic back tests
        if (time < Instant.now() - 1.hours) throw UnsupportedException("Cannot place old orders")

        // Validation of the supported order types
        require(orders.all { it is MarketOrder })

        // Store them in the internal account. Orders should never be (temporary) lost
        iAccount.initializeOrders(orders)

        // Process the orders
        for (order in orders.filterIsInstance<MarketOrder>()) {
            // Fictitious API call
            api.placeMarketOrder(order.asset.symbol, order.size.toBigDecimal(), order.id)
        }


    }

}