Kdb+tick captures, publishes and analyzes millions of streaming ticks as well as the accumulating terabytes of historical data. The realtime and historical databases are in kdb+. Realtime analytic/alert clients can be written in java, .net, c or q. Kdb+tick includes some example clients. see below
Each tickerplant and rdb(realtime db) can handle 100,000 records per second -- more than enough to deal with all trades and quotes, level2 quotes or options. The tickerplant, rdb and hdb(historical) are all 24/7. The latency between feed and rdb is less than one millisecond. Real-time analytic/alert clients can be written in java, c or q. The rdb resets the hdb once a day. See http://kx.com/q/d/taq.htm for a hdb example: nyse trade and quote
The kdb+ storage factor is about 1. Get 2 times as much disk for raid, staging and scratch space. In production (staying less than 50% cpu) the user should have 4 cpus(feed,tick,rdb,hdb) per 4GB of intraday data(with peak 50000 records per second). Backup and development can run with half. The rdb and hdb should be 64bit cpu's.
All US equities per day(2004):
Install kdb+. Unzip tick.zip in q/
A tickerplant system usually has the tickerplant, realtime db, historical db, one or more feeds and several clients, e.g.
All programs run 24/7. The logs are updated immediately. The clients are updated on a timer loop.
All messages are lists of records -- for higher throughput. The schemas must be
The feeds must call
is set by the tickerplant.
The historical database port should be 1+realtime port -- e.g. 5012 -- so the tickerplant can send the reset message.
We provide triarch/ssl and bloomberg feed handlers. The handlers can be on any machine. They can parse about 100,000 records per second -- i.e. more than the feeds can send.
For testing and replay(http://kx.com/q/tow) don't provide a DST, e.g.
The incoming messages are
i.e. in general many records at once.
In java and c#: (using http://kx.com/q/c/c.jar or http://kx.com/q/c/c.cs)
Watch out: a bad client(doesn't keep up) will cause the tickerplant to buffer up all the messages.
Possibly reset feed subscription list every day.
these have to run 32bit because triarch is 32bit.
the tickerplant can be anywhere/anything.
options and underlying: two tick's. one client subscribes to both.
rolling vwap, e.g. every 10 seconds
We can also keep track of nbbo (or the entire book) if there are multiple market makers,
Minimal Ticker Plant Memory usage
The kdb+ TP holds only one second's worth of data in memory. It instantly logs incoming data.
Kdb+ has built-in daily rollover mechanisms to eliminate downtime. All processes can run forever.
Scalability and Resilience
Real Time subscribers can subscribe at any time during the day without overloading or delaying the plant. This is because the rdb loads the day's tick data from disk and then continues to receive updates via tcp/ip.
Kdb+ plants allow table subscription by symbol - i.e. the plant sends only symbol data that the client has requested. This allows slower (desktop java/.net/excel) processes to be connected without the client thrashing to filter unwanted data.
Increased Memory Space
Processes can hold more data in memory - allows process types (depth/stock/option) for a region to be combined into a single kdb+ process, reducing duplication of data across databases, simplifies query writing, and eases configuration management/support.
Simpler Query Language
The new unified query language, Q, is much simpler and more flexible to write queries.
The old kdb is no longer the best product for processing tick data. Kdb+ is.