Data | Data Tool

Modified on 2020/09/15 11:47 by Eugene — Categorized as: Knowledge Base, Misc, Providers

What it is

This tool integrates with Data Manager, allowing to perform certain operations on entire DataSets which would otherwise take certain effort:

Truncate last or first N bars (or N days of intraday data)
Wipe the entire DataSet data (to help mass reload)
Remove inactive (dead) symbols
Remove selected symbols
Remove bad bars (e.g. weekends)
Fix OHLC out of bounds
Change symbol
Check data for validity

Compatibility

Supported are static data providers that rely on Wealth-Lab to maintain their data in the local storage of .WL files. Only their DataSets will appear in the DataSets tree on the left and allow themselves to be processed.

Compatible: Alphavantage, CBOE, Cryptocurrency, Dukascopy, Finam, Forexite, Free COT, IQFeed, Market Sentiment, Nasdaq, Quandl, QuoteMedia, Random, Stooq, Tiingo, Yahoo.
Incompatible: ASCII, Bloomberg Static, COTCollector, Database, Excel, Metastock, MetaTrader, Taipan, Telecharts, Watchlist, Wealth-Data* and other.

* You shouldn't truncate data by Wealth-Data provider using the Data Tool. Otherwise it's compatible.

Quotes pane

From this handy list you can take a quick peek at the market dynamics of the selected DataSet. The pane displays a table with the DateTime + OHLCV data for all symbols of the highlighted DataSet. For convenience, the table includes absolute and percentage change from the previous bar for each instrument as well as if the symbol has hit a 52-week high/low today. The data is displayed with respect to the number of default decimal places specified in Wealth-Lab's Preferences dialog.

By default, the table is sorted in descending order on the Bars.Count column to instantly show symbols with zero bars. It's possible to sort it in either ascending or descending order by repeatedly clicking on any column.

Below the Quotes pane, there are four tabs which group the actions you can perform on DataSets using the Data Tool:

Data truncation
Inactive symbols removal
Data checker
Data cleaner

Datasource Truncation

The following data truncation options apply to all symbols of highlighted DataSet:

The "Truncate" button makes possible to selectively remove a group of bars for symbols of the selected DataSet. If symbols are selected, the button changes to "Truncate Selected" and acts accordingly. (To apply to all DataSet symbols, click outside or change DataSet.) When "After" is selected in the suitable dropdown box, clicking "Truncate" deletes the last N bars of data. Selecting "Before" will remove the penultimate N bars. Activating the checkbox near the Date field sets the start date to truncate the data before or after. For intraday data it's also possible to delete last (or previous) N days by toggling the radio button (for non-intraday scales, last N bars will be removed). Requesting to delete more bars than a symbol has is equivalent to wiping out its entire historical data.
"Remove all data" is helpful if you want to reload the historical data of an entire DataSet but are reluctant to perform "Reload All History" manually. We know how tedious it could be when many symbols are involved. Click this button to nullify the accumulated data in highlighted DataSet, and then use Data Manager's "Update" button as you would do normally - the entire DataSet will be reloaded from scratch.

Dead symbol removal

To clear highlighted DataSet from dead (inactive) symbols with 0 (zero) bars, click "Remove inactive". To additionally remove any symbols that stopped trading before some date, pick a date and enable the checkbox before clicking on "Remove inactive".

Note: Before making changes, close all Strategy windows using the DataSet you're removing inactive symbols from.

Data validity checker

To quickly verify a DataSet's integrity, click "Check data integrity" from the "Data checker" tab. The following errors are detected in historical data:

Erronenous bars (Close < Low, Open < Low, Open > High, Close > High, Low < 0, and Volume < 0)
Zero bars (OHLC = 0)
Duplicate dates
Out of sequence dates
Gap of N days that are not weekends or holidays
Data spikes (when close-to-close change exceeds a specified percentage e.g. +/- 20%). To avoid false positives, this option should be turned off on Yahoo! DataSets with split/dividend adjustment applied.
Weekend bars
No trading (volume = 0)
Synthetic bars

Synthetic bars are inserted by some data vendors for days on which no trades took place (e.g. stock halt). The Data Tool can detect the following types of these bars:

OHLC are all the same as the previous Close and the Volume is 0
OHLC are the same as the previous bar's OHLC and the Volume is 0

Due to the fact that some conditions like weekend bars are perfectly correct in some international markets, some options can be turned on or off.

Clean bad data

As bad bars may produce inaccurate backtesting results, the Data Tool can perform various data cleanup tasks like:

Remove weekend bars
Remove zero bars
Remove synthetic bars
Fix Open/Close above High or below Low (except for bars where either price is zero)

The "Clean data" button makes possible to clean up for symbols of the selected DataSet. If symbols are selected, the button changes to "Clean selected" and acts accordingly. (To apply to all DataSet symbols, click outside or change DataSet.)

Right click menu

Context menu features

The right click menu serves two purposes:

Change symbol: renames a single highlighted symbol. Simply type in the new symbol name and hit "Enter". Both the symbol name and its data file will be renamed. Helpful when symbol name changes for whatever reason.
Remove highlighted symbols from the current DataSet without affecting their data.

Limitations

By design, Data Tool works with raw data. Consequently, it's not possible to precisely determine market holidays when detecting for "Gap of N Days" errors, and "Spike Detection" will not work for Yahoo!, AlphaVantage and Tiingo DataSets with split/dividend adjustment applied.

A symbol has to have at least 2 bars of data to be visible in the Data Tool - required to calculate Change and Change %.

Fixed in v2020.08 DataSets created by discontinued data providers (like Google or Fidelity) or providers uninstalled/not installed may break the Data Tool. Delete them prior to using it. If provider has been uninstalled but a DataSet leftover exists, remove its XML file.