The amount of data that is being created in today’s age is almost incomprehensible.

It’s estimated that in just the last two years that 90% of the world’s data has been created.

Each minute there were 3,607,080 Google searches conducted worldwide in 2017. Since 2013, the number of Tweets each minute has increased 58% to more than 455,000 Tweets per minute in 2017.

In the investment world knowledge is power. The proliferation of data is in the early stages of upending the hedge fund industry and rewarding those with the ability to access and make sense of large and alternative data sources.

Historically, data was structured and neatly organised in databases, and then came along the internet.

The Internet of Things (IoT) phenomenon is the major driver behind the shift, with sensors now in everything from home appliances through to the more sophisticated like the satellite orbiting earth. All the while, capturing data and transmitting it to the internet. Most of this is in an unstructured and more complex format than traditional datasets.

internet of things hedge fund

In theory, this allows investors to access market relevant data in real time. For instance, at Imbue Capital we are utilising satellite and weather station data to monitor the health of the major US agricultural crops and provide production estimates that are used in sophisticated trading algorithms.

Satellite imagery can also be used to monitor the activity of oil rigs and GPS sensors can track the movement of ships exporting commodities to the major importing countries.

Sophisticated quantitative hedge funds are making use of this data rather than waiting months for the release of official government data.

Outside of sensors, data can also originate from individual processes (e.g. google search traffic or sentiment of news/blog articles) or business processes (e.g. credit card spending data tracking consumer spending behaviour).

Historically, similar data may have only been available at very low frequencies, if at all. This may have been at monthly intervals like the US Government’s Monthly Agricultural Supply and Demand Estimates or weekly US oil rig counts.

Data Science Has Drastically Improved

For hedge funds the value now lies in mining and harvesting data, storing this largely unstructured data, and most importantly making sense of this data with the aid of machine learning and artificial intelligence to provide clarity of vision in identifying trading opportunities.

Not only has the amount of data simply blown up but there have been significant improvements in machine learning capabilities to levels well beyond any human.

It has come as a result of the exponential improvement in computing power and the plummeting of the costs associated with the technology. Delivering the ‘perfect storm’ for hedge fund disruption.

While machine learning and AI isn’t an exactly new phenomenon, quantum leaps in the development of data science has been achieved in recent times.

These advancements in the machine learning space (particularly deep learning) have been led by the technology industry with major breakthroughs being registered by companies such as Google, Baidu, and Facebook. At Imbue Capital we’re taking those cutting-edge advancements and combining that with our large alternative data sets and academic theory to trade global commodity markets.

Traditional Human Based Discretionary Traders Will Be Left Behind

While hedge fund closures have been ticking up since 2014, this year has been different: the names have gotten bolder, their time in the industry longer and their assets under management greater.

The big data ‘revolution’ is and will no doubt continue to erode the value of traditional human-based traders who rely on these traditional, low-frequency public datasets interpreted by their human ‘intuition’ that is subject to behavioural biases.

With a new generation of tech-focused hedge funds adopting alternative and often private datasets, the market is beginning to react faster to and will increasingly anticipate traditional or ‘old’ data sources (e.g. quarterly corporate earnings or monthly USDA agricultural supply and demand updates).

This provides an edge to sophisticated quant managers and those that fail to innovate risk becoming obsolete.

Ultimately, the big data ‘revolution’ is likely to upend the industry, shifting investment approaches away from human discretionary trading to a quantitatively focused investment style.

What an exciting time.