Rewriting the cornerstone of the Mann Group Data platform with a powerful C++ engine
Gary Collier.
A group of people
Steadfast. A group of people
Project: Improving the human team’s data-science engine to take in large amounts of data
Chief Executive Officer: Gary Collier, Chief Technology Officer of Alpha Technology at Mann Group
Data is the lifeblood of any investment firm. And at Mann Group — one of the largest listed hedge funds, with $142 billion in assets under management — everything hangs in the balance.
The firm was inspired to build Arctic, a Python-centric data-science system used by Mann Group investment analysts to generate alpha, perform risk analysis and fuel machine learning applications. The system was made open source in 2015 and has had more than 1 million downloads to date, Collier said.
“We routinely process billions of data points every day, and the platform’s strength and flexibility enables us to deliver 9,500 GB of net position, risk, trade and market data,” said Collier, who oversees technology used by investment managers. he told Insider via email. A key component of Arctic is its seamless integration with the Python class language widely used in financial services, as it is designed to serve as a natural extension to what Python Stack Man Group teams use every day.
“When you’re dealing with relevant data to model financial markets, you’ll quickly find yourself working with data frames of any type and shape—number, text, image—basically, a very large matrix,” Collier said. “Arctic provides the ability to store, query, and manipulate data frames at the required industrial scale—think billions of rows and hundreds of thousands of columns.”
In the year In late 2017, Mann Group began rewriting Arctic “to ensure it is ready for the next generation of industry challenges.” Arctic evolved into ArcticDB, which has the same user-friendly Python interface, but with a supercharged C++ engine. That engine provides an order-of-magnitude improvement in the scale of data the system can handle, how quickly queries arrive, and how efficiently the data is stored.
“A concrete example of this is that Arctic makes it easy to handle very large data frames, such as a 400,000-column data frame,” Collier said. “Arctic is addressing these types of data challenges now and in production.”
ArcticDB now manages hundreds of terabytes of data in research and production, Collier added. And the rewrite started in 2017, it’s still in active development, with Mann Group adding capabilities to improve performance and efficiency.
“It’s often more useful to think of transformative technology projects as ‘waves of change,’ as opposed to a binary start, implement, stop initiative,” Collier said.
