The main US derivatives regulator is set to share resources with the US Treasury as it struggles to get a firmer oversight on the mushrooming volumes of swaps data, and spot market participants’ risk exposures in global derivatives markets.

The Commodity Futures Trading Commission has been working on taking staff expertise from the Treasury’s Office of Financial Research, Commissioner Scott O’Malia confirmed on Tuesday. The two bodies have been working on a memorandum of understanding, he said.

The OFR was set up in the wake of the financial crisis by Congress to use Big Data and other cutting-edge computing techniques to monitor financial flows.

“OFR has long recognised that granular swap data are essential to assess exposures, interconnectedness and concentration in the market. Their assistance and work in this area will be a tremendous help,” he told a conference on data standards in London.

“Together, by leveraging our resources, we will be able to start doing some of the heavy lifting of our big data problem,” he said.

The joint effort by the regulator and the Treasury highlights how regulators are becoming concerned that efforts to make global markets more transparent are being undermined by the scale and complexity of the task.

Regulators have struggled to implement a political mandate to have swaps trades in the vast, opaque $700tn over-the-counter derivatives market reported to data repositories. They are being operated by private market infrastructure companies such as IntercontinentalExchange, CME Group, the London Stock Exchange Group and the Depository Trust and Clearing Corporation.

Efforts to use the new data warehouses as the window for analysing potential systemic risk have been undermined by local privacy laws and differing and highly complex local data standards. That has resulted in millions of trades being reported daily.

Alongside the regulatory push, technological advances allow traders to make and cancel thousands of orders per second on futures markets, throwing up further clouds of data.

The difficulty was underlined by the CFTC’s inability to find the JPMorgan “London Whale” trade. In response, the US regulator last week commenced a review of how swaps data are reported by market participants.

Mr O’Malia, also head of the CFTC’s Technology Advisory Committee, said he had called on counterparts in Brussels to focus on recognising each other’s swaps data warehouses and develop a means to share trade information. He has asked that Europe and the US should harmonise the form and format of data being reported.

Sharing information between the two economic blocs – which account for the vast majority of the global swap market – has been undermined by EU unease over Washington intruding on to its patch to collect highly sensitive banking and markets data. The issue has taken on new significance in the wake of US spying revelations.

“I urge international regulators to quickly and – dare I say – swiftly come together to sign an agreement that will rely on substituted compliance and allow for the standardisation and sharing of swaps data,” Mr O’Malia said.

He also called for the CFTC to automate market surveillance, rather than rely on “stale” transaction data.

“In a world where data are king and traders increasingly rely on cutting-edge technologies and automated strategies, regulators must also become early adopters. The technology of yesterday will not be able to keep up with the market realities of today. Call it Data Darwinism,” he said.

Copyright The Financial Times Limited 2018. All rights reserved.