Machine Learning and Markets

Markets can undergo significant shifts due to unforeseen events. An adaptive quantitative model must distinguish between genuine changes in market dynamics and random noise to avoid overreacting and making poor investment decisions. One modern way to ensure proper adaptation of a quantitative strategy is to use Machine Learning (“ML”) for its constant feature optimization and adjust their weights in the decision-making scheme. However, common machine learning algorithms have inherent problems that might lead to unexpected losses and poor performance.

• The efficacy of ML-based models is heavily influenced by the quality and quantity of data used to train them. When dealing with financial data, challenges such as noise, incompleteness, or bias may arise, which can lead to unreliable predictions. Additionally, gathering continuous, high-quality data can be costly and difficult.
• It’s important to note that any biases in the training data will also affect the models’ decisions. This can result in skewed or inaccurate investment choices.
• One issue with complex ML models, especially those utilizing deep learning algorithms, is their lack of interpretability. Understanding how the model arrives at its predictions can be challenging, raising concerns about transparency and potential biases.
• Overfitting is another risk when using ML models in finance, where the model performs well on training data but struggles with unseen data in real-world situations. This reduces its ability to generalize and make reliable decisions.
• Financial markets are dynamic and can experience sudden shifts due to unforeseen events. As a result, an ML model must be able to adapt (known as concept drift) to avoid making poor investment decisions based on outdated patterns.
• Financial markets can, at times, be highly efficient and reflect most of the available information in asset prices. It is extremely challenging for ML models to identify undiscovered inefficiencies and stable exploitable patterns.
• Training and running complex ML models can also incur significant computational costs, making it difficult for some investment firms to utilize them effectively.
• Finally, ethical considerations must be considered when using ML models in finance. There is a risk of manipulation by investors and perpetuating algorithmic bias in decision-making processes.

There are three major schools of thought in modern Machine Learning:

Symbolic AI, also known as Logical Reasoning, posits that intelligence can be achieved through computer programs that encode knowledge and logical rules. These programs utilize symbols and logic to represent and solve problems in the world.

Connectionism, also known as the Artificial Neural Networks school of thought, suggests that intelligence can be achieved by mimicking the human brain’s intricate structure and function. These programs, inspired by our neural pathways, are called artificial neural networks (ANNs). They comprise a network of interconnected nodes, or artificial neurons, which work together to process information and adapt based on experience. Like a living brain, these networks constantly evolve and develop over time.

Evolutionary Computation believes that intelligence can be attained by emulating the principles of natural selection. Evolutionary algorithms create a diverse population of potential solutions through this approach and gradually refine them through selection, mutation, and crossover cycles. Much like the myriad species that evolve through adaptation and survival of the fittest, these candidate solutions develop and improve until they ultimately reach their optimal form.

Both the Symbolic and Connectionism schools achieved impressive results. For example, Symbolic AI gave us Expert Systems such as medical diagnoses software (MYCIN) and a financial analyst support system (XCON). It also pioneered chess and checkers-playing programs (Deep Blue and Chinook) that played those games superhumanly. Early chatbots and language understanding systems were also made possible through Natural Language Processing (NLP), which utilized rule-based methods.

On the other hand, Connectionists have given us modern image recognition abilities, speech generation, machine translation, and, lately, large language models (LLMs), which we have used in ChatGPT, Gemini, and other ground-breaking AI tools.

Unfortunately, despite their impressive achievements, Symbolic and Connectionism-based machine learning systems fell short when researchers tried to apply them to Quantitative Investment Systems. Both approaches are extremely sensitive to noisy data, could deadlock themselves during the feature optimization process, hallucinate, confabulate, and become extremely opaque, blocking any attempts to understand how and why they came up with a particular investment decision.

Evolutionary algorithms take a unique approach to problem-solving, relying on the evaluation and refinement of a diverse population of candidate solutions. Their population-based dynamics allow them to excel in scenarios where traditional methods may struggle, such as problems with elusive underlying relationships or complex optimization challenges.

Evolutionary Computing extends far beyond mere problem-solving, with applications ranging from scheduling tasks to strategic gameplay. Unlike other approaches, it is not limited by data format and can effectively handle continuous and discrete values.

One of the major strengths of evolutionary, nature-inspired algorithms is their resilience in the face of noise and errors within the data. By leveraging a population-based process, these algorithms avoid overreliance on any single solution. Even if some candidates are swayed by erroneous information, others often lead to successful outcomes.

A crucial advantage of Evolutionary Systems is their ability to produce an acceptable solution without becoming trapped in an endless cycle of refinements or deadlock. This unique ability ensures progress towards a viable solution, even in complex and unpredictable environments.

**********

Dr. Alex Bogdan is Chief Scientific Officer at Castle Ridge Asset Management

Adrian de Valois-Franklin is CEO at Castle Ridge Asset Management

***

The views expressed in this article are those of the author and do not necessarily reflect the views of AlphaWeek or its publisher, The Sortino Group