Algorithms in the financial services industry

The ultimate goal of every software product is to convert inputs(provided by end users or automatically received from external systems) in valuable outputs (insights).
This is usually done via processing logic (= algorithms), which transform a number of inputs into a set of outputs.

In my experience, these process logic components can be divided into 5 categories:

  • Calculation: a calculation is a clear mathematical formula, which allows to calculate a specific value. This category can be identified by the fact that all the required information is available as input and the output is accurate, i.e. there is only one correct result and this is perfectly reproducible. Examples are calculating the average price of a share over the past year, calculating the time-weighted rate of return on an investment portfolio, the repayment schedule for a loan, or the amount of interest. to be paid into a savings account each quarter.

  • Data searches: a data search consists in retrieving a result in its raw form or in a processed form (for example sorted, filtered, aggregated, etc.) from a database. This is usually a query executed to retrieve specific data. Examples are finding data for a specific customer, retrieving all customer transactions for the past few months in descending order of transaction amount or the number of customers and total transaction amount generated last year grouped by client country.

  • Expert systems: they are trees of (nested) “If-Then-Else” clauses (or a similar conditional logic), resulting in a workflow, a flowchart or a decision tree. This means that the algorithm will check one or more conditions to determine which path in the business logic should be taken (i.e. to determine the next step). Examples are decision rules to determine the type of analysis required for a credit file, rules for protection against fraud and the fight against money laundering in the area of ​​payments, etc.

  • Optimization issues: these are mathematical calculations with, in principle, only one solution, but it is often impossible (or too complex or too greedy in resources / time) to know if the perfect solution is obtained. Usually, these algorithms work with a cost function that needs to be optimized, but it is unclear whether a local minimum or the absolute minimum is reached. Nevertheless, all the rules are clearly defined and with sufficient processing (computing) power an exact result can be obtained. Examples are resource scheduling issues, portfolio rebalancing based on the number of investment constraints (client) and recommendation lists or portfolio templates… More information can also be found in my blog ” Optimization Problems – Far From Being a Commodity “(https: // bankloch / 2020/05 / optimization-problems-far-from-being.html).

  • System identification (= SI) and machine learning (= ML) issues: these are problems for which it is difficult or even impossible to describe the rules of behavior of the system (the algorithm). Instead, the algorithm is configured as a black box, for which a large number of coefficients / parameters (usually elements in a matrix) are estimated by training the model. This is done by adjusting these coefficients, so that they best match the input and output test data provided to the model. Typical examples are fraud / AML detection, recommendation engines and engines to define the next best action for users, anomaly detection, speech and image recognition …

Obviously, many algorithms are a combination of these categories. Ex: a price calculation engine (see my blog “Calculation engines in financial services – A key differentiator in business strategy” – in-financial.html) can combine a number of categories, for example * Data Lookup: to retrieve certain configurable values, which can be easily configured by business users (for example the percentage of discount granted for specific types of credit) , but also to retrieve the data necessary to feed the engine (such as customer segmentation for example) * Expert system: set of rules to decide which pricing regime should be applied * Calculation: obviously a certain number of calculations must be applied, such as calculating the price according to different factors (for example, applying a percentage to the amount of the transaction), applying minimum and maximum thresholds… * SI & ML: allow ent to set up dynamic pricing models, which evolve automatically depending on whether or not customers accept products at a proposed price.

When defining business logic (or an algorithm), it is important to choose the right category (or the right combination of categories).
It is generally a compromise between:

  • Flexibility to change the behavior of the algorithm

  • Operational complexity to explain / understand a result, but also to validate if the algorithm behaves as expected

  • Complexity of implementation, that is, the complexity of implementing the algorithm in code form.

  • Knowledge rules that drive logic and the ability to describe it easily and with it the number of factors to take into account

For example: a “calculation” type algorithm is very inflexible and requires very precise knowledge and description of all governance rules, but once defined, it is generally easy to explain a result and test it. Since a certain set of inputs will always lead to the same outputs, it’s also easy to set up automated regression tests.
On the other hand, an “SI & ML” type algorithm can be very flexible (often it will even adapt automatically) and requires less knowledge of the associated rules, but it is often impossible to explain / understand a result and it is almost impossible to fully validate the algorithm, i.e. to ensure that the algorithm will not give very bad results in some peak cases.

The implementation of each category of algorithm is further supported by specific implementation software, i.e.

  • Calculation: hundreds of mathematical libraries (often open source) exist to calculate any type of result (ex: libraries with statistical formulas or libraries with financial calculations)

  • Data search: SQL is the standard for interacting with a database, but often all kinds of abstraction layers have been built on it to support easier complex data searches, like Hibernate, QueryDSL, jOOQ, Spring Data …

  • Expert systems: obviously every programming language supports If-Then-Else and Case clauses, but there are also hundreds of abstraction layers, helping to implement this kind of algorithms, like BPMS systems (like TIBCO ActiveMatrix BPM , IBM BPM, Oracle BPM, Camunda BPM, jBPM…), workflow systems (like Nintex, Zapier, ProcessMaker…) and business rule engines (like Kissflow Process, IBM Operational Decision Manager, Drools, Red Hat Decision Manager – formerly JBoss BRMS), Progress Corticon Business Rules Engine, SAS Business Rules Manager, Hyperon …)

  • Optimization issues: this is still a somewhat unexplored and immature area, with few (user-friendly) tools available, as I also mentioned in one of my previous blogs. The interesting names to look at are JuMP (based on the Julia language), ADMB, GLPK, OpenMDAO, Motulus, OptaPlanner… However, all of these tools are still quite complex and therefore still difficult to use for non-specialist developers.

  • SI and ML: in this space TensorFlow is the best known abstraction for implementing such algorithms, but there are obviously many alternatives such as PyTorch, Keras, Amazon SageMaker, IBM Watson Studio…

With the popularity of AI / ML, people are trying to implement many algorithms with AI / ML logic. While AI / ML is a powerful tool, it also has a number of drawbacks. As such, it is best to use it only when the majority of the rules for describing the relationship between inputs and outputs are unknown. If a part is known, it is probably best to start with other categories of algorithms and possibly refine the result with AI / ML.

It is therefore important to approach the categories in the order defined above, that is, use an exact calculation if possible, otherwise use rules derived through an expert system, data search or a problem. ‘optimization. AI / ML should be the last resort, when it is not possible to properly define the guiding business rules.
This is important, because too often people today see AI / ML as the first solution to any problem. For example, in recommendation engines, people are pushing AI / ML more and more, although this is not always the best idea, as a lot of rules for recommending a product are known, often there are no rules for recommending a product. not enough data available to properly train the AI ​​/ ML model and being able to explain recommendations is often important because a vendor needs to support the process or regulators require proof of certain proposals / decisions.

As always in IT, there is no silver bullet and no one size fits all to rule the world, but instead a deliberate choice must be made, where the pros and cons are weighed against each other. compared to others, instead of surfing the latest buzz.

Check out all my blogs at

About Scott Conley

Check Also

New organization to provide access to fair and affordable financial services

Stephen Pearson, Chairman of Financial Inclusion for Scotland and Sharon MacPherson, CEO of Scotcash A …

Leave a Reply

Your email address will not be published.