Introduction
The heart and soul of your program is your BSA (Bank Secret Act)/AML This article will define what an AML model validation is and it’s importance. We will also delve into our top ten findings of AML model validations and ways to avoid them.
What is an AML Model Validation?
An AML model validation is a process with the intent to test that your model is performing as expected and that the style is in line with the financial institutions (FIs) objects and business uses. An effective model validation helps an FI ensure that their current models are sound; it also identifies any potential limitations and assumptions while assessing the potential impact of those limitations. If the model validation results show there are any gaps, identifying them is important so that they can be remediated properly and quickly.
The four components of a model include Governance – how the system is running or who is controlling the system, Input – what gets put into the system, Processes – baseline rules/profiles, and Output – what is of use. The combination of these components is where the model validation takes place. Model validations can apply to transaction monitoring/BSA, sanctions screening/OFAC, Know You Customer (KYC)/Customer Due Diligence (CDD)/Enhanced Due Diligence (EDD), and 314(a). A model validation can include a combination of all the aforementioned models, or just a single model.
The Importance of Conducting an AML Model Validation
AML-related models are complex entities or programs. These models are collecting data from different sources to detect pre-determined patterns that may indicate the presence of a violation of AML/Counter Terrorist Financing (CFT)/Sanctions issues. These pre-determined patterns may have been set by your financial institution when your system was implemented. If these systems are not regularly reviewed, errors can have an easy time getting by. A model may, also, have a fundamental error that may produce inaccurate outputs when viewed against the design and objective and intended business uses. People may try to use the model for something that it was not intended to be used for; this can lead to difficulties on its own.
What The Regulators Are Saying About AML Model Validations?
The regulators are not saying much specifically, in the way of regulations, about model validations, but there is an expectation. As we all know, there are expectations of compliance against the pillars of Section 352 stating that BSA/AML programs must be tested independently. The Fed, Office of the Comptroller of the Currency (OCC), and Federal Deposit Insurance Corporation (FDIC) all have guidance that spells out these expectations; and if you are under the jurisdiction of the New York Department of Financial Services (NYDFS), they have their own rule, Rule 504. While these validations can be done both internally and externally, those who are conducting the validations must be doing so correctly.
The 10 Ten Findings in AML Model Validations and What They Mean
In our experience of the volume of model validations we have performed on various AML systems throughout the United States, we have identified the ten most common model validation findings. These findings are identified and written up in a report so that the FI has the documentation of what we found and an actionable way to remediate it. We have broken our findings into three parts related to validations: overall model governance, BSA/transaction monitoring, and OFAC/sanctions screening.
Findings Related to Overall Model Governance
- Missing/Inadequate Documentation – Since Governance is one of the key components of a model, an important part of that is the documentation. This documentation would explain why the model was chosen, why the parameter/thresholds were selected, what risks the model covers, and who is responsible for that
- Missing/Inadequate Policies and Procedures – The Policies and Procedures should cover the regulatory framework around AML/CTF/OFAC, define roles and responsibilities, and explain Compliance’s methods of oversight and reporting.
Findings Related to BSA / Transaction Monitoring
- Missing/Partial Master Mapping Documentation – We stated earlier that models are complex entities, but with the incorporation of Applied Learning or Artificial Intelligence into models, they have grown even more complex and more capable of providing more accurate results. BUT the input has to be accurate. Having an accurate mapping document is the roadmap to ensuring that data going in has the correct format and is applied correctly in the Transaction Monitoring system (TMS).
- Update/Tune Rule Thresholds – Updating and Tuning Thresholds is an exercise your financial institution should be conducting on a defined risk-based interval. This applies to any rule that is not producing any or is producing an excessive number of false positives.
- Missing Policies and Procedures –
- Fields Mapped Incorrectly from Core System to Transaction Monitoring System (TMS) – Every data piece that goes into the TMS must meet the standard data set that the TMS expects. It’s common to see incorrect data formats, core transaction codes misrepresented in the TMS, debits treated as credits and vice versa, impossible values being used as thresholds, among other things.
- No Reconciliation Procedure – For TMS, institutions have ignored the fact that there is not reconciliation between the system that is feeding into the TMS and if the data is getting there right. It is something that needs to be acknowledged in writing. Saying that you reconciled is not the same thing as doing the reconciliation in writing.
- Missing Rules – For every AML risk, there should be some way of analyzing the data to determine if there are transactions that match the defined risk. It depends on your institutions risk: if you don’t have the risk you don’t need the rule, but there needs to be some way of monitoring this.
- Missing Rules / Multiple Jurisdictions – If your institution acts as an intermediary for foreign wires, or processes a good number of foreign wires, this is an important rule – and often is not found. A typical international wire probably will pass through three countries, the originator’s, the intermediary’s, and the receiver’s. But if there are multiple jurisdictions in a wire, it may be time to ask “Why are there multiple jurisdictions?”.
Findings Related to OFAC / Sanctions Screening
- OFAC Threshold Tuning – OFAC Filtering programs generally run on some matching form of algorithm or a combination of algorithms, and all systems have their limitations. Most institutions try to maintain a balance between finding true matches while avoiding false positives.
Tips on How to Avoid These Findings
It is important to keep certain documents on hand to avoid missing/inadequate documentation. This includes but is not limited to:
- Up-to-date Risk Assessment
- Business Requirements Document (BRD), which can help explain why the model was chosen
- Pre- and Post-Implementation testing
- Scenarios/Rules Document, which can explain why the parameters and thresholds were selected, the risks covered, how it’s covering those risks, and the expectations
- Security explaining who has access, including levels of access
- Change Management describing how changes to the model are processed
- Vendor Management explaining the management of third parties related to the model in use
To avoid missing/inadequate policies and procedures, the Policies and Procedures in place describing, in sufficient detail, all aspects of the BSA/AML/CFT/OFAC regulations as expected in the FFIEC Bank Secrecy Act / Anti-Money Laundering Examination Manual[1]. This manual may be written for examiners but knowing what the examiners are looking for will help you to know what to get in front of them. Your FIs policies and procedures should also describe Compliance Oversight, including whom it reports and the nature of a BSA Compliance Committee. This needs to describe what the Compliance Committee’s responsibilities are. It is also important to include the Roles and responsibilities of BSA/OFAC staff and all who have BSA/OFAC functions regardless of their locations in the institution, as well as, BSA/OFAC training requirements for staff. There should also be a policy and procedure that describes the proper use of the model so anyone who is using it, understands and knows how to use it.
In order to avoid missing/partial master mapping documentation, a good mapping document is a collaborative effort from Compliance, IT Support, and the Vendor. These three need to work together otherwise the mapping document will not be complete. Each and every field needs to be properly mapped. On the vendor side, they need to receive the proper data, so they know if the data is going in the right way. Your FI must clearly define the source of data, all application codes, etc.
Developing a risk-based Policy and Procedure to conduct a review of the efficiency of each and every rule/scenario/agent the institution has enabled will help your institution avoid updating/tuning rule thresholds. It is important to look at the number of alerts, alerts to cases, cases to SARs, and thresholds. And, if possible, test raised and lowered thresholds to see what results occur. Whatever the risk of your institution is will determine how often you should seek an outside opinion on Tuning Thresholds.
Although mapping and reconciliation are not easy and require assistance from technical staff, and probably your vendor, it is important to build this to avoid missing policies and procedures. Building this may require assistance from multiple departments. A Scenarios Document should include details for each rules/scenario/agent, the reason why it was enabled by your institution, what BSA/AML risk it covers, what the thresholds are, why your institution chose them, etc. This will allow us to know why the threshold for that particular rule/scenario/agent was chosen, as opposed to another threshold, specifically what AML risk is being covered, and the expected results.
Generally speaking, field mapping takes a lot of planning and testing. A model implementation plan that includes both pre- and post-implementation testing provides a strong method of avoiding fields mapped incorrectly from source systems to the TMS. This testing would include testing the documentation that goes with it. Making sure everything is going into the right place at the right time is difficult, but if it’s not done right the system is going to fail.
Another difficult area, that may require assistance from technical staff, is how to avoid no reconciliation procedure. The data sets that transfer need to be clearly defined and written down; not every core customer transaction record needs to go into the TMS, e.g., fees do not measurably impact a customer’s transaction volume. It is important to know which records must go in to the TMS, and to make sure that they have actually gone in and gone in correctly. It’s not just the number of records that go from one system to another, it’s the physical data from the record. It’s important to know if the total number of records are the same and the total dollars are the same. If there is a difference, it’s important to find it and act on it right away. On a positive note, Compliance doesn’t need to do the recon – they only need to check it. This is something that is best to be done by those processing the data.
To avoid missing rules, use your BSA/AML Risk Assessment to list out the risks and what the coverage is for each. Although the coverage does not have to be in the TMS, it should exist someplace where your institution knows they can control it. Remember: all systems have limitations and not every system can analyze every risk. You need to know what your system can not do so your institution can map out ways some rules that may need to be covered.
First thing to do to avoid missing rules and multiple jurisdictions is to make sure your TMS can handle it. If not, there are other ways to handle it. The difficulty that comes with avoiding this is finding all the country codes in the wire.
An OFAC Threshold Tuning is the attempt to reduce false positives to as few as possible without taking the risk of missing a true match. To avoid OFAC threshold tuning, develop a risk-based policy and procedure to conduct a review of the efficiency of the algorithm in the filtering program. In addition to the policies and procedures, develop a set of names to be tested that includes exact matches and variations on each. Keep in mind that with some algorithms, the scoring threshold is not the only variable – there may be moving parts that can be changed.
Conclusion
It is recommended that in alternating years, you institution should consider contracting with an independent third party to do a BSA/AML and OFAC Threshold Tuning and periodic testing. Periodic testing will allow you know to know what thresholds are working as they are supposed to work. Using an independent third party can also provide you the advantage to see how others are doing it.
[1] FFIEC Bank Secrecy Act / Anti-Money Laundering Examination Manual, https://bsaaml.ffiec.gov/manual
