Abstract

XCRO currently supports the ability to create automated scheduled instructions for long-term deals, both, deals without an end date, and deals over multiple years. There are defects identified with some of the possible configurations as explained in this LBN.

When the deal has several accounts/instructions/sub-instructions, the deals take long to approve, occasionally resulting in erroneous scenarios where schedules are not created.

To prevent this situation, please follow the steps below to scale up the number of execution pods, increase the memory of the MongoDB database and increase the tolerance limit for the default of 2 minutes.

Pod scaling

Scale the pods up using the following commands

1. kubectl scale deployment execution -n <namespace> --replicas=10

2. kubectl scale deployment deal -n <namespace> --replicas=10

3. kubectl scale deployment dealdraft -n <namespace> --replicas=5

4. kubectl scale deployment cbs -n <namespace> --replicas=5

5. kubectl scale deployment dms -n <namespace> --replicas=5

6. kubectl scale deployment bff -n <namespace> --replicas=5

Note: The CBS pod must be completely stateless i.e. should not run any schedulers/timers.

MongoDB indexes

Login to the MongoDB shell and execute the following commands to create indexes

7. use xcro-db

8. db.execution.executions.createIndex({ scheduledOn: 1 }, { name: "idx_scheduledOn" } );

9. db.execution.executions.createIndex({ refId: 1 }, { name: "idx_refId" } );

10. db.execution.executions.createIndex({ status: 1 }, { name: "idx_status" } );

11. db.execution.executions.createIndex({ transactionType: 1 }, { name: "idx_transactionType" } );

12. db.execution.executions.createIndex({ sourceAccountNumber: 1 }, { name: "idx_sourceAccountNumber" } );

13. db.execution.executions.createIndex({ destinationAccountNumber: 1 }, { name: "idx_destinationAccountNumber" } );

Increase the MongoDB memory

Step 1: Execute the below command to get a list of parameters

db.adminCommand({getParameter: ‘*’})

Step 2: Check the value of the field "internalQueryExecMaxBlockingSortBytes", it defaults to be about 33,554,432 bytes (32MB)

Step 3: Increase the memory using the below command

db.adminCommand({"setParameter":1,"internalQueryExecMaxBlockingSortBytes": 104857600})

Tolerance window

Increase the tolerance window for executions

To increase time-frame of execution between getting picked up and actual scheduled time before it is failed (tolerance window), follow the steps below.

Step 1: Edit the config map

kubectl edit cm -n <namespace>

Step 2. Edit the value of the TOLERANCE_LIMIT_MINUTES. E.g. to set it to 10 minutes:

  TOLERANCE_LIMIT_MINUTES: "10"

Step 3: Save the config map and scale down + scale up the eceproducer and ececonsumer pods

Impact and Resolution

• After scaling up the pods, the creation of deals and the execution of schedules will be faster.

• Increasing the MongoDB memory will result in larger reports being downloadable

• Indexes help deal approval, reports and execution perform better by 2x to 5x

• Increasing the tolerance window will ensure slower executions are still picked up and executed