Christof Donat

ProFed is an open source project, developing a federated system for professional social networking. Additionally to features of other ActivityPub systems, like e.g. Mastodon, Friendica, or Mobilizon, it will include features, like the users CV, and a mechanism to query for open positions, or matching candidates. It is meant to be installed in very small setups, beginning with a single user, as well as big cloud setups with many small services.

The solution, that I found is an event sourcing modular monolith architecture. Components are independent of each other and only communicate through events. The admin can decide to start all components at once for a small setup, or pick any collection of them.

That is, how ProFed will be able to scale to the limits of the message bus, while keeping simple setups very simple. Development is ongoing.

SwissRe has a very big collection of data from varous sources. That data would be useless without the ability to analyze it, and extract intelligence from it. Therefore SwissRe has a sophisticated data processing pipeline built with PySpark on DataBricks, running in an Azure cloud environment. Though that is a particularly powerfull data processing environment, the process still took over eight hours for each run. To improve the experience of the users, the goal was to speed up that process significantly.

Usually only data from the current quarter, and sometimes the previous, or the next are changed. I implemented the necessary logic to detect which quarters are affected, and process only those. I also took some measurements, to optimize the cluster size, and the node size for the specific workload. Apart from only performance oriented optimizations I also developed several approaches to improve the over all code quality.

Together with some changes, other team members implemented in parallel, my changes reduced the processing time to around two hours.

Many Credit Suisse customers seek advice for their wealth management. To support them, we developed a portfolio optimization system that uses mathematical optimizations with high dimensional functions to determine an optimal investment strategy.

We developed a microservice system for reliability and scalability. I was responsible for some of the more mathematically challenging quantitative computing services, as well as for the framework and deployment infrastructure for those services written in Python.

The project was successfully delivered.

PPRO had a lot of data spread over several systems and aspired to analyze it in order to improve its marketing and product development decisions.

As technical lead, I set up a data warehouse with my team in PostgreSQL because PostgreSQL allows to leverage remote tables from other database systems such as Cassandra. This in turn allowed for a smooth transition at a later date when PostgreSQL was no longer sufficient. For scheduling, I decided to use Azkaban, and the collection of data from all the systems was done with Pentahoo Kettle.

The database had three schemata: a staging area, the data warehouse itself, and several data marts. With that setup we were able to feed Tableau Dashboards for decision support and allow the data scientists to run their data mining without disturbing the production processes.

The BI system, as a data-driven decision support tool, was highly regarded by management.

As a financial institution, PPRO is obliged to obey the sanction regimes in all jurisdictions it is operating in. The authorities distribute the list of sanctioned people as XML files, which contain the sometimes sparse information available, to identify sanctioned persons, or organizations.

In this project I developed not only the import and update of these sanction lists, but also the matching of customer requests to entries on these lists. The result was a risk score, which reflected the probability that the customer might match an entry on one of the lists. With that risk score, the system either decided to allow, or deny the customer, or to delegate the decision to a human in unclear cases.

The system was applied to every new customer, and regularly to the whole customer base. For the customer base, I stored a list of human decisions, so that we didn’t have human staff members decide again.

With this decision support system, PPRO was able to reduce the work load on the staff, while at the same time reducing the risk of failing to comply with the sanction regimes.

PPRO decided to develop a payment service for the German market to complete their portfolio. The idea was that the user would enter his login for his online banking, and then the service would trigger a transaction to the shop. Since the service could view historical transactions, we were able to tell the shop, if he can be sure that the money is coming, or if he should wait for it to arrive at his account.

Together with a second developer, I analyzed the online banking systems of the biggest banks, banking conglomerates, and online banking providers for small banks. That was enough to cover over 90% of all German bank customers.

Since my coworker and I implemented this system in a short time, PPRO was able to improve its position in the market significantly.

SIX Telekurs (now SIX Financial Information) is one of only a few companies that can provide marked data from almost every exchange around the world to customers in a standardized format with low latency. The goal of this project was to replace the central market data processing system with a decentralized one, for even lower latency.

We were able to leverage the domain-specific language for parser generators from the existing system. These parsers had to interpret the different stream formats from the different exchanges, and therefore there were several hundreds of them already implemented.

Not only have I written big parts of the code. I was also responsible for the design of the main components and, together with the technical lead, the main architecture decisions. We developed an asynchronous system that used a reliable multicast protocol (LBM by 29West) for hot failover as well as for delivery to the customer. The code had to be very close to the bare metal, considering aspects like the NUMA structure, memory layout and cache strategies, and user space direct network card access to reduce the number of context switches. The applications I designed basically took over most of the machine from the operating system with CPU binding and CPU shielding. The whole system was designed lock-free to avoid any unnecessary latency, and handled all issues that were not part of the processing itself asynchronously, like e.g. logging, or metrics collection. To give the compiler the best chances to optimize the code, I relied on the most modern C++ techniques of the time, like template meta programming.

While I was on the project, we managed to successfully deploy a prototype to production.