Data flowing the easy way

María Arias de Reyna

@delawen

Senior Software Engineer at

About me

19🐱🐱 Crazy Cat Lady
Free and Open Source Advocate
2008 Computer Science Graduate (Software Engineer)
2009 Dijkstra A* Witch
2012 Metadata Wrangler
2013 OSGeo Charter Member
2017-2019 OSGeo President
2016 Women in Tech
Senior Software Engineer at
2019 Integration Druid
2020 Apache Software Foundation

"We may say most aptly that the Analytical Engine
weaves algebraical patterns just as
the Jacquard loom weaves flowers and leaves."

"The Analytical Engine might act upon other things besides number,
were objects found whose mutual fundamental relations
could be expressed by those of the abstract science of operations,
and which should be also susceptible of adaptations to
the action of the operating notation and mechanism of the engine"

- Ada Lovelace

Software is a collection of instructions and data

that tell the computer how to work.

- Wikipedia

Complex software architectures are made of several components you have to interconnect, like a database, a user interface, an analysis tool,... The pieces of software that connects them are the middleware, the "glue", the orchestration between those components. These interactions between different unit operations are what make your usecase work.

Usually you design a flow that defines how each component interact with the rest. This flow can be linear or circular, split into several threads, have conditional steps,... and that's what you have to implement to make your software adapts to your usecase.

Integration Frameworks, can help us orchestrate and automate the interaction between different applications, APIs, third party services or any other software piece we may have to connect.

A proper integration tool should provide us with the following features: transformation, integration patterns and connectors to existing protocols and components.

Let's start with transformations. When we connect different components of an architecture, they rarely speak the same languages or, on this case, data formats. Some components will output an xml that has to be fed to the following component on a json form. Maybe we even need to add or remove some attributes on that json data. An integration framework allows us to define what is the mapping between the output of one component and the input of the following so we can forget about the explicit implementation. The less code we have to maintain, the better.

Now, being able to transform data on each step is not enough if we can’t connect to (and from) the specific component we need. Our ideal integration framework should offer support for common protocols like ftp, http, jdbc,… Also it should offer support to connect to common components like a mail server, messaging services, atom,… We could claim even that no integration tool would be good if it doesn’t also support specific well known services like being able to send a message through a Telegram bot or store information on Elastic Search. Being able to seamlessly connect from one component to the next without having to worry about the specifics of their interfaces is what distinguishes an average integration tool from a good integration tool.

And finally, not all flows will be lineal (->). Some of the steps will require broadcasting, some of them will require conditional flowing. Some will require waiting the output of different components to conflate the data. These action patterns are something that have been studied for a long time. And as with software development patterns, you can classify them and study them to create better integrations. You can find them as "Enterprise Integration Patterns".

Adopt a cat

On this case, we decided that we need a database to store the data of the cats waiting to be registered. We also decided that the input data is going to be in JSON format and that we will receive the inputs via kafka brokers.

Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. Or put it simply: Kafka is a very light messaging service that allows clients to receive messages.

This means that we have two different components, the database and Kafka brokers, that we want to connect with some logic. We could do it by implementing all the workflow from scratch, adding connector libraries as dependencies and handling secrets and listeners.

On this talk I am going to show how to use one of those Integration Frameworks to connect different pieces of your software architecture with minimal effort.

The framework we are going to use is Apache Camel. Camel is not only one of the most active projects inside the Apache Software Foundation, it is also the lightest and most complete integration framework available. And on top of it, it is also Free and Open Source Software! Which is always a warrantee of quality and good practices.

Camel is already an old actor on the integration world. It has support for hundreds of components, protocols and formats. Some of these protocols come very handy allowing the user, for example, to connect to any REST API that they need.

Camel uses its own DSL, a simplified language to define easily the workflows step by step.

Camel is available in Knative with the name CamelK. And Camel is also available over Quarkus, the supersonic subatomic Java.

The exchange that carries the data has many elements. Obviously it carries the data itself, which we call the body of the message. But similarly as how the HTTP protocol works, you can add headers to that message that can give context to your flow. For example, you can store credentials on those headers or any other value that can be used on a later step but is not included explicitly on the data being carried forward.

If there's any error that throws an exception during the routing or the processing of the message, it will also be stored on the exchange, making it easier to recover from errors. For example, if some component returned an exception because the external system is unreachable, you may be prepared for it with some conditional step that checks that exception and sends an email to the administrator to warn.

Demo time

Questions?

You can find more on

https://camel.apache.org

https://syndesis.io

https://www.redhat.com/en/technologies/jboss-middleware/fuse

This presentation is on