How camel breeding
can help you in IoT

María Arias de Reyna

@delawen

Senior Software Engineer at

About me

19🐱🐱 Crazy Cat Lady
Free and Open Source Advocate
2008 Computer Science Graduate (Software Engineer)
2009 Dijkstra A* Witch
2012 Metadata Wrangler
2013 OSGeo Charter Member
2017-2019 OSGeo President
2016 Women in Tech
Senior Software Engineer at
2019 Integration Druid
2020 Apache Software Foundation

Suppose we are designing a new software project. Normally, the software can be defined with the classic UML diagrams where you have a series of events or triggers, which usually bring data input, and that chains a series of instructions that process this initial data until an output is obtained.

That is, we have a workflow in which there are a series of steps.

If the architecture is fairly complex, we will have a series of external components (databases, APIs, file repositories, web services, ...). These components interact with each other throughout this workflow.

If we define this software project from scratch, we will find that we first have to read the documentation and understand how to interact with each component. Then we will surely define a module or class, depending on how complex the component is, which will communicate with said component. We can make use of external libraries, for example use hibernate to connect with a database.

Once we are clear about how to interact with each component, we will have to add the logic and data routing from one component to another. For example, let's imagine that we are modeling a simple REST API. When making a request to said REST API, we could have to translate that request into a query in the database, convert the result of that query to modeling the response of this REST API and return said result as a response to the original request.

It could also be that we are listening to a messaging channel like Kafka and that when a message arrives through that channel, we process the message and generate a response that is sent as an email.

→

Apache Camel is an integration framework. Which means that it can be used to connect different components easily. Low coupling and easy maintenance are one of the biggest advantages of using these frameworks.

Camel allows you to abstract from almost all the details of how to interact with each external component so that you focus on the logic and modeling of the specific data of your use case. It is often compared to block construction games because the idea is that each step of the workflow is easily interchangeable for a different step and all the pieces would still fit together perfectly.

You could implement the REST API example from above with only Camel without knowing how to connect a database or how to implement a REST service.

Integration Frameworks are specially well-designed for dynamic and changing architectures. This is exactly what happens for example on IoT environments: you have usually a complex architecture with lots of sensors. Some of those sensors are active only during specific times of the year. Some others are temporary sensors during specific events (like a flooding). Some other types of sensors are quickly replaced every year because technology advances and we can get better hardware in short spans of time.

This means that you need to extend and modify your architecture and how your components interact with each other on a fast and easy to maintain way.

→

Integration Frameworks are the middleware, the glue that interconnects the different components. On this talk I am going to talk to you about Apache Camel, one of the (if not the) most used integration frameworks. It is Free and Open Source, which means there's not a single company maintaining it, but dozens of them. It is the most active project of the Apache Software Foundation.

To implement an integration in Apache Camel, a domain specific language (DSL) is used which, although it is not natural language, is very easy to understand. This language uses the integration patterns to carry out advanced steps such as a broadcast, a loop, a conditional, a data transformation, ...

You can find all of these patterns prettily explained in the classic Enterprise Integration Patterns book.

If we look at the entrails of Camel, there are three types of basic objects.

On the one hand we have the Endpoints. These types of objects are specialized in a specific type of external component: database, ftp, mail server, telegram bot, Twitter, ServiceNow, SalesForce, ... There are more than 300 different types of components. And also, we do not have to worry about adding any type of dependencies because Camel already has everything necessary to connect to these services included. Endpoints perform all inbound and outbound communication with external components and generate what is called Exchange.

The Exchange is the datagram object that contains all the data that goes through the workflow. This exchange is similar to how an HTTP request would be because it has its message body, its headers, ... but it also has attributes and has a space to store errors and exceptions that may occur throughout the workflow. This Exchange message that leaves the endpoint reaches what would be the router.

The router decides what will be the next step in the workflow. It is in the router where the workflow logic is and where the integration patterns are applied. For example, in the image we see how the next step is divided into two flows, each going to a different Endpoint.

The output Exchange of one step is always the input message of the next. Sometimes these steps do not connect with any external component, but are dedicated to transforming the message to prepare it for the next step. For example, it could be that one step works with XML and the next one works with JSON and you have to convert it from one to the other. It could also be something more complex such as that you have to modify the data model by adding or removing attributes.

Hello World


            # Write your routes here, for example:
            - from:
                uri: "timer:yaml"
                parameters:
                  period: "1000"
                steps:
                  - set-body:
                      constant: "Hello Camel K from yaml"
                  - to: "log:info"


<?xml version="1.0" encoding="UTF-8"?>
<routes xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xmlns="http://camel.apache.org/schema/spring"
        xsi:schemaLocation="
            http://camel.apache.org/schema/spring
            http://camel.apache.org/schema/spring/camel-spring.xsd">

	<!-- Write your routes here, for example: -->
    <route id="xml">
        <from uri="timer:xml?period=1000"/>
        <setBody>
            <simple>Hello Camel K from ${routeId}</simple>
        </setBody>
        <to uri="log:info"/>
    </route>
</routes>


          import java.util.Random;

          import org.apache.camel.builder.RouteBuilder;

          public class HelloWorld extends RouteBuilder {
            @Override
            public void configure() throws Exception {
                from("timer:java?period=1000")
                  .setBody()
                    .simple("Hello World!")
                  .to("log:info");
            }
          }


                from("timer:java?period=1000")
                  .setBody()
                    .simple("Hello World!")
                  .to("log:info");


               
from("timer:java")
    .choice()
        .when(someCondition)
          .setBody().simple("Hello World!")
        .when(anotherCondition)
          .setBody().simple("¡Hola Mundo!")
      .otherwise()
        .setBody().simple("Salut Monde!")
    .end()
    .to("log:info");

Camel Quarkus

Camel K

Kamelets

Camel is a surprisingly light and fast library that runs on a Java virtual machine. But still, it's still Java, which may not be palatable for people who want to use serverless.

→

That is why we have had Camel Quarkus available for a couple of years. Quarkus is a native Kubernetes Java stack specifically designed to run GraalVM and OpenJDK. The idea of Camel Quarkus is to bring together the power of Apache Camel and its entire library of components and run it on a Quarkus runtime. This allows users to benefit from optimization improvements in code execution and being KNative, it also makes it easy to containerize.

Camel Quarkus also benefits from many of the improvements introduced in Camel 3: lower memory footprint, fewer reflections (which always make code more obtuse to run), and ridiculous initialization times. Going from a couple of seconds to milliseconds.

→

But of course, when introducing knative and containers it seems that the devops part gets complicated. This is when Camel K appears. Camel K is a toolbox that allows you to easily deploy Apache Camel integrations (both those generated for normal Java and for Camel Quarkus) easily and quickly.

→

As if all this were not enough to simplify our lives, in the last year the concept of Kamelets has appeared. Kamelets are route snippets that allow users to connect to external systems using a very simplified interface. If Camel used small building blocks before, a Kamelet could be a meta block made up of those little blocks. Kamelets hide all the small implementation details so that the workflow designer can focus even more on what the workflow itself is.

A Kamelet can act as a "source" of data or as a "sink". These are concepts that will sound familiar to those familiar with KNative: a source consumes data from an external system while a sink sends data to an external system or executes a specific action and returns a result.

Repository of Kamelets

Coding time


                from("timer:java?period=1000")
                  //.setBody()
                  //  .simple("Hello World!")
                  // Generate random name
                  .process(processor)
                  .to("log:info");


                  Processor processor = new Processor() {

                    String[] firstname = new String[] { "Aleja", "Almerinda", "Ambrosia", "Benilda",
                    "Bercia", "Cayetana", "Ermisinda", "Escolástica", "Esmaragda", "Esmerencia"};
                    String[] lastname = new String[] {"Zuzunaga", "Sorní", "Sandemetrio", "Bonachera",
                    "Sazón", "Piesplanos", "Parraverde", "Alcoholado" };

                    Random r = new Random();

                    @Override
                    public void process(Exchange exchange) throws Exception {
                      StringBuilder name = new StringBuilder("Hello ");
                      name.append(firstname[r.nextInt(firstname.length)]);
                      name.append(" ");
                      name.append(lastname[r.nextInt(lastname.length)]);
                      name.append("!");
                      exchange.getMessage().setBody(name.toString());
                    }

                  };

Coding time


   Exchange[
         ExchangePattern: InOnly, 
         BodyType: String, 
         Body: 
            {"coord":
              {"lon":-5.9761,"lat":37.3824},
              "weather":[
                {
                  "id":804,
                  "main":"Clouds",
                  "description":"overcast clouds",
                  "icon":"04d"
                }],
              "base":"stations",
              "main":
                {
                  "temp":293.64,
                  "feels_like":293.68,
                  "temp_min":292.59,
                  "temp_max":294.26,
                  "pressure":1003,
                  "humidity":74
                },
              "visibility":7150,
              "wind":{"speed":3.13,"deg":225,"gust":8.49},
              "clouds":{"all":100},
              "dt":1619448268,
              "sys":{
                "type":3,"id":2011488,
                "country":"ES","sunrise":1619415275,
                "sunset":1619464109},
              "timezone":7200,"id":2510911,"name":"Seville","cod":200}
   ]


from("timer:weathertest")
  .to("weather:foo?location=Sevilla,Spain&appid={{TOKEN}}&geolocationAccessKey={{TOKEN}}")
  .to("log:info");

Coding time


                public class ChatBotLogic {
                    public String chatBotProcess(String message) {
                        if( "do-not-reply".equals(message) ) {
                            return null; // no response in the chat
                        }

                        return "Received " + message;
                    }
                }


from("telegram:bots?authorizationToken={{TELEGRAM_BOT_TOKEN}}")
  .bean(ChatBotLogic.class)
  .to("telegram:bots?authorizationToken={{TELEGRAM_BOT_TOKEN}}");

In this example we have implemented a simple telegram bot that responds with the same string that you send it. In order to implement this, we only need the token that telegram assigns us when creating a bot. We don't need to know how telegram bots work or what kind of calls to make to the API. If the API changes in the future, just update Camel and it will continue to work transparently.

The nice thing about using Camel in this case is that we can even forget that it is a bot specifically from Telegram and focus on what is specific to our use case, which is what to answer. If tomorrow instead of a telegram bot we want it to be a bot on a website, we would only have to change the from and to, the rest would remain the same. We cannot ask for it to be more decoupled.

Coding time


from("paho-mqtt5://{{APP_ID}}/devices/{{DEVICE_ID}}/up?host=" + broker)
  .unmarshal().json(JsonLibrary.Gson, Device.class)
  .setBody().simple("The device ${body.name} is a ${body.description}.")
  .setHeader("CamelTelegramChatId", constant("{{MY_CHAT_ID}}"))
  .to("telegram:bots?authorizationToken={{TELEGRAM_BOT_TOKEN}}");


   {
  "ids": {
    "device_id": "my-device",
    "dev_eui": "0102030405060708",
    "join_eui": "0102030405060708"
  },
  "name": "My Device",
  "description": "Living room temperature sensor",
  "lorawan_version": "MAC_V1_0_2",
  "lorawan_phy_version": "PHY_V1_0_2_REV_B",
  "frequency_plan_id": "EU_863_870_TTN",
  "supports_join": true,
  "root_keys": {
    "app_key": {
      "key": "01020304050607080102030405060708"
    }
  }
}

In this case we have created an integration that is listening on a typical IoT message queue like The Things Network. The Things Network (TTN) is an open global LoRaWAN network that if you do not know it and are interested in this whole world of IoT, it is interesting to take a look. In addition to being an open collaborative network where the information of the devices is shared, it is intended that everything that connects to said network is open source and with free hardware.

When an event arrives, it parses that response and sends the name and description of the device that generated the event to a specific telegram chat. As you can see, we have not even needed to implement a process since what Camel receives is a JSON that knows how to parse it and can generate the body of the message to send to Telegram using attributes of said JSON.

This is an example that Apache Camel can be used even if you don't know how to program. And if you want to change the telegram bot by saving the data in a database or sending an email or whatever comes to mind, simply by changing that line of the workflow, everything would continue to work just as well and decoupled. We have assembled with five lines of code something that without Apache Camel there would be a minimum of 5 code files between connectors, parsers and others.

Questions?

You can find more information about Apache Camel at

https://camel.apache.org

This presentation is published on

(you can see the notes by pushing 's')

How camel breeding can help you in IoT

María Arias de Reyna

@delawen

Senior Software Engineer at .b185c6fe-daea-408b-9fe9-61f9c6069fbe{fill:#e00;}RedHat

About me

Hello World

Coding time

Coding time

Coding time

Coding time

Questions?

How camel breeding
can help you in IoT

Senior Software Engineer at