Mobilization advanced c   swift java kowalczyk header

Combining Swift/Kotlin with C/C++

The appearance of Swift and Kotlin is a great change in mobile world and there are a number of advantages to using them. Still, one thing didn’t change: as embedded programmers, in order to provide better performance, reduce battery use, etc., we sometimes need to operate on a lower level. During this talk from Mobilization 2017, Michał Kowalczyk provides you with ready-to-use tools that will let you utilize a huge codebase of C and C++ in your project, and discuss differences between them.


Introduction

My name is Michal Kowalczyk and I work for TomTom. I want to talk about combining Swift and Kotlin, with old languages like C and C++.

Why C++???

C++ is full of dangerous features - it’s a dangerous language and powerful language that you should not use. This talk will cover some of the reasons why you would.

TomTom & C++

At TomTom, we produce navigational maps. The company has been around for over 25 years, and the oldest code I’ve seen is 25 years old. Because we have been around for long, much of the codebase is written in C or C++.

Now that we’re in a mobile age, our clients want us to run our code on Android and iOS devices. They want APIs in Java and Objective-C.

In this case, should we translate all the code that we have written in C++ to Java or Objective-C? If we do that, it would lead to a lot of code duplication, and it would not be the best approach.

A solution to interoperability is to use language bindings. Code written with C++ bindings can be used in Objective-C, this can also be done in Java.

How about Kotlin/Swift

What if the client one day asks us for Swift or Kotlin implementations?

Wrappers can use a small bridging header for communication with Swift, then we can also use this C++ in Swift code. This is the same for Java and Kotlin.

Example App

I would like to create an app that will let you count the number of repetitions of push-ups for Android and iOS.

First, we want to detect a single push-up. To accomplish this, we can use image processing for detection of a raised and lowered position. In this example, I used an existing algorithm for face detection.

Let’s think about architecture.

Platform Specific Libraries

We have two platforms: Android and iOS. We may find an image processing library for Android, and the other for iOS. Then the business logic and UI can be done on the respective platforms.

Because we use two different libraries, one for Android and one for iOS, there may be differences in the implementation of the face detection algorithm, providing an unequal user experience.

Get more development news like this

As a solution, we might put business logic on the native side that is written once, suppose in C++, with image processing library.

The only code that I would like to leave on a platform-specific layer is a code for UI, as it’s difficult to write it in C++.

The other layer is a platform agnostic layer. There’s a written code that doesn’t know what platform it executes on, and it does not use any platform-specific API’s.

OpenCV

We can use OpenCV as the image processing library, and it will satisfy our requirements. It is written in C and, partially, in C++. It has interfaces in C++ and it has also wrappers for Python and Java.

With those wrappers, we might prototype our solution then write the final production code. But, there are no Objective C++ wrappers; so using those wrappers won’t be a solution for iOS. Though you can use the wrappers to fully implement OpenCV in Java, there are some performance drawbacks.

Each Java native interface needs cores, needs processor cycles to execute and, it is much more than executing a single method. So, when using Java native interface with methods from OpenCV, you lose all that OpenCV brings to you.

Another reason to use C++ is that on StackOverflow, most OpenCV questions and answers are in C++.

Glue Code Generators

There are glue code generators that will save us from having to hand write code. There are two approaches available on the market.

  • Swig - it has existed for over twenty-two years and it has opportunities to generate a language binding to many, many languages like Java, PHP, and JavaScript.

  • Djinni - which is most mobile, and currently only supports Objective C, Java and Python.

The idea of generating glue code for both is the same. The glue code is generated from a configuration file called IDL.

Djinni’s IDL

An IDL may contain three types of structures:

  • Abbreviations
  • Records - wrappers for data.
  • Interfaces, which are sets of methods

PushUpPalApp = interface +c {
start();
stop();
reset();
isStarted(): bool;
setListener(listener: PushUpListener);
}
PushUpListener = interface +j +o {
onPushUp(rep: i32);
}

We will need at least four methods for starting, pausing, resetting and checking if we already started. Then we need a listener for informing, Swift or Kotlin code, that this push up was detected.

The last hurdle is to figure out how to create an instance of this push-up/pull up that will be implemented in C++. We have a static factoring method that may create this interface. It should accept all things that are specific to a platform. Every parameter that differs between platforms should be passed to such a method.

Run Djinni


deps/djinni/src/run-assume-built \
    --java-out android/app/src/main/java/pl/ekk/mkk/pushuppal/gen \
    --java-package pl.ekk.mkk.pushuppal.generated \
    --jni-out "native/PushUpPal/glue-code/jni/generated" \
    \
    --cpp-out "native/PushUpPal/glue-code/interfaces/generated" \
    --cpp-namespace generated \
    \
    --objc-out "native/PushUpPal/glue-code/objc/generated" \
    --objc-type-prefix PUP \
    --objcpp-out "native/PushUpPal/glue-code/objc/generated" \
    --objc-swift-bridging-header "PushUpPal-Bridging-Header" \
    \
    --idl PushUpPal.djinni

Djinni accepts a number of different parameters. The above is from the command line tool and, it accepts parameters specific to Java, e.g. where this code should be generated, or what package should be used in this project.

C++ has no packages, but there are namespaces and, we have the same for Objective-C, Objective C++ but here there are prefixes. Note the special flag that is needed for Swift. Thanks to this bridging header, we may reuse this Objective-C++ code into Swift.

Android - Kotlin

Let’s create an Android project. First, create a project and include C++ support. Then, specify where the Native Development Kit (NDK) is located and set it in local properties.


class MainActivity : Activity() {
    private var mPushUpPalApp: PushUpPalApp? = null
    public override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        mPushUpPalApp = PushUpPalApp.create(
            ResourcePathAccessor.getClassifierFilePath(this@MainActivity))
        mPushUpPalApp!!.setListener(object : PushUpListener() {
            override fun onPushUp(rep: Int) {
                runOnUiThread {
                    repsTextView.text = Integer.valueOf(rep)!!.toString()
                }
            }
        }
        startStopButton.setOnClickListener {
            if (mPushUpPalApp!!.isStarted) {
                mPushUpPalApp!!.stop()
                startStopButton.text = "Start"
            } else {
                mPushUpPalApp!!.start()
                startStopButton.text = "Stop"
            }
        }
        resetButton.setOnClickListener {
            mPushUpPalApp!!.reset()
            startStopButton.text = "Start"
        }
    }
    companion object {
        init {
            System.loadLibrary("native-pushuppal")
        }
    }
}

We start with an empty activity, then we need to create this instance of push up/pull up. Next, set a listener. This is a way to inject Kotlin code into C++, because C++ is calling this listener. On Push up, we may execute some Kotlin code in C++, and C++ into Kotlin code.

There is one thing specific in developing Android apps or Java apps you will need to do. When you want to use native code, you need to load a shared library and we do it in companion objects in it, and shared libraries are loaded in RAM time.

CMake

Cmake is a tool that we use in C and C++ for creating build systems to make them more generic. Android Studio started to support Cmake, and now we can also use it in Android.

A Cmake file has a special command that loads all the sources that you wrote in C++. It has also included directories that states where different dependencies for our project located. We have our own headers here, placed in the native folder and we have some different dependencies from Djinni, from Objective-C, etc.

The next thing is to say that we want to create a share library from this source.


file(GLOB_RECURSE SRC_FILES FOLLOW_SYMLINKS
    ../../../native/PushUpPal/src/*.cpp
    ../../../native/PushUpPal/glue-code/jni/*.cpp
    ../../../deps/djinni/support-lib/*.cpp)

include_directories(native/PushUpPal/src
                    native/PushUpPal/glue-code/interfaces/generated
                    native/PushUpPal/glue-code/jni
                    deps/djinni/support-lib
                    deps/djinni/support-lib/jni
                    deps/OpenCV-android-sdk/sdk/native/jni/include)

add_library(native-pushuppal SHARED ${SRC_FILES})

add_library(lib-opencv SHARED IMPORTED)
set_target_properties(lib-opencv PROPERTIES IMPORTED_LOCATION
                      ${CMAKE_SOURCE_DIR}/src/main/jniLibs/${ANDROID_ABI}/
                          libopencv_java3.so)

target_link_libraries(native-pushuppal
                      lib-opencv)

You can use this code and now you have this Android app ready.

iOS - Headers, Sources, Bridging Header, Swift

There is one marked difference between iOS and Android when doing this. In Android, you need to load this shared library at the RAM time, and on iOS, you need to do it during compilation time or link time.

We need to specify where are all the C++ and Objective-C++ sources are.

Lastly, set up the bridging header - and thanks to that, you can use Objective-C++ code in Swift.

We start with an empty view controller. When we create this instance of push up/pull up, it looks like we are writing code in Swift. We also create a listener, and we inject Swift into C++ code.


class ViewController: UIViewController, PUPPushUpListener {
  @IBOutlet weak var repsLabel: UILabel!
  @IBOutlet weak var startStopButton: UIButton!
  @IBOutlet weak var resetButton: UIButton!

  var pushUpPalApp: PUPPushUpPalApp?

  override func viewDidLoad() {
    super.viewDidLoad()

    let classifierFilePath = Bundle.main.path(forResource: “haarcascade_frontalface_alt2",
                                              ofType: "xml", inDirectory: "")!
    pushUpPalApp = PUPPushUpPalApp.create(classifierFilePath)!

    pushUpPalApp!.setListener(self)
  }

  func onPushUp(_ rep: Int32) {
    DispatchQueue.global(qos: .userInitiated).async {
        DispatchQueue.main.async {
            self.repsLabel.text = rep.description
        }
    }
  }

  @IBAction func startStopButtonOnTouchUpInside(_ sender: UIButton, forEvent event: UIEvent) {
    if pushUpPalApp!.isStarted() {
      pushUpPalApp!.stop()
      startStopButton.setTitle("Start", for: .normal)
    } else {
      pushUpPalApp!.start()
      startStopButton.setTitle("Stop", for: .normal)
    }
  }

  @IBAction func resetButtonOnTouchUpInside(_ sender: UIButton, forEvent event: UIEvent) {
    pushUpPalApp!.reset()
    startStopButton.setTitle("Start", for: .normal)
  }
}

Native - C++

Djinni generated to me generated this C++ code.


#include "generated/PushUpPalApp.hpp"

class PushUpPalAppImpl : public generated::PushUpPalApp {
public:
    PushUpPalAppImpl(const std::string& classifierFilePath);

    ~PushUpPalAppImpl() override;

    void start() override;

    void stop() override;

    bool isStarted() override;

    void reset() override;

    void setListener(const std::shared_ptr<generated::PushUpListener>&
        listener) override;

private:
    class Impl;
    std::unique_ptr<Impl> impl_;
};

std::shared_ptr<generated::PushUpPalApp>
generated::PushUpPalApp::create(const std::string& classifierFilePath)
{
    return std::make_shared<PushUpPalAppImpl>(classifierFilePath);
}

Djinni uses modern C++. It has new features that make writing C++ code fun, to include smart pointers that reduce problems with memory.

Display Camera Image

Open CV has some solutions for us to display the camera image. We can use a Java surface view for displaying it on Android or, CV video camera for displaying it on iOS. With iOS, you need to pass the image view. With the Java surface view, you use it in your layout file. After that, it will be the same for both platforms: register for camera updates. You will get a frame, do something with it and, it will display the modified frame.


PushUpPalApp = interface +c {

    start();

    stop();

    reset();

    isStarted(): bool;

    setListener(listener: PushUpListener);

    onFrame(frame: Mat);

    static create(classifierFilePath: string): PushUpPalApp;
}

PushUpListener = interface +j +o {

    onPushUp(rep: i32);
}

There are two approaches to accomplishing this. You can pass to the native code, but that would require creating two different methods for Android and for iOS.

The other approach is to pass through this native layer a mat. A mat is a matrix - a frame from the cameras. We will have a method that will accept it and draw all these things on it and then, we can display it on our screens.

Djinni: supported types

Djinni supports some types - enumerations, records, and interfaces but, besides built-in types (e.g. integer flow), it doesn’t support any other types by default.

Djinni requires a YAML file to use this mat, then it can generate the code to convert a type between C++ and Java or C++ and Objective-C.

Djinni YAML file

It will look like this:


name: Mat
typedef: 'record'
params: []
prefix: ''
cpp:
    typename: '::cv::Mat'
    header: '<opencv2/core/mat.hpp>'
    byValue: false
objc:
    typename: 'OpenCvMat'
    header: '"OpenCvMat.h"'
    boxed: 'OpenCvMat'
    pointer: true
    hash: '%s.hash'
objcpp:
    translator: '::cv::djinni::objc::OpenCvMat'
    header: '"OpenCvMat+Private.h"'
java:
    typename: 'org.opencv.core.Mat'
    boxed: 'org.opencv.core.Mat'
    reference: true
    generic: true
    hash: '%s.hashCode()'
jni:
    translator: '::cv::djinni::jni::NativeMat'
    header: '"NativeMat.hpp"'
    typename: jobject
    typeSignature: 'Lorg/opencv/core/Mat;'

Mat on Java side

A mat looks like the following on Java side.


package org.opencv.core;

public class Mat {
    public final long nativeObj;

    public Mat(long addr) {
        if (addr == 0)
            throw new
                UnsupportedOperationException(
                "Native object address is NULL");
        nativeObj = addr;
    }

    // ...
}

It has one long field called native object. This is a primitive, but, when we take a look at this constructor of mat, we realize that this long other that is probably something created with an address that is a pointer from C++ code. To initialize a mat you need to pass a pointer and native object will be initialized with it.

Java Native Interface

Java native interface code:


#include <opencv2/core/mat.hpp>
#include "djinni_support.hpp"

class NativeMat final :
    ::djinni::JniInterface<::cv::Mat, NativeMat> {
public:
    using CppType = ::cv::Mat;
    using JniType = jobject;
    using Boxed = NativeMat;

    ~NativeMat();

    static CppType toCpp(JNIEnv* jniEnv, JniType matObj) {
        auto matClass = jniEnv->GetObjectClass(matObj);
        jfieldID nativeObj = jniEnv->GetFieldID(
            matClass, "nativeObj", "J");
        long matPtr = jniEnv->GetLongField(matObj, nativeObj);
        CppType& mat = *((::cv::Mat*)matPtr);
        return mat;
    }

    static ::djinni::LocalRef<JniType> fromCpp(
        JNIEnv* jniEnv, const CppType& c) {
        // ...
    }
};

Conclusions

If you have C, C++ code in your organization that is liked by some programmers, you can reuse it in your Swift and Kotlin applications, as well as in Java and Objective-C.

You can also take advantage of native libraries, such as Open CV.

Djinni produces this glue code that works well about 95% of the time.

If you tried writing native apps on the Android, please go to this Android developer website for NDK. Cmake is available in Android studio for a few releases. You’ll find other options to build the shared libraries, but that is not a big deal.

Tools:

If you want to try this code that I showed to you, please visit my account on Github.

Next Up: New Features in Realm Java

General link arrow white

About the content

This talk was delivered live in October 2017 at Mobilization. The video was transcribed by Realm and is published here with the permission of the conference organizers and speakers.

Michał Kowalczyk

Michał has worked in various sectors of IT for last 12 years. He has created web applications as well as embedded systems. On his job, he likes to combine various technologies. For the last couple of years, he has used C++ with Java, Python and Swift.

Currently, he works at TomTom developing a multiplatform engine for map visualisation.

4 design patterns for a RESTless mobile integration »

close