Zaki is realm magic header

Is Realm Magic? Bytecode Manipulation in Realm Java

Introduction

Today I will talk about the implementation detail of the Realm Java. I’m Makoto Yamazaki, Realm Java Team, and I’m living in Japan. Realm has very nice features. This is from my website, and today I’m focusing on the base.

Magic in Simplicity

This code is just creating the object and then assigning the value to the field, and it looks like it’s just assigning the value to the field, but actually it’s passing data into the database file. In addition, just reading the field also reads the data from the database file.


Realm realm = Realm.getDefaultInstance();

realm.executeTransaction(r -> {
    final Person person = r.createObject(Person.class);
    person.name = "Zaki";
});

Person person = realm.where(PErson.class).findFirst();
Log.d("realm", person.name); // Zaki

realm.close()

The model definition is very simple – this is just defining a field, and it extends a RealmObject. Nothing special, but there’s a lot of magic behind these things.


public class Person extends RealmObject {
    public String name;
}

@RealmClass
public class Person implements RealmModel {
    public String name;
}

Realm has a lazy loading feature - that means that Realm doesn’t load any data if the user doesn’t need it. For example, this code is creating a query and then adding one condition. Realm does almost nothing, just adding a query condition. findAll() execute the actual query, but it doesn’t load the actual data, it just creates the row index from the table. When the person object is created, still the actual data is not loaded, it’s just creating the proxy object. On the last line the actual value, the name value, is read from the database file. This is lazy loading.


RealmQuery<Person> query = realm.where(Person.class);
                           .beginsWith("name","Z");

RealmResults<Person> results = query.findAll();

Person person - results.first();

String name = person.name;

Realm Java’s Old Implementation

The old implementation contained some limitations. This was the model definition below – the user needed to follow some rules:

Get more development news like this

  1. The field must be private
  2. The user must define getters and setters

public class Person extends RealmObject {
  private String  name;


  public String getName() { return name; }

  public void setName(String name) { this.name = name; }
}

Against that model definition, we create the proxy class by using the annotation processor.

The generated proxy is a subclass of the model class, in this case, person, and it contains a row object. Row is a pointer to the internal database.


public class PersonRealmProxy extends Person {
  private Row row =...;
  private PersonColumnInfo columnInfo = ...;

  @Override
  public String getName() {
    return row.get(columnInfo.columnNameIndex);
  }

  @Override
  public void setName(String value) {
    row.set(columnInfo.columnNameIndex, value);
  }
}

This implementation was very simple – it just generates the subclass as a proxy by annotation processor and it overrides getters and setters. Realm returns a managed object, meaning an object that is bound to the database. That is our proxy object, so then proxy just steals access to the properties and reads and writes database instead.

Limitations of the Old Way

That is the basic strategy, and the current implementation is almost the same as that, but we needed to remove some limitations because there are many restrictions in this way:

  1. Fields must be private. It’s actually okay to public, but many users make mistakes when the field is public, so we prohibit public fields.
  2. You must have getters and setters, and because we need to override the getters and setters,
  3. Getters and setters must not have a logic, just read and write the fields.
  4. No custom methods other than getters and setters, because users can easily make mistakes – for example accessing the fields from custom methods.
  5. Users must extend a RealmObject class.

Removing Limitations with Bytecode Manipulation

To remove those limitations we use the bytecode manipulation. Bytecode manipulation is modifying the bytecode at compile time or loading time. We are currently modifying them at compile time.

Current Implementation

In our current implementation, users can define the model the same as the old implementation. Currently we’ve removed the private field and getters and setters, but we use the same definition.

Then we generate two proxies, an interface and a class. The interface has the getters and setters for the field, but the name is special, realmGet$name. The setter has the name realmSet$name. The subclass implements those special getters and setters in the proxy class.


public interface PersonRealmProxyInterface {
  public String realmGet$name();
  public void realmSet$name(String value);
}
public class PersonRealmProxy extends Person implements PersonRealmProxyInterface {
  private Row row = ...;
  private PersonColumnInfo columnInfo = ...;

  public String realmGet$name() {
    return row.get(columnInfo.columnNameIndex);
  }
  public void realmSet$name(String value) {
    row.set(columnInfo.columnNameIndex, value);
  }
}

We apply bytecode manipulation after compile. First, we add the interface to the model classes to access the property directory from our implementation. By doing this, we can cast the model object to the interface and directly call the getters setters defined by us, and add the getters setters implementation. These getters and setters are just reading or writing the field directory.


public class Person extends RealmObject implements PersonRealmProxyInterface {
  private String  name;
  public String   getName() { return realmGet$name(); }
  public void     setName(Stringname) {realmSet$name(name); }

  @Override
  public String realmGet$name() { return name; }

  @Override
  public void realmSet$name(String value) { this.name = name; }
}

The important thing is replacing all accesses to the model’s fields (such as reading or writing the name) with getters or setters calls. By doing this, every access to the fields will use the getters and setters. By doing this, we can allow this definition below.


public class Person extends RealmObject {
  public String  name;
}

If a user directly accesses the name, we can steal that access by the getters and setters defined in the proxy interface or proxy class.


RealmResults <Person> results = realm.where(Person.class).findAll();
Person person = results.first();

Log.d("realm", person.name);

For example if you have code just accessing a name like the code above, bytecode manipulation replaces the field’s access to the getter call:


Log.d("realm", person.realmGet$name());

How it’s Done

So how do we actually do that? We use two technologies: One is Transform API, provided by Google and Android Gradle plugin, and the other one is Javassist, a library to modify bytecode.

Transform API

When you make app which uses Realm, you can see this line :app:transformClassesWithRealmTransformerForDebug. This means that the transformer is applied to your application.

To apply the transformer, we execute project.android.registerTransform(new RealmTransformer(project)), and the Gradle plugin has an API to register the transformer. When the transformer is registered, the transformer is called at compile time, and then we receive some information for transforming.

Transformer can give some information with context like input classes, referenced libraries, or output directories. In our transformer we add the accessors to the classes and also add the proxy interface to the classes. Each class’s name contains all classes using the application code. Then we modify every field access in every class using methods.


@Override
void transform(...) throws IOException, ... {

  // Create and populate the Javassist class pool
  ClassPool classPool = createClassPool(inputs, referencedInputs)

  inputModelClasses.each {
    BytecodeModifier.addRealmAccessors(it)
    BytecodeModifier.addRealmProxyInterface(it,classPool)
  }

  inputClassNames.each {
    def ctClass = classPool.getCtClass(it)
    BytecodeModifier.useRealmAccessors(ctClass, allManagedFields, allModelClasses)
    ctClass.writeFile(getOutputFile(outputProvider).canonicalPath)
  }
}

There are many bytecode manipulation libraries - some are low level and some of them are very high level. High level libraries are made for very specific purposes and they’re easy to use. Retrolambda and AspectJ are two common high-level bytecode manipulation libraries. Low level libraries are very powerful, and can do everything the Java virtual machine can do, but they are very hard to use, and require very deep knowledge about commands, bytecodes, and virtual machines.

Realm uses Javassist, which is middle level. There’s no need to know all the details of the bytecode, but we can still use it for very generic purposes. For example, Javassist can add classes and fields. Defining a field or defining class is very easy. When we define a field we just create a Ctfield instance and set the modifier. To define a class we just call makeClass. Adding a field to a class is very easy with addField. You can use Javassist with knowledge of the Java language, but not the bytecode.

Peep the Code

Then this is the actual code in Realm Java:


public static void addRealmAccessors(CtClass clazz) {
  def methods = clazz.getDeclaredMethods()*.name
    clazz.declaredFields.each { CtField field ->
        if (isModelField(field)) {
            if (!methods.contains("realmGet\$${field.name}".toString())) {
                clazz.addMethod(CtNewMethod.getter("realmGet\$${field.name}", field))
            }
            if (!methods.contains("realmSet\$${field.name}".toString())) {
                clazz.addMethod(CtNewMethod.setter("realmSet\$${field.name}", field))
            }
        }
    }
}

We call the addRealmAccessors from the transformer, and the first line collects all the names of the declared methods. Then we get the declared fields in the class, and if it’s a modelfield, we add the special getters and setters to that class.

This is the code to add the accessors:


public static void useRealmAccessors (CtClass clazz, List<CtField> managedFields) {
    clazz.getDeclaredBehaviors().each { behavior ->
        if (
              (
                  behavior instanceof CtMethod &&
                  !behavior.name.startsWith('realmGet$') &&
                  !behavior.name.startsWith('realmSet$')
              ) || (
                  behavior instanceof CtConstructor
              )
        )  {
              behavior.instrument(
                  new FieldAccessToAccessorConverter(managedFields, clazz, behavior))
        }
    }
}

We’re getting the declared behaviors. If it’s a method but the name of the method doesn’t start with realmGet or realmSet, the constructor will replace their access to the fields.

This is actual converter code:


private static class FieldAccessToAccessorConverter extends ExprEditor {
  @Override
  void edit(FieldAccess fieldAccess) throws CannotCompileException {
    def isRealmFieldAccess = managedFields.find {
        fieldAccess.className.equals(it.declaringClass.name) && fieldAccess.fieldName.equals(it.name)
    }
    if (isRealmFieldAccess != null) {
      def fieldName = fieldAccess.fieldName
      if (fieldAccess.is.Reader()) {
        fieldAccess.replace('$_=$0.realmGet$' + fieldName + '();')
      } else if (fieldAccess.isWriter()) {
        fieldAccess.replace('$0.realmSet$' + fieldName + '($1);')
      }
    }
  }
}

If it’s access to the model, and if it’s a reader, we replace the read access to the method call realmGet$ + fieldName. If it’s a field write, we replace it with a setter call.

Conclusion

You can find everything here in our repository because Realm is open source. Everything I’ve explained is all open, and you can read the code yourself.

Next Up: New Features in Realm Java

General link arrow white

About the content

This content has been published here with the express permission of the author.

Makoto Yamazaki

Makoto Yamazaki is a software engineer with extensive experience in Java and Android. He is a member of Realm Java team and also CTO of uPhyca Inc.