Tuesday, December 20, 2016

Quick Tip To Prevent Your Caches From Exploding

There are many scenarios when you can benefit from caching commonly used objects in your application, especially in web and micro-service oriented environments. The most simple type of caching you can do in Java is probably to introduce a private HashMap that you query before calculating an object to make sure you don't do the job twice.

Here is an example of that:


public class PrimeService {

    private Map<Long, BigInteger> cache = new HashMap<>();
    
    public BigInteger getPrime(long minimum) {
        return cache.computeIfAbsent(minimum, 
            m -> BigInteger.valueOf(m).nextProbablePrime()
        );
    }
    
}

This is a quick solution to the problem, but sadly not very efficient. After a few months of people entering all kinds of crazy numbers into the service, we will have a very large HashMap that might cause us to run out of memory.

Here is a quick trick to solve that. Instead of using a HashMap, you can use a LinkedHashMap and simply override the method removeEldestEntry. That allows you to configure your own limit function to prevent the map from exploding.


public class PrimeService {

    public final static int CACHE_MAX_SIZE = 1_000;

    private Map<Long, BigInteger> cache = new LinkedHashMap<>() {
        @Override
        protected boolean removeEldestEntry(Map.Entry<ID, Boolean> eldest) {
            return size() > PrimeService.CACHE_MAX_SIZE;
        }
    };
    
    public BigInteger getPrime(long minimum) {
        return cache.computeIfAbsent(minimum, 
            m -> BigInteger.valueOf(m).nextProbablePrime()
        );
    }
    
}

We have now successfully limited the cache, preventing it from explosion. Old entries will be removed as new ones are added. It should be noted that this soluton works well in small applications, but for more advanced scenarious you might want to use an external caching solution instead. If you are interested in caching entries in a SQL database without writing tons of SQL code, I would recommend Speedment since it is lightweight and has a very fluent API.

Until next time!

Friday, December 9, 2016

Creating a REST API with Speedment and Spring

Happy Spring Wishes from Speedment and Crew

With the 4th release of Spring Boot, developing enterprise applications for the web have become so much easier. Something that still requires a lot of time on the developer's behalf is modelling an existing database in for an example Hibernate to get an object-oriented view of the data. In this tutorial we are going to explore how to use the open source tool Speedment together with Spring to generate entities, managers and controllers with the press of a button, enabling you to get started with the development so much faster.

About Speedment

Speedment is an open source java toolkit that makes it possible for a developer to rapidly generate all the glue required to communicate with a database. Using a graphical tool, you can connect to a database and generate java sources in seconds. Speedment is built in a modular fashion, just like Spring, making it easy to learn and use only the parts you are interested in. In this article we are going to use a plugin for Speedment to generate Spring controllers in addition to the standard files.

Step 1: Creating a New Spring Boot Project

Spring Boot consists of a number of templates that make it easy to get started with a new application. We are going to use one called “spring-boot-starter-web” to set the stage for our web application.

Begin by creating a new Maven Project and add the following to your “pom.xml”-file:



<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>1.4.1.RELEASE</version>
    </parent>
    
    <groupId>com.github.pyknic</groupId>
    <artifactId>speedment-spring-example</artifactId>
    <version>1.0.0-SNAPSHOT</version>
    <packaging>jar</packaging>
    
    <properties>
        <java.version>1.8</java.version>
        
        <speedment.version>3.0.1</speedment.version>
        <mysql.version>5.1.39</mysql.version>
    </properties>
    
    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
            
            <plugin>
                <groupId>com.speedment</groupId>
                <artifactId>speedment-maven-plugin</artifactId>
                <version>${speedment.version}</version>
                
                <dependencies>
                    <dependency>
                        <groupId>mysql</groupId>
                        <artifactId>mysql-connector-java</artifactId>
                        <version>${mysql.version}</version>
                    </dependency>
                    
                    <dependency>
                        <groupId>com.speedment.plugins</groupId>
                        <artifactId>spring-generator</artifactId>
                        <version>${speedment.version}</version>
                    </dependency>
                </dependencies>
                
                <configuration>
                    <components>                       
<component>com.speedment.plugins.spring.SpringGeneratorBundle</component>
                    </components>
                </configuration>
            </plugin>
        </plugins>
    </build>
    
    <dependencies>
        <dependency>
            <groupId>com.speedment</groupId>
            <artifactId>runtime</artifactId>
            <version>${speedment.version}</version>
            <type>pom</type>
        </dependency>
        
        <dependency>
            <groupId>com.speedment.plugins</groupId>
            <artifactId>spring-generator</artifactId>
            <version>${speedment.version}</version>
        </dependency>
        
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>${mysql.version}</version>
        </dependency>
        
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
    </dependencies>
</project>

This will configure your project as a Spring Boot application and tell it to use Speedment with the Spring Generator Plugin.

Step 2: Using Speedment to Generate Sources

Once the pom-file has been modified, a number of new Maven goals will be available in the IDE. Speedment can be used both graphically or from the command line. In this tutorial, we are going to use the UI. To start the Speedment Tool, execute the following Maven goal on the project:

mvn speedment:tool

A dialog will open that let you connect to a database. Once connected, you will see a window with an overview of the database on the left and various configuration options in the center. For this tutorial the default settings will suffice, so simply press “Generate” in the toolbar.

Screenshot of the Speedment Open Source Tool

If you switch back to the IDE you will see the new generated sources. You will notice that every class exists in two copies, one of them with the “Generated”-prefix. The reason for this is to allow modifications without the risk of overwriting your changes should you need to regenerate the sources at some point. Files with the “Generated”-prefix will always be overwritten and files without it will only be created once.

Step 3: Create a Main File

Speedment has generated a complete object-oriented model of the database, but we still need to create an entry point for the application. We will put this in the main package and call it Main.java.

Main.java

package com.github.pyknic.spring;

import com.speedment.common.logger.Level;
import com.speedment.common.logger.LoggerManager;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class Main {
    
    public static void main(String... args) {
        SpringApplication.run(Main.class, args);
    }
}

If we start the application, Spring Boot will setup a self-contained web application with a generated controller for every table in the database. We can try it out by going to the following path in a browser:

http://localhost:8080/hare/

A JSON representation of the “hare” table in my database will now be returned.


[
    {"id":1, "name":"Harry", "color":"Gray", "age":3},
    {"id":2, "name":"Henrietta", "color":"White", "age":2},
    {"id":3, "name":"Henry", "color":"Black", "age":9}
]

Note #1: If you get an exception that says something in style of this...

There was an unexpected error (type=Internal Server Error, status=500). 
Could not write content: No value present (through reference chain: 
java.util.ArrayList[0]...

...it probably means that you have nullable columns in your database that Speedment chooses to implement as OptionalLong, OptionalInt etc. You can turn this feature off in the Speedment Tool by setting the "Nullable Implementation" field to WRAPPER instead of OPTIONAL for those columns.

Note #2: If you get an exception in the style of this...

java.sql.SQLException: Access denied for user 'root'@'localhost' 
(using password: YES)

...you will need to create an application.properties-file in the root of the project and add the authentication details for your database.

application.properties

jdbc.username=root
jdbc.password=password

Summary

In this article we have used Speedment and the Spring Generator plugin to automatically create a complete Spring Boot Application. Speedment has generated entities, managers and REST controllers for communicating with the database. If you want to know more about Speedment and how you can control the generated code, check out the many examples at the Speedment GitHub page!

Monday, November 21, 2016

Hack Speedment into Your Own Personal Code Generator

Spire and Duke dressed as Neo from Matrix

Speedment is an Open Source toolkit that can be used to generate Java entities and managers for communicating with a database. This is great if you need an Object Relational Mapping of the domain model, but in some cases, you might want to generate something completely different using your database as the template. In this article I am going to show you a hack that you can use to take over that code generator in Speedment and use it for your own personal purposes. At the end of the article we will have a completely blank code generator that we can program to do our bidding!

Background

Speedment is designed to operate as a plugin to Maven. By invoking various new Maven goals we can instruct Speedment to connect to a database, generate source code and also remove any generated files from our project. It also contains a graphical user interface that makes it easy to configure the generation job based on metadata collected from our database. Now, imagine all this information that we can collect from analyzing that metadata. We know what tables exist, we know of all the constraints they have and what types the individual columns have. There are probably millions of use-cases where we can benefit from automatically generating stuff from that metadata. Following the steps in this article, we can do all those things.

Screenshot of the Speedment User Interface

Step 1: Set Up a Regular Speedment Project

Create a new Maven Project and add the following to the pom.xml-file:

pom.xml

<properties>
    <speedment.version>3.0.1</speedment.version>
    <mysql.version>5.1.39</mysql.version>
</properties>


<dependencies>
    <dependency>
        <groupId>com.speedment</groupId>
        <artifactId>runtime</artifactId>
        <version>${speedment.version}</version>
        <type>pom</type>
    </dependency>
</dependencies>


<build>
    <plugins>
        <plugin>
            <groupId>com.speedment</groupId>
            <artifactId>speedment-maven-plugin</artifactId>
            <version>${speedment.version}</version>
            <dependencies>
                <dependency>
                    <groupId>mysql</groupId>
                    <artifactId>mysql-connector-java</artifactId>
                    <version>${mysql.version}</version>
                </dependency>
            </dependencies>
        </plugin>
    </plugins>
</build>

We have added Speedment as a runtime dependency and configured the Maven plugin to use the standard MySQL JDBC driver to connect to our database. Great! You now have access to a number of new Maven goals. For an example, if we wanted to launch the Speedment UI we could do that by running:


mvn speedment:tool

If we did that now, Speedment would launch in normal mode allowing us to connect to a database and from it generate entities and managers for communicating with that database using Java 8 streams. That is not what we want to do this time around. We want to hack it so that it does exactly what we need it to do. We therefore continue to modify the pom.

Step 2: Modify the Plugin Declaration

Speedment is built up in a modular fashion with different artifacts responsible for different tasks. All the pre-existing generator tasks are located in an artifact called "com.speedment.generator:generator-standard". That is where we are going to strike! By removing that artifact from the classpath we can prevent Speedment from generating anything we don’t want it to.

We modify the pom as follows:


...
<plugin>
    <groupId>com.speedment</groupId>
    <artifactId>speedment-maven-plugin</artifactId>
    <version>${speedment.version}</version>
    <dependencies>
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>${mysql.version}</version>
        </dependency>
       
        <!-- Add this: -->
        <dependency>
            <groupId>com.speedment</groupId>
            <artifactId>tool</artifactId>
             <version>${speedment.version}</version>
             <type>pom</type>
             <exclusions>
                 <exclusion>
                     <groupId>com.speedment.generator</groupId>
                     <artifactId>generator-standard</artifactId>
                 </exclusion>
            </exclusions>
        </dependency>
    </dependencies>
</plugin>
...

What is that? We exclude a dependency by adding one? How can that even work? Well, Speedment is designed to include as little code as possible unless it is explicitly needed by the application. The "com.speedment:tool-artifact" is a dependency to the maven plugin already, and by mentioning it in the <dependencies>-section of the maven plugin we can append settings to its configuration. In this case, we say that we want the plugin to have access to the tool except we don’t want the standard generator.

There is a problem here though. If we try and launch the speedment:tool goal, we will get an exception. The reason for this is that Speedment expects the standard translators to be on the classpath.

Here is where the second ugly hack comes in. In our project, we create a new package called com.speedment.generator.standard and in it define a new java file called StandardTranslatorBundle.java. As it turns out, that is the only file that Speedment really needs to work. We give it the following content:

StandardTranslatorBundle.java

package com.speedment.generator.standard;


import com.speedment.common.injector.InjectBundle;
import java.util.stream.Stream;


public final class StandardTranslatorBundle implements InjectBundle {
    @Override
    public Stream<Class<?>> injectables() {
        return Stream.empty();
    }
}

Next, we need to replace the excluded artifact with our own project so that the plugin never realizes that the files are missing. We return to the pom.xml-file and add our own project to the <dependencies>-section of the speedment-maven-plugin. The full pom file looks like this:

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" 
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
    http://maven.apache.org/xsd/maven-4.0.0.xsd">
    
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.github.pyknic</groupId>
  <artifactId>speedment-general-purpose</artifactId>
  <version>1.0.0-SNAPSHOT</version>
  <packaging>jar</packaging>
    
  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <maven.compiler.source>1.8</maven.compiler.source>
    <maven.compiler.target>1.8</maven.compiler.target>
    <speedment.version>3.0.1</speedment.version>
  </properties>
    
  <dependencies>
    <dependency>
      <groupId>com.speedment</groupId>
      <artifactId>runtime</artifactId>
      <version>${speedment.version}</version>
      <type>pom</type>
    </dependency>
  </dependencies>
    
  <build>
    <plugins>
      <plugin>
        <groupId>com.speedment</groupId>
        <artifactId>speedment-maven-plugin</artifactId>
        <version>${speedment.version}</version>
        <dependencies>
          <dependency>
            <groupId>com.speedment</groupId>
            <artifactId>tool</artifactId>
            <version>${speedment.version}</version>
            <type>pom</type>
            <exclusions>
              <exclusion>
                <groupId>com.speedment.generator</groupId>
                <artifactId>generator-standard</artifactId>
              </exclusion>
            </exclusions>
          </dependency>
          <dependency>
            <groupId>com.github.pyknic</groupId>
            <artifactId>speedment-general-purpose</artifactId>
            <version>1.0.0-SNAPSHOT</version>
          </dependency>   
          <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>5.1.39</version>
          </dependency>
        </dependencies>
      </plugin>
    </plugins>
  </build>
</project>

If we now build our project and then run the goal speedment:tool, we should be able to launch the graphical user interface without problem. If we connect to the database and then press "Generate", nothing will happen at all! We have successfully hacked Speedment into doing absolutely nothing!

Step 3: Turn Speedment Into What You Want It To Be

Now when we have a fresh, clean Speedment, we can start turning it into the application we want it to be. We still have a powerful user interface where we can configure the code generation based on a database model. We have an expressive library of utilities and helper classes to work with generated code. And above all, we have a structure for analyzing the database metadata in an object-oriented fashion.

To learn more about how to write your own code generation templates and hook them into the platform, check out this article. You should also check out the Speedment GitHub page to learn how the existing generators work (the ones we just disabled) and maybe get some inspiration on how you can build your own.

Until next time, continue hacking!

Monday, November 14, 2016

Auto-Generate Optimized Java Class Specializations

Spire and Duke are playing with specialized geometric shapes

If you visited JavaOne this year you might have attended my presentation on “How to Generate Customized Java 8 Code from your Database”. In that talk I showcased how the Speedment Open Source toolkit is used to generate all kinds of Java code using a database as the domain model. One thing we didn’t have time to go into though is the fact that Speedment is not only making code generation easier, it is also itself made up by generated code. In this article I will show you have we set up Speedment to generate specialized versions of many classes to reduce the memory footprint of performance critical parts of the system.

Background

As you might know, Java has a number of built-in value types. These are bytes, shorts, integers, longs, floats, doubles, booleans and characters. Primitive value types are different from ordinary objects primarily in that they can be allocated directly on the memory stack, reducing the burden on the Garbage Collector. A problem with not inheriting from Object is that they can’t be put into collections or passed as parameters to methods that take object parameters without being wrapped. Typical wrapper classes are therefore “Integer”, “Double”, “Boolean” etc.

Wrapping a value type is not always a bad thing. The JIT (Just-In-Time) compiler is very good at optimizing away wrapper types if they can safely be replaced with primitive values, but that is not always possible. If this occur in a performance critical section of your code like in an inner loop, this can heavily impact the performance of the entire application.

That’s what happened to us when working on Speedment. We had special predicates and functions that contained metadata about their purpose. That metadata needed to be analyzed very rapidly inside an inner loop, but we was slowed down by the fact that most of this metadata was wrapped inside generic types so that they could be instantiated dynamically.

A common solution to this problem is to create a number of “specializations” of the classes that contain the wrapper types. The specializations are identical to the original class except that they use one of the primitive value types instead of a generic (object only) type. A good example of specializations are the various Stream interfaces that exist in Java 8. In addition to “Stream” we also have an “IntStream”, a “DoubleStream” and a “LongStream”. These specializations are more efficient for their particular value type since they don’t have to rely on wrapping types in objects.

The problem with specialization classes is that they add a lot of boilerplate to the system. Say that the parts that need to be optimized consist of 20 components. If you want to support all the 8 primitive variations that Java has you suddenly have 160 components. That is a lot of code to maintain. A better solution would be to generate all the extra classes.

Template Based Code Generation

The most common form of code generation in higher languages is template based. This means that you write a template file and then do keyword replacement to modify the text depending on what you are generating. Good examples of these are Maven Archetypes or Thymeleaf. A good template engine will have support for more advanced syntax like repeating sections, expressing conditions etc. If you want to generate specialization classes using a template engine you would replace all occurrences of “int”, “Integer”, “IntStream” with a particular keyword like “${primitive}”, “${wrapper}”, “${stream}” and then specify the dictionary of words to associate with each new value type.

Advantages of template based code generation is that it is easy to setup and easy to maintain. I think most programmers that read this could probably figure out how to write a template engine fairly easy. A disadvantage is that a templates are difficult to reuse. Say that you have a basic template for a specialized, but you want floating types to also have an additional method. You could solve this with a conditional statement, but if you want that extra method to also exist in other places, you will need to duplicate code. A typical example of code that often need to be duplicated is hashCode()-methods or toString(). This is where model based code generation is stronger.

Model Based Code Generation

In model based code generation, you build an abstract syntax tree over the code you want generated and then render that tree using a suitable renderer. The syntax tree can be mutated depending on the context that it is being used in, for an example by adding or removing methods to implement a certain interface. The main advantage of this is higher flexibility. You can dynamically take an existing model and manipulate which methods and fields to include. The downside is that model based code generation generally take a bit longer to set up.

Case Study: Speedment Field Generator

At Speedment, we developed a code generator called CodeGen that uses the model based approach to automatically generate field specializations for all the primitive value types. In total around 300 classes are generated on each build.

Speedment CodeGen uses an abstract syntax tree built around the basic concepts of object oriented design. You have classes, interfaces, fields, methods, constructors, etc that you use to build the domain model. Below the method level you still need to write templated code. To define a new main class, you would write:


import com.speedment.common.codegen.model.Class; // Not java.lang.Class

...

Class createMainClass() {
  return Class.of("Main")
    .public_().final_()
    .set(Javadoc.of("The main entry point of the application")
      .add(AUTHOR.setValue("Emil Forslund"))
      .add(SINCE.setValue("1.0.0"))
    )
    .add(Method.of("main", void.class)
      .public_().static_()
      .add(Field.of("args", String[].class))
      .add(
        "if (args.length == 0) " + block(
          "System.out.println(\"Hello, World!\");"
        ) + " else " + block(
          "System.out.format(\"Hi, %s!%n\", args[0]);"
        )
      )
    );
}

This would generate the following code:


/**
 * The main entry point of the application.
 * 
 * @author Emil Forslund
 * @since  1.0.0
 */
public final class Main {
  public static void main(String[] args) {
    if (args.length == 0) {
      System.out.println("Hello, World!");
    } else {
      System.out.format("Hi, %s!%n", args[0]);
    }
  }
}

The entire model doesn’t have to be generated at once. If we for an example want to generate a toString()-method automatically, we can define that as an individual method.


public void generateToString(File file) {
  file.add(Import.of(StringBuilder.class));
  file.getClasses().stream()
    .filter(HasFields.class::isInstance)
    .filter(HasMethods.class::isInstance)
    .map(c -> (HasFields & HasMethods) c)
    .forEach(clazz -> 
      clazz.add(Method.of("toString", void.class)
        .add(OVERRIDE)
        .public_()
        .add("return new StringBuilder()")
        .add(clazz.getFields().stream()
          .map(f -> ".append(\"" + f.getName() + "\")")
          .map(Formatting::indent)
          .toArray(String[]::new)
        )
        .add(indent(".toString();"))
      )
    );
}

Here you can see how the Trait Pattern is used to abstract away the underlying implementation from the logic. The code will work for Enum as well as Class since both implement both the traits “HasFields” and “HasMethods”.

Summary

In this article I have explained what specialization classes are and why they are sometimes necessary to improve performance in critical sections of an application. I have also showed you how Speedment use model based code generation to automatically produce specialization classes. If you are interested in generating code with these tools yourself, go ahead and check out the latest version of the generator at GitHub!

Monday, October 31, 2016

What’s New in Speedment 3.0

Spire standing in front of a sign that says Forest

If you have followed my blog you know that I have been involved in the open-source project Speedment for a while. During the summer and fall I have worked a lot with finishing up the next big 3.0.0 release of the toolkit. In this post I will showcase some of the cool new features we have built into the platform and also explain how you can get started!

New Module System

The biggest change from the previous version of Speedment, and the one that took us most time to get right, is the new module system. If you have been following the progress of the new JDK 9 project Jigsaw, you will recognize this subject. Previously, Speedment consisted of one big artifact called com.speedment:speedment. In addition to this, we had a few minor projects like the speedment-maven-plugin and speedment-archetypes that made the tool easier to use. There was several issues with this design. First, it was very tedious to develop in it since we often needed to rebuild the entire thing multiple times each day and every build could take minutes. It was also not very plugin-friendly since a plugin had to depend on the entire code base, even if it only modified a small group of classes.

In 3.0 however, com.speedment is actually a multi-module pom-project with a clear build order. Inside it are groups of artifacts, also realized as multi-module projects, that separate artifacts based on when they are needed. We now have the following artifact groups:

  1. common-parent contains artifacts that are mature, reusable in a number of situations and that doesn’t have any dependencies (except on our own lightweight logging framework). Here you will find some of the core utilities of Speedment like MapStream and CodeGen.
  2. runtime-parent contains artifacts that are required by the end-user during the runtime of their application. We wanted to separate these into their own group to make sure that the final jar of the user’s app has as small footprint as possible.
  3. generator-parent contains artifacts related to the code generation and database analyzation parts of Speedment. These classes doesn’t require a graphical environment which is useful if you want to use Speedment as a general purpose code generator in a non-graphical environment.
  4. tool-parent contains all the artifacts used by the graphical Speedment tool. Here we put all our home-brewed JavaFX-components as well as resources like icons used by the UI.
  5. build-parent is a meta group that contains various artifacts that we build simply to make Speedment easier to use for the end user. Here we for an example have a number of shaded artifacts that you can use when you deploy your application on a server and the Maven Plugin that users use to launch Speedment as a Maven goal.
  6. plugins-parent is a whole new group where we put official plugins for Speedment that doesn’t quite fit into the general framework but that many users request. This makes it possible for us to automatically rebuild them in the general build cycle, making sure they are always up-to-date with the latest changes in the platform.
  7. archetypes-parent is a group of all the official Maven Archetypes for Speedment. This was previously a separate project but has now been lifted into the main project so that they can be automatically reinstalled every time Speedment is built.

All these groups are built in the same order as specified above. This makes it much easier to keep dependencies single-directional and the overall design of the system more comprehensive.

So how do I use it?

The beautiful thing is that you barely have to change a thing! We automatically build an artifact that is called com.speedment:runtime that you can depend on in your project. It contains transitive dependencies to the exact set of artifacts that are required to run Speedment.


<dependency>
    <groupId>com.speedment</groupId>
    <artifactId>runtime</artifactId>
    <version>3.0.1</version>
    <type>pom</type>
</dependency>

When it is time for deployment, you simply replace this dependency with com.speedment:runtime-deploy and you will get a shaded jar with all the Speedment-stuff bundled together and ready to ship!


<dependency>
    <groupId>com.speedment</groupId>
    <artifactId>runtime-deploy</artifactId>
    <version>3.0.1</version>
</dependency>

For more details about the new release, go to this official GitHub page and fork it!

Monday, October 24, 2016

Database CRUD Operations in Java 8 Streams

Spire and Duke in front of the letters CRUD

The biggest obstacle to overcome when starting out with a new tool is to get your head around how to do the little things. By now you might feel confident in how the new Java 8 Stream API works, but you might not have used it for database querying yet. To help you get started creating, modifying and reading from your SQL database using the Stream API, I have put together this quick start. Hopefully it can help you take your streams to the next level!

Background

Speedment is an Open Source toolkit that can be used to generate java entities and managers for communicating with a database. Using a graphical tool you connect to your database and generate a complete ORM tailored to represent your domain model. But Speedment is not only a code generator but also a runtime that plugs into your application and makes it possible to translate your Java 8 streams into optimized SQL queries. That is the part that I will focus on in this article.

Generate Code

To begin using Speedment in a Maven project, add the following lines to your pom.xml-file. In this example I am using MySQL, but you can use PostgreSQL or MariaDB as well. Connectors to proprietary databases like Oracle are available for enterprise customers.

pom.xml

<properties>
  <speedment.version>3.0.1</speedment.version>
  <db.groupId>mysql</db.groupId>
  <db.artifactId>mysql-connector-java</db.artifactId>
  <db.version>5.1.39</db.version>
</properties>

<dependencies>
  <dependency>
    <groupId>com.speedment</groupId>
    <artifactId>runtime</artifactId>
    <version>${speedment.version}</version>
    <type>pom</type>
  </dependency>
        
  <dependency>
    <groupId>${db.groupId}</groupId>
    <artifactId>${db.artifactId}</artifactId>
    <version>${db.version}</version>
  </dependency>
</dependencies>

<build>
  <plugins>
    <plugin>
      <groupId>com.speedment</groupId>
      <artifactId>speedment-maven-plugin</artifactId>
      <version>${speedment.version}</version>

      <dependencies>
        <dependency>
          <groupId>${db.groupId}</groupId>
          <artifactId>${db.artifactId}</artifactId>
          <version>${db.version}</version>
        </dependency>
      </dependencies>
    </plugin>
  </plugins>
</build>

You now have access to a number of new Maven Goals that make it easier to use the toolkit. The launch the Speedment UI, execute:

mvn speedment:tool

This will guide you through the process of connecting to the database and configure the code generation. The simplest way in the beginning is you just run along with the default settings. Once you press “Generate”, Speedment will analyze your database metadata and fill your project with new sources like entity and manager classes.

Initialize Speedment

Once you have generated your domain model, setting up Speedment is easy. Create a new Main.java-file and add the following lines. All the classes you see are generated so their names will depend on the names of your database schemas, tables and columns.

Main.java

public class Main {
  public static void main(String... param) {
    final HaresApplication app = new HaresApplicationBuilder()
      .withPassword("password")
      .build();
  }
}

The code above creates a new application instance using a generated builder pattern. The builder makes it possible to set any runtime configuration details like database passwords.

Once we have an app instance, we can use it to get access to the generated managers. In this case, I have four tables in the database; “hare”, “carrot”, “human” and “friend”. (You can see the entire database definition here).


final CarrotManager carrots = app.getOrThrow(CarrotManager.class);
final HareManager hares     = app.getOrThrow(HareManager.class);
final HumanManager humans   = app.getOrThrow(HumanManager.class);
final FriendManager friends = app.getOrThrow(FriendManager.class);

These managers can now be used to perform all our CRUD operations.

Create Entities

Creating entities is very straight forward. We use the generated implementation of our entities, set the values we want for columns and then persist it to the data source.


hares.persist(
  new HareImpl()
    .setName("Harry")
    .setColor("Gray")
    .setAge(8)
);

The persist method returns a (potentially) new instance of Hare where auto-generated keys like “id” has been set. If we want to use Harry after persisting him we should therefore use the instance returned by persist.


final Hare harry = hares.persist(
  new HareImpl()
    .setName("Harry")
    .setColor("Gray")
    .setAge(8)
);

If the persistence fails, for an example if a foreign key or a unique constraint fails, a SpeedmentException is thrown. We should check for this and show an error if something prevented us from persisting the hare.


try {
  final Hare harry = hares.persist(
    new HareImpl()
      .setName("Harry")
      .setColor("Gray")
      .setAge(8)
  );
} catch (final SpeedmentException ex) {
  System.err.println(ex.getMessage());
  return;
}

Read Entities

The coolest feature in the Speedment runtime is the ability to stream over data in your database using Java 8 Streams. “Why is that so cool?” you might ask yourself. “Even Hibernate have support for streaming nowadays!”

The beautiful thing with Speedment streams is that they take intermediary and terminating actions into consideration when constructing the stream. This means that if you add a filter to the stream after it has been created, it will still be taken into consideration when building the SQL statement.

Here is an example. We want to count the total number of hares in the database.


final long haresTotal = hares.stream().count();
System.out.format("There are %d hares in total.%n", haresTotal);

The SQL query that will be generated is the following:


SELECT COUNT(*) FROM hares.hare;

The terminating operation was a .count() so Speedment knows that it is a SELECT COUNT(...)-statement that is to be created. It also knows that the primary key for the “hare” table is the column “id”, which makes it possible to reduce the entire statement sent to the database down into this.

A more complex example might be to find the number of hares that has a name that ends with the letters “rry” and an age greater or equal to 5. That can be written as this:


final long complexTotal = hares.stream()
  .filter(Hare.NAME.endsWith("rry"))
  .filter(Hare.AGE.greaterOrEqual(5))
  .count();

We use the predicate builders generated to us by Speedment to define the filters. This make it possible for us to analyze the stream programmatically and reduce it down to the following SQL statement:


SELECT COUNT(id) FROM hares.hare
WHERE hare.name LIKE CONCAT("%", ?)
AND hare.age >= 5;

If we add an operation that Speedment can’t optimize to the stream, it will be resolved just like any Java 8 stream. We are never limited to the use of the generated predicate builders, it just makes the stream more efficient.


final long inefficientTotal = hares.stream()
  .filter(h -> h.getName().hashCode() == 52)
  .count();

This would produce the following extremely inefficient statement, but it will still work.


SELECT id,name,color,age FROM hares.hare;

Update Entities

Updating existing entities are done very similar to how we read and persist entities. Changes made to a local copy of an entity will not affect the database until we call the update()-method in the manager.

In this case we take the hare Harry created earlier and we want to change his color to brown:


harry.setColor("brown");
final Hare updatedHarry = hares.update(harry);

The manager returns a new copy of the hare if the update is accepted, so we should continue using that instance after this point. Just like in the “create”-example, the update might fail. Maybe color was defined as a “unique” column and a “brown” hare already existed. In that case, a SpeedmentException is thrown.

We can also update multiple entities at the same time by combining it with a stream. Say that we want to make all hares named “Harry” brown. In that case, we do this:


hares.stream()
  .filter(Hare.NAME.equal("Harry"))
  .map(Hare.COLOR.setTo("Brown"))
  .forEach(hares.updater()); // Updates remaining elements in the Stream

We should also wrap it in a try-catch to make sure we warn the user if a constraint failed.


try {
  hares.stream()
    .filter(Hare.NAME.equal("Harry"))
    .map(Hare.COLOR.setTo("Brown"))
    .forEach(hares.updater());
} catch (final SpeedmentException ex) {
  System.err.println(ex.getMessage());
  return;
}

Removing Entities

The last CRUD operation we need to know is how to remove entities from the database. This is almost identical to the “update”. Say that we want to remove all hares older than 10 years. We then do this:


try {
  hares.stream()
    .filter(Hare.AGE.greaterThan(10))
    .forEach(hares.remover()); // Removes remaining hares
} catch (final SpeedmentException ex) {
  System.err.println(ex.getMessage());
  return;
}

Summary

In this article you have learned how to set up Speedment in a Maven project and how to create, update, read and delete entities from a database using Java 8 Streams. This is only a small subset of all the things you can do with Speedment, but it serves as a good introduction to start getting your hands dirty. More examples and more advanced use-cases can be found on the GitHub-page.

Until next time!

Tuesday, October 11, 2016

Event-Sourcing and CQRS in Practise

Sauna.png

Anyone that has tried to implement a fully ACID compliant system knows that there are a lot of considerations you have to do. You need to make sure database entities can be freely created, modified and deleted without the risk of errors, and in most cases, the solution will be at the cost of performance. One methodology that can be used to get around this is to design the system based on a series of events rather than mutable states. This is generally called Event Sourcing.

In this article I will showcase a demo application that uses the Open Source toolkit Speedment to rapidly get a scalable event-sourced database application up and running. Full source code for the example is available here.

What is Event Sourcing?

In a typical relational database system you store the state of an entity as a row in a database. When the state changes, the application modifies the row using an UPDATE or a DELETE-statement. A problem with this method is that it adds a lot of requirements on the database when it comes to making sure that no row is changed in a way that puts the system in an illegal state. You don’t want anyone to withdraw more money than they have in their account or bid on an auction that has already been closed.

In an event-sourced system, we take a different approach to this. Instead of storing the state of an entity in the database, you store the series of changes that led to that state. An event is immutable once it is created, meaning that you only have to implement two operations, CREATE and READ. If an entity is updated or removed, that is realized using the creation of an “update” or “remove” event.

An event sourced system can easily be scaled up to improve performance, as any node can simply download the event log and replay the current state. You also get better performance due to the fact that writing and querying is handled by different machines. This is referred to as CQRS (Command-Query Responsibility Segregation). As you will see in the examples, we can get an eventually consistent materialized view up and running in a very little time using the Speedment toolkit.

The Bookable Sauna

To showcase the workflow of building an event sourced system we will create a small application to handle the booking of a shared sauna in a housing complex. We have multiple tenants interested in booking the sauna, but we need to guarantee that the shy tenants never accidentally double-book it. We also want to support multiple saunas in the same system.

To simplify the communication with the database, we are going to use the Speedment toolkit. Speedment is a java tool that allows us to generate a complete domain model from the database and also makes it easy to query the database using optimized Java 8 streams. Speedment is available under the Apache 2-license and there are a lot of great examples for different usages on the Github page.

Step 1: Define the Database Schema

The first step is to define our (MySQL) database. We simply have one table called “booking” where we store the events related to booking the sauna. Note that a booking is an event and not an entity. If we want to cancel a booking or make changes to it, we will have to publish additional events with the changes as new rows. We are not allowed to modify or delete a published row.


CREATE DATABASE `sauna`;

CREATE TABLE `sauna`.`booking` (
  `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
  `booking_id` BIGINT NOT NULL,
  `event_type` ENUM('CREATE', 'UPDATE', 'DELETE') NOT NULL,
  `tenant` INT NULL,
  `sauna` INT NULL,
  `booked_from` DATE NULL,
  `booked_to` DATE NULL,
  PRIMARY KEY (`id`)
);

The “id” column is an increasing integer that is assigned automatically every time a new event is published to the log. The “booking_id” tells us which booking we are referring to. If two events share the same booking id, they refer to the same entity. We also have an enum called “event_type” that describes which kind of operation we were trying to do. After that comes the information that belongs to the booking. If a column is NULL, we will consider that as unmodified compared to any previous value.

Step 2: Generating Code using Speedment

The next step is to generate code for the project using Speedment. Simply create a new maven project and add the following code to the pom.xml-file.

pom.xml

<properties>
  <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  <maven.compiler.source>1.8</maven.compiler.source>
  <maven.compiler.target>1.8</maven.compiler.target>
  <speedment.version>3.0.0-EA2</speedment.version>
  <mysql.version>5.1.39</mysql.version>
</properties>

<build>
  <plugins>
    <plugin>
      <groupId>com.speedment</groupId>
      <artifactId>speedment-maven-plugin</artifactId>
      <version>${speedment.version}</version>

      <dependencies>
        <dependency>
          <groupId>mysql</groupId>
          <artifactId>mysql-connector-java</artifactId>
          <version>${mysql.version}</version>
        </dependency>
      </dependencies>
    </plugin>
  </plugins>
</build>

<dependencies>
  <dependency>
    <groupId>mysql</groupId>
    <artifactId>mysql-connector-java</artifactId>
    <version>${mysql.version}</version>
  </dependency>

  <dependency>
    <groupId>com.speedment</groupId>
    <artifactId>runtime</artifactId>
    <version>${speedment.version}</version>
    <type>pom</type>
  </dependency>
</dependencies>

If you build the project, a new maven goal called speedment:tool should appear in the IDE. Run it to launch the Speedment user interface. In there, connect to the Sauna database and generate code using the default settings. The project should now be populated with source files.

Tip: If you make changes to the database, you can download the new configuration using the speedment:reload-goal and regenerate sources using speedment:generate. No need to relaunch the tool!

Step 3: Creating the Materialized View

The materialized view is a component that regularly polls the database to see if any new rows have been added, and if so, downloads and merges them into the view in the correct order. Since the polling sometimes can take a lot of time, we want this process to run in a separate thread. We can accomplish that with a java Timer and TimerTask.

Polling the database? Really? Well, an important thing to take into consideration is that it is only the server that will poll the database, not the clients. This gives us very good scalability since we can have a handful of servers polling the database that in turn serve hundreds of thousands of tenants. Compare this with a regular system where every client would request a resource from the server, that in turn contacts the database.

BookingView.java

public final class BookingView {

  ...

  public static BookingView create(BookingManager mgr) {
    final AtomicBoolean working = new AtomicBoolean(false);
    final AtomicLong last  = new AtomicLong();
    final AtomicLong total = new AtomicLong();
        
    final String table = mgr.getTableIdentifier().getTableName();
    final String field = Booking.ID.identifier().getColumnName();

    final Timer timer = new Timer();
    final BookingView view = new BookingView(timer);
    final TimerTask task = ...;

    timer.scheduleAtFixedRate(task, 0, UPDATE_EVERY);
    return view;
  }
}

The timer task is defined anonymously and that is where the polling logic will reside.


final TimerTask task = new TimerTask() {
  @Override
  public void run() {
    boolean first = true;

    // Make sure no previous task is already inside this block.
    if (working.compareAndSet(false, true)) {
      try {

        // Loop until no events was merged 
        // (the database is up to date).
        while (true) {

          // Get a list of up to 25 events that has not yet 
          // been merged into the materialized object view.
          final List added = unmodifiableList(
            mgr.stream()
              .filter(Booking.ID.greaterThan(last.get()))
              .sorted(Booking.ID.comparator())
              .limit(MAX_BATCH_SIZE)
              .collect(toList())
            );

          if (added.isEmpty()) {
            if (!first) {
              System.out.format(
                "%s: View is up to date. A total of " + 
                "%d rows have been loaded.%n",
                System.identityHashCode(last),
                total.get()
              );
            }

            break;
          } else {
            final Booking lastEntity = 
              added.get(added.size() - 1);

            last.set(lastEntity.getId());
            added.forEach(view::accept);
            total.addAndGet(added.size());

            System.out.format(
              "%s: Downloaded %d row(s) from %s. " + 
              "Latest %s: %d.%n", 
              System.identityHashCode(last),
              added.size(),
              table,
              field,
              Long.parseLong("" + last.get())
            );
          }

          first = false;
        }

        // Release this resource once we exit this block.
      } finally {
        working.set(false);
      }
    }
  }
};

Sometimes the merging task can take more time to complete than the interval of the timer. To avoid this causing a problem, we use an AtomicBoolean to check and make sure that only one task can execute at the same time. This is similar to a Semaphore, except that we want tasks that we don’t have time for to be dropped instead of queued since we don’t really need every task to execute, a new one will come in just a second.

The constructor and basic member methods are fairly easy to implement. We store the timer passed to the class as a parameter in the constructor so that we can cancel that timer if we ever need to stop. We also store a map that keeps the current view of all the bookings in memory.


private final static int MAX_BATCH_SIZE = 25;
private final static int UPDATE_EVERY   = 1_000; // Milliseconds

private final Timer timer;
private final Map<Long, Booking> bookings;

private BookingView(Timer timer) {
  this.timer    = requireNonNull(timer);
  this.bookings = new ConcurrentHashMap<>();
}

public Stream<Booking> stream() {
  return bookings.values().stream();
}

public void stop() {
  timer.cancel();
}

The last missing piece of the BookingView class is the accept()-method used above in the merging procedure. This is where new events are taken into consideration and merged into the view.


private boolean accept(Booking ev) {
    final String type = ev.getEventType();

    // If this was a creation event
    switch (type) {
        case "CREATE" :
            // Creation events must contain all information.
            if (!ev.getSauna().isPresent()
            ||  !ev.getTenant().isPresent()
            ||  !ev.getBookedFrom().isPresent()
            ||  !ev.getBookedTo().isPresent()
            ||  !checkIfAllowed(ev)) {
                return false;
            }

            // If something is already mapped to that key, refuse the 
            // event.
            return bookings.putIfAbsent(ev.getBookingId(), ev) == null;

        case "UPDATE" :
            // Create a copy of the current state
            final Booking existing = bookings.get(ev.getBookingId());

            // If the specified key did not exist, refuse the event.
            if (existing != null) {
                final Booking proposed = new BookingImpl();
                proposed.setId(existing.getId());

                // Update non-null values
                proposed.setSauna(ev.getSauna().orElse(
                    unwrap(existing.getSauna())
                ));
                proposed.setTenant(ev.getTenant().orElse(
                    unwrap(existing.getTenant())
                ));
                proposed.setBookedFrom(ev.getBookedFrom().orElse(
                    unwrap(existing.getBookedFrom())
                ));
                proposed.setBookedTo(ev.getBookedTo().orElse(
                    unwrap(existing.getBookedTo())
                ));

                // Make sure these changes are allowed.
                if (checkIfAllowed(proposed)) {
                    bookings.put(ev.getBookingId(), proposed);
                    return true;
                }
            }

            return false;


        case "DELETE" :
            // Remove the event if it exists, else refuse the event.
            return bookings.remove(ev.getBookingId()) != null;

        default :
            System.out.format(
                "Unexpected type '%s' was refused.%n", type);
            return false;
    }
}

In an event sourced system, the rules are not enforced when events are received but when they are materialized. Basically anyone can insert new events into the system as long as they do it in the end of the table. It is in this method that we choose to discard events that doesn’t follow the rules setup.

Step 4: Example Usage

In this example, we will use the standard Speedment API to insert three new bookings into the database, two that are valid and a third that intersects one of the previous ones. We will then wait for the view to update and print out every booking made.


public static void main(String... params) {
  final SaunaApplication app = new SaunaApplicationBuilder()
    .withPassword("password")
    .build();

  final BookingManager bookings = 
    app.getOrThrow(BookingManager.class);

  final SecureRandom rand = new SecureRandom();
  rand.setSeed(System.currentTimeMillis());

  // Insert three new bookings into the system.
  bookings.persist(
    new BookingImpl()
      .setBookingId(rand.nextLong())
      .setEventType("CREATE")
      .setSauna(1)
      .setTenant(1)
      .setBookedFrom(Date.valueOf(LocalDate.now().plus(3, DAYS)))
      .setBookedTo(Date.valueOf(LocalDate.now().plus(5, DAYS)))
  );

  bookings.persist(
    new BookingImpl()
      .setBookingId(rand.nextLong())
      .setEventType("CREATE")
      .setSauna(1)
      .setTenant(2)
      .setBookedFrom(Date.valueOf(LocalDate.now().plus(1, DAYS)))
      .setBookedTo(Date.valueOf(LocalDate.now().plus(2, DAYS)))
  );

  bookings.persist(
    new BookingImpl()
      .setBookingId(rand.nextLong())
      .setEventType("CREATE")
      .setSauna(1)
      .setTenant(3)
      .setBookedFrom(Date.valueOf(LocalDate.now().plus(2, DAYS)))
      .setBookedTo(Date.valueOf(LocalDate.now().plus(7, DAYS)))
  );

  final BookingView view = BookingView.create(bookings);

  // Wait until the view is up-to-date.
  try { Thread.sleep(5_000); }
  catch (final InterruptedException ex) {
    throw new RuntimeException(ex);
  }

  System.out.println("Current Bookings for Sauna 1:");
  final SimpleDateFormat dt = new SimpleDateFormat("yyyy-MM-dd");
  final Date now = Date.valueOf(LocalDate.now());
  view.stream()
    .filter(Booking.SAUNA.equal(1))
    .filter(Booking.BOOKED_TO.greaterOrEqual(now))
    .sorted(Booking.BOOKED_FROM.comparator())
    .map(b -> String.format(
      "Booked from %s to %s by Tenant %d.", 
      dt.format(b.getBookedFrom().get()),
      dt.format(b.getBookedTo().get()),
      b.getTenant().getAsInt()
    ))
    .forEachOrdered(System.out::println);

  System.out.println("No more bookings!");
  view.stop();
}

If we run it, we get the following output:

677772350: Downloaded 3 row(s) from booking. Latest id: 3.
677772350: View is up to date. A total of 3 rows have been loaded.
Current Bookings for Sauna 1:
Booked from 2016-10-11 to 2016-10-12 by Tenant 2.
Booked from 2016-10-13 to 2016-10-15 by Tenant 1.
No more bookings!

Full source code for this demo application is available on my GitHub page. There you can also find many other examples on how to use Speedment in various scenarios to rapidly develop database applications.

Summary

In this article we have developed a materialized view over a database table that evaluates events on materialization and not upon insertion. This makes it possible to spin up multiple instances of the application without having to worry about synchronizing them since they will be eventually consistent. We then finished by showing how the materialized view can be queried using the Speedment API to produce a list of current bookings.

Thank you for reading and please checkout more Speedment examples at the Github page!

Wednesday, August 10, 2016

How To: Insert and Read From a Database using Json

Spire and Duke downloading stuff from a database

In this article we will create a plugin for Speedment that generates serialization and deserialization logic using Gson to make it super easy to map between database entities and JSON strings. This will help to showcase the extendability of the Speedment code generation while at the same time explore some of the cool features of the Gson library.

Speedment is a code generation tool for java that connects to a database and use it as a reference for generating entity and manager files for your project. The tool is very modular, allowing you to write your own plugins that modify the way the resulting code will look. One thing several people have mentioned on the Gitter chat is that Speedment entities are declared abstract which prevents them from being automatically deserialized. In this article we will look at how you can deserialize Speedment entities using Gson by automatically generating a custom TypeAdapter for each table in the database. This will not only give us better performance when working with JSON representations of database content, but might also serve as a general example on how you can extend the code generator to solve your problems.

Step 1: Creating the Plugin Project

In a previous article I went into detail on how to create a new plugin for Speedment, so here is the short version. Create a new maven project and set Speedment and Gson as dependencies.

pom.xml

<name>Speedment Gson Plugin</name>
<description>
    A plugin for Speedment that generates Gson Type Adapters for every 
    table in the database.
</description>

<properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <maven.compiler.source>1.8</maven.compiler.source>
    <maven.compiler.target>1.8</maven.compiler.target>
    <speedment.version>2.3.7</speedment.version>
</properties>

<dependencies>
    <dependency>
        <groupId>com.speedment</groupId>
        <artifactId>speedment</artifactId>
        <version>${speedment.version}</version>
    </dependency>

    <dependency>
        <artifactId>gson</artifactId>
        <groupId>com.google.code.gson</groupId>
        <version>2.6.2</version>
    </dependency>
</dependencies>

Step 2: Create a Translator Class for the Type Adapter

Next we need to create the translator that will generate the new type adapter for us. A translator is a class that describes what name, path and content a generated file will have. To that it has a lot of convenience methods to make it easier to generate the code. The basic structure of the translator is shown below.

GeneratedTypeAdapterTranslator.java

...
public GeneratedTypeAdapterTranslator(
       Speedment speedment, Generator gen, Table table) {
    super(speedment, gen, table, Class::of);
}

@Override
protected Class makeCodeGenModel(File file) {
    return newBuilder(file, getClassOrInterfaceName())
        .forEveryTable((clazz, table) -> {
            // Code generation goes here
        }).build();
}

@Override
protected String getClassOrInterfaceName() {
    return "Generated" + getSupport().typeName() + "TypeAdapter";
}

@Override
protected String getJavadocRepresentText() {
    return "A Gson Type Adapter";
}

@Override
public boolean isInGeneratedPackage() {
    return true;
}
...

Every translator is built up using a builder pattern that can be invoked using the newBuilder()-method. This becomes important later on when we want to modify an existing translator. The actual code is generated inside the builder’s forEveryTable()-method. It is a callback that will be executed for every table that is encountered in the scope of interest. In this case, the translator will only execute on one table at a time so the callback will only be executed once.

For complete sources for the GeneratedTypeAdapterTranslator-class, please go to this github page.

Step 3: Create Decorator for Modifying the Manager Interface

Generating a bunch of TypeAdapters is not enough though. We want to integrate the new code into the already existing managers. To do this, we need to define a decorator that will be applied to every generated manager after the default logic has been executed.

GeneratedManagerDecorator.java

public final class GeneratedManagerDecorator 
implements TranslatorDecorator<Table, Interface> {

    @Override
    public void apply(JavaClassTranslator<Table, Interface> translator) {
        translator.onMake((file, builder) -> {
            builder.forEveryTable(Translator.Phase.POST_MAKE, 
            (clazz, table) -> {
                clazz.add(Method.of(
                    "fromJson", 
                    translator.getSupport().entityType()
                ).add(Field.of("json", STRING)));
            });
        });
    }
}

A decorator is similar to a translator, except it only defines the changes that should be done to an existing file. Every decorator executes in a specific phase. In our case we want to execute after the default code has been generated, so we select POST_MAKE. The logic we want to add is simple. In the interface, we want an additional method fromJson(String) to be required. We don’t need to define a toJson since every Speedment manager already has that from an inherited interface.

Step 4: Create Decorator for Modifying the Manager Implementation

The manager implementation is a bit trickier to modify. We need to append it with a Gson instance as a member variable, a implementation for the new interface method we just added, an override for the toJson-method that uses Gson instead of the built-in serializer and we need to modify the manager constructor to instantiate Gson using our new TypeAdapter.

GeneratedManagerImplDecorator.java

public final class GeneratedManagerImplDecorator 
implements TranslatorDecorator<Table, Class> {

    @Override
    public void apply(JavaClassTranslator<Table, Class> translator) {
        final String entityName = translator.getSupport().entityName();
        final String typeAdapterName = "Generated" + entityName + 
            "TypeAdapter";
        final String absoluteTypeAdapterName =
            translator.getSupport().basePackageName() + ".generated." + 
            typeAdapterName;

        Final Type entityType = translator.getSupport().entityType();
        
        translator.onMake((file, builder) -> {
            builder.forEveryTable(Translator.Phase.POST_MAKE, 
            (clazz, table) -> {
                
                // Make sure GsonBuilder and the generated type adapter 
                // are imported.
                file.add(Import.of(Type.of(GsonBuilder.class)));
                file.add(Import.of(Type.of(absoluteTypeAdapterName)));
                
                // Add a Gson instance as a private member
                clazz.add(Field.of("gson", Type.of(Gson.class))
                    .private_().final_()
                );
                
                // Find the constructor and define gson in it
                clazz.getConstructors().forEach(constr -> {
                    constr.add(
                        "this.gson = new GsonBuilder()",
                        indent(".setDateFormat(\"" + DATE_FORMAT + "\")"),
                        indent(".registerTypeAdapter(" + entityName + 
                            ".class, new " + typeAdapterName + "(this))"),
                        indent(".create();")
                    );
                });
                
                // Override the toJson()-method
                clazz.add(Method.of("toJson", STRING)
                    .public_().add(OVERRIDE)
                    .add(Field.of("entity", entityType))
                    .add("return gson.toJson(entity, " + entityName + 
                         ".class);"
                    )
                );
                
                // Override the fromJson()-method
                clazz.add(Method.of("fromJson", entityType)
                    .public_().add(OVERRIDE)
                    .add(Field.of("json", STRING))
                    .add("return gson.fromJson(json, " + entityName + 
                        ".class);"
                    )
                );
            });
        });
    }
}

Step 5: Install All the New Classes into the Platform

Once we have created all the new classes, we need to create a component and a component installer that can be referenced from any project where we want to use the plugin.

GsonComponent.java

public final class GsonComponent extends AbstractComponent {

    public GsonComponent(Speedment speedment) {
        super(speedment);
    }

    @Override
    public void onResolve() {
        final CodeGenerationComponent code = 
            getSpeedment().getCodeGenerationComponent();

        code.put(Table.class, 
            GeneratedTypeAdapterTranslator.KEY, 
            GeneratedTypeAdapterTranslator::new
        );
        code.add(Table.class, 
            StandardTranslatorKey.GENERATED_MANAGER, 
            new GeneratedManagerDecorator()
        );
        code.add(Table.class,
             StandardTranslatorKey.GENERATED_MANAGER_IMPL, 
             new GeneratedManagerImplDecorator()
        );
    }

    @Override
    public Class<GsonComponent> getComponentClass() {
        return GsonComponent.class;
    }

    @Override
    public Software asSoftware() {
        return AbstractSoftware.with("Gson Plugin", "1.0", APACHE_2);
    }

    @Override
    public Component defaultCopy(Speedment speedment) {
        return new GsonComponent(speedment);
    }
}
GsonComponentInstaller.java

public final class GsonComponentInstaller 
implements ComponentConstructor<GsonComponent> {

    @Override
    public GsonComponent create(Speedment speedment) {
        return new GsonComponent(speedment);
    }
}

Usage

When we want to use our new plugin in a project, we simply add it as a dependency both in the dependency section in the pom and as a dependency under the speedment maven plugin. We then add a configuration tag to the plugin like below:


<plugin>
    <groupId>com.speedment</groupId>
    <artifactId>speedment-maven-plugin</artifactId>
    <version>${speedment.version}</version>

    <dependencies>
        <dependency>
            <groupId>com.speedment.plugins</groupId>
            <artifactId>gson</artifactId>
            <version>1.0.0</version>
        </dependency>

        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>5.1.39</version>
        </dependency>
    </dependencies>

    <configuration>
        <components>
            <component implementation="com.speedment.plugins.gson.GsonComponentInstaller" />
        </components>
    </configuration>
</plugin>

We can then regenerate our code and we should then have access to the new serialization and deserialization logic.


final String pippi = "{" + 
    "\"id\":1," +
    "\"bookId\":-8043771945249889258," +
    "\"borrowedStatus\":\"AVAILABLE\"," + 
    "\"title\":\"Pippi Långström\"," + 
    "\"authors\":\"Astrid Lindgren\"," + 
    "\"published\":\"1945-11-26\"," + 
    "\"summary\":\"A story about the world's strongest little girl.\"" + 
    "}";

books.fromJson(pippi).persist();

Summary

In this article we have created a new Speedment plugin that generated Gson TypeAdapters for every table in a database and integrates those adapters with the existing manager generation. If you want more examples on how you can use the Speedment code generator to increase your productivity, check out the GitHub page!

Until next time!

Tuesday, April 12, 2016

How to Generate Customized Java 8 Code with Plugins

One thing most programmers hate is to write boilerplate code. Endless hours are spent setting up entity classes and configuring database connections. To avoid this you can let a program like Speedment Open Source generate all this code for you. This makes it easy to get a database project up and running with minimal manual labour, but how do you maintain control of the written code when large parts of it is handed over to a machine?

Say that you have a database with a table "user" which has a column "gender", and you want that implemented as an enum in java. If you run Speedment and use it to generate code, the "gender" field will be represented as a String. The reason for this is that there isn’t any built-in mappers for converting between database ENUMs and custom java classes. This is one of those cases when you might feel that the generator is taking away control for you. Well, fear not, for since the 2.3 Hamilton release, you can get this same control by creating your own plugin for Speedment!

The Goal of this Article

In this example we have a database schema with a table called "Person". A person has an id, a name and a gender. The gender is declared as an ENUM with three possible values: "Male", "Female" and "Other". If we use the default settings in Speedment to generate this class, Speedment will consider the ENUM a String. There are some issues with this however. For an example, if you want to persist a new person into the database, there is nothing that prevents you from spelling a gender wrong and getting an exception when you do the insert. Instead, we want to define a java enum with the specified alternatives as constants. What would make the generated code more secure and easier to use.

We can achieve this using a plugin for Speedment!

Creating the Plugin Project

To do any custom modifications to the Speedment platform we will need to define a plugin. A plugin is a piece of software that can be plugged into the Speedment runtime from the pom.xml-file. The plugin resides in its own maven project and can be shared between projects.

Begin by creating a new Maven Project and declare Speedment as a dependency. You will not need the speedment-maven-plugin in this project.


<dependency>
    <groupId>com.speedment</groupId>
    <artifactId>speedment</artifactId>
    <version>${speedment.version}</version>
</dependency>

The plugin system revolves around two interfaces; Component and ComponentConstructor. A Component is a pluggable piece of software that can be executed as part of the Speedment lifecycle. Every component has a number of stages in which it is allowed to execute. These are "initialize", "load", "resolve" and "start".

The ComponentConstructor is a lightweight type that has a default constructor and a method for initializing new instances of the custom component. This is used by the maven plugin to setup the new code.

Here is how our two implementations will look:

CustomMappingComponent.java

public final class CustomMappingComponent 
extends AbstractComponent {
    
    CustomMappingComponent(Speedment speedment) {
        super(speedment);
    }

    @Override
    public void onResolve() {
        // Resolve logic here...
    }

    @Override
    public Class<CustomMappingComponent> getComponentClass() {
        return CustomMappingComponent.class;
    }

    @Override
    public Software asSoftware() {
        return AbstractSoftware.with(
            "Custom Mapping Component", 
            "1.0", 
            APACHE_2
        );
    }

    @Override
    public Component defaultCopy(Speedment speedment) {
        return new CustomMappingComponent(speedment);
    }
}
CustomMappingComponentInstaller.java

public final class CustomMappingComponentInstaller 
implements ComponentConstructor<CustomMappingComponent> {

    @Override
    public Component create(Speedment speedment) {
        return new CustomMappingComponent(speedment);
    }
}

We now have a bare-bone plugin that can be added to a Speedment project. The next step is to define the logic that maps between strings and genders. To this this, first we need the Gender enum.

Gender.java

public enum Gender {
    MALE   ("Male"), 
    FEMALE ("Female"),
    OTHER  ("Other");

    private final String databaseName;

    Gender(String databaseName) {
        this.databaseName = databaseName;
    }

    public String getDatabaseName() {
        return databaseName;
    }
}

If you store the enum values in upper-case in the database, this class could be much shorter since you could simply use the Enum.name()-method to get the database name, but this approach is better if you want flexibility in naming the constants.

Now for the final piece. We need to declare a type that implements the TypeMapper-interface in Speedment. A type mapper is really simple. It contains two methods for mapping to and from the database type as well as methods for retrieving the java class of both types.

StringToGenderMapper.java

public final class StringToGenderMapper implements TypeMapper<String, Gender> {
    
    @Override
    public Class<Gender> getJavaType() {
        return Gender.class;
    }

    @Override
    public Class<String> getDatabaseType() {
        return String.class;
    }

    @Override
    public Gender toJavaType(String value) {
        if (value == null) {
            return null;
        } else {
            return Stream.of(Gender.values())
                .filter(g -> g.getDatabaseName().equals(value))
                .findAny()
                .orElseThrow(() -> 
                    new UnsupportedOperationException(
                        "Unknown gender '" + value + "'."
                    )
                );
        }
    }

    @Override
    public String toDatabaseType(Gender value) {
        if (value == null) return null;
        else return value.getDatabaseName();
    }

    @Override
    public boolean isIdentityMapper() {
        return false;
    }
}

This new mapper also need to be installed in the Speedment platform. We can do that from the component we created earlier by modifying the onResolve()-method:

CustomMappingComponent.java

@Override
public void onResolve() {
    // Resolve logic here...
    getSpeedment().getTypeMapperComponent()
        .install(StringToGenderMapper::new);
}

Our new plugin is now done! Build the project and you are set to go!

Using the Plugin

To use a plugin in a project, you only need to modify the pom.xml-file of that project. Open up an existing Speedment project and locate the pom.xml-file. In it, you should be able to find the speedment-maven-plugin. To make your own plugin accessible for the maven plugin, you need to add it as a dependency inside the <plugin>-tag and add the ComponentInstaller to the configuration. Here is an example of how it can look:

pom.xml

<plugin>
    <groupId>com.speedment</groupId>
    <artifactId>speedment-maven-plugin</artifactId>
    <version>${speedment.version}</version>

    <dependencies>
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>${mysql.version}</version>
        </dependency>

        <!-- Our plugin project -->
        <dependency>
            <groupId>com.github.pyknic</groupId>
            <artifactId>custom-mapping-component</artifactId>
            <version>1.0.0-SNAPSHOT</version>
        </dependency>
    </dependencies>
 
    <configuration>
        <components>
            <!-- Path to the component installer -->
            <component implementation="
com.github.pyknic.salesinfo.plugin.CustomMappingComponentInstaller
            " />
        </components>
    </configuration>
</plugin>

You also need to add the project as a runtime dependency since the new Gender-enum must be accessible from the generated code.


<dependencies>
    ...
    <dependency>
        <groupId>com.github.pyknic</groupId>
        <artifactId>custom-mapping-component</artifactId>
        <version>1.0.0-SNAPSHOT</version>
    </dependency>
    ...
</dependencies>

Trying it out

That’s it! The plugin is installed! If you want to a particular column to be mapped to a Gender instead of a String, you can go into the User Interface, navigate to the particular column in the "Project Tree" and select your new Type Mapper in the dropdown list.

screenshot of the Speedment user interface

If you want to see a list of all the components and/or type mappers loaded into the platform, you can also go to "About" → "Components..." in the UI. There you should see the new component.

screenshot of the components dialog in the Speedment user interface

Summary

In this article you have learned how to create a custom plugin for Speedment that integrates a new Type Mapper from a String to a Gender enum. You have also learned how you can see which components are loaded into the platform and select which type mapper you want to use for each column.

PS: If you create some cool new mappers for your Speedment project, consider sharing them with the community in our Gitter chat!

Friday, April 8, 2016

Java 8: Bye Manual SQL, Hello Speedment!

Spire says Goodbye to the old way of writing
database applications in Java.

Most applications written in Java require some form of data storage. In small applications this is often realized using a primitive JDBC-connection that is queried using ordinary SQL. Larger systems on the other hand often use an Object Relational Mapping (ORM) frameworks to handle the database communication. There are pro’s and con’s with both of these approaches, but both tend to involve writing a lot of boilerplate code that looks more or less the same across every codebase. In this article I will showcase another approach to easy database communication using an open source project called Speedment.

What is Speedment?

Speedment is a developer tool that generates java classes from your SQL metadata. The generated code handles everything from setting up a connection to data retrieval and persistence. The system is designed to integrate perfectly with the Java 8 Stream API so that you can query your database using lambdas without a single line of SQL. The created streams are optimized in the background to reduce the network load.

Setting Up a Project

In this article I will write a small application that asks for the user’s name and age and persist it in a MySQL database. First of, we will define the database schema. Open up your MySQL console and enter the following:

CREATE DATABASE hellospeedment;
USE hellospeedment;

CREATE TABLE IF NOT EXISTS `user` (
    `id` bigint(20) NOT NULL AUTO_INCREMENT,
    `name` varchar(32) NOT NULL,
    `age` int(5) NOT NULL,
    PRIMARY KEY (`id`),
    UNIQUE KEY `name` (`name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=1;

Next we will create our java project. Fire up your favorite IDE and create a new Maven Project from Archetype. Archetypes are template projects that can be used to quickly define new maven projects. Exactly how they are used differ between different IDEs. The following information will have to be entered:

Repository https://repo1.maven.org/maven2
GroupId com.speedment
ArtifactId speedment-archetype-mysql
Version 2.3.0

Similar archetypes are available for PostgreSQL and MariaDB as well.

On NetBeans, the archetype is usually found among the default ones indexed from the Maven Central Repository. When the project is created you should have something like this:

Launching the Speedment UI

Now then the project has been created it is time to start up the Speedment User Interface. This is done by executing the speedment:gui-maven goal. In NetBeans and IntelliJ IDEA, a list of available maven goals can be found from within the IDE. In Netbeans this is found in the Navigator window (often located in the bottom-left of the screen). The project root node must be selected for the goals to appear. In IntelliJ, the goals can be found under the "Maven Projects"-tab in the far right of the screen. You might need to maximize the "Project Name", "Plugins" and "speedment-maven-plugin"-nodes to find it. In Eclipse, you don’t have a list of goals as far as I know. Instead you will have to define the goal manually. There is a tutorial for doing this on the Speedment GitHub wiki.

When the user interface starts the first time it will ask for your email address. After that you can connect to your database.

The connection dialog will only allow you to choose between databases that you can connect to using the loaded JDBC-drivers. If you for example want to use a PostgreSQL-database, you should add the PostgreSQL-driver to the <dependencies>-tag of the speedment-maven-plugin section in the pom.xml-file and the re-run the UI.

Once you have connected to the database, the main window opens. On the left side you can see a tree-view of the database. In the middle is the workspace where things like database connection, code generation and entity naming can be configured. You can select what part of the project to configure by selecting other nodes in the tree.

In this case, we will simply press the "Generate"-button in the toolbar to generate a project using the default settings. We can now close the UI and return to our IDE.

Write the Application

Now when Speedment has generated all the boilerplate code required to communicate with the HelloSpeedment database we can focus on writing the actual application. Let’s open the Main.java-file created by the maven archetype and modify the main() method.


public class Main {
    public static void main(String... params) {
        Speedment speedment = new HellospeedmentApplication()
            .withPassword("secret").build();
        Manager<User> users = speedment.managerOf(User.class);
    }
}

In Speedment, an application is defined using a builder pattern. Runtime configuration can be done using different withXXX()-methods and the platform is finialized when the build()-method is called. In this case, we use this to set the MySQL password. Speedment will never store sensitive information like your database passwords in the configuration files so you will either have to have a unprotected database or set the password at runtime.

The next thing we want to do is to listen for user input. When a user starts the program, we should greet them and then ask for their name and age. We should then persist the user information in the database.


final Scanner scn = new Scanner(System.in);

System.out.print("What is your name? ");
final String name = scn.nextLine();

System.out.print("What is your age? ");
final int age = scn.nextInt();

try {
    users.newEmptyEntity()
        .setName(name)
        .setAge(age)
        .persist();
} catch (SpeedmentException ex) {
    System.out.println("That name was already taken.");
}

If the persistence failed, a SpeedmentException is thrown. This could for example happen if a user with that name already exists since the name column in the schema is set to UNIQUE.

Reading the Persisted Data

Remember I started out by telling you how Speedment fits in nicely with the Stream API in Java 8? Let’s try it out! If we run the application above a few times we can populate the database with some users. We can then query the database using the same users manager.


System.out.println(
    users.stream()
        .filter(User.ID.lessThan(100))
        .map(User::toJson)
        .collect(joining(",\n    ", "[\n    ", "\n]"))
);
This will produce a result something like this:

[
    {"id":1,"name":"Adam","age":24},
    {"id":2,"name":"Bert","age":20},
    {"id":3,"name":"Carl","age":35},
    {"id":4,"name":"Dave","age":41},
    {"id":5,"name":"Eric","age":18}
]

Summary

This article has showcased how easy it is to write database applications using Speedment. We have created a project using a maven archetype, launched the Speedment UI as a maven goal, established a connection with a local database and generated application code. We have then managed to do both data persistence and querying without a single row of SQL!

That was all for this time.

PS: Speedment 2.3 Hamilton was just released the other day and it contains a ton of really cool features for how you can manipulate the code generator to fit your every need. Check it out!