Designing a RESTful API Framework

As the principal responsible for the design of middleware software at Zebrium, I’m writing to share some of the choices we made and how they have held up. Middleware in this context means the business-logic that sits between persistent storage and a web-based user interface.

Goals

Here are some of the Zebrium Middleware design goals:

• DRY (Don’t Repeat Yourself) - Limit the amount of boiler-plate code that is copied from one API to another
• Provide quick turn around for testing and deployment
• A language that has broad support and is deployed widely in similar applications

Technologies

Early on we made several selections of key technology that has proved mostly successful:

• SQL Databases, Postgres for user/session management, Vertica for analytics
• Go for middleware software
• Containers for deployment

SQL Databases

Having worked with both no-SQL and SQL databases in the past it is clear that the promise of no-SQL scaling and reliability has not born out. The vast majority of database operations are read-only and using SQL allows us to push the cache of data needed to make complex correlations down to the database software that is optimized for the task. The reliability of SQL has also been addressed, either in clustered server implementations or hosted services with replication.

Go

Coming from C/C++, I have been repeatedly impressed with Go. The execution speed allows me to introduce complex data transformations without significant overhead, while the speed of compiling renders the test cycle close to interactive. And the scope and variety of open-source support libraries is impressive; more on that later.

Containers

Building the middleware server into a container allows us to test it in simple platform independent docker-compose configurations as well full-stack Kubernetes staging and production. For example, our UI engineers can deploy the container on their MacBooks using Docker.

Templates

We used Go templates effectively for two areas of the middleware:

• APIs to perform CRUD (create, read, update, delete) on generic database tables
• Interpolating arguments into complex SQL queries

CRUD with go-raml

I was looking for an API framework that allowed me to reduce the amount of repeated specifications. RAML and go-raml (github.com/Jumpscale/go-raml) fit the bill. Though not as mature as Swagger/OAS, RAML provides more expressions on types that can be shared between APIs. And, because go-raml is written in Go and open-source, I was able to fix bugs and write my own extensions. Here is a sample RAML type:

  Vote:
    properties:
      id:
        type: string
        generated: true
      createTime:
        type: datetime
        required: false
      createUserId:
        type: string
        generated: true
      modifyTime:
        type: datetime
        required: false
      modifyUserId:
        type: string
        generated: true
      name:
        type: string
        pattern: "^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$"
        db: "rid"
      thumb:
        enum: [ up, down ]
      desiredPriority:
        enum: [ high, med, none ]

The framework will validate inputs and translate JSON camel-case tags into SQL snake-case, with the “db” override above for name. With this type, I can describe various HTTP methods that act on it. The following is an HTTP PUT which creates or updates a vote by name. The responses document the possible return values.

/vote:
  dataSource: "odbc/t_votes"
  put:
    securedBy: [ Token ]
    description: Create or update a vote
    body:
      application/json:
        type: Vote
    responses:
      200:
        body:
          application/json:
            type: Vote
      201:
        body:
          application/json:
            type: Vote
      400:
        body:
          application/json:
            type: Meta
      500:
        body:
          application/json:
            type: Meta

The above text is translated by go-raml and our templates into Go code that serves the request. We have templates for the following HTTP operations:

We also have callbacks embedded into the generated code to deal with pre- and post-processing, allowing us to adapt this model to a wide variety of object types. About half of our APIs are handled with the above CRUD operations and callbacks, while the other half are handled with custom actions, the last HTTP pattern above.

SQL Templates

The other use of Go templates in our framework involves adding parameters to complex queries. The following is an example an SQL template:


The formatting of a Go string array into a SQL list is accomplished with a template function (https://golang.org/pkg/text/template/#Template.Funcs):

        "csv": func(in []string) string {
            out := ""
            for _, v := range in {
                if out != "" {
                    out += ","
                }
                out += "'" + v + "'"
            }
            return out
        },

This design provides a separation of concerns between the Go code that provides parameters to the often complex SQL that extracts the data.

Conclusions

To review our goals from the top:

• RAML and go-raml, together with pre- and post-processing filter functions generate about half of our API code
• Container deployment and quick compile times provide fast and flexible dev/test cycles
• Go and it’s collection of open-source libraries cover relevant functionality and enjoy broad support

But most of all, this has helped the company I work for (Zebrium) to build and deploy versions of software rapidly and with high quality.