As the principal responsible for the design of middleware software at Zebrium, I’m writing to share some of the choices we made and how they have held up. Middleware in this context means the business-logic that sits between persistent storage and a web-based user interface.
Here are some of the Zebrium Middleware design goals:
Early on we made several selections of key technology that has proved mostly successful:
Having worked with both no-SQL and SQL databases in the past it is clear that the promise of no-SQL scaling and reliability has not born out. The vast majority of database operations are read-only and using SQL allows us to push the cache of data needed to make complex correlations down to the database software that is optimized for the task. The reliability of SQL has also been addressed, either in clustered server implementations or hosted services with replication.
Coming from C/C++, I have been repeatedly impressed with Go. The execution speed allows me to introduce complex data transformations without significant overhead, while the speed of compiling renders the test cycle close to interactive. And the scope and variety of open-source support libraries is impressive; more on that later.
Building the middleware server into a container allows us to test it in simple platform independent docker-compose configurations as well full-stack Kubernetes staging and production. For example, our UI engineers can deploy the container on their MacBooks using Docker.
We used Go templates effectively for two areas of the middleware:
I was looking for an API framework that allowed me to reduce the amount of repeated specifications. RAML and go-raml (github.com/Jumpscale/go-raml) fit the bill. Though not as mature as Swagger/OAS, RAML provides more expressions on types that can be shared between APIs. And, because go-raml is written in Go and open-source, I was able to fix bugs and write my own extensions. Here is a sample RAML type:
Vote:
properties:
id:
type: string
generated: true
createTime:
type: datetime
required: false
createUserId:
type: string
generated: true
modifyTime:
type: datetime
required: false
modifyUserId:
type: string
generated: true
name:
type: string
pattern: "^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$"
db: "rid"
thumb:
enum: [ up, down ]
desiredPriority:
enum: [ high, med, none ]
The framework will validate inputs and translate JSON camel-case tags into SQL snake-case, with the “db” override above for name. With this type, I can describe various HTTP methods that act on it. The following is an HTTP PUT which creates or updates a vote by name. The responses document the possible return values.
/vote:
dataSource: "odbc/t_votes"
put:
securedBy: [ Token ]
description: Create or update a vote
body:
application/json:
type: Vote
responses:
200:
body:
application/json:
type: Vote
201:
body:
application/json:
type: Vote
400:
body:
application/json:
type: Meta
500:
body:
application/json:
type: Meta
The above text is translated by go-raml and our templates into Go code that serves the request. We have templates for the following HTTP operations:
Method | URL Pattern | Description |
---|---|---|
GET | {object} | Get all objects |
GET | {object}/{id} | Get object by ID |
POST | {object} | Create object |
PUT | {object} | Create/update object by name |
PUT | {object}/{id} | Create/update object by ID |
PATCH | {object}/{id} | Update fields in object by ID |
DELETE | {object}/{id} | Delete object by ID |
POST | {object}/{action} | Custom request/response API |
We also have callbacks embedded into the generated code to deal with pre- and post-processing, allowing us to adapt this model to a wide variety of object types. About half of our APIs are handled with the above CRUD operations and callbacks, while the other half are handled with custom actions, the last HTTP pattern above.
The other use of Go templates in our framework involves adding parameters to complex queries. The following is an example an SQL template:
The formatting of a Go string array into a SQL list is accomplished with a template function (https://golang.org/pkg/text/template/#Template.Funcs):
"csv": func(in []string) string {
out := ""
for _, v := range in {
if out != "" {
out += ","
}
out += "'" + v + "'"
}
return out
},
This design provides a separation of concerns between the Go code that provides parameters to the often complex SQL that extracts the data.
To review our goals from the top:
But most of all, this has helped the company I work for (Zebrium) to build and deploy versions of software rapidly and with high quality.