DADI API 3.1

As a side note, let me talk about something that developers love: quick wins. Sometimes, you’re able to deliver a shiny new feature or a huge enhancement simply by making small changes to a codebase. This release is probably the exact opposite of that, as I worked on a huge piece that drove me absolutely insane to produce a result that, when not looked at closely, may look thin. Let me show you why it’s not.

🔗Field types

Creating a collection involves specifying the fields and their types. API supports the usual primitive types, like String, Number or Boolean, as well as some compound ones like Mixed, Object or Reference. Each of these fields requires a different treatment from API when it’s processing a query or a document, as each of them will have their own validation, sanitization and formatting routines.

Previously, all this logic lived in the core of the application, in large modules holding all the rules and particularities of every field type. These modules would process documents like an assembly line, spitting out on the other side a document that is ready to be saved to the database or delivered to the consumer.

The problem with this approach is that it became difficult to customize the behavior of a particular field type at a project level, and even more difficult to create a completely new field type. The bulk of the work around the 3.1 release was (the very unsexy task of) breaking that logic out into several field modules, one for each type, compartmentalizing all the validation, sanitization and formatting involved with each type.

In the immediate term, this makes the application easier to maintain and reason about, but what’s really great about it is that it paves the way for the introduction of user-defined fields in a future release. The idea is that API will still ship with a set of pre-defined field types, but allow developers to extend them or build with their own custom implementations as well.

For maximum flexibility and customization, this will take place in the form of modular plug-ins, inline with the approach we’ve taken on data connectors, custom endpoints and collection hooks.

🔗Reference fields

Perhaps the most powerful field implemented by API is called Reference. It allows a collection to reference one or multiple documents from another collection, allowing the system to represent complex one-to-one, one-to-many and many-to-many relationships.

For example, if we were to create a collection of movies, we could represent the cast as a Reference field, indicating the actors collection as the source of the referenced documents.

collection.movies.json

  "fields": {
    "title": {
      "type": "String"
    },
    "cast": {
      "type": "Reference",
      "settings": {
        "collection": "actors"
      }
    }
  }
}

We can associate one or more actors with a movie by setting their IDs in the cast field of a document in the movies collection (acting as a foreign key in relational database terms).

POST /v1/testdb/movies

  "title": "Casablanca",
  "cast": [
    "5ac16b70bd0d9b7724b24d3e",
    "5ac16b70bd0d9b7724b24d3f"
  ]
}

GET /v1/testdb/books/5ac16a2c16ce7b7710dbb799?compose=true

  "title": "Casablanca",
  "cast": [
    {
      "_id": "5ac16b70bd0d9b7724b24d3e",
      "name": "Humphrey Bogart"
    },
    {
      "_id": "5ac16b70bd0d9b7724b24d3f",
      "name": "Ingrid Bergman"
    }
  ]
}

🔗Multi-collection references

API 3.1 expands on this concept and introduces the possibility for a single field to reference documents from multiple collections. We could introduce a crew field which references documents from multiple collections such as writers, directors and producers.

PUT /v1/testdb/movies/5ac16a2c16ce7b7710dbb799

  "crew": [
    {
      "_collection": "writers",
      "_data": "5ac16b70bd0d9b7724b24a41"
    },
    {
      "_collection": "directors",
      "_data": "5ac16b70bd0d9b7724b24a42"
    },
    {
      "_collection": "producers",
      "_data": "5ac16b70bd0d9b7724b24a43"
    }
  ]
}

GET /v1/testdb/movies/5ac16a2c16ce7b7710dbb799?compose=true

  "title": "Casablanca",
  "cast": [
    {
      "_id": "5ac16b70bd0d9b7724b24d3e",
      "name": "Humphrey Bogart"
    },
    {
      "_id": "5ac16b70bd0d9b7724b24d3f",
      "name": "Ingrid Bergman"
    }
  ],
  "crew": [
    {
      "_id": "5ac16b70bd0d9b7724b24a41",
      "name": "Julius J. Epstein"
    },
    {
      "_id": "5ac16b70bd0d9b7724b24a42",
      "name": "Michael Curtiz"
    },
    {
      "_id": "5ac16b70bd0d9b7724b24a43",
      "name": "Hal B. Wallis"
    }
  ]
}

We’ve also introduced a new syntax that allows the value of multi-collection reference fields to be part of query filters. For example, we could ask for movies where the crew field includes a director with curtiz in the name:

GET /v1/testdb/movies?filter={“crew.name@directors”:{“$regex”: “curtiz”}}

🔗Fine-grained composition level

Referenced documents aren’t composed by default, as it can be an expensive operation, so the reference IDs are shown instead. This can be changed by setting compose=true in the URL. However, this will compose the first level of references only, unless all the collections in the reference chain have compose explicitly set to true in the collection schema. This sounds confusing, but bear with me.

Imagine that the directors collection above had a Reference field of its own, pointing to a collection holding the awards won by each director. If a document is requested in the movies collection with compose=true, the directors in the crew field would be composed to include the content of the director document, but the awards field would be left as a series of IDs, because that’s an extra level of references.

In API 3.1, the compose URL parameter now accepts numeric values which specify exactly how many levels of references will be resolved. In the example above, compose=2 would resolve directors and their awards. You can ask for all references to be infinitely composed with compose=all (that sounds scary, proceed with caution).

🔗New Model API

If you’re an advanced DADI API user (we ❤️you) who ventured into using collection hooks or custom endpoints, you probably found yourself having to access the Model class directly in order to find, insert, update or delete documents. We’re rolling out a major facelift to the ModelAPI, introducing support for Promises and named parameters.

Before:

  [{name: 'Doc 1'}, {name: 'Doc 2'}],
  {_apiVersion: '1.0'},
  function (err, result) {
    if (err) return err

    mod.find({name: 'Doc 1'}, {}, function (err, result) {
      if (err) return err

      console.log(result)
    })
  },
  req
)

After:

  documents: [{name: 'Doc 1'}, {name: 'Doc 2'}],
  internals: {_apiVersion: '1.0'},
  req
}).then(response => {
  return model.find({
    query: {name: 'Doc 1'}
  })
}).then(response => {
  console.log(response)
})

For backward-compatibility, we’re retaining support for the deprecated syntax, but we encourage all developers to update their code as the legacy version will be removed sometime in the future (don’t worry, we’ll give you plenty of notice).

We’re also working on some exciting changes to API wrapper to make it easier to interact with API, regardless of whether you’re accessing it remotely or from within your API project. Watch this space!

🔗Parting thoughts

This is a release with important structural improvements that will pave the way for exciting new things for API. It was a huge refactoring that was only possible thanks to the 616 unit tests in the codebase, giving us a high level of confidence in the code we’re shipping.

We’ll publish a release candidate for version 3.1 very soon, which you will be able try by installing API with npm install @dadi/api@beta.