Weekly Update: W/C 12th August, 2019

Hello everyone 👋

I’ve been on holiday this week, so this will be a fairly quick update.

Of course while I’m away I like nothing more than to work! So I Spent a bit of time updating foundthings.com, which is now running in full on the network (DNS + Content distribution).

Setup was easy and it’s pretty performant 🙂

As I mentioned last week, we’ve now added Edge-Device alongside Edge-Host in the headers returned by the network.

DEVICE_ID is the ID given to the device by Agent, the application that manages authentication and software updates. When Agent registers the device with Consul, it registers using this ID to create a Device service.

HOST_ID is the Consul ID for the host application. When Agent authenticates on the network it downloads Host and Host connects to Consul using the HOST_ID.

Why don’t we have a single ID? Because each Consul service need a unique ID. It’s important to know that if Host goes offline, the Agent application is still running.

For Telemetry it’s also important to be able to distinguish between the resources used by Device Agent, and by Host.

I see the following network headers for foundthings.com:

Edge-Device => 9e82bf98-d6f7-4559-b91c-95c578eb74a2

Edge-Gateway => 455106e7-612e-411b-9ded-38bfce8f59ee

Edge-Host => 82a3ce68-d489-4fab-afed-7046e3988752

+We also return some cache control headers:

Etag => "11ec4-wBAM3fVvut8jqaPsB2ab15I2Zng"

X-Cache => HIT

Aside from this update to the network, the network team have been working on improving the performance of the Gateway HTTP request queue, and have also been building out the transfer queue for Edge Storage. The response time for requests has been reduced by around 40% on the back of this work.

That’s from very fast, to extremely fast folks.

To achieve the performance improvement we eliminated one of the RPC (Remote Procedure Call) methods.

Up until now there were three RPC calls performed by Host:

• Check, opens an http/2 stream and waits until there is a request
• Read triggers when Check closes and fetches the request
• Write returns the response

After some benchmarking and a lot of experimentation, Check was dropped and Read replaced it as the long polling stream.

This means that rather than having to make a call to Check, then Read and then Write, Host now only calls Read and Write. Which means far fewer requests in production.

Another optimization was made to the Protobuf which are the blueprints for RPC interactions. They tell the Gateway and the Host how to convert byte data into meaningful objects, rather than the traditional approach of including keys and values in the payload.

To reduce the payload size, we attributed shorthand values to some of the stricter keys, for example:

1. Previous version of method transaction

message Request {
  string method = 1;
  bytes body = 2;
}

1. New version using enum for keys:

message Request {
  enum Method {
    GET = 0;
    PUT = 1;
    POST = 2;
    DELETE = 3;
  }
  Method method = 1;
  bytes body = 2;
}

These small changes make a huge difference, and there are plenty more to come!

The team have also been cracking through earning/payout routines, which we expect to be releasing at the end of the month. This means we’ll be moving from monthly to much more regular payouts very soon.

And work on CLI for self onboarding has been ongoing.

We released CDN 3.6.1 with a fix for a bug raised by the Edge team. You can see this here: https://github.com/dadi/cdn/releases/tag/v3.6.1

(Note that we’ve not migrated the web services to our new Git org yet. This will happen in the coming weeks.)

And we added new functionality to the API validator module in preparation for some changes to API authentication. The new functionality will also be used in our SaaS platform.

The team also made a start and significant progress on a new authentication method for API, called access keys. This will solve the requirement from the Edge network to have bearer tokens that don’t expire, and will also allow us to persist Publish sessions for as long as we need. It will also be a cool way for people to integrate third-party apps with their Edit installations, with the ability to view, edit and revoke any issued keys.

They also made some structural changes to Publish routing, in preparation for the SaaS product of what were our web services. This adds the property name to the URL if more than one property is defined. it also stops reference selection from pushing a new URL to the history, as discussed previously, which solves a bunch of smaller issues.

We have some new business through an existing customer, which we expect to be kicking off in the next few weeks.

And we spent a bunch of time with the PPA, working up future workstreams and assisting them with their 2020 roadmaps.

Tiipr is also now ready to submit to the Apple store, a little later than planned on the back of some last minute changes, but it’s great to have it so close to the line. I’ll let you know when it moves to live.

And that’s about it for this week!

Enjoy your weekends.