Building the OpenTaxi App — Part 2: Connecting Riders to Drivers

10 min readJun 22, 2018

This series of articles, will take us through a prototyping process: designing, architecting and coding; including wireframes, deployment templates, source code (Python and JavaScript), and references to tools/documentation we used.
Our focus is not on optimising user experience, but rather on how efficiently we can build something end-to-end. We’re going to try to be innovative, so we are likely to make some mistakes along the way.

In Part 1, we covered the concept with quick wireframes and identified some architecture considerations that are important to us.

We’ll build functionality incrementally, identifying needs and making decisions as we go along, and we’ll start where we have the most uncertainty: the back-end service architecture.

Functionality

For every prospective rider there could be multiple available drivers. Similarly, for every driver there could be multiple riders searching at the same time. We will need to have a process to select the first/best match, but we needn’t fuss about the details of that process yet. We just need to ensure the list of candidate drivers can get to a single place where selection can happen. I’m happy to assume that to be the rider’s device.

The flow would be something like this:

But how will the availability messages from drivers reach only the relevant riders? In a typical location-based directory, we’d store location data in a database and query it to let the database perform some form of optimised sort based on distance calculation on a geographic index. Our locations will be changing all the time and updating an index all the time can’t be cheap. We want this to be cheap, so let’s try avoiding using a database for this part.

And how do we connect them more efficiently? We’re going to go with asynchronous Pub/Sub messaging over MQTT, using AWS IoT Core. It allows us to publish messages that fan-out to multiple subscribers and supports wildcard filtering on topic name. If we can map topics to regions we can achieve targeted delivery. Hopefully we won’t run into a limit on the number of topic names; I couldn’t find any mention of such limits in the documentation. We need to work out how to name our topics based on location.

Our first code

A while ago I stumbled upon the rather clever Google’s S2 library, a port of it that’s part of the golang/geo library. In short, it is way of slicing up the sphere that is planet Earth into a hierarchy of regions called “cells” to efficiently do things like area calculation, measuring distances and geographic querying. It is used by many, even by MongoDB, and I found an excellent article by Sidewalk Labs, with an useful online tool to help us explore the functionality.

The S2 hierarchy has 30 levels of cells ranging in shape and size. At each level cells contain smaller cells, like Russian dolls. To match contained cells, we’ll always need to subscribe at the same or broader level than where we are publishing. As users move around, topics would change, and we can limit the churn of topics by not using the smallest cells. We also said that we don’t want to reveal the rider’s exact location too early. The S2 website has some cell size statistics can help us decide which levels to use.

We are unlikely to ever be searching at a range much wider than a small multiple of 9km x 9km, which means our broadest level can be level 10. I’d be happy to publish on level 16, about 130m x 130m.

Are we ready to code? Well, yes. The sooner we can see stuff working the better. Let’s just confirm how topic naming could work…

Hitting the terminal, we can run our code to help us see that when riders are publishing at a topic named 487605/487604d/487604cf/487604ce3, the drivers could be subscribing to 487605/487604d/*wildcard* to receive the messages.

$ python
Python 3.6.3 (v3.6.3:2c5fed86e0, Oct  3 2017, 00:32:08)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin>>> import geo
>>> broadcast_cell_id = geo.get_cellid_at_level(51.507351, -0.127758, 16)
>>> geo.get_hierarchy_as_tokens(broadcast_cell_id)
['487605', '487604d', '487604cf', '487604ce3']
>>>

Defining the Service Infrastructure

Before we can create our infrastructure, we should elaborate on how we expect data to flow through the platform.

1. Rider’s device publishes their location and availability status.

2. If they are marked ‘searching’, we’ll asynchronously trigger a function to calculate the topics they should subscribe on, and let them know via their reply topic.

3. The rider’s device will subscribe to the recommended topics.

4. The driver’s device publishes their location, preferences and searching status.

5. If they are marked as ‘available’, we’ll trigger a function that calculates topics to publish on, and publishes an availability message on these.

How will we trigger the functions? IoT Topic Rules allow us to trigger actions base on a SQL filter on the stream of messages. One such action type is a Lambda function trigger. We will also need to think about access control. IoT Core with IAM policies allows us to specify who can publish/subscribe where:

Riders topic: ot/riders/broadcast
Subscribe: Only our topic rules
Publish: Any Rider
Reply topic: ot/replies/{client_id}
Subscribe: Only the client connected with this client_id
Publish: Anyone
Drivers topic: ot/drivers/broadcast
Subscribe: Only our topic rules
Publish: Any Driver
Rider locations topics: ot/drivers/available/{level_10_cellid}/{level_12_cellid}/…
Subscribe: All riders
Publish: Only our Lambda function

We can also cover off a latent part of this functionality: the publishing of riders’ locations to drivers (so they can see activity on the map, for example.)

Drivers looking where about the riders are

We’ll add this topic:

Driver locations topics: ot/drivers/available/{level_10_cellid}/{level_12_cellid}/…
Subscribe: All drivers
Publish: Only our Lambda function

Coding the Infrastructure

We want infrastructure as code. I like the Serverless framework for helping me configure and deploy Lambdas, but we don’t need the multi-cloud support and want to reduce the number of tools to learn. We can stick to AWS CloudFormation with AWS SAM (Serverless Application Model) extensions for this project. The make utility is already installed on most *nix-based systems, so we can configure a makefile to help us with our development workflow.

$ make clean    # if previously built
$ make build    # after code changes
$ make package  # after code or CloudFormation changes
$ make deploy

Reading through the documentation, I see we will likely need these:

AWS::IoT::TopicRule for invoking our Lambdas.
AWS::Serverless::Function for our Lambdas.

Is that all? Well, the bulk of the template is going to be security related. We’ll need these:

AWS::IAM::Role for drivers, riders, and our functions (they have an execution role.)
AWS::IAM::Policy for our common user and function permissions.

Take a look at our template.yaml.

Security

In conjunction with IAM, we have the option of using AWS Cognito to serve as a authentication and user management platform. I created a test user pool with the AWS console, and found that implementing it for our use case is likely to increase complexity. That said, not using it means we lose out on the benefits of a fully managed service. We will have to create our own authentication service, which means more custom source code and more chances for something to go wrong.

I’m not going to fully commit to a custom solution yet. We will build a first version within a module that we use when during development. We can create rider and driver roles and restrict the permissions our MQTT client by generating temporary credentials with the STS AssumeRole API.

We need to restrict access to some topics to specific users. When our script connects, we can specify a client id that for the connection. We can tie the reply topic (see above) to the connected client id using policy variables, e.g. arn:aws:iot:*:*:topic/ot/replies/${iot:ClientId} as resource name.

We also want to be sure that the connected client id belongs to the STS session. We can achieve this by passing a modified, more restrictive, policy document. To ensure only a specified client id can connect with our temporary credentials, we need to modify the iot:Connect policy statement to specify the client id as the resource.

And voilà! We’ll now have a topic that anyone can publish to, but only a specific, verified user can subscribe to.

However, when a message is received by a client, there’s no way to tell who the message is from, apart from the body of the message. Unless we can control the body of the message, any client could pretend to be someone else. With IoT Rules SQL we can either inject the clientid() or we can verify that an identification already within the body of the message is true. If the reply topic can be considered identification of the sender, then we could verify that.

Completed, the flows could look something like this:
(Note: I’ve included the sending of the SMS auth code using SNS)

Coding our Service Functions

We’ll develop the service functions with Python, instead of (my common first choice) JavaScript. I haven’t used Python that much, but we don’t have a particularly nice S2 library in NodeJS. I’ve added some commands to our makefile to help me remember how to create a virtualenv, install dependencies and bundle non-development dependencies with the code we’ll be deploying to AWS Lambda. Here’s what we’ll use:

$ make activate
$ source ./.venv/bin/activate
$ make install$ make test  # runs our tests once
$ make watch # runs our tests on source file changes

In the absence of an all-singing all-dancing local development environment (traditionally a partial replica of our production environment,) I find a test runner to be indispensable. I’m still getting used to Python and its tools, so I expect to be getting better at mocking and stubbing the AWS SDKs. In an environment with minimal state and where the behaviour of your APIs are so consistent, it seems pretty straightforward.

Well-defined interfaces are particularly necessary with distributed architectures. We should do some validation of incoming messages to avoid functions crashing upon hitting a missing or mistyped values. Uncaught crashes will mean our function gets retried — in the case of stream-based event sources for the life of the source message, which could be days. We’ll use JSONSchema to validate our incoming messages, failing with a warning instead of a crashing error. It is predictable and and JSON definitions work in most programming languages.

To test in ‘production’, we don’t have a proper client app yet, so I’ve created some command-line example scripts simulating a rider and driver. When we run them in separate terminal windows, we can see messages coming through as expected. We made the driver go along a path, and we can see them meeting with the range of the rider a few broadcasts in.

(.venv) ➜  open-taxi git:(master) ✗ python examples/rider.py
subscribing to ot/replies/example-rider
published to ot/riders/broadcast
subscribing to 19 topics and unsubscribing from 0
published to ot/riders/broadcast
published to ot/riders/broadcast
published to ot/riders/broadcast
ot/replies/example-driver [51.511115, -0.096459] 0 secs ago
published to ot/riders/broadcast
ot/replies/example-driver [51.510974, -0.107735] 0 secs ago
published to ot/riders/broadcast
ot/replies/example-driver [51.509745, -0.118489] 0 secs ago

And for the driver:

(.venv) ➜  open-taxi git:(master) ✗ python examples/driver.py
subscribing to ot/replies/example-driver
published [51.509541, -0.076598] to ot/drivers/broadcast
subscribing to 20 topics and unsubscribing from 0
published [51.509804, -0.088039] to ot/drivers/broadcast
subscribing to 10 topics and unsubscribing from 10
published [51.511115, -0.096459] to ot/drivers/broadcast
subscribing to 8 topics and unsubscribing from 8
published [51.510974, -0.107735] to ot/drivers/broadcast
subscribing to 15 topics and unsubscribing from 18
ot/replies/example-rider ?,? 0 secs ago
published [51.509745, -0.118489] to ot/drivers/broadcast
subscribing to 13 topics and unsubscribing from 12
ot/replies/example-rider ?,? 0 secs ago
published [51.507234, -0.126293] to ot/drivers/broadcast
subscribing to 16 topics and unsubscribing from 14
ot/replies/example-rider ?,? 0 secs ago

Next Up

We now know quite a bit more about how all of this will work. We should tackle these components:

Part 3 — Authentication: Will we expose a function through API Gateway, sending SMS auth codes using SNS, or will we go full OAuth 2.0?

After that…

Client App: We have some wireframes to code up into something that looks respectable. I’m thinking React Native would be fine.
Client Library: If we’re using React Native, it would be great to create a JavaScript library that will handle the interface with our API and the AWS IoT MQTT SDK. We may even be able to reuse it to create a web-only interface.

You can check out the source code at: https://github.com/jupiter/open-taxi.