We are just selling a few products to our
customers, it can’t be that hard, can it?
Background
We were a number of months into designing and building a micro service
based platform to replace a number of monolithic B2B applications (each had
their own business organisations, products and services), when it was
discovered that no one in the business had considered how the
products might be sold and billed from a single place, they just assumed we
would do it the same as we had always done.
This seemed like a simple issue, cross selling products to our customers is
what we had always done, but in a very manual way.
We were planning on combining each business into a single data-driven
platform where we would link a product to a customer and invoice them for the
number of users that use the product.
Investigations
However after doing some research around the different business heads and
sales teams, it turned out that there wasn’t a lot of common ground to start
from.
In reality:
- Customers were being charged different prices for
the same products.
- Prices were time boxed or volume controlled.
- Some sales were based on sliding scales.
- Contracts sometimes didn’t mention pricing.
- Price changes were difficult due to no end dates in
contracts.
- Forecast were impossible as we sometimes billed in
arrears.
- The number of drivers changed on a nearly daily
basis.
There were also a few additional wrinkles that I found out about that
sat outside of what we knew:
- Pricing, some customers were being charged in Euros
and a few in other currencies.
- Content, was predominantly in English, but we had
one product that supported 5 foreign languages.
- Versioning, some content was updated constantly and
only some customers would be eligible to see it.
- Users being added at any time should get the latest
content their company has access to.
- Users being added or removed might trigger a charge
or refund depending on the contract.
This new information left us a with a lot of new technical challenges
that had not been picked up in the investigation stages.
What we did as a business
Our generic business model was simple, our customers have drivers and
their drivers have risk associated with them. We mapped and showed companies
where that risk was and how to mitigate it.
Based on a bespoke customer algorithm they might have a range of products added
to their user account depending on what they do:
- a regular licence check
- elearning courses
- online assessments
- on road training
- telematics installed in their vehicle
We charged the customers for each product their drivers completed.
The billing was very ad-hoc and was purely manual
Our monthly bill run was taking weeks of time and no one really had any solid
idea of where the money was coming from or when it might arrive.
Automation
I really wanted to limit the manual work needed to run the new platform
and decided to introduce a workflow process that would cover the assignment and
billing for products to drivers.
In later iterations the workflow became intrinsic and central to how the entire
platform functioned as we linked any repeatable task as a step in a workflow
somewhere.
I tasked one of the senior engineers to find some options and after some demos
and discussions we ended up using an off the shelf c# open source platform
called elsa: https://v2.elsaworkflows.io/
It had hooks in and out of our systems for triggers and had its own admin UI
tooling, this saved us a lot of development time.
Mobile Contracts
I decided that the best solution for "how will we sell and bill
products to our customers" was to build it like a mobile phone
bundle/contract.
Everyone understands how these work and I had done this before in the telco
world.
I was hopeful it would allow us to sell any number of items bundled together to
any number of customers at any price.
I set out the technical business requirements for the Offer Package.
It would be a structural wrapper around our content, products, prices,
policies, templates and any other data linked to
what we would “sell” to a customer.
The technical design sessions did become busy over time as the team and I
were modelling against ideas that were evolving.
To help we developed proof of concepts and mock-ups as we went to make sure
we were on the right track.
We were able to hide the complexity and repetitive work inside the workflow
engine.
Data Structure
Working closely with the my data team, we worked out a base data
structure that would work within a relational database and fit into the overall
tech roadmap we had in place.
Our biggest goal was that we didn’t want any data duplication anywhere, as it
was a huge issue in the current monoliths.
The Offer package structure ended up a:
Each Offer Package had many Products and each Product had versioned content
in different languages with a default price for each available currency.
We added more availability dates after modelling this out against what the
sales teams would need to sell we realised we had to be able to control when
the Offer package would be available.
These were then added to Products, Prices, Pricelists and in the end every
entity in the database had them, these became some of our core fields that we
used for all sorts of reporting and automation checks.
Building the validation for this to work in code was a headache for the
developers and a lot of unit tests were created to cover a very large amount of
use-cases.
The validation had to consider cases like: An offer package could be assigned
to a customer at any time, but it would only be visible to a workflow to be
assigned to a user after its available date was passed. However, if the
products or prices inside the Offer Package were not available then the
workflow would error and the users would not be assigned the right products and
no charges would be raised.
As the complexity grew so did the technical workshops and whiteboard
sessions to make sure we were not missing anything.
Educating the developers on what the architecture looked like was
imperative, proof of concept models and UI were being drafted/built on a
sprintly basis.
Once the team had agreed on a model that felt we were on the right track
and user stories were created
Demos were done to the stakeholders at the beginning of each testing
release.
Any ideas from these stored for a later development phase to get the core
done as quickly as possible.
Versioning
The next challenge was how to store the versioned and multi-lingual
product content, each type of content had a different data structure; text,
videos, images, files, full assessments, 3rd party data,
external API calls.
A standard relational database model really wasn’t the best option for storing
this mixed data set very efficiently, but we couldn’t move from this.
One piece of content like a word document or PDF
for example, would need to be made available to a driver in their own language,
this content would always need to be up to date when they read it the first
time. However, if it was something they signed and agreed to, they should never
see the updated version of the same document as it would not be valid.
But if its not something that’s been agreed, then maybe they should see the
latest version every time, even if its changed.
This gets more complicated when managing elearning content, if you have started
the course and it gets updated after you started, should you get the new
version or continue on the old?
All of these sorts of use cases had to be managed via flags in the data,
every workflow decision needed to be able to be answered via a data query.
Complex Content Storage
Each product was available in multiple languages, each language could have
multiple versions.
Some versions were ready for the future, some had been available in the
past but only one was ever available ‘now’.
Now being relative to when the user wanted to view it.
My initial solution was to use a SQL field to store serialised and
structured model JSON for the content and standard fields for the generic data
name, dates, version ID, etc.
I had done this sort of thing before for health data models in XML in the past
and although the developers weren’t overly comfortable with storing the data
like this, once we ran a proof of concept through they saw the benefits.
Each product type would have its own model in c# that would allow the system to
store and read the JSON instantly based on a lookup type stored in the data.
It also enabled us to update the structure of the models by either migrating
the JSON itself or simply creating a new model type to save and read the new
data type.
Unfortunately, when we started volume testing this as part of a proof of
concept, it didn’t work exactly as we expected.
For smaller pieces of content, like a URL to a file etc, both reads and writes
were fast. The serialised JSON would be ingested by the correct class and it
worked really well.
However when working with complex data like assessments/surveys, large amounts
of HTML this read time became too long.
We had to split the versions of each content out so these also had available
from/to dates as well.
This wasn’t a huge amount of extra work, just an extra layer of data and it now
left us with a much cleaner data structure, we still used the JSON concept, but
now we could find the just content we needed ‘now’ much faster than opening a
large piece of JSON, parsing it and finding the right ‘now’ content.
Cache time
After the table adjustment we ran some more efficiency/benchmarking
tests on the new data layers and found that storing the complete UI view in the
table as serialised JSON was extremely fast.
Adding a caching layer on top of the database behind the API gave us even more
flexibility.
If we needed [/product/123/content/568] from the API it returned the JSON
object directly from the cache if it was available, or from the database if
not, while storing the result in the cache for next time.
The UI could render the object without any computation time.
A workflow service monitored the cache and clearing out anything that was older
than a set time frame.
We added a separate mechanism that reacted to changed data in the database and removed dirty
cache entries.
Take away
Looking back there were times when I am sure my team thought I was losing the
plot with some of the decisions that were being made, however the end result
was a slick system with the smallest amount of human admin as necessary to keep
things running.
There are parts of this I have not mentioned, that were just as big a challenge
as offer packages, like billing cross currencies, the definition of ‘now’ for 2
users in different timezones etc.
The Offer Packages and Workflow were a turning point in the evolution of how
software was being developed.
Abstracting business ideas away into generic data structures with automation
and solid technology behind them leaves everyone in a place where any
mis-understandings or changes can be made without re-developing code and moves
closer to my personal goal of configuration
Don’t get me wrong, it wasn’t perfect, but it was a lot cleaner and more
importantly leaner than the systems we were leaving behind.
Complexity
The underlying data over time became very simple and was everything
ended up being based on a generic table structure.
The models we mapped out in the database were mapped directly to real world
values, giving the data meaning and context, end to end across the platform and
its services.
When we reviewed the system when it was up and running with real data in
it, we could see that things made sense to both developers and business
stakeholders alike.
Any changes to the system now had to have a reason for being made, new
fields have to have a value for being added.
We were finally only storing the data we really needed.
What I didn’t get time to do
This wasn’t a true micro service when I left because although the
services themselves (Offer Package, Prices, Billing, etc) were independent
pieces of functionality running on different azure pieces, there was still only
a single database.
Workflow was the only true microservice, although we shared c# code with it
so it could understand our logic.
The SQL database schema was set out to allow splitting or sharding, but we
were waiting until we had the data volumes in before we started doing this as
the cost was just too high to do on a small customer base.
Future thoughts
Abstract data-driven systems are powerful, but they are also very complicated
to put together.
We could have built a very specific piece of software, this would give us a
mirror of what we were replacing, it was a bold move to do and took longer than
we wanted it to, but the end result is a data driven platform, that with a few changes
of config could be used for a completely different business, without
redevelopment.
