Co-op's own online grocery website, shop.coop.co.uk, started life as a proof of concept in a handful of stores in 2019. A year later, the Covid pandemic completely changed the online grocery market, and this proof of concept had expanded to cover the majority of the UK population and was served from ~1500 of Co-op's stores.
Increasing Scale and Table Stakes
With the unexpected expansion of the shop.coop.co.uk service we suffered our fair share of growing pains. While not unexpected with such a rapid expansion, it was an early indication that the platform might not meet the longer-term ambitions of Co-op's Food business.
There were some features and capabilities that our customers and colleagues expected from a modern eCommerce website, that we weren't able to effectively deliver.
As we emerged from the Covid pandemic and the growth slowed we took the opportunity to assess the headless eCommerce platform landscape, specifically we were looking for a more composable architecture. After much discussion we decided on Salesforce Commerce Cloud (and associated Salesforce Order Management system).
If you were looking for a post about why we chose Salesforce, this wasn't it. It was about how we turned it on without anyone noticing.
Working out a plan
Now our engineering teams could get a really good look at the challenge ahead.
We had an existing eCommerce Platform that sold tens of thousands of unique SKUs, from 1500 stores, often within 60 minutes to hundreds of thousands of customers, and we were looking to swap it for an entirely new platform.
I set out some guidelines to support any decisions the teams needed to make:
-
The existing customer experience should, where possible, be unchanged. Our Shopfront (shop.coop.co.uk) website had in the preceding 12 months been completely re-written from the ground up. That didn't mean that the shop.coop.co.uk site wouldn't be changed - far from it - but we would anchor our technical design around providing consistency.
-
Our store colleagues shouldn't notice. With 1500 stores and tens of thousands of extremely hard working store colleagues we had to aim to not change any of their day to day processes.
-
Existing head office colleague processes should be optimised. The nature of the proof of concept expanding into a nationwide service has resulted in processes and systems being developed that are far from the most efficient. We had an opportunity to take everything we'd learned running the service to date and optimise the hell out of it.
Once we'd laid out the constraints on what we could and, more importantly, what we could not change, the next thing we'd need to work out was how we'd actually get the new system live.
Working backwards from go‑live
A small aside, I've worked on a few very large, often multi-year, projects in my engineering career and it still surprises me how many of them had seemingly given no thought to the actual mechanics of going live in production before the milestone was on the horizon in a project plan. It always seemed to come as a big shock that we might need to change what we'd designed and built because someone had an existing system in place.
I wanted to understand what people were expecting on the first day of this new platform going live. One response stood out:
Nobody notices and it's a complete non-event.
To achieve this we had to work out a rollout plan. These generally fell into 3 categories:
- A big bang - everything goes live all at once.
- A progressive store by store rollout.
- A progressive rollout based on the customer ordering.
The Big Bang approach
This was by far the riskiest way to launch any service. With the best QA team in the world you would still run into unexpected edge-cases or design and implementation oversights. By switching the whole service in one go we'd risk the service as a whole if there was an issue. Given the number of customers, colleagues and stores involved this was completely unpalatable.
A Store‑by‑Store approach
By building on the principle that "our store colleagues shouldn't notice" we would be able to switch over the backend of an entire store from the existing to the new platform with zero store colleague or customer impact. We also benefited from being able to cherry pick different types of stores to work through edge-cases (e.g. inner-city stores had very different order sizes and volumes compared to our rural stores). In addition we could easily identify if there were systemic issues with a specific store if all orders were from one platform or another.
Customers are the trigger
Building on our first principle "the existing customer experience should, where possible, be unchanged" we could have customers allocated to a platform, similar to an A/B test, and as the experiences between both platforms were designed to match each other the customer would have no idea they were using either platform. This approach did however muddy the waters when it came to investigations as there were orders from both platforms going into stores.
The chosen approach
We opted for the Store-by-Store approach. It allowed us to roll out at a pace that we felt comfortable with, to have operational visibility if there were issues with the roll out, and to split the rollout into multiple phases.
The other massive advantage with this approach was that we could roll back immediately if we spotted something unexpected. The existing platform remained in place for the entire roll out and with the toggle of a setting a store would return to handling orders from the existing platform.
At this stage both the technical and business teams fully bought into this approach, as the next stage would be designing the technical approach to achieve that Store-by-Store rollout.
Defining our (technical) contracts
Here were our three guiding principles:
- The existing customer experience should, where possible, be unchanged.
- Our store colleagues shouldn't notice.
- Existing head office colleague processes should be optimised.
Well the first two principles sure looked like contracts to me. Breaking them down made them more explicit.
To retain the existing customer experience we had user journeys already built, they required certain pieces of data, certain behaviours, expected interactions. These would form the outward facing contract of our system design. Any radical departure from the existing data structures would mean reworking large parts of a live site.
To avoid impacting colleagues we needed the in-store colleague experiences to both look and behave the same. Our primary in-store interactions were: picking customer's orders from the shelves and dispatching those orders to our last-mile courier partners. This made another contract in our system design.
Another hidden layer was our Integration Services that sat between the services required to support our Shopfront whilst abstracting away all of the internal complexities of Co-op's big systems (the big ERP, enterprise-y scary things). This abstraction kept our teams nimble, able to adapt much more quickly than waiting for changes in other Co-op systems.
I can sense you're looking at the diagram and thinking, well this is really easy. You just want to pop the new platform...there.
And whilst that was, in fact, not far from what the finished migration looked like, there was a fairly substantial middle step where we needed to be dual-running the old and new platforms. This required us to add some complexity upfront to de-risk the overall rollout. This was what we ended up building in diagram form; I'll go into detail of each element to explain what we did and why.
The Shopfront
With the customer experience remaining unchanged, the goal with all of the development within the Shopfront was to reuse all of the visual elements and UX patterns but have them completely rewired behind the scenes.
As a starting point our Shopfront codebase heavily utilised feature flags; these allowed us to deploy new features and capabilities to production without customers actually seeing them. This allowed us to safely deploy regularly as well as test in production. We introduced a new platform feature flag which was the gate for running the dual-running checks.
The next thing the Shopfront was responsible for was determining which platform should actually handle your order. This was the only part of the solution where users were interacting with both old and new platforms. The first step of an online shopping journey with Co-op started with entering your postcode, which allowed us to determine which store would deliver your order. To support dual running we needed to ask both the old and new systems if the store that was servicing that postcode had been migrated yet. If the new platform said "Yes" we favoured this response and started the user's shop in the new platform. If not, we dropped the user back into the existing journey.
One of the key things that changed between the old and new platforms was the unique IDs the systems used to find products and categories. However, customers and search engines wouldn’t know that we were replatforming, so we had to build a mechanism to reroute requests. For example, your favourite crisps could be purchased from /product/co-op-irresistible-hand-cooked-sea-salt-chardonnay-wine-vinegar-crisps-150g--f935925c-11fa-43ab-a7db-b2bc9121b2e6 on the old platform or /product/co-op-irresistible-hand-cooked-sea-salt-chardonnay-wine-vinegar-crisps-150g--5000128605113 on the new platform.
It’s a small change, but using the product’s well-understood identifier rather than an internal ID makes linking to products much easier. Our category pages have seen a similar glow-up; this time, we took an interim step to support backwards compatibility where we use both the new slug and the old category names in the url, we'll move to just the slugs in the future. The old platform exposed category IDs, whereas the new platform uses customisable URL slugs, which provide greater SEO benefits.
To retain the current user behaviour we had to compromise on the implementation of some features, using some of the new platform's features before a complete rollout would cause the user experiences to diverge. We had built up a collection of things we would revisit after the rollout, a conscious choice to smooth the transition to adopt some short-term technical baggage.
Salesforce (the Shiny New Headless Commerce Platform)
The overwhelming majority of the focus for this re-platform obviously sat in this component. But for the topic of this post it was one of the less interesting bits.
We chose to delay adopting new capabilities in the platform for the initial launch. However, we did lay solid foundations and laid out a roadmap of future capabilities. This was a really difficult decision because as an organisation we were investing in a huge platform because of its new features and were choosing not to use them.
The key was that we had not implemented the current capabilities in a way that would constrain us in future.
In‑store Picking App
Our store colleagues shouldn't notice. The hardest work that no one knew about. When we decided that we were going to roll out store by store we knew that we'd have to do some work to make the transition seamless.
Our in-store colleagues are notified when an order arrives, they open the order up, are presented with a list of products from around the store, scan all of the products into bags, pop the order number on the bag, and move on to their next task. In the background the picked order is sent back to the Commerce Platform to trigger payment, email updates and driver orchestration.
To keep that experience identical we had to update the existing picker application to take orders through an adaptor. This would allow the current platform to continue to send orders to the store; the new platform then used that same adaptor to send and receive updates when it took over.
Integration Services (Data)
Developing the current platform from a proof of concept had meant that the processes for getting product, category, availability and pricing data in had morphed and changed over time. They were, however, still full of manual steps and spreadsheets *shudders*.
We used this re-platform as an opportunity to refine all of these processes. This was in line with our third principle to optimise the head office colleague processes. We built the new processes from the ground up to fetch data from the "Big Corporate Systems", reshape that data for Salesforce, and push it in automatically. Our internal platforms provided key events that we could listen to like: new product available in store, product on a promotion, product price change. We could use these events to trigger changes in the new platform.
We also updated business processes to have our users working with the data within the Commerce Platform itself, this gave much greater flexibility when it came to the number of elements that could be controlled. It also made the processes much more responsive: changes could appear on site within seconds rather than the hours it might have previously taken.
Integration Service (Payment)
The touchpoints between shop.coop.co.uk and the Big Corporate Systems were kept to a minimum. We had the Data Integration Services we had just discussed. This got data out of the Big Corporate Systems into the commerce platforms. And there was the Payments Integration Service, which took transaction information and pushed it back into the Big Corporate Systems.
We could have introduced a new service here, but that would have required us to completely revalidate that the transaction workflows worked in the same way not just with shop.coop.co.uk but also with every one of the transaction-adjacent systems within Big Corporate Systems. We opted to reuse the existing integration. One big benefit with this approach was that from the perspective of every other system - nothing had changed. Existing reporting would be unaffected, stock movement records would be unaffected, financial updates would be unaffected.
Big Corporate Systems
I kept this as a big red box, this box contained all of the large corporate systems that were known for being hard to change but were also the foundations of the organisation. Most of these systems had three-letter acronyms or initialisms for names. An unwritten principle we had was that we didn't change things in here.
Let's do it again
So, did it work? Yup. We migrated all 1500 stores. The plan was eight weeks and we did it in six, not because we rushed but because the early weeks raised no critical defects and that gave us the confidence to pick up the pace.
Those early decisions we took - the dual-running, feature flagging, the picking adaptors, keeping the old platform alive the whole time - they all gave us the boring rollout we wanted. Given the chance, I'd make exactly the same boring choices again. We do have to spend time unpicking the bits of technical baggage we took on to smooth the rollout, and that's planned on the roadmap.
Up next is the interesting bit: we get to take advantage of the new capabilities the platform offers to us. Also one of the best things about being an engineer is deleting all of the old code that you've grown to dislike over time, so the teams are queueing up to strip out the old platform and associated plumbing.