The RichRelevance Ops Team is Coming to Town

The RichRelevance Ops Team is Coming to Town: They work when you’re sleeping, they work when you’re awake…

The week leading up to Thanksgiving most people in the US are getting ready for Thanksgiving and shopping for turkey and all the fixin’s, the RichRelevance ops team is stocking up on coffee, snacks and tunes to get through the busiest shopping time of the year.  This time each year, I wonder what we will see over the upcoming days… how much traffic and sales will we see across our customer sites, will there be record breaking numbers or will shopping trends plateau? And every year, I wonder if this is the year the buzz around the holiday shopping weekend will wear off. What year will people opt out of shopping during this 5-day period?  I’m sure it comes as no shock, that this was not the year that bucked the trend.

This is my 8th Black Friday at RichRelevance, and despite the many changes in the industry, a lot remains the same.  This year, like all the other years, what I knew for sure is that RichRelevance was prepared, we worked hard to ensure that we could and would deliver.

The RichRelevance Ops team tracks a number of metrics like requests per second, response times, feeds, model builds, transfer rates and system loads during the shopping rush to ensure we are on par with how retailers expect performance.  Here’s a brief recap of our ride as we scaled the peaks of the Devil’s Horns in 2017 – climbing these peaks never seems to get old and the views at top are still spectacular.

  • 2.4B pageviews
  • 4.3B placement views (vs 3.4B last year)↑
  • 38.1B recommendations served (vs 29.7B last year) ↑
  • 100% uptime on all 14 of our data centers.
  • An RPS peak of 18,600; that’s 67M requests per hour, more than 1.1M a minute. (vs 18,330 in 2016, 16,500 in 2015, and 17,134 in 2014) ↑

As usual, Black Friday and Cyber Monday were great shopping days that brought new trends.  Mobile made another move to shopping domination accounting for 46% of visits to e-commerce websites and apps on Thanksgiving and one third of all Thanksgiving Day sales. This year, Mobile drove a record $2B on Cyber Monday alone. (

We had zero severities from customers, no major issues. The only thing we saw was an issue with the internet in APAC for a couple hours.  We saw this before it affected our customers, we simply shifted the APAC Singapore traffic to Sydney and kept humming along with no service being affected.  This spider web below, thanks to ThousandEyes, is the internet connections between our datacenter worldwide to Singapore, red lines and circles are heavily dropped packets on the public internet.

As I reflect on this year’s holiday shopping weekend, I realize that I am truly thankful to be blessed with one of the best Operations Teams in my career who are like family to me.  Marry that to our Support and Client Services team and you have a World Class service.  This team works tirelessly over the holiday weekend and remains laser focused to ensure our customers showcase their best work during their busiest time of year.   Thanks to this team’s dedication, confidence and bit of risk taking, I can say we were 100% for 8 years and counting.  That’s 96 months! Maybe I should have 100 at 100 party in 4 months? Until next year, happy shopping and happy holidays.

Share :

This post was written by Kevin Duffey

ABOUT Kevin Duffey
As Vice President of IT Operations, Kevin brings more than 30 years of engineering, operations and management experience from both the public and private sector to RichRelevance. He is responsible for meeting RichRelevance’s customer needs by aligning business requirements with IT operations, and managing complex budgets and vendors. Under Kevin’s leadership, the IT team maintains 100% uptime and performance of fourteen global datacenters that deliver more than 1 billion recommendations each day on more than 500 retail websites. These results are due in part to continuous IT innovation using the latest technology, including multi-tenant Hadoop leveraging Hive, Parquet, Spark, Hbase and Cassandra. Prior to RichRelevance, Kevin worked in the public, private and entrepreneurial sectors for diverse companies such as the General Services Administration, AARP, Sonopia and Marketo.
Related Posts