The Most Wonderful Time of the (Ops Team’s) Year!

While most people were getting ready to indulge for the Thanksgiving Holiday here in the U.S., the {rr} Ops Team was stocking up on their Mountain Dew supply and compiling lengthy playlists to weather the busiest shopping period of the year. As we geared up for these events, I wondered what we would see over the upcoming days given the heavy focus on ‘Cyber Week’ in lieu of in-store Thanksgiving shopping, how much would traffic and sales spike across our 240+ retailers come Black Friday and Cyber Monday? The anticipation was exhausting. We added 20+% in capacity, a new datacenter in London, England as well as Sydney, Australia. We doubled our network capacity, and so much more to prepare for these 6 days.

The {rr} Ops team tracks a number of metrics like requests per second, response times, model builds, transfer rates and system loads during the shopping rush to ensure we are on par with how retailers expect performance. Records were quickly broken and as in years past, the numbers spoke loud and clear. In the past six days, we’ve delivered more than 38.9 billion recommendations (up from our previous record of 19.5 billion), over 630 million page views on Black Friday (up from 532M last year) and managed more than 18,330 peak requests per second on our servers. In general, we saw much higher overnight (USA) troughs in our charts, showing a larger engagement overseas during Thanksgiving night and early Black Friday. In fact we saw our EU datacenters enjoying high numbers until 1130p to midnight GMT and USA shopping up to 2am PT. Through all the holiday extended weekend we enjoyed 100% uptime as well.

Here’s a quick recap of our ride as we scaled the peaks of the Devil’s Horns in 2016. My seventh at RichRelevance – climbing these peaks never become old.

November 23: T minus 1 to Thanksgiving

  • In the two-day ramp up to Black Friday, we saw a 20+% increase in requests per second (from an average of 10,000 RPS last year to over 11,975 RPS this year).
  • On Monday of Thanksgiving week, we saw over 343M page views, compared to 300M page views last year
  • Traffic doubled and ran higher and longer globally starting in early November. A normal day from September, as shown on the below left, would show a defined growth to a peak, short idle, then a drop —think of a hill or small mountain’s shape. Whereas our troughs (nights) now rested around a sustained 6000 requests globally per second, a month ago they were below 3000. Seems like those earlier, longer holiday shopping periods sure are evident online.

 

 

November 24: Turkey with a side of shopping?

  • In all the years my team has been tracking this, we typically we see higher page views on Thanksgiving compared with the rest of the weekend, but more purchases on Black Friday and Cyber Monday. Until last year. And now…
  • Today, we hovered around 12,500 to 13,800 RPS all day.  We peaked at 13,950K RSP which means, Thanksgiving 2016 had a lot more Pageviews and clicks than last year!

 

 

November 25: Black Friday is Upon Us

  • At 7:30am PT on Black Friday, we hit over 18,3000 requests per second, breaking the record of 17,240 set in 2014.  The long high flat plateaus of Thanksgiving, Black Friday and Cyber Monday signal lots of traffic. 300RPS brings more than 1M more requests in an hour – all those clicks and views add up.
  • Page views for Black Friday were 16% higher than last year at >630M.  Our former high was previous black Friday with about 532M page views. After two years of this pattern, we are converting our thoughts to Black Friday being the king of Page Views, usurping Thanksgiving from its former glory.

 

In the graph below, the purple shows current page views in millions up through the end of day black Friday. Orange is the projections through the end of the year. Each color below represents the previous 8 years (2008 to current).

 

 

The Weekend

Time for resting, restocking our Dew, and refreshing our playlists. The typical “eye of the storm” Cyber weekend occured. Saturday’s and Sunday’s high is down well below the trough of Thursday night. We just floated around 10-11K on the Load Balancers(edge) RPS all day.

November 30: Cyber Monday

  • Our high for Cyber Monday Page Views were decent at 503M (503,359,235 to be exact).
  • Monday beat Black Friday on Units (total aggregated items placed in purchased carts over the day) inferring possibly more value/’Dollars’ on the day, but lower on clicks, pageview and the rest of the stats.
  • We didn’t break our 18,300 RPS peak from Black Friday as I had thought we would, hitting just 17,140 RPS, but still up from 2015 Cyber Monday RPS of 16,530.
  • Below is the Devil’s Horns for Cyber Monday.   As I mentioned above, we hit 503M page views as we continue our upward trend year over year (499M on Cyber Monday 2015 and 215M on Cyber Monday 2014)

 

 

The shopping season is in full force, and almost a week longer this year with the early Thanksgiving, but to summarize a great holiday shopping season opening weekend, in 6 days we had:

  • 2.7B page views,
  • 4.4B placement views,
  • 38.9B recommendations served,
  • 100% uptime on all 14 of our front-end data centers,

And a RPS peak of 18,330; that’s 65M requests per hour, more than 1M a minute.

Some things just don’t change. Black Friday 2016 was a great shopping day as expected, but brought whole new twists to existing patterns. Traffic climbed earlier in the day, plateauing high across the day and ran high for longer periods; the troughs were high, higher than any past year. We saw more traffic from overseas, and not just from growth in our client base ­ it was also much longer and much higher in EU than any year past, even APAC shopped tons. Interestingly, mobile made up a much larger percentage of sales, the National Retail Federation said more than half of all smartphone and tablet owners used those devices to shop over the holiday weekend. With Black Friday 2016 over, we prepare to wash, rinse, repeat.  2017, bring it.

I do get a bit reflective this time of year. I’m thankful to be blessed with one of the best Operations Team in my career who are like family to me.  My team has four members with more than 7+ years with RichRelevance, and we have an average of four or more years with the Company. You’d think this would just be second hand for them. But these highly skilled people worked tirelessly over each of the Thanksgiving/Black Friday/Cyber Monday weekends together and remained as focused as our first time around. Just the right amount of caution, risk, nervousness and confidence ensured that we were up 100% again, for seven years and counting.

 


Share :

This post was written by Kevin Duffey

ABOUT Kevin Duffey
As Vice President of IT Operations, Kevin brings more than 30 years of engineering, operations and management experience from both the public and private sector to RichRelevance. He is responsible for meeting RichRelevance’s customer needs by aligning business requirements with IT operations, and managing complex budgets and vendors. Under Kevin’s leadership, the IT team maintains 100% uptime and performance of fourteen global datacenters that deliver more than 1 billion recommendations each day on more than 500 retail websites. These results are due in part to continuous IT innovation using the latest technology, including multi-tenant Hadoop leveraging Hive, Parquet, Spark, Hbase and Cassandra. Prior to RichRelevance, Kevin worked in the public, private and entrepreneurial sectors for diverse companies such as the General Services Administration, AARP, Sonopia and Marketo.
Related Posts
Study: 58% of shoppers say deal days like Black Friday don’t matter
100% Uptime Makes for a Holly, Jolly Cyber Week
Archives