How to scale 10M+ Search Request with ElasticPress

NefisYemekTarifleri.com (Shortly we call as NYT. (not meaning NYTimes for us 🙂 ) is largest recipe site of TURKEY. We have more than 180k recipes that reaching millions of users. We switched our search infrastructure to Elasticsearch at the end of 2015. (I wrote about it.) Our cluster running smoothly for months.

Why ElasticPress?

Actually, there is no other choice for me. Look here; (seriously 😱 thousands of plugins in the repository and just 8 about Elasticsearch) 😂 (just kidding!)

I always appreciate 10up for open source contributions and they have great developers making awesome WordPress plugins. When I’ve seen ElasticPress, directly jumped into code and it seemed written for me👌 then decided to use it.

Custom wrapper

We can’t use ElasticPress directly because ElasticPress designed for standard WordPress installations and it doesn’t fit directly to our posts. Also, we have a few edge case to fix.

That’s why I wrote a custom wrapper.

Our wrapper fixing these problems by touching almost everything (analyzer to data sync)

What else?

We are using ElasticPress for finding the related post and some custom (expensive) queries.

We are using categories and ingredients to find the best matches. Elasticsearch really fasts at this point and far far better than YARPP or any other plugin.

ElasticPress has own Related Posts plugin.

Upgrading ElasticPress

When a new version of ElasticPress released, I check diff and start bulk-indexing process locally. If we don’t see any problem, we upgrade on production as well, otherwise reporting an issue on GitHub. (like this)

Large datasets can help to detect performance problems.

Statistics and some facts

• 180k+ published posts. I’ve checked large WordPress installs like mashable, techcrunch, fusion etc.. and we are probably bigger than them and continue to grow faster 😉
• 2 index (posts and trending searches)
• 2 Elasticsearch server with HA setup. (using nginx for auth, reverse proxy to ES)
• ~300k search request per day
• a few million related post queries per month (estimated)
• we didn’t open advanced filtering features to end users yet.
• kopf,bigdesk and head are useful plugins.

Our MySQL loads dramatically (half-and-half) decreased after we started to use Elasticsearch.

Feel free to ask any questions.