How to scale 10M+ Search Request with ElasticPress

NefisYemekTarifleri.com (Shortly we call as NYT. (not meaning nytimes for us🙂 ) is largest recipe site of TURKEY. We have more than 180k recipes that reaching millions of users.We switched our search infrastructure to Elasticsearch at the end of 2015. (I wrote about it.) Our cluster running smoothly for months.

Why ElasticPress?

Actually there is no other choice for me. Look here; (seriously 😱 thousands of plugins in repository and just 8 about Elasticsearch) 😂 (just kidding!)

wordpress-org-es.png

WordPress.org should improve searching

I always appreciate 10up for open source contributions and they have great developers making awesome WordPress plugins.When I’ve seen ElasticPress, directly jumped into code and it seemed written for me👌 then decided to use it.

Custom wrapper

We can’t use ElasticPress directly because ElasticPress designed for standard WordPress installations and it doesn’t fit directly to our posts. Also we have a few edge case to fix.

That’s why I wrote custom wrapper.

search-about.png

Slide from Greg’s WordCamp US 2015 presentation

Our wrapper fixing these problems by touching almost everything (analyzer to data sync)

What else?

We are using ElasticPress for finding related post and some custom (expensive) queries.

ep-related-post.png

Ex: Related pizza recipes

We are using categories and ingredients to find best matches. Elasticsearch really fast at this point and far far better than YARPP or any other plugin.

ElasticPress has own Related Posts plugin.

Upgrading ElasticPress

When new version of ElasticPress released, I check diff and start bulk-indexing process locally. If we don’t see any problem, we upgrade on production as well, otherwise reporting issue on github. (like this)

Large datasets can help to detect performance problems.

Statistics and some facts

  • 180k+ published posts. I’ve checked large WordPress installs like mashable, techcrunch, fusion etc.. and we are probably bigger than them and continue to grow faster😉
  • 2 index (posts and trending searches)
  • 2 Elasticsearch server with HA setup. (using nginx for auth, reverse proxy to ES)
  • ~300k search request per day
  • a few million related post queries per month (estimated)
  • we didn’t open advanced filtering features to end users yet.
  • kopf,bigdesk and head are useful plugins.

Our mysql loads dramatically (half-and-half) decreased after we started to use Elasticsearch.

Feel free to ask any questions.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s