Mustafa Uysal

I'm traveling light, it's au revoir…

How to scale 10M+ Search Request with ElasticPress

NefisYemekTarifleri.com (Shortly we call as NYT. (not meaning NYTimes for us 🙂 ) is largest recipe site of TURKEY. We have more than 180k recipes that reaching millions of users. We switched our search infrastructure to Elasticsearch at the end of 2015. (I wrote about it.) Our cluster running smoothly for months.

Why ElasticPress?

Actually, there is no other choice for me. Look here; (seriously 😱 thousands of plugins in the repository and just 8 about Elasticsearch) 😂 (just kidding!)

wordpress-org-es.png
WordPress.org should improve searching

I always appreciate 10up for open source contributions and they have great developers making awesome WordPress plugins. When I’ve seen ElasticPress, directly jumped into code and it seemed written for me👌 then decided to use it.

Custom wrapper

We can’t use ElasticPress directly because ElasticPress designed for standard WordPress installations and it doesn’t fit directly to our posts. Also, we have a few edge case to fix.

That’s why I wrote a custom wrapper.

search-about.png
Slide from Greg’s WordCamp US 2015 presentation

Our wrapper fixing these problems by touching almost everything (analyzer to data sync)

What else?

We are using ElasticPress for finding the related post and some custom (expensive) queries.

ep-related-post.png
Ex: Related pizza recipes

We are using categories and ingredients to find the best matches. Elasticsearch really fasts at this point and far far better than YARPP or any other plugin.

ElasticPress has own Related Posts plugin.

Upgrading ElasticPress

When a new version of ElasticPress released, I check diff and start bulk-indexing process locally. If we don’t see any problem, we upgrade on production as well, otherwise reporting an issue on GitHub. (like this)

Large datasets can help to detect performance problems.

Statistics and some facts

  • 180k+ published posts. I’ve checked large WordPress installs like mashable, techcrunch, fusion etc.. and we are probably bigger than them and continue to grow faster 😉
  • 2 index (posts and trending searches)
  • 2 Elasticsearch server with HA setup. (using nginx for auth, reverse proxy to ES)
  • ~300k search request per day
  • a few million related post queries per month (estimated)
  • we didn’t open advanced filtering features to end users yet.
  • kopf,bigdesk and head are useful plugins.

Our MySQL loads dramatically (half-and-half) decreased after we started to use Elasticsearch.

Feel free to ask any questions.

7 responses

  1. […] use ElasticPress to handle 10M+ requests. To scale this platform, we use various tools which we share at Stackshare. We share some of our […]

  2. Gaurav Tewari Avatar

    Hi, Is it possible you to share the plugin you have developed for ElasticSearch.

    1. Mustafa Uysal Avatar

      Hello, we are using ElasticPress. Our custom wrapper just fixes some edge cases, you can use ElasticPress directly.

  3. Gaurav Tewari Avatar

    HI, Can you help me to setup Fuzzy search on Elasticpress, it doesn’t seems to be working.

    1. Mustafa Uysal Avatar

      Hello, unfortunately I can’t help directly. Have you checked issues?

      If the problem is general, you can create new issue as well 🙂

  4. […] of millions of search […]

  5. […] milyonlarca […]

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.