Connect Django Haystack to Solr Cloud

At BV FAPESP (www.bv.fapesp.br) we use Solr as the searchengine backend, and a library called Haystack to tie Solr to Django.

In 2018, me and my team wrote a Python/Django library to use with Apache Solr in cloud mode. We were avoiding the use of Django/Haystack library, since there were some features not supported, like grouping, Streaming Expressions, Graph Analysis.

So far so good, before the end of the project I had in production environment Solr Cloud running smoothly, but I still had a single Solr running with Haystack, because we didn't re-code the whole system, and there still exist a legacy using Haystack.

To turn-off the single Solr, we moved all documents to Solr Cloud and connected Haystack to it. This is what I documented here, for myself and maybe you, trying to make the same.

Step-by-step

There is Solr Cloud python backend for Haystack, that you can find here:
https://github.com/django-haystack/django-haystack/pull/1580/commits/13df4a9e69ececd5567636085df4e353ce540a35
Copy this file to your Haystack env/virtualenv folder, like this:
/lib/python2.7/site-packages/haystack/backends/solrcloud_backend.py

Use this pysolr.py
https://github.com/django-haystack/pysolr/blob/master/pysolr.py


Zookeeper / Kazoo

You must use Zookeeper in production environment
pip install kazoo

Configuring

For the configurations bellow, follow this comments here:

Put those lines on your settings/local_settings.py with your infrastructure settings.
https://github.com/django-haystack/django-haystack/pull/1580#issuecomment-399378902

HAYSTACK_CONNECTIONS = {

    'default': {

        'ENGINE': 'solrcloud_backend.SolrCloudEngine',

        'URL': '127.0.0.1:9983',  # this is the ZooKeeper

        'COLLECTION': 'gettingstarted',  # example SolrCloud collection

    },

}
Configure this settings in pysolr.py to:

[...]

ZooKeeper.CLUSTER_STATE = '/collections/{}/state.json'.format(connection_options['COLLECTION'])

zookeeper = ZooKeeper(connection_options['URL'])

[...]

wsgi.py

If you use uwsgi, just take care when you change your environmet / virtualenv, to set PATH and PYTHONPATH correctly.

This is for me to remember, when creating a new virtualenv and copy wsgi.py already configured.

Popular posts from this blog

Atom - Jupyter / Hydrogen

Image
I will show you how to use the Atom editor with Jupyter Notebook to debug Python/Django views using Hydrogen plugin. On the Internet you can find some examples using this setup to debug Python but I found nothing related to Django, so I decided to publish this notes. This is not a high performance setup, because it involves many technologies at several layers. This is more a proof of concept of nowadays technologies which are available to the programmer's toolbox. This notes has been written during the development and improvement of the automated tests of the Virtual Library of FAPESP ( http://www.bv.fapesp.br ). The environment The diagram below shows an actual infrastructure to use this setup in a development environment. Technologies It is not expected that you know all of the technologies in detail, but there are some trick points. This paper is oriented for Python programmer who already use Atom and Jupyter and enjoy explore the edges of these tools.
Read more

Design Patterns

Design Patterns Creational Patterns These patterns focus on object creation mechanisms, providing flexibility and decoupling the client code from the specific classes instantiated. Some examples of creational patterns are: Singleton: Ensures that only one instance of a class exists throughout the program. Factory: Provides an interface for creating objects, but allows subclasses to decide which class to instantiate. Builder: Separates the construction of complex objects from their representation, allowing the same construction process to create different representations. Abstract Factory: Provides an interface for creating families of related or dependent objects without specifying their concrete classes. Prototype: Creates new objects by cloning existing ones and modifying them as needed. Structural Patterns These patterns deal with the composition of classes and objects, focusing on how they can be organized to form larger structures while keeping them flexible and ef
Read more

Robson Koji Moriya disambiguation name

Image
Robson Koji Moriya This page is dedicated to my name and other similar names found on the Internet. The purpose of this page is to assist search engines like Google, Bing, Yahoo, and others in understanding that I am a distinct individual and that there are other people with similar names. I will also share some interesting facts with you. Search Engines It is interesting to note that Bing does a much better job than Google when it comes to indexing my name. When I search for "Robson Koji Moriya" on both search engines, Google displays a lot of other individuals with the names "Robson Koji" and "Moriya," but it doesn't show most of my own online publications. Even this blog, which has my complete name in its domain, is hidden by Google. On the other hand, Bing correctly displays most of my information. Yahoo search engine is powered by Bing through a partnership. Google and Ask.com produce very similar results, and unfortunately, they are mostly incorr
Read more