Connect Django Haystack to Solr Cloud

At BV FAPESP (www.bv.fapesp.br) we use Solr as the searchengine backend, and a library called Haystack to tie Solr to Django.

In 2018, me and my team wrote a Python/Django library to use with Apache Solr in cloud mode. We were avoiding the use of Django/Haystack library, since there were some features not supported, like grouping, Streaming Expressions, Graph Analysis.

So far so good, before the end of the project I had in production environment Solr Cloud running smoothly, but I still had a single Solr running with Haystack, because we didn't re-code the whole system, and there still exist a legacy using Haystack.

To turn-off the single Solr, we moved all documents to Solr Cloud and connected Haystack to it. This is what I documented here, for myself and maybe you, trying to make the same.

Step-by-step

There is Solr Cloud python backend for Haystack, that you can find here:
https://github.com/django-haystack/django-haystack/pull/1580/commits/13df4a9e69ececd5567636085df4e353ce540a35
Copy this file to your Haystack env/virtualenv folder, like this:
/lib/python2.7/site-packages/haystack/backends/solrcloud_backend.py

Use this pysolr.py
https://github.com/django-haystack/pysolr/blob/master/pysolr.py


Zookeeper / Kazoo

You must use Zookeeper in production environment
pip install kazoo

Configuring

For the configurations bellow, follow this comments here:

Put those lines on your settings/local_settings.py with your infrastructure settings.
https://github.com/django-haystack/django-haystack/pull/1580#issuecomment-399378902

HAYSTACK_CONNECTIONS = {

    'default': {

        'ENGINE': 'solrcloud_backend.SolrCloudEngine',

        'URL': '127.0.0.1:9983',  # this is the ZooKeeper

        'COLLECTION': 'gettingstarted',  # example SolrCloud collection

    },

}
Configure this settings in pysolr.py to:

[...]

ZooKeeper.CLUSTER_STATE = '/collections/{}/state.json'.format(connection_options['COLLECTION'])

zookeeper = ZooKeeper(connection_options['URL'])

[...]

wsgi.py

If you use uwsgi, just take care when you change your environmet / virtualenv, to set PATH and PYTHONPATH correctly.

This is for me to remember, when creating a new virtualenv and copy wsgi.py already configured.

Popular posts from this blog

Atom - Jupyter / Hydrogen

Image
I will show you how to use the Atom editor with Jupyter Notebook to debug Python/Django views using Hydrogen plugin. On the Internet you can find some examples using this setup to debug Python but I found nothing related to Django, so I decided to publish this notes. This is not a high performance setup, because it involves many technologies at several layers. This is more a proof of concept of nowadays technologies which are available to the programmer's toolbox. This notes has been written during the development and improvement of the automated tests of the Virtual Library of FAPESP ( http://www.bv.fapesp.br ). The environment The diagram below shows an actual infrastructure to use this setup in a development environment. Technologies It is not expected that you know all of the technologies in detail, but there are some trick points. This paper is oriented for Python programmer who already use Atom and Jupyter and enjoy explore the edges of these tools. ...
Read more

Metodologias em ação

Image
O objetivo desse artigo é apresentar algumas metodologias atuais relacionadas à TI, na perspectiva das suas aplicações dentro de uma empresa. A estrutura de cada empresa pode variar muito, dessa maneira irei adotar uma estrutura clássica em formato de pirâmide, representada na figura abaixo para efeitos da nossa análise. As metodologias apresentadas, dificilmente serão utilizadas exatamente como estão sendo estudadas aqui pois não é comum a necessidade de utilizá-las em uma empresa todas ao mesmo tempo, apesar de ser possível. Isso dependerá do tipo de empresa, da estrutura que a mesma possui e como a empresa aplica a TI nos seus negócios. Também será feita uma breve análise do relacionamento entre as diversas metodologias quando adotadas em conjunto. A pirâmide representa a hierarquia em uma empresa tradicional, com o board of directors ficando no topo, abaixo a gerência senior com os CEO, CIO, CSO, CFO etc, abaixo a gerência operacional e abaixo os departamentos operacionais. Qual o ...
Read more

Design Patterns - Observer

Design Patterns - Observer A simple Python code example to illustrate the Observer Pattern for an ETL-like scenario. The "Extract Microservice" observes changes in a data source and notifies its registered observers (data extractors) to perform data extraction. We'll use a basic Publisher-Subscriber pattern to demonstrate the concept of the Observer Pattern. Observer interface class Observer: def update(self, data): pass Extract Microservice (Subject) class ExtractMicroservice: def __init__(self): self.observers = [] def register_observer(self, observer): self.observers.append(observer) def unregister_observer(self, observer): self.observers.remove(observer) def notify_observers(self, data): for observer in self.observers: observer.update(data) def start_extraction(self): # Simulate data extraction process data = ["Data 1", "Data 2", "Data 3",...
Read more