Extensão da Memória

About the Moving Linear Regression This type of regression to be used when data of sub periods are different inside periods. For example, hours of day, days of week, months of year. For example, the variance of  the frequency of people in a commercial mall varies when comparing the days of the week. The same applies to an amount of cars on downtown or roads when comparing the days in a week, or even when comparing months, due to holidays and vacancies. The chart bellow displays a seven year Moving Linear Regression applied to each month, for the number of ongoing research projects.  About the data The data is public available on the open government initiative of the State of São Paulo - Brazil. State of São Paulo / Brazil - Transparency Portal Disclaimer - The sole purpose of this presentation is to carry out tests with machine learning, using public and open data to improve the use of technology for future application in the FAPESP Virtual Library. ...

Sometimes you have to detect search engine crawling activity on your system, to handle the workflow for this type of request. On our website, there is a high crawling activity, because BV FAPESP (https://bv.fapesp.br) provides useful information abour reaserch for science, technology and academy in Sao Paulo state of Brazil. Recently, I deployed a feature on our system to allow people to store their queries and filters they do on our website. This is an open feature for everybody who navigates on our system. It is based on the HTTP session of the brower, which is stored on Python/Django server-side. When we were tunning Django session table in the database, we found out that there were a lot of sessions created by crawling activities. To take control of the session creation, I wrote the middleware that follows.  The list of crawlers, I get from ngix logs, on a very small timeframe, so maybe there are some more search engines that are not listed. The use-case presented, ...

At BV FAPESP (www.bv.fapesp.br) we use Solr as the searchengine backend, and a library called Haystack to tie Solr to Django. In 2018, me and my team wrote a Python/Django library to use with Apache Solr in cloud mode. We were avoiding the use of Django/Haystack library, since there were some features not supported, like grouping, Streaming Expressions, Graph Analysis. So far so good, before the end of the project I had in production environment Solr Cloud running smoothly, but I still had a single Solr running with Haystack, because we didn't re-code the whole system, and there still exist a legacy using Haystack. To turn-off the single Solr, we moved all documents to Solr Cloud and connected Haystack to it. This is what I documented here, for myself and maybe you, trying to make the same. Step-by-step There is Solr Cloud python backend for Haystack, that you can find here: https://github.com/django-haystack/django-haystack/pull/1580/commits/13df4a9e69ececd5567636085df4...

I have been using d3.js in a production environment for about 3 years. And now due to some features upgrade on our website some changes need to be made on our charts. I decided to put this on paper to document and share this experiments. To play with d3.js, you have to deal with some technologies: Javascript Css Ajax Server-side (Python/Django) for me For this system I also use Solr to deliver graph data. System overview The portal ( https://bv.fapesp.br ) is a standard web site that loads dynamic data from relational database and nosql database. To display analytics charts, we use d3.js. The diagram below shows the communication layer between the components, to load d3.js charts on the front-end of BV FAPESP. Essentially, we use d3.js just like that, a javascript framework to display chart data. However, when you have to load data dynamically in a couple of charts, these things need to be a bit elaborated. BV FAPESP portal makes use of some d3.js charts seemin...

In this document I will show you how I am using some distinct technology layers to display graph data from BV FAPESP, stored inside Solr. FAPESP´s virtual library (BV FAPESP) is the information system, source of the referential data, of funded projects from São Paulo funding agency. These projects holds relation to each other in a three structure, a specialized form of a graph structure. BV FAPESP data projects structure In the relational database, the projects are linked by foreign keys, the most simple form to store the three structure relationship. But the main data source of BV FAPESP, is not a relational database, but the search engine Solr. Solr is a search engine, that uses the inverted index structure which is very fast at query time. The initial approach At the early stages of BV FAPESP, the associated projects were indexed with some kind of information from their parent, to allow the inverted search to get all children of a subset of projects. Everything is...

I will show you how to use the Atom editor with Jupyter Notebook to debug Python/Django views using Hydrogen plugin. On the Internet you can find some examples using this setup to debug Python but I found nothing related to Django, so I decided to publish this notes. This is not a high performance setup, because it involves many technologies at several layers. This is more a proof of concept of nowadays technologies which are available to the programmer's toolbox. This notes has been written during the development and improvement of the automated tests of the Virtual Library of FAPESP ( http://www.bv.fapesp.br ). The environment The diagram below shows an actual infrastructure to use this setup in a development environment. Technologies It is not expected that you know all of the technologies in detail, but there are some trick points. This paper is oriented for Python programmer who already use Atom and Jupyter and enjoy explore the edges of these tools. ...

O objetivo deste texto é documentar algumas dificuldades encontradas para substituir o framework Google Maps pelo Open Street Map. Este texto não é um tutorial do OSM, seja no nível de instalação, configuração ou uso; e se limita a apresentar os detalhes de implementação específicos ao caso de uso do portal web Biblioteca Virtual da FAPESP (BV FAPESP - http://bv.fapesp.br ). Caso de uso A BV FAPESP utiliza mapas georreferenciados para apresentar informações de concessão de fomentos de Projetos de Pesquisa (Auxílios à Pesquisa e Bolsas de Pesquisa) relacionados às cidades de origem das Instituições-sede de pesquisa e instituições no exterior, quando o Projeto de Pesquisa se relaciona com alguma instituição no exterior. Deste modo é necessário que o sistema de mapas tenha uma abrangência global. A BV FAPESP sempre utilizou o framework de mapas Google Maps, porém o mesmo passou a cobrar pelos seus serviços à partir de Junho de 2018. Isso levou a uma decisão interna do projeto ...

Instale o Jenkins de acordo com o seu sistema operacional O Jenkins é uma aplicação web JAVA autocontida, que utiliza o servlet container/HTTP server Jetty. O Jenkins roda como serviço, o que significa que uma vez instalado, basta você rodar o comando de acordo com o seu sistema operacional que o mesmo sobe tudo o que for necessário e fornece uma interface web para ser acessada via browser. Uma vez que o Jenkins está instalado e inicializado corretamente, basta acessar a página de adminstração do Jenkins na porta que você subiu a aplicação. A interface abaixo é uma tela da versão 2.7.4, que requer o Java jdk 1.7. Gitlab plugin Este plugin permite que o Jenkins faça os builds automaticamente quando houver um push ou merge de código no Gitlab. Instalar O mesmo deve ser instalado, pois não vem instalado por padrão no Jenkins. Para instalar plugins, clique no item de menu “Jenkins” > “Gerenciar Jenkins” > “Gerenciar Plugins”, conforme a figura abaixo. Certifique...

Nota de esclarecimento Eu estou publicando esse artigo em inglês na tentativa de alcançar uma audiência maior. O objetivo é expandir e não limitar o acesso, e c aso haja demanda eu escreverei em português. O assunto desse artigo é de interesse global e tem um viés mais técnico do que outros textos que eu tenho publicado. Before I start, I would like to tell you that english is not my primary language so you will see creative english, non standard english also known as errors.  I will try my best to explain the idea of “making software”. Introduction Technology is evolving faster and faster day after day. Some advocates that this process draw a exponential line, and by 2045 technology will have reached a new singularity that human bean will have difficulties to comprehend the surrounding technological environment. I will present a software that is far from this reality, but it innovates in the sense that it carries facility to the end user, so he or she can write...

Sistemas de apoio a decisão são utilizados para auxiliar na resolução de problemas complexos e específicos, problemas que lidam com volumes elevados de informações e que exijem tratamentos e interpretações para a obtenção das respostas esperadas. A sua utilização pode se dar em qualquer tamanho de escala, desde a otimização de uma equipe de garçons em uma festa, ou a melhoria no desempenho de distribuição de energia elétrica. A base de dados de tais sistemas precisa ser atualizada constantemente, porque as situações analisadas são dinâmicas e podem se alterar continuamente, como no caso do gerenciamento da frota de ônibus em uma metrópole, ou da análise de ações da bolsa de valores. Todos esses dados existem de alguma maneira em formato digital, e para alimentar um sistema de apoio a decisão, pode-se utilizar robôs que executem essa tarefa em tempo real, ou processos em lote quando o processamento for assíncrono. Um sistema de grande complexidade geralmente envolve processos síncronos ...