Posts

Showing posts with the label BV FAPESP

Machine Learning - Moving Linear Regression

Image
About the Moving Linear Regression This type of regression to be used when data of sub periods are different inside periods. For example, hours of day, days of week, months of year. For example, the variance of  the frequency of people in a commercial mall varies when comparing the days of the week. The same applies to an amount of cars on downtown or roads when comparing the days in a week, or even when comparing months, due to holidays and vacancies. The chart bellow displays a seven year Moving Linear Regression applied to each month, for the number of ongoing research projects.  About the data The data is public available on the open government initiative of the State of São Paulo - Brazil. State of São Paulo / Brazil - Transparency Portal Disclaimer - The sole purpose of this presentation is to carry out tests with machine learning, using public and open data to improve the use of technology for future application in the FAPESP Virtual Library. ...

Dynamic data use-case for d3.js

Image
I have been using d3.js in a production environment for about 3 years. And now due to some features upgrade on our website some changes need to be made on our charts. I decided to put this on paper to document and share this experiments. To play with d3.js, you have to deal with some technologies: Javascript Css Ajax Server-side (Python/Django) for me For this system I also use Solr to deliver graph data. System overview The portal ( https://bv.fapesp.br ) is a standard web site that loads dynamic data from relational database and nosql database. To display analytics charts, we use d3.js. The diagram below shows the communication layer between the components, to load d3.js charts on the front-end of BV FAPESP. Essentially, we use d3.js just like that, a javascript framework to display chart data. However, when you have to load data dynamically in a couple of charts, these things need to be a bit elaborated. BV FAPESP portal makes use of some d3.js charts seemin...

Atom - Jupyter / Hydrogen

Image
I will show you how to use the Atom editor with Jupyter Notebook to debug Python/Django views using Hydrogen plugin. On the Internet you can find some examples using this setup to debug Python but I found nothing related to Django, so I decided to publish this notes. This is not a high performance setup, because it involves many technologies at several layers. This is more a proof of concept of nowadays technologies which are available to the programmer's toolbox. This notes has been written during the development and improvement of the automated tests of the Virtual Library of FAPESP ( http://www.bv.fapesp.br ). The environment The diagram below shows an actual infrastructure to use this setup in a development environment. Technologies It is not expected that you know all of the technologies in detail, but there are some trick points. This paper is oriented for Python programmer who already use Atom and Jupyter and enjoy explore the edges of these tools. ...

Jenkins Gitlab Django

Image
Instale o Jenkins de acordo com o seu sistema operacional O Jenkins é uma aplicação web JAVA autocontida, que utiliza o servlet container/HTTP server Jetty. O Jenkins roda como serviço, o que significa que uma vez instalado, basta você rodar o comando de acordo com o seu sistema operacional que o mesmo sobe tudo o que for necessário e fornece uma interface web para ser acessada via browser. Uma vez que o Jenkins está instalado e inicializado corretamente, basta acessar a página de adminstração do Jenkins na porta que você subiu a aplicação. A interface abaixo é uma tela da versão 2.7.4, que requer o Java jdk 1.7. Gitlab plugin Este plugin permite que o Jenkins faça os builds automaticamente quando houver um push ou merge de código no Gitlab. Instalar O mesmo deve ser instalado, pois não vem instalado por padrão no Jenkins. Para instalar plugins, clique no item de menu “Jenkins” > “Gerenciar Jenkins” > “Gerenciar Plugins”, conforme a figura abaixo. Certifique...