Posts

ETL, SOLID and Design Patterns

ETL, SOLID and Design Patterns Let's discuss here the best practices to implement a ETL - Extraction Transform Load process using SOLID principles and Design Patterns. Pipeline Pattern One design pattern commonly used in ETL (Extract, Transform, Load) processes is the "Pipeline" pattern. The Pipeline pattern is a software architectural pattern where a series of processing steps are connected together to form a data processing pipeline. Each step in the pipeline performs a specific operation on the data, such as extracting data from a source, transforming it into a desired format, and loading it into a target destination. By using the Pipeline pattern, you can modularize and organize your ETL process into smaller, reusable components. Each component focuses on a specific task, making the overall process more manageable and easier to maintain. Additionally, the Pipeline pattern allows for flexibility and scalability, as you can easily add or remove steps in the pipeli
Read more

Robson Koji Moriya disambiguation name

Image
Robson Koji Moriya This page is dedicated to my name and other similar names found on the Internet. The purpose of this page is to assist search engines like Google, Bing, Yahoo, and others in understanding that I am a distinct individual and that there are other people with similar names. I will also share some interesting facts with you. Search Engines It is interesting to note that Bing does a much better job than Google when it comes to indexing my name. When I search for "Robson Koji Moriya" on both search engines, Google displays a lot of other individuals with the names "Robson Koji" and "Moriya," but it doesn't show most of my own online publications. Even this blog, which has my complete name in its domain, is hidden by Google. On the other hand, Bing correctly displays most of my information. Yahoo search engine is powered by Bing through a partnership. Google and Ask.com produce very similar results, and unfortunately, they are mostly incorr
Read more

Backpropagation

Backpropagation Backpropagation algorithm involves several steps. Here is a high-level overview of the process Steps 1. Forward Pass Input: Start by feeding the input data through the neural network layer by layer. Weighted Sum: Each layer performs a weighted sum of the inputs from the previous layer, which is calculated as the dot product of the input vector and the weight matrix. Activation Function: Apply an activation function to the weighted sum to introduce non-linearity and produce the output of the layer. 2. Loss Calculation Compare Predictions with True Labels: Calculate the discrepancy between the predicted outputs obtained from the forward pass and the true labels associated with the input data. Loss Function: Use a suitable loss function to quantify the error between the predictions and true labels. The choice of the loss function depends on the task at hand (e.g., mean squared error for regression, cross-entropy loss for classification). 3. Backward Pass
Read more

Tensorflow - Computer Vision

Read more

SOLID - Training models in TensorFlow

SOLID - Training models in TensorFlow The SOLID principles can be applied to the software architecture surrounding the machine learning models. Here's how the principles can be relevant: Single Responsibility Principle (SRP) Each class or module in your TensorFlow project should have a clear and single responsibility. For example, you can have separate modules for data preprocessing, model training, model evaluation, and model deployment. This promotes modularity and makes it easier to understand, test, and maintain each component. Open-Closed Principle (OCP) By designing your TensorFlow project with the OCP in mind, you can make it easier to extend the functionality without modifying existing code. For example, you can define abstract base classes or interfaces that define the common behavior expected from different models, allowing you to add new models by implementing these interfaces without modifying the existing code that consumes them. Liskov Substitution Principle (LS
Read more

Hiring Process Interviews

Hiring Process Interviews First Round Hiring Process IT team structure Second Round IT team structure Job description Have to talk to management? Have to gather requirements? Code only? Nature of the system? Web Large System Distributed Code what? Infrastrucure? System Architect? Design Patterns Creational Patterns Builder class Pizza: def init(self): def str(self): class PizzaBuilder def init(self): def set_size(self, size): def add_cheese(self): def add_pepperoni(self): def add_bacon(self): def build(self): pizza = builder.set_size("Large").add_cheese().add_pepperoni().build() Factory class DataProcessor def process(self) class DatabaseExtractor(DataProcessor) def process(self): class APIExtractor(DataProcessor) def process(self) class FileExtractor(DataProcessor) def process(self) class DataProcessorFactory: def create_data_processor(source) Structural Patterns Adapter class MediaPlayer: def play(self
Read more

Design Patterns - Observer

Design Patterns - Observer A simple Python code example to illustrate the Observer Pattern for an ETL-like scenario. The "Extract Microservice" observes changes in a data source and notifies its registered observers (data extractors) to perform data extraction. We'll use a basic Publisher-Subscriber pattern to demonstrate the concept of the Observer Pattern. Observer interface class Observer: def update(self, data): pass Extract Microservice (Subject) class ExtractMicroservice: def __init__(self): self.observers = [] def register_observer(self, observer): self.observers.append(observer) def unregister_observer(self, observer): self.observers.remove(observer) def notify_observers(self, data): for observer in self.observers: observer.update(data) def start_extraction(self): # Simulate data extraction process data = ["Data 1", "Data 2", "Data 3",
Read more