Bijit Deka

Data Science Interview Question: Q. What's the difference between orthogonality and orthonormality in 2 vectors? Answer: Let's assume we have 2 vectors a and b. 1. Orthogonal - a and b are orthogonal if the dot product between them is zero. This means that there is no projection of one vector onto the other, implying that they are at a right angle (90 degrees) to each other. Mathematically - a⋅b=0 Orthogonality does not concern the magnitude of the vectors, only their angle. 2. Orthonormal - Two vectors are orthonormal if they are both orthogonal and each vector is of unit length. This means in addition to being orthogonal, each vector must have a magnitude (or norm) of 1. Mathematically, if a and b are orthonormal, they satisfy: a⋅b=0 and ∥a∥=1,∥b∥=1 Orthonormal vectors are a special case of orthogonal vectors where the additional condition of having unit norm is met.

125

4 Comments

#Amdari21DaysDataChallenge Machine Learning for Jewelry Price Optimization In my deliverable today, I performed #Spearman #correlation #analysis using heatmap to visualise it. # Feature correlation correlations = real_categories.drop(labels = ['Order_datetime','User_ID', 'Order_ID', 'product_ID'], axis = 1).corr(method = 'spearman') # Correlation heatmap plt.figure(figsize = (8, 8)) sns.heatmap(correlations, annot = True, center = .3) plt.show(); plt.close() # set correlation limit upper = 1 lower = .2 # Explicitly obtain correlated features correlated_columns = {   column: correlations[(correlations[column]> lower) & (correlations[column]< upper)].index.tolist()   for column in correlations.columns } correlated_columns #Output {'SKU_Quantity': [], 'Category_ID': [], 'Category': ['Price_USD'], 'Brand_ID': ['Target_Gender'], 'Price_USD': ['Category'], 'Target_Gender': ['Brand_ID'], 'Main_Color': [], 'Main_metal': [], 'Main_gem': []} #CasmirAAmdari21DDC #Amdari21DaysDataChallenge

4

Another way of handling missing values in your modelling projects!! In data science, handling missing values is a common and crucial task. Common methods include imputation, deletion, or using algorithms that can handle missing values natively. However, another effective way is using missing value indicators. Scenario: Imagine we are modeling loan approval of loan applicants based on marital status, as one of the features. In our sample, 30% of individuals are unmarried, thus their marital status is missing. This missingness isn't random; it indicates that these individuals are actually unmarried. Instead of just imputing these missing values with the mode, we can use an indicator to signify missingness (1 for unmarried and 0 for married). This provides the model with additional information. More info and practical example here: https://lnkd.in/drMhfbHz #machinelearning #datascience #missingvalues #missingvalueimputation #handlingmissingvalues #statistics

17

Summer Learnings!!✨ ML System Design Case study 01- Scaling ML Systems at DoorDash - Key Takeaways Recently, I delved into an insightful ML system design case study by Raghav Ramesh, an ML Architect at DoorDash. The case study detailed how DoorDash scaled their ML solutions to meet growing demands, focusing on these key strategies and below is my detailed notes on the case study: 1. Common Feature Extractor: Standardizing feature extraction to improve consistency and reliability across models. 📊 2. Experimentation and Shadowing: Continuously testing new models in parallel with existing ones to ensure optimal performance before deployment. 🔍 3. Templatization and Code Consistency: Using templates and enforcing coding standards to maintain high-quality, maintainable, and collaborative codebases. 📝 4. Scalable Infrastructure: Leveraging distributed computing and cloud solutions to handle increasing data loads and optimize real-time performance. ☁️ These principles are invaluable for anyone looking to build and scale robust ML systems. Inspired by DoorDash's journey, I’m excited to apply these learnings in my own projects. 🚀 #MachineLearning #AI #TechInnovation #DoorDash #MLArchitecture #DataScience #SystemDesign Feel free to connect and share your insights! 💬

66

10 Comments

This is the future of building RAGs. Let me introduce you to DSPy today! In terms of analogy, DSPy is to RAG as PyTorch is to DNNs. DSPy : RAG :: PyTorch : DNNs To understand DSPy, we need to grasp three key concepts: - Signatures - Modules - Optimizers In the images below I've explained everything step-by-step! 🚀 #linkdin #follow #llm #ai #artificialintelligence #machinelearning #ml #connect #connection #largelanguagemodels #dspy #trending #recuriter #recuriters

9

Question : Why does an AI product manager need to monitor #AI model❓ Once AI model is developed and deployed, the job doesnt end there. Model encodes pattern of the data it has seen during training phase. There may be several patterns or edge cases which are not part of training data. If model encounters any such patterns in testing or production phase, it will produce results which wont make sense without giving any warning about it. 👉 Data drift is a common challenge faced by AI-powered products. Over time, the distribution of input data can shift, leading to a mismatch between the model's training data and the real-world data it encounters. This can result in degraded model performance, unexpected outputs, and even complete model failure. 👉 The consequences of data drift can be severe, ranging from suboptimal user experiences to regulatory compliance issues. Product managers must be vigilant in detecting and addressing these challenges to ensure the continued success of their AI-powered offerings. 👉 Sudden changes in metrics can indicate input data drift. Additionally, monitoring the distribution of model outputs can reveal output drift, which may signal a need to retrain or fine-tune the model. Detecting data drift early is crucial. Product managers can leverage techniques such as A/B testing, anomaly detection, and continuous monitoring to identify and address these issues. Once detected, the product team can take appropriate actions, such as retraining the model, adjusting the data pipeline, or even considering a model architecture change. 💡 Quick Tip: Integrate model monitoring into your product roadmap and allocate dedicated resources to ensure the long-term success of your AI-powered offerings. Establish a comprehensive model monitoring dashboard to gain real-time visibility into the health and performance of your AI models. #artificialintelligence #productmanagement #datascience #productmanager

7

My first Kaggle Notebook! 📔✔ Problem - We need to rank various suppliers based on their overall performance/fit for the business. Approach Used - 1. Try to predict one of the given features to get some initial weights. 2. Check correlation against the target var(quality is a good option) 3. Based on model weights & correlation matrix, choose the best features & create a weighted sum(the score) 4. Aggregation is done in a way, where we add the values for features that we want to maximize(e.g. durability, ratings) & subtract/penalize the features that we want to minimize(e.g. price, delivery time) 4. Sort based on score & return the top 5 suppliers Let me know what you think about this approach, & if there is something I overlooked. 🤗 #kaggle #datascience #EDA #supplychain #business #machinelearning

10

Day 14 - 30-Day Data Engineering Interview Preparation Series PySpark (#Day14): 15 PySpark scenario based interview questions that can be relevant for a data engineering role 👇 📍 How do you deploy PySpark applications in a production environment? 📍 What are some best practices for monitoring and logging PySpark jobs? 📍 How do you manage resources and scheduling in a PySpark application? 📍 Write a PySpark job to perform a specific data processing task (e.g., filtering data, aggregating results). 📍 You have a dataset containing user activity logs with missing values and inconsistent data types. Describe how you would clean and standardize this dataset using PySpark. 📍 Given a dataset with nested JSON structures, how would you flatten it into a tabular format using PySpark? 📍 Your PySpark job is running slower than expected due to data skew. Explain how you would identify and address this issue. 📍 You need to join two large datasets, but the join operation is causing out-of-memory errors. What strategies would you use to optimize this join? 📍 Describe how you would set up a real-time data pipeline using PySpark and Kafka to process streaming data. 📍 You are tasked with processing real-time sensor data to detect anomalies. Explain the steps you would take to implement this using PySpark. 📍 Describe how you would design and implement an ETL pipeline in PySpark to extract data from an RDBMS, transform it, and load it into a data warehouse. 📍 Given a requirement to process and transform data from multiple sources (e.g., CSV, JSON, and Parquet files), how would you handle this in a PySpark job? 📍 You need to integrate data from an external API into your PySpark pipeline. Explain how you would achieve this. 📍 Describe how you would use PySpark to join data from a Hive table and a Kafka stream. 📍 You need to integrate data from an external API into your PySpark pipeline. Explain how you would achieve this. 🚨 We have launched our most affordable and industry-oriented "Complete Data Engineering 3.0 With Azure" bootcamp and ADMISSIONS ARE OPEN! 🔥 Only 5 days left and 70 seats remaining! 🚀 👉 Enroll Here (Limited Seats): https://lnkd.in/dZSAi93i 🔗 Code "DE300" for my Linkedin connections 🚀 Live Classes Starting on 1-June-2024 📲 Call/WhatsApp on this number for career counselling and any query +91 9893181542 Cheers - Grow Data Skills Shashank Mishra 🇮🇳 SHAILJA MISHRA🟢 Sahil Choudhary Shubhankit Sirvaiya Aman Kumar 🙂 #DataEngineering #SQL #Hive #Hadoop #Kafka #Pyspark #AdvancedSQL #DataScience #BigData #TechInterview #InterviewPreparation #CareerGrowth #TechSkills #SQLQueries #azure

121

5 Comments

ArmoRM : Interpretable Reward modeling approach via Multi-Objective Reward Modeling and Mixture-of-Experts Conventional Reward Models (RMs) are trained on pairwise responses to the same user request, with relative ratings indicating which response humans prefer. The trained RM serves as a proxy for human preferences. However, due to the black-box nature of RMs, their outputs lack interpretability, due to lacking intuition on why a RM thinks a response is good or not In this approach, authors propose ArmoRM (Absolute-Rating Multi-Objective Reward Model (ArmoRM) to build RMs with interpretable preferences. 𝗡𝗲𝗲𝗱 𝗳𝗼𝗿 𝗜𝗻𝘁𝗲𝗿𝗽𝗿𝗲𝘁𝗮𝗯𝗹𝗲 𝗥𝗲𝘄𝗮𝗿𝗱 𝗠𝗼𝗱𝗲𝗹𝘀 - when applying RLHF for LLM alignment, the phenomenon of reward hacking is widely observed, where the aligned LLMs generate high-reward responses (rated by the RM) that do not align with actual human preferences - For mitigating the reward hacking issue, one solution is to make the reward model more interpretable and debuggable.  - Enhancing the interpretability of RMs also allows humans to verify whether RMs have similar internal decision processes to humans when acting as proxies for human preferences 𝗧𝘄𝗼-𝘀𝘁𝗮𝗴𝗲 𝗔𝗽𝗽𝗿𝗼𝗮𝗰𝗵 i) Multi-Objective Reward Modeling - train an Absolute-Rating Multi-Objective Reward Model (ArmoRM) with multi-dimensional absolute-rating data, each dimension corresponding to a human-interpretable objective (e.g., honesty, verbosity, safety) ii) Mixture-of-Experts Aggregation of Reward Objectives - employ a Mixture-of-Experts (MoE) strategy with a gating network that automatically selects the most suitable reward objectives based on the context 𝗘𝗺𝗽𝗶𝗿𝗶𝗰𝗮𝗹 𝗥𝗲𝘀𝘂𝗹𝘁𝘀: 𝗦𝗼𝗧𝗔 𝗼𝗻 𝗥𝗲𝘄𝗮𝗿𝗱-𝗕𝗲𝗻𝗰𝗵 -  final reward model, ArmoRM-Llama3-8B-v0.1, ranks first on the leaderboard of RewardBench, a benchmark evaluating RMs for language modeling - performance of trained model surpasses the LLM-as-a-judge approach using GPT-4 and the common Bradley-Terry modeling approach with Llama3-8B or Yi-34B by a margin. 𝗥𝗲𝗹𝗲𝗮𝘀𝗲 𝗕𝗹𝗼𝗴: https://lnkd.in/e-qtGyrN 𝗖𝗼𝗱𝗲: https://lnkd.in/ecCXNKvz 𝗠𝗼𝗱𝗲𝗹: https://lnkd.in/ezQGdxC8 #generativeai #rlhf

26

الحمد لله ch8 Done : Dimensionality Reduction هل كل ما اقلل الفيتشر هاخد دقة اعلي للموديل؟ طيب هل اختلاف الموديل بنفس عدد Features هيديني نفس ال accuracy? طيب هل الفيتشر ال راحت ينفع ترجع؟ طيب لو رجعت هتكون 100% هي ال اتعملها دروب؟ استخدمت 2 Model : 1-Random Forest Classifier 2 - SGDClassifier  The author suggest that you should train your model in all the trainning set before using Dimensionality Reduction as it may make your system perform slightly worse. It also makes your pipelines a bit more complex and thus harder to maintain just like compressing an image to JPEG can degrade its quality. اتعرفت علي PCA: is a linear dimensionality reduction technique that transforms the data into a new coordinate system such that the greatest variance by any projection of the data comes to lie on the first coordinate (called the first principal component), the second greatest variance on the second coordinate, and so on. واتعرفت علي نوع جديد لل DM وهو Manifold Learning : is a type of non-linear dimensionality reduction that seeks to uncover the low-dimensional structure embedded within high-dimensional data. وشفت نوع من انواعه ال هو Locally Linear Embedding (LLE) ده اختصار للشابتر علي كاجل فيها ملاحظات والكودينج ال في الشابتر

2

Hey folks sharing a game-changing paper, "Attention is All You Need," which introduced the Transformer model. This work transformed NLP by using self-attention mechanisms to process sequences efficiently, eliminating the need for recurrence and convolutions. Key innovations: 🔍 Self-Attention: Captures dependencies across the sequence. 🔄 Multi-Head Attention: Focuses on different parts of the sequence simultaneously. 📐 Positional Encoding: Adds order information to tokens. This paper laid the groundwork for models like BERT and GPT. A must-read for anyone in AI! #NLP #MachineLearning #AI #Transformers

13

🚨 ACL 2024 paper alert! 🚨 Is your NLP model 𝙤𝙫𝙚𝙧𝙘𝙤𝙣𝙛𝙞𝙙𝙚𝙣𝙩? It might not be as robust to adversarial attacks as you think! We find that highly over/underconfident models appear robust by disrupting the search process of NLP adversarial attacks. However, an adversary with logit access can negate this "defense" just by calibrating the logits. We call this phenomenon an 𝙄𝙡𝙡𝙪𝙨𝙞𝙤𝙣 𝙤𝙛 𝙍𝙤𝙗𝙪𝙨𝙩𝙣𝙚𝙨𝙨. Check out the paper for different ways to create 𝗮𝗻𝗱 pierce such illusions at inference time! Inspired by how we induced this illusion, we further demonstrate a simple training-time modification that boosts robustness against unseen attacks. I had a lot of fun uncovering this phenomenon over the summer with Vyas Raina and Volkan Cevher, Aditya Rawal, Sheng Z., and George Karypis @ Amazon Science Vyas and I will also be at ACL in August! 📜: https://lnkd.in/gseBFDEn 🐤🧵: https://lnkd.in/gwMwS9s9

82

5 Comments

2 Reasons Why You Need to Implement RAG 1. RAG REDUCES HALLUCINATION : Hallucination, or the generation of incorrect or misleading information, is a common challenge in large language models. The RAG framework significantly reduces this issue by combining the power of retrieval and generation. The RAG's retrieval component helps mitigate this problem by grounding generated content, ensuring the responses it generates are well-grounded and therefore minimising the chances of hallucination. Example: In the financial services industry, providing accurate information on investment options is crucial because it directly impacts customers' purchasing decisions and financial well-being. RAG can help ensure that the information generated about stocks, bonds, or mutual funds is well-grounded in factual data, reducing the risk of misleading clients and potentially harmful financial decisions. 2. COST-EFFECTIVE ALTERNATIVE : Fine-tuning a pre-trained language model can be a resource-intensive process. However, RAG offers a cost-effective alternative by leveraging an existing model and augmenting it with retrieval mechanisms; this can save businesses both time and money while still achieving excellent results. Example: Banks often need to access the creditworthiness of potential borrowers. Fine-tuning pre-trained language models to analyse credit histories can be resource-intensive. RAG architecture offers a cost-effective alternative by retrieving relevant financial data and credit history information. from existing databases, combining this with pre-trained language models to generate accurate credit assessments. This approach saves both time and money while still providing reliable results for banks.

2

Why do activation functions exist in neural networks? To answer this, let's do the following experiment: Let's say we don't need activation functions. We remove all activation functions and create a simpler neural network. We have a dataset which is given to us. The dataset is a simple sinusoidal function. You construct a neural network with no activation functions. It has 2 layers and 10 neurons in each layer. You try to approximate the dataset by optimizing the weights and biases of this neural network. It will be impossible to do this. The only curve you will approximate through such a neural network is a straight line. In fact, even if you have 100 neurons, you will only get as far as a straight line. One small change will help you to approximate the underlying dataset. What's that small change? Take the outputs of all neurons and if they are negative, clip them to zero. If they are positive, you let them be whatever they are. It turns out: this small change works! This small change helps you approximate the underlying dataset. Why does it work? Because you introduce non-linearity. And when you add non-linear functions emerging from different neurons, you can approximate very complex functions. The small change you did is exactly what the ReLU activation function does. That's why you need activation functions. They add non-linearity and give neural networks the power to approximate complex functions. Without activation functions, a neural network is like a bird waiting to fly, but can't! Adding activation functions gives wings to neural networks. Here are 2 videos I recorded to understand everything about activation functions: (a) Activation functions theory: https://lnkd.in/gF7krnbM (b) Code activation functions from scratch: https://lnkd.in/g9SESjU6

42

Hey Data Enthusiasts! Excited to share my latest blog post on ensemble techniques in machine learning. 🎉 In our previous blog, we delved into the intricacies of decision trees and their power in handling non-linear data. But what if we could take it a step further? Imagine combining multiple decision trees to enhance their predictive power—this is where ensemble techniques like bagging and boosting come into play. In this blog, we’ll explore: 🔹The concepts behind bagging and boosting 🔹Detailed examples and practical applications 🔹A comparative analysis to help you choose the right technique for your data Check out the blog and let me know your thoughts and questions in the comments! Looking forward to your feedback. 📊✨ Read here: https://lnkd.in/gGrn6s5C Happy learning and exploring! #MachineLearning #DataScience #EnsembleLearning #Boosting #Bagging #AI #ML #DataScienceCommunity #TechBlog #DataAnalysis

2

Apache Airflow + Papermill = Efficiency Using Apache Airflow and Papermill, I've automated the daily updates of a risk model. With Papermill's parameterization, I've created a DAG to execute Jupyter notebooks for model training and inference. Below is an over simplified setup I'm curious to hear about your experiences with automation in AI/ML workflows. How do you handle automation challenges in your projects? #Automation #DataScience #MachineLearning #ApacheAirflow #Papermill #AI #ML #Workflow #Python #Fintech #DataArchitect

5

Post 1/2: General guidelines for your ML Technical interviews: Before Applying: Research the company and team in advance. Check out their blogs and the prospective team members. Try to align and highlight your skills and key projects that are aligned to the company or the team or the role you are applying to. For example: Let's say you have have completed a variety of ML projects in both computer vision and NLP. If you are applying for an LLM engineer role, you might want to highlight your courseworks, projects, etc. related to NLP. Try to highlight those libraries that you have used for such projects. While writing CVs, try to write concisely in 1-2 lines and use bullet points if possible, instead of writing long sentences and paragraphs. Don’t Inflate your skills and contributions: I have noticed that a few candidates mention their group projects, without having core contributions and passing it as their own, while their contributions are almost marginal. During interviews, when asked about such group projects and their details, the candidates bail out saying they don’t know because they were not working on that particular aspect and they had worked on other aspects. This may bring out a bad impression to your interviewer. Now, the question is should you include or not such projects in your CV? Personally, I think it is fine to add such projects, as long as you have good understanding and you are able to explain the underlying concepts and all the aspects of the projects, while clearly mentioning your contributions. Putting lots of jargon and keywords: Your resume might get noticed in the initial stages, when someone might be searching for potential candidates and the keywords in your resume might get you lucky. However, on the long run they might hurt The more keywords you add and highlight, there is a high chance the interviewer may assume that you have thoroughly understood or applied those concepts. Unless you are prepared to explain those concepts in detail along with their intuition and understanding, I would suggest not using unnecessary jargon. Continue reading in Post 2/2... https://lnkd.in/gQGjMGJ9 P.S. This post does not generalize the interview process and varies from interviewer to interviewer. Each organization may have their own interview policies and may have their own style of interview process. #machinelearning #interviews #interviewpreparation #interviewtips

60

2 Comments