Kouame Kouassi

Day 44: Unveiling Local Structures with Locally Linear Embedding (LLE) #100DaysOfML Challenge Today on the #100DaysOfML Challenge, we explore Locally Linear Embedding (LLE), a dimensionality reduction technique that tackles high-dimensional data by focusing on local neighbourhoods. Unlike PCA or Isomap, which target global structures, LLE excels at preserving the local geometry of the data. Imagine a dataset scattered on a bumpy surface - LLE captures how each data point relates to its immediate neighbours, rather than the entire landscape. Here's a simplified look at LLE's approach: Nearest Neighbors: Similar to Isomap, LLE identifies the k nearest neighbours for each data point. Local Reconstruction: It assumes each data point can be reconstructed as a linear combination of its neighbours. Optimizing Weights: LLE finds the optimal weights for each neighbour that minimize the reconstruction error, essentially capturing the most important local relationships. Low-Dimensional Embedding: Finally, LLE projects the data points into a lower-dimensional space while trying to maintain these reconstructed relationships. LLE proves valuable for tasks like: Visualizing high-dimensional data with localized patterns: Explore how data points cluster or interact within their neighbourhoods. Dimensionality reduction for machine learning: Prepare data for algorithms that work better in lower dimensions, while preserving local relationships important for tasks like classification. Feature extraction: Identify underlying features within the data based on the learned local reconstructions. I'm interested in applying LLE to datasets with potential local variations and structures. Let's discuss! Have you used LLE before? What other dimensionality reduction techniques are you curious about? #MachineLearning #DataScience #LLE #DimensionalityReduction #LocalGeometry #100DaysOfMLChallenge

3

Text embeddings have become crucial for transforming textual data into dense vector representations (with context), enabling efficient processing across various modalities like text, images, and audio. This article provides a good summary on some of these (not all) embeddings available for text https://lnkd.in/gKp4ta9a 1. Gensim Word Embeddings (Free) Why Use It: Specializes in NLP with open-source models like Word2Vec and FastText, promoting transparency and ease of custom training. 2. Facebook Embeddings (Free) Why Use It: Trained on extensive corpora and supports over 100 languages, making it versatile for global applications. 3. AllenNLP Embeddings (Free) Why Use It: Provides NLP-focused embeddings like BERT and ELMo, with tools for fine-tuning and visualization. 4. OpenAI Embeddings (Paid) Why Use It: Comprehensive training on massive datasets allows for effective semantic understanding, making it suitable for advanced NLP tasks. Ideal for tasks requiring zero-shot learning. 5. MultiLingual BERT (Free) Why Use It: Handles multilingual datasets effectively, offering robust support for multiple languages. Perfect for applications requiring consistent performance across different languages, such as global customer support systems. 6. DistilBERT Base Uncased (Free) Why Use It: Smaller and faster than BERT, maintaining near-comparable performance with reduced resource usage. 7. MPNet V2 (Free) Why Use It: Excels in text similarity tasks with a Siamese architecture, providing accurate similarity measurements. 8. HuggingFace Embeddings (Free/Paid) Why Use It: Offers versatility with a wide range of embeddings (text, image, audio) and the ability to fine-tune on custom data. HF also provides wrappers to almost all open sourced models. 9. Longformer Base 4096 (Free) Why Use It: Supports long texts up to 4096 tokens, making it unique in handling lengthy documents without truncation. 10. RoBERTa (2022) (Free) Why Use It: Offers updated text representations with comprehensive training on recent data. 11. GTE-Base (Free) Why Use It: Offers a good balance of performance and resource requirements with a 768-dimensional embedding. 12. GTE-Large (Free) Why Use It: Provides high-quality embeddings with more detailed representations, useful for nuanced text analysis. Best for applications requiring in-depth text understanding, such as detailed content recommendations. 13. GTE-Small (Free) Why Use It: Optimized for speed with a smaller model size, making it efficient for applications needing quick processing. Useful for real-time systems where fast text similarity searches are critical. 14. E5-Small (Free) Why Use It: Combines compact size with good performance, making it suitable for fast, general-purpose text embedding tasks.

10

There are a lot of things "old school" machine learning models (which, by the way, are still pretty powerful tools) and LLM models have in common: They can be a NIGHTMARE to deploy and maintain in a production environment! So, as you can imagine, I'm always trying to find approaches that make my life and my clients' lives easier. Recently, NVIDIA and Google Cloud released the NVIDIA NeMo framework on Google Cloud Marketplace which utilizes everyone's favorite "powerlifter" GKE for model training and distributed deployment. Three things that make this worth checking out: ☑ Creates stronger unity between GCP infra and the foundational hardware needed for scaled gen-AI systems (i.e., NVIDIA GPUs) ☑ The NeMo framework is delivered as a set of microservices allowing developers to "mix-and-match" what they need for specific use cases ☑ Greater abstraction! Has anyone had a chance to try this out yet? I'd love to hear your experiences! #gke #gcp #nvidia #gpu #llm

6

1 Comment

Many years ago, when I first encountered Data science, especially on its claims on possible sentiency, I thought it was all stupid and I even wrote LinkedIn articles as such (since then deleted those articles to avoid confusion!). Ignoring data science, I learnt Cloud technologies, Kubernetes, DevOps, Blockchain etc. Subsequently, when I happened to study data science from job search perspective, and when I started studying Deep Learning, I realized the huge potential of AI, and wrote an article avowed by its future possibilities. My intuition was proved when ChatGPT arrived. Though ChatGPT was far beyond the expectations, it was not totally surprising. However, owing to its phenomenal capabilities, the doubt remained as to whether an 'artificial consciousness' has been created - implying a greatest technological feat of humanity, or was it just a pseudo-science, involving external consciousness, such that in its absence, the AI programs will become useless, chaotic and at least not as good as they are now. I would like to continue to believe that it is not external consciousness but the concept of emergent abilities are indeed true (refer to my articles on this topic), and that we are indeed actually making digital brains. When I listen to AI leaders talking about its potential, I think they are talking from the perspective of creation of digital brain (that can even somehow harness cosmic consciousness 😀 ) and when I write here about AI, I write it from this perspective only! However, if we assume that the second premise is true - that only external consciousness is involved, there is another interesting thing I thought of writing. Long time ago, I used to frequently receive emails regarding "Leadership program for Ethical AI" from a reputed institution. I used to wonder why am I receiving promotion for this program as against any other AI program! Now, I guess there could even be a misunderstanding with regard to 'what really is ethics in AI' that many talk about! 😀 Assuming that AI is not any great technological feat of humanity and that it functions solely, by capturing / deploying external consciousness that could even exhibit 'power of consciousness', the question of ethics can have different connotation. If we go by the definition of consciousness that says emotions and feelings are integral to consciousness, and also by the tenet that consciousness cannot exist independent of a soul, the question arises how ethical is it to deploy external soul consciousness into machines? Are we torturing innocent consciousness / souls by deploying them into machine programs for our selfish material benefits? Is it sustainable at all in the long term? May be when we talk about Ethics in AI, we should be talking about this! Not about using AI ethically - that is completely under our control. I continue to hope that we have indeed achieved a greatest scientific milestone, and hence there is no sufferings involved in this AI phenomenon.

4

2 Comments

Discovering the world of data science and machine learning doesn't always require enrolling in specialized courses or earning diplomas or even masters. Many have embarked on this journey armed with nothing but curiosity and gained essential technical knowledge from a few well-chosen books. Knowledge knows no bounds, and with the right resources, anyone can dive into the exciting realm of data science and machine learning. 📚💻 #DataScience #MachineLearning #SelfLearning

3

1 Comment

Are words actually that Powerful Let me take you down the line to my junior secondary days.   It was time to pick prefects, so the selected few were called to the principal's office. We started the interview, and then we were asked: "If you were to be a prefect, which one would you be?" From the left, people started answering, then it came to my turn.   OK, some backstory: In junior secondary school, I was almost always the latest to come. I think I can even count how many times I attended assembly 😥😥.   Okay, let's continue: When it came to my turn, I said, "I would love to be the punctuality prefect." No, I was not prophesying about my life. The environment was quite tense, so I just wanted to lighten the mood a bit 😁.   Well, I was not chosen for any prefectship post due to some factors.   Fast forward to secondary school—SS3 to be precise. We were called for another interview, and we were told to choose the positions we wanted. Guess what I said?   Punctuality prefect.   No, nobody laughed this time. The whole panel nodded their heads in affirmation; it just had to be me. Even before the choosing of prefects, it was so clear that it would be me.   What happened? Yes, I believe words are powerful, but other things played a factor.    I was in boarding school at this point, so I knew I had to take on responsibilities myself. I did not like bathing with others, so I had to wake up much earlier than others. I did not really like crowds, so I went to school much earlier. I had other commitments apart from school—competitions and other things I had to learn—so I knew I had to make a plan. This habit accumulated, and all of a sudden, it seemed I was just the best at managing my time.     What is the point of the whole story?    Time management starts with a purpose. You must have a reason why you want to manage your time. The reason we manage money is so we can use it for something more important. One reason you might not be managing your time well is that you do not have a vision. You do not have something you are pursuing.   Know your purpose, know the steps to achieve it, know the disadvantages of not following those steps and follow through.   That is not all, I also said I wanted to be the sports prefect—if you know, you know (IYKYK). Let's see how that goes 😂.  

61

8 Comments

Week 4 of DataTalksClub LLM-zoomcamp was rich in content. This week the focus was: 1) Setting up an LLM powered Streamlit app. The app provides users the capability to ask questions. We capture feedback from the user as part of the monitoring workflow. All data is stored and in a Postgres database. A visualisation dashboard is used for monitoring model performance (Grafana Labs). 2) We also explored how to use cosine similarity to deliver a verdict on how well an LLM is responding to user queries. A large language model was then used as a secondary evaluation method to guage quality of our user facing model. As always a big thanks to Alexey Grigorev and the team for making these available to many of us who enjoy collaborative learning and hands on projects. https://lnkd.in/drN-v7xv

11

1 Comment

🔥 Hot off the press... Ingest your PDFs into RAG pipelines - using the power of an LLM - with LlamaIndex's LlamaParse and KDB.AI! #LlamaParse is a document parsing tool that revolutionizes how you interact with your complex digital documents. Whether you're dealing with PDFs, PowerPoint slides, Word documents, or more, LlamaParse has got you covered. 📄 Supported Formats: PDF, .pptx, .docx, .rtf, .pages, .epub, and more... 🔄 Transform to: Markdown or plain text 🛠️ Extracts: Text, tables, images, graphs, comic books, and mathematical equations. 💡 Customized Parsing: Thanks to its LLM capabilities, just prompt LlamaParse like you would an any LLM. Add context, dictate output style, or even ask for language translations and sentiment analysis as you parse! 📊 JSON Mode: Dive deep into document structure with metadata-rich JSON outputs—ideal for custom RAG applications and improved citations. And as always, we conclude with a code walk-through 👨‍💻 Read the full article here: https://lnkd.in/gG57r789

88

1 Comment

This month Microsoft made GraphRAG available on github (https://lnkd.in/g3Jc4fj7), a graph-based approach to retrieval-augmented generation (RAG) that enables question-answering over private or previously unseen datasets. Some key difference between 𝗚𝗿𝗮𝗽𝗵𝗥𝗔𝗚 and 𝗡𝗮𝗶𝘃𝗲 𝗥𝗔𝗚. Information Structure: - GraphRAG: Uses a knowledge graph to represent structured information from documents. - Naive RAG: Relies on simple vector representations of text chunks. Retrieval Approach: - GraphRAG: Leverages graph-based retrieval, considering relationships between entities. - Naive RAG: Uses vector similarity search to find relevant text chunks. Global Question Handling: - GraphRAG: Excels at answering global questions about the entire dataset. - Naive RAG: Struggles with global questions, often providing incomplete or misleading answers. Content Overview: - GraphRAG: Provides hierarchical summaries of the dataset without specific queries. - Naive RAG: Requires specific queries to retrieve information. Comprehensiveness: - GraphRAG: Offers more comprehensive and diverse answers. - Naive RAG: Limited to information in top-k similar chunks. GraphRAG's structured approach allows for more nuanced and complete information retrieval, making it particularly effective for complex queries and dataset exploration. #GraphRAG #Microsoft #MachineLearning #ArtificialIntelligence #QuestionAnswering #KnowledgeGraph #DataRetrieval #InformationRetrieval #AI #DataScience #NLP #Innovation #TechNews #DeepLearning #TechTrends

44

Andrej Karpathy - Storyteller AI Large Language Model (LLM). 📚 Syllabus: - Chapter 01 Bigram Language Model (language modeling) - Chapter 02 Micrograd (machine learning, backpropagation) - Chapter 03 N-gram model (multi-layer perceptron, matmul, gelu) - Chapter 04 Attention (attention, softmax, positional encoder) - Chapter 05 Transformer (transformer, residual, layernorm, GPT-2) - Chapter 06 Tokenization (minBPE, byte pair encoding) - Chapter 07 Optimization (initialization, optimization, AdamW) - Chapter 08 Need for Speed I: Device (device, CPU, GPU, ...) - Chapter 09 Need for Speed II: Precision (mixed precision training, fp16, bf16, fp8, ...) - Chapter 10 Need for Speed III: Distributed (distributed optimization, DDP, ZeRO) - Chapter 11 Datasets (datasets, data loading, synthetic data generation) - Chapter 12 Inference I: kv-cache (kv-cache) - Chapter 13 Inference II: Quantization (quantization) - Chapter 14 Finetuning I: SFT (supervised finetuning SFT, PEFT, LoRA, chat) - Chapter 15 Finetuning II: RL (reinforcement learning, RLHF, PPO, DPO) - Chapter 16 Deployment (API, web app) - Chapter 17 Multimodal (VQVAE, diffusion transformer) #llm #AI #datascience

1

While exploring various transformer-based architectures, I stumbled upon a great blog titled: "Understanding Large Language Models" by Sebastian Raschka, PhD The blog provides a concise overview of the key papers that have shaped the field of large language models (LLMs) in recent years. Sebastian mentioned the papers to read in chronological order to understand the evolution of LLM architectures and training techniques. The post starts with the original transformer paper from 2017 and covers influential works on attention mechanisms, layer normalization, and the bifurcation of LLMs into encoder-style models for predictive tasks and decoder-style models for generative tasks. It also discusses the ULMFiT approach for finetuning language models on specific tasks. The blog concludes by covering the topic of alignment, which aims to steer LLMs towards intended goals and interests. Key papers discussed include the InstructGPT approach that led to ChatGPT and a survey on parameter-efficient finetuning methods. Overall, the blog provides a well-curated reading list for researchers and practitioners looking to understand the rapid progress in LLMs over the past few years. Here's the link: https://lnkd.in/dEuq4e8p Happy Learning! #LLMs #LargeLanguageModels #GenAI #NLP #NaturalLanguageProcessing #TransformerArchitecture #ScalingLaws #ParameterEfficientFinetuning #LanguageModelAlignment #AIResearch #MachineLearning #ComputerScience

15

Is GraphRAG worth the hype ? This weekend I explored much-trending "GraphRAG" and analysed feasibility to include it in a LLM production pipeline. While GraphRAG is a substantial contribution to better retrievals, I still am concerned about its inefficiency and cost implications. - Implementing GraphRAG can be prohibitively expensive. For example, processing a single book of 32,000 words through an LLM to utilize GraphRAG's functionalities can cost about $10. Scale this to thousands of documents, and the financial burden becomes unsustainable for most operations. - Additionally, because information is stored in nodes, intricate details, specifically reasoning part, can be lost, leading to incomplete data retrieval. This lack of detail can significantly impact applications where nuance and completeness are critical. - Considering these factors, the tangible improvement in system performance ,in practice, is often marginal—around 5-10%. This leads us to question whether the effort and resources required to implement GraphRAG could be better spent enhancing existing systems. Alternatives like refining embedding models or improving data curation strategies might offer similar benefits at a fraction of the cost and complexity. Thoughts ? Alok Abhishek Hamza Farooq Victor Calderon, Ph.D. Tiffany Teasley Vasanth Raghu Nair

24

12 Comments

Learning to code is dead. Learning what to code is the future: With AI chatbots like ChatGPT, Copilot and Gemini you can generate code to solve problems instantly. So whilst you still need to understand & edit code generated by AI, you can devote more energy to deciding what problems you want to solve. At the end of the day, this is what businesses hire Data Scientists for - to solve their problems. Not to piss about in VSCode for hours. This is what makes Tailored Projects unique - great care is placed on finding a problem relevant to the industry you want to work in, and then executing it to perfection. Focusing on the right projects is what makes you stand out to employers, especially if you have limited Data Science experiment. Whilst if you only work on beginner datasets from Kaggle, your resume isn't going far. So first focus on solving problems relevant to the jobs you want, then worry about how you'll code the solutions. I'm curious, what do you guys think of this approach? Let me know in the comments below ⬇

6

[LLM.int8() tech summary] Ever wondered how LLM int8 inference in HuggingFace/bitsandbytes works or just in general? How can you run a 175B param models with no degradation while using only [-128, 127] (int8) range? First of all for some background context that helps appreciate the fact that this even works: LLMs nowadays are mostly trained in fp16/bf16/fp32 mixed precision. That means they're "used to seeing" numbers up to e.g. 3.39*10^38 (in bf16), but during inference you force the activations & weights into the [-128, 127] range, and things...just work? :) The answer is: well yes, but it's not trivial to pull it off, esp. for LLMs that have a lot of params. The main reason why previous quantization methods didn't work in the large scale regime is that fact that around ~7B param threshold (6.7B in the paper) a phase shift happens. What happens is an emergence of outlier features that are e.g. 20x+ larger than other features in other dimensions. And they're extremely sparse (e.g. 7 out of 4096 features are affected). If you try something like absmax quantization those features will make all the other features collapse basically to zero. That happens because we divide the vector by the absmax value and then multiply by 127 & round to nearest int in order to bring them into int8 range. If most numbers were around e.g. 0.5-3.5 and your outlier is 70.3 you crush those other features after the quantization. So what's the solution? There are 2 parts to it. First of all we mostly care about matmul operation as that's where most of the flops & params are. How matrix multiplication computation is done on the actual GPU is in a embarrassingly parallel fashion where you do dot products between rows and columns at the same time. (obv. this is a simplification, there is tiling and there is caching to GPU's shared memory (SRAM) but this level of understanding is good enough for the sake of my explanation). Back to what the 2 parts are: 1) We first do the so-called vector-wise quantization. Where you absmax normalize all rows & columns of your 2 matrices independently (as opposed to having e.g. a single scalar for the whole matrix). You then convert them into the int8 range, you do the dot product and accumulate into int32 (so you don't overflow). To dequantize you just undo the normalization (elementwise product with the outer product of column and row scaling coefficients you previously computed). The problem is this approach works only until we hit ~2.7B param scale! 2) The second crucial part is "mixed-precision decomposition", i.e. do the same computation except that this time you split your rows & columns such that the ones that have outliers (big numbers) are extracted and they undergo 16 bit dot-product - otherwise 8 bit. Most dot products will be 8-bit (as outliers are extremely sparse) so we still get ~2x memory saving! More in the comments! :)

364

19 Comments

Excited to launch my first blog on document chat with advanced RAG! ⛃🔗 Discover the power of Retrieval-Augmented Generation (RAG) in my latest project. From retrievers that pinpoint the most relevant docs to stuff chains that optimise and rank results for top-notch accuracy, all turbocharged by the FAISS vector database! 🧠💥 Ready to dive into these game-changing concepts? Check out my blog post and stay tuned for more exciting insights on LangChain and beyond. This is just the beginning of an incredible journey! 😊 Blog Post: https://lnkd.in/gYbBMSfi #AI #MachineLearning #NLP #DocumentSearch #RAG #FAISS #LangChain #Innovation #DataScience #BlogPost

20

Just have an experience with streamlit. I've built a document summarizer 📄🙌... The LLM i used is PaLM 2 and for the frontend obviously Streamlit. it's quite interesting and easy to test the working of ML models 🧑‍💻 #DataScience #NLP #PaLM2 #Transformers #GenAI #Streamlit #Document

28

6 Comments

Can Large Language Models put 2 and 2 together? Probing for Entailed Arithmetical Relationships From the paper: "We empirically demonstrate that although LLMs make steady progress in knowledge acquisition and (pseudo)reasoning with each new GPT release, their capabilities are limited to statistical inference only." #LLMs #AI #Reasoning #Knowledge #Approximation Let me know what you think about the paper! Yours, Salman   https://lnkd.in/dyVQ3c9D PS. Thanks to Walid Saba for sharing the paper 🙏

3

3 Comments