Natalia Neverova

Were you unable to attend this year’s 🔥 ICLR 2024 🔥 at the beginning of May? As part of our work at Echo Analytics , we went on a trip to get you some surprising findings or trends you might have missed ⬇ ✨ On Large Language Models✨ ➡Using LLMs to check and correct their own outputs works well to improve their accuracy: https://lnkd.in/ebGGPtjz https://lnkd.in/e4SveSxA https://lnkd.in/ehYagWc8 https://lnkd.in/ekiNkDYa ➡A multi-agent role-based collaborative framework for LLMs: https://lnkd.in/eEWXWNPc ➡In context Learning in LLMs (most probably) guesses the rules of the task on the fly, and doesn't just recognize the task from a set of previously seen tasks: https://lnkd.in/eRH_vHnK ➡Contrastive instruction tuning, where we maximize (“probability of the right response” - “probability of a response to the opposite instruction”) outperforms vanilla instruction tuning: https://lnkd.in/ekByaUrH ➡Generating task demonstrations with LLMs for LLM few shot prompting is (almost) as good as human-written demonstrations: https://lnkd.in/edaxiEdY ✨ On Time series modeling✨ ➡Time series modeling (even with transformers) does improve if the input series is decomposed: https://lnkd.in/eYBdh8wm https://lnkd.in/eE9Rp7fV https://lnkd.in/eRQbJvGU ➡Foundation Models for time-series are (surprisingly) a valid concept: https://lnkd.in/em49ZsDF https://lnkd.in/ex3VWdAc ✨ Miscellaneous highlights ✨ ➡ Subsampling Data from a data set (e.g. 60%) by only keeping examples that are harder (not hardest) according to a silver labeler can significantly improve results: https://lnkd.in/eRCKB6T6 ➡Training S4 models from scratch leads to better results for S4 than Transformers. BUT, adding Self-Supervised Denoising training for both evens out the results, even when it’s only done on the (small) dataset of the target task: https://lnkd.in/eH6d5xnu It’s also worth mentioning that Kingma and Welling’s work “Auto-Encoding Variational Bayes” recieved the “Test of Time” Award, a well deserved consecration for Variational Autencoders over the last decade 🎊

60

1 Comment

CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts https://lnkd.in/gb78Y4bn In the latest foray into the world of multimodal language models, researchers have unveiled an intriguing new approach they've dubbed "CuMo." This novel technique aims to harness the power of Mixture-of-Experts (MoE) architectures, incorporating Co-upcycled Top-K sparsely-gated MoE blocks into the vision encoder and the multi-layer perceptron connector of multimodal LLMs. The incorporation of these MoE components is a bold move, promising to enhance the capabilities of multimodal LLMs in a significant way. However, the researchers didn't stop there. They've also adopted a three-stage training approach that employs auxiliary losses to help stabilize the training process and ensure a balanced distribution of workload across the expert modules. One of the most noteworthy aspects of this project is that CuMo has been exclusively trained on open-source datasets, a commendable decision that promotes transparency and accessibility in the field of AI research. Overall, this project presents an exciting exploration of MoE architectures in the context of multimodal language models. While the true impact of CuMo remains to be seen, the innovative approaches employed and the promising early results make this a development worth keeping an eye on in the rapidly evolving field of AI.

3

A couple of quick experiments with Lens Studio 5.0's new GenAi Suite. ML models trained directly in Lens Studio!  Low poly effect inspired by recent visit to ACMI Beings by Universal Everything, and Lego effect (needs more work!) inspired by The White Stripes - Fell In Love With A Girl directed by Michel Gondry. Would be great if you could combine text and image prompting, for greater control, but overall pretty exciting to see all these tools in one easy to use framework. Link to Lego effect post: https://shorturl.at/5u0zT

5

🔥ICML-2024 Oral 📢🔥for our work "PRISE: LLM-Style Sequence Compression for Learning Temporal Action Abstractions in Control", one of the 🔥1.5% of submissions🔥 honored this way! Paper 📰: https://lnkd.in/ggAUMMA7 Code 👩‍💻: https://lnkd.in/gsWWVpg8 In a nutshell, PRISE tackles the issue of large planning horizons in decision-making, which complicates training models for robot control. 💡The paper's key insight is that large language models (LLMs) have been secretly using a suitable approach to this issue for a long time.💡 PRISE can be understood via the following analogy... 👇 LLMs are models for text generation. Text is a sequence of byte values -- let's call these 256 possible values "codes" -- but handling it byte-by-byte is too expensive. Instead, LLMs treat it as a sequence of less granular units, 🔶 tokens 🔶, each of which corresponds to a sequence of codes. LLMs learn to generate token streams and decode them into (much longer) codes streams. Thus, tokens are temporal abstractions for text. Continuous control 🤖, like text generation, involves making a sequence of highly granular decisions ("which control inputs to send to the robot's motors in the current state?") that result in a trajectory -- a sequence of state-action pairs. Like LLM training, behavior cloning (BC), a staple of pretraining for robotics, also faces major challenges when learning from granular trajectories, as this involves long planning horizons even for basic tasks. Temporal abstractions in the form of tokens decodable into blocks of several low-level decisions seem like they could be helpful here... 👉 Our PRISE (PRImitive Sequence Encoding) method builds on this intuition by transplanting LLM-style tokenization to BC-based learning to control. LLM training commonly constructs a mapping from tokens to code sequences using a method called Byte Pair Encoding (BPE). BPE can be applied to code sequences over any finite codebook, but state-action pairs in control trajectories are from a continuous space. To overcome this problem, we observe that robotic manipulation policies typically consist of switching among a handful of low-level primitives: free-space moves (FSM), lowering the end-effector (LEEF), etc. PRISE learns a "codebook" of these primitives using VQ-VAE and labels every state-action pair in our training set's trajectories with the code of the primitive that generated it. 🎉Voila, our trajectories are ready for tokenization with BPE! 🎉 The "policy tokens" BPE constructs are sequences of primitive policy applications, e.g., FSM-FSM-LEEF. Our experiments show that training hierarchical state-to-token and token-to-control policies over tokens and primitives induced by PRISE results in significantly higher task success rates than using either raw BC or BC with action chunking (https://lnkd.in/gnuemsgn). Happy to chat about the details (oh yes, there are many more details!) at ICML-2024 in Vienna! ✈

59

5 Comments

A very short backlog on AGI. 1. Implement hypernetwork in llama3 2. Implement queue Augmented llama 3. with push, pop, nop. how : use bidirectional encoder like Deberta for getting embedding of the queue, and concatenate to input. and output layer for action of push, pop, nop 3. neural projection operator in transformer block, for learning physics constraints on embedding. 4. Pretrained using PPO. Sebastian Raschka, PhD how much effort is this?

18

2 Comments

🌟Announcing ROLEBENCH, a framework designed to evaluate the performance of Role-Prompting across various datasets and Large Language Models (LLMs). The results across some benchmark datasets demonstrates better performance than results mentioned in model cards [Refer to Github Repository for results]. A free-tier colab notebook to test ROLEBENCH on Llama 3-8B: https://lnkd.in/g4zDRHZf Role-Prompting assigns specific roles to the AI, such as acting as a teacher, a historian, or a technical expert, to see how well the model can adapt its responses based on these roles. ROLEBENCH assess how well different LLMs perform when given these role-specific prompts. This tool will offer valuable insights into the capabilities and limitations of current LLMs. ROLEBENCH currently supports Llama 3-8B, Phi-3-mini, Mistral-7B and Gemma-7B across some benchmark datasets. Github : https://lnkd.in/gbsax-hX #AI #MachineLearning #RolePrompting #AIResearch #LLM #DataScience

80

4 Comments

Spatial Reasoning did not ‘emerge’ spontaneously in Large Language Models (LLMs) the way so many reasoning capabilities did. Humans have specialized, highly capable spatial reasoning capabilities that LLMs have not replicated. But every subsequent release of the major models — GPT, Claude, Gemini- promises better multimedia support, and all will accept and try to use uploaded graphics along with texts. #llms #chatgpt4 #gemini #llmops #aiml #ml https://lnkd.in/dFSWKqKs

2

Just read Farquhar et al's Nature paper on detecting hallucinations in large language models using semantic entropy. Interesting result on quantifying uncertainty in LLMs. When you ask an LLM a question, its answers may look correct, but be made up - a confabulation. For example, the LLM may answer incorrectly that the STARD10 protein is a "negative regulator of the mTOR pathway" instead of its true role as a lipid transfer protein. When an answer is confabulated, multiple subsequent generations will have different meanings. For example, a second answer of the LLM may incorrectly say that STARD10 is a "negative regulator of the meiotic recombination process". The variability in meaning can be quantified using semantic entropy, which measures variability over multiple generated answers. Higher entropy suggests greater variability and potential confabulation. The LLM can then refuse questions with high semantic entropy. To calculate semantic entropy, you generate 5-10 answers. You then cluster the answers based on semantic meaning using bi-directional entailment. Two answers are part of the same cluster if they entail - or imply - each other (e.g. "Joe buys a red apple" implies "A red apple was bought by Joe" and vice-versa). The entailment is calculated using another, possibly less powerful LLM, so it's an interesting applications of using language models to correct other language models. After clustering, the probabilities of each cluster are calculated from the output probabilities of the LLM, and then the resulting entropy. Thresholding the entropy can classify a question as likely to produce confabulations with an AUROC of around 0.75. Link to paper below.

63

2 Comments

An easy, intuitive video to understand RESNET. Throughout my career I find the coolest thing about maths is that it all come back to fundamental ideas. Neural Nets are just approximation functions RESNET has an equivalence in Numerical Integration as a EULER step https://lnkd.in/gyNidB5A

4

After reading Anthropic and OpenAI success stories with sparse autoencoders for interpretability of text, I was curious about its application to images... After a quick search, I found out that Hugo Fry wrote a blogpost in LessWrong: https://lnkd.in/dA33fWqE which also provides some playground! Of course part of the success is due to the ImageNet structure itself (as many papers out there) but still is an interesting exercise. It enables lots of experiments, and I already can think about practical usage for improving image search in marketplaces. #interpretability #image #CLIP

7

3 Comments

You Only Cache Once(YOCO): Decoder-Decoder Architectures for Large Language Models The KV (key-value) cache improves inference speed by eliminating the need to re-encode historical data for each token. However, as the number of tokens processed increases, the memory demand of the KV caches also rises, making the operation of large language models heavily dependent on available GPU memory. To address this, the authors of the paper introduce a decoder-decoder architecture named YOCO, which efficiently caches KV pairs a single time. 🔍 Key Highlights: Innovative Caching Method: YOCO introduces a novel decoder-decoder architecture that caches key-value pairs only once. This method significantly reduces the GPU memory requirements, which is a breakthrough for running large models efficiently. Performance Efficiency: The architecture not only saves memory but also manages to deliver competitive performance, rivaling traditional Transformers across various benchmarks. This makes YOCO a potent tool for complex language tasks. Scalability: It scales beautifully with increased model size and training tokens, demonstrating robustness and adaptability, crucial for future AI models. 📉 Challenges and Considerations: Training Complexity: The architecture, while efficient, is complex and might pose challenges during implementation, requiring precise tuning to achieve the best results. Broad Applicability: Extending the architecture to applications beyond language modeling, such as multimodal tasks, remains a challenge and an exciting area for future research. 🔮 Future Directions: The paper suggests potential adaptations and optimizations that could pave the way for YOCO's integration into more mainstream AI applications, promising substantial reductions in computational costs. I encourage everyone interested in the latest AI technologies to delve into this paper. The innovative approaches and potential applications of YOCO could be transformative for the field. Check out their Paper link: https://lnkd.in/gp5HjWg5 Github link: https://lnkd.in/g-QKEJyX 👀 What are your thoughts on these advancements? Could YOCO be the future of efficient AI modeling? #AI #MachineLearning #LanguageModels #Innovation #TechnologyNews #YOCO

13

If you are benchmarking Computer Vision models and you use random inputs because you don’t have input data yet, be careful! If your model includes preprocessing, you may be getting higher times. Why? Usually the most expensive operation in preprocessing is the resize, and there’s a catch: It takes more time to resize a completely random image (noise) than a natural image! For example, here it is an example with the Pillow Python library: https://lnkd.in/d3RHYxEe In the notebook, you see that it takes almost twice the time for the random image when compared to a natural image or a monochromatic one. The solution is always use images that follow the expected distribution in production. For example in Nvidia Model Analyzer there is the “input-data” option (https://lnkd.in/dum_eRXF) where you can set the directory of input images or a JSON with the base64 string of the images. The exception is if you are analyzing just the model and you are happy with providing random images at the exact size the model is expecting (i.e. 224x224), so there is no resize involved.

8

Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? Just came across this paper discussing the challenges of integrating new factual knowledge through fine-tuning in large language models (LLMs). The research indicates that LLMs have a harder time learning new information through fine-tuning, absorbing it at a slower pace compared to pre-existing knowledge. Intriguingly, once the model does grasp the new knowledge, it tends to hallucinate. Seems like a good read 🚀 #AI #LLMs #ai #machinelearning

16

DRaFT+: a simple text-2-image alignment technique that actually works

24

Very happy to present our latest paper: Transformer Layers as Painters Through our paper, we aim to understand the flow of information in a pretrained transformer. We present a series of experiments for both decoder-only and encoder-only frozen transformer models. **Note that we do not perform any kind of fine-tuning on these pretrained models.** With a series of experiments done on a diverse set of datasets with both types (decoder-only e.g. Llama, Mistral like models) and (encoder-only e.g. BERT), we answer the following questions: 1. Do layers speak the same language? 2. Are all layers necessary? 3. Are the middle layers all doing the same thing? 4. Does the layer order matter? 5. Can we run the layers in parallel? 6. Does the order matter for some tasks more than others? 7. Does looping help parallelized layers? 8. Which design variants are least harmful? I would provide the link where for the full summary, and the paper in the comments section. This was an fun collaboration between Sakana AI and Emergence, and for me I take immense pride in collaborating with Marc Pickett Llion Jones and Qi sun. Enjoy reading the paper! 🍻

172

4 Comments

In-Context Learning at Extreme Scales: My Thoughts As large language models (LLMs) continue to expand their context capacities, the domain of in-context learning (ICL) at extreme scales has piqued my interest. A recent paper by researchers at Carnegie Mellon University and Tel Aviv University provides a detailed exploration into this area, unveiling both impressive capabilities and notable challenges. 🔍 Key Insights: Performance Improvement: The study demonstrates that ICL can improve significantly by integrating hundreds to thousands of examples into the context window—far beyond traditional few-shot settings. This is particularly impactful in complex tasks like 77-way banking intent classification, where long-context ICL matches or surpasses traditional finetuning. Input Sensitivity: Long-context ICL shows reduced sensitivity to input shuffling compared to shorter contexts. However, organizing examples by label dramatically reduces performance, emphasizing the need for diverse context configurations. Source of Gains: Performance gains primarily arise from accessing more relevant examples, rather than aggregating knowledge across many examples during encoding. 📉 Challenges & Drawbacks: Sensitivity to Example Order: Despite reduced sensitivity, the performance still varies with input order, indicating potential instability in practical scenarios. Performance Saturation: The models often do not fully leverage their extended context capacities, showing performance saturation before reaching the maximum context length. Computational Cost: The increased computational cost associated with using large context windows could limit practical deployment, particularly in resource-constrained environments. Efficiency Issues: There is a noted inefficiency in how models utilize the encoded information within large datasets, suggesting that current architectures may need refinement to better use available data. 🚀 Future Directions: The paper suggests several exciting avenues for future research, including optimizing the trade-offs between computational costs and performance, exploring model adaptation strategies, and refining ICL approaches for specific applications or datasets. As we push the boundaries of what LLMs can achieve, effectively leveraging their full context could unlock new capabilities. This area is ripe for further exploration and could lead to major advancements in how we deploy AI. For those at the cutting edge of language model development, I highly recommend this paper for its thorough analysis and forward-looking perspective. Check out the full work at https://lnkd.in/d7AmGh4y. What are your thoughts on the future of in-context learning at such scales and the challenges discussed? #AI #MachineLearning #DeepLearning #LanguageModels #InContextLearning

17

Exploring the Pros and Cons of Multi-token Prediction in Language Models : A recent study, "Better & Faster Large Language Models via Multi-token Prediction," by researchers from FAIR offers a fascinating look at how predicting multiple tokens simultaneously can significantly enhance the efficiency of large language models like GPT and Llama. The researchers of the paper claim that the models developed via the technique discussed in the paper solves 12 % more problems on HumanEval and 17 % more on MBPP than comparable next-token models Key Highlights: Enhanced Efficiency: The multi-token prediction method boosts sample efficiency without additional training time or memory overhead, presenting a compelling case for its adoption in larger models. Improved Performance: This technique has shown to dramatically improve performance, particularly in complex coding tasks and natural language processing, consistently outperforming traditional models. Pros of Multi-Token Prediction: Scalability and Versatility: The method scales impressively with larger models and demonstrates benefits across various data types, including both code and natural language. Inference Speed: Models trained with this method can be up to 3× faster at inference, making them not only more intelligent but also quicker. Potential Challenges: Complexity in Training Dynamics: The method involves multiple output heads which could complicate model training dynamics and require careful tuning. Optimal for Large-Scale Models: While the benefits are clear for large models, smaller models may not see as significant improvements, possibly limiting its applicability in resource-constrained projects. Generalization across Tasks: The effectiveness across all types of tasks is not fully explored, which may limit its applicability in certain scenarios. Future Directions: The paper discusses promising future research paths, including automating the selection of the number of tokens to predict and optimizing vocabulary sizes for multi-token prediction. This approach represents an exciting advancement in AI training techniques. However, it also poses new challenges that necessitate further exploration to fully understand and mitigate potential drawbacks. Paper link : https://lnkd.in/d69Je8wU #AI #MachineLearning #LanguageModels #Innovation #Research

14