Mark Papermaster

100,000 #H100 Clusters: Power, Network Topology, Ethernet vs InfiniBand, Reliability, Failures, Checkpointing 💡Frontier Model Scaling Challenges and Requirements, Fault Recovery through Memory Reconstruction, Rack Layouts There is a camp that feels #AI capabilities have stagnated ever since GPT-4’s release. This is generally true, but only because no one has been able to massively increase the amount of compute dedicated to a single model. Every model that has been released is roughly GPT-4 level (~2e25 FLOP of training compute). This is because the training compute dedicated to these models have also been roughly the same level. In the case of Google’s Gemini Ultra, NVIDIA Nemotron 340B, and Meta LLAMA 3 405B, the FLOPS dedicated were of similar magnitude or even higher when compared to GPT-4, but an inferior architecture was utilized, resulting in these models falling short of unlocking new capabilities. While OpenAI has gotten more #compute, they have mostly directed it at bringing smaller, overtrained, cheaper to inference models such as GPT-4 Turbo and GPT-4o. OpenAI admits they only just started training the next tier of model recently. The obvious next step for #AI is to train a multi-trillion parameter multimodal transformer with massive amounts of #video, #image, #audio, and text. No one has completed this task yet, but there is a flurry of activity in the race to be first. Multiple large AI labs including but not limited to OpenAI/Microsoft, xAI, and Metaare in a race to build #GPU clusters with over 100,000 GPUs. These individual training clusters cost in excess of $4 billion of server capital expenditures alone, but they are also heavily limited by the lack of #datacenter capacity and power as #GPUs generally need to be co-located for high-speed chip to #chip networking. A 100,000 GPU cluster will require >150MW in datacenter capacity and guzzle down 1.59 terawatt hours in a single year, costing $123.9 million at a standard rate of $0.078/kWh. A very big thank you again to Dylan Patel , Daniel Nishball, CFA and SemiAnalysis for the full article with more background, charts and insights via the link below 💡🙏👇 https://lnkd.in/eM_CBmM8 #semiconductorindustry #semiconductors #semiconductors #technology #tech #innovation #artificialintelligence #foundry #tsmc #datacenter #it #ai #cloudcomputing #computing #computer #hpc

51

2 Comments

https://lnkd.in/gppV4nKd very interesting news around glass substrates for semiconductor packaging I was speaking with a friend today when it came out that while interposer based solutions are ruling the high end right now at NVIDIA and AMD (and Intel Corporation has its own #EMIB technology which is also on offer to Intel Foundry customers for packaging-only - customers can buy wafers from other foundries and package at Intel), the cost will be prohibitive for many applications. Biggest market is in substrate based multi die packaging (in fact I spoke with Blue Cheetah Analog Design, Inc. CEO sometime ago and he mentioned the same. Eliyan Corporation is also trying in the same space) and interposer is a fraction of that market size. but organic substrate has it's limitations glass will mitigate several problems enabling smaller bump and route pitches and closing the gap with interposer based designs https://lnkd.in/gppV4nKd

6

How to improve spectral efficiency and overall performance per watt? In this blog, the approach of accelerated computing and how it applies to RAN and spectral efficiency is presented. Many wireless algorithms and techniques exist that improve wireless performance and hence spectral efficiency; however, they need lots of compute power. This is where Aerial on NVIDIA Accelerated Computing solves these challenges. The outcome is a better TCO for the telco. Thanks to the team on putting this together. Jim Delfeld Yan Huang Rajesh Gadiyar CC Chong Emeka Obiodu, PhD https://lnkd.in/g6fsFPap

49

2 Comments

The attacched report talks about foundries ranking. In recent months I have been asked which foundries I would choose for a new chip design. While TSMC remains a popular choice, exploring options like Samsung Semiconductor, GlobalFoundries, and Intel Corporation (Missing from the attached list) can offer unique benefits ( specialized processes in III-V, cost models, and scheduling). Today, designing for multiple foundries has been simplified by leveraging IP from suppliers like Cadence Design Systems, Synopsys Inc, and other. #ChipDesign #Foundries #TechnologyIndustry

11

2 Comments

Tech company earnings calls happened this week and NVIDIA, Intel, and AMD were bullish on the impact their AI chips are making on bottom lines. AI solutions for data centers are impressive, and the data center is currently a large market but analysts at OMDIA, Gartner, and others believe this market will pale in comparison to the trillion-dollar potential of Edge AI. Innovative companies focused on developing chips for this space believe AI at the Edge is going to be a revolution that impacts people around the world and I wholeheartedly agree! I envision a world where agricultural robots harvest more abundant crops effortlessly and farmers from different regions can communicate using AI-enabled voice translation on their mobile phones. I envision their children accessing learning through connected, affordable, low-powered, AI-enabled laptops that run on solar. Edge AI has the potential to improve the lives of billions by bridging the education gap and giving people access to affordable or no-cost tools to lift them up and out of poverty. This is why I believe so strongly that Edge AI is part of responsible, global AI. One company that is SO close to bringing this vision to life is Blaize, which raised an additional $106 million this week. CEO Dinakar Munagala and his co-founders Satyaki Koneru and Ke Yin are all experienced in the world of high-powered compute applications for the data center, which ultimately inspired them to form Blaize. They realized that building technology that affects people’s DAILY lives is where great value creation and impact will happen when compared to say, helping the enterprise create better chatbots. Munagala describes that Edge AI requires a radically new approach to compute, one that factors for low power envelopes, near-zero latency, and hyper-efficient compute. He also believes customers shouldn’t be encumbered by deployments that require expensive upgrades, so Blaize’s hardware is re-programmable, easy to deploy and easy to upgrade its software. They already have customer engagements in the automotive and machine vision markets, and if their team can nail the challenges in other markets through their solution stack - which is part of their roadmap - they’ll unlock unprecedented opportunities. The recent funding will help Blaize commercialize and scale, which will likely mean hiring. They’ve also made it clear they intend to enter the public market via SPAC but that could create headwinds for hiring simply because SPACs are lesser known than conventional exits like acquisition or an IPO. In my experience, employees who are looking to join want clarity, and while SPACs have many benefits to employees, they can also create uncertainty about the long-term ramifications to individuals. My advice to Blaize’s leaders, be ready to have clear conversations about not only your mission but your comp structure, etc., to win the best talent #semiconductorindustry #artificialintelligence #edgecompute

39

1 Comment

Great article with insights by Bernard Murphy at SemiWiki.com, "Arteris Frames Network-On-Chip Topologies in the Car." "... while there is plenty of complexity in evolving car architectures and the consequent impact on subsystem chip/chiplet designs, connected on-chip through both coherent and non-coherent networks, the intent behind these systems is quite clear. We just need to start thinking about total car system architecture rather than individual chip functions." #Arteris #NetworkOnChip #Automotive https://bit.ly/3UiKqls

25

AI workflows are driving demand for memory and storage to prepare data, train the model and create useful inference engines.  This is driving announcements of storage and memory products that support the various elements of these AI workflows.

51

4 Comments

With throughput reaching up to 115 Tokens Per Second (TPS) on a single-node configuration, the inference speeds underscore the suitability of Ampere A1 serving for production workloads. Moreover, the widespread availability of Ampere shapes across OCI regions ensures accessibility and scalability for users worldwide.

58

1 Comment

Digital storage demand is increasing and this creates a need for more advanced storage technologies to support a robust digital mass storage hierarchy.  The IEEE IRDS Mass Storage Roadmap outlines projections for NAND flash, emerging non-volatile memories, HDDs, magnetic tape, optical recording and DNA storage development. https://lnkd.in/g8qwRvC7

40

1 Comment

Automotive camera processors continue to struggle to deliver real time and accurate color reproduction. This problem is compounded while dealing with sensors designed to deliver superior low light performance eg RCCB, RYYCy, RCCG etc. Our next generation video processor iND880 puts a complete end to these issues. As we take automotive video processing to next level, iND880 handles several cameras simultaneously with different frame rates, different resolutions and different contexts without dropping any frame. #Automotive #ADAS #CameraProcessing #AutomotiveSensing #indie

73

2 Comments

SK hynix reports record Q1 with soaring demand for #HBM #chips #SKhynix saw first 2024 quarter revenues soar 144 percent as demand for high-bandwidth memory (HBM) chips rocketed. The #DRAM and #NAND manufacturer reported ₩12.43 trillion ($9.03 billion) in revenues with a ₩1.92 trillion ($1.4 billion) profit compared to last year’s ₩2.6 trillion ($1.9 billion) loss. This was a record first quarter revenue amount. DRAM accounted for 61 percent of its revenue and NAND 35 percent, with the remainder coming from other activities. Thanks again to Chris Mellor and Blocks and Files for the full article with more background and insights via the link below 💡🙏👇 https://lnkd.in/ejUBVCMT #semiconductorindustry #semiconductors #semiconductor #technology #ai #tech #ai #artificialintelligence #semiconductormanufacturing #chips

110

2 Comments

Ampere announces 256-core 3nm #CPU, unveils partnership with Qualcomm 💡 Also looking at #chiplets, integration of third-party #IP Ampere Computing introduced its roadmap for the coming years, including new #CPUs and collaborations with third parties. In particular, the company said it would launch its all-new 256-core AmpereOne #processor next year, made on TSMC's N3 process #technology. Also, Ampere is teaming up with Qualcomm to build #AI inference #servers with the company's accelerators. Apparently, Huawei is also looking at integrating third-party UCIe-compatible #chiplets into its own platforms. Ampere has begun shipping 192-core AmpereOne processors with an eight-channel DDR5 memory subsystem it introduced a year ago. Later this year, the company plans to introduce 192-core AmpereOne CPUs with a 12-channel DDR5 memory subsystem, requiring a brand-new platform. Thanks again to Anton Shilov and Tom's Hardware for the full article with more background and insights via the link below 💡🙏👇 https://lnkd.in/e6kMBR2Q #semiconductorindustry #semiconductors #semiconductor #tech #cpu #it #ic #semiconductormanufacturing #technology #chip #foundry #ai

215

3 Comments

KIOXIA Group plots tech route to 1,000 layer 3D #NAND by 2027 #Kioxia outlined a tech roadmap to 1,000 layer 3D NAND at the IWM 2024 conference in #Seoul against a backdrop of statements from NAND fab JV partner Western Digital that #manufacturing costs are rising and returns on #investment are falling. The #Japanese PC Watch media outlet covered Kioxia’s pitch, which predicted that NAND die density would reach 100 Gbit/mm2 with 1,000 wordline (#memory cell) layers by 2027 by extrapolating past trends and improving NAND cell #technology. Thanks again to Chris Mellor and Blocks and Files for the full article with more background and insights via the link below 💡🙏👇 https://lnkd.in/dw3gjJDJ #semiconductorindustry #semiconductors #semiconductor #technology #nandflash #flashmemory #ssd #ssds #semiconductormanufacturing #ai

62

The true innovation in cpu design and use will come from from the whole world participating in such efforts. It is great to see RISC-V gathering tremendous momentum. We expect to see innovations and use cases that we did not even imagine. This is true #democracyofSilicon

32

2 Comments

Lisa Su says AMD is on track to a 100x power efficiency improvement by 2027 — CEO outlines AMD’s advances during keynote at imec’s ITF World 2024 The problem is only intensifying as the compute required to train models increases. Su pointed out that the size of the first image and speech recognition AI models used to double every two years, largely matching the pace of advances in computing power over the last decade. The size of generative AI models is now growing at 20x per year, however, outstripping the pace of computing and memory advancements. Su said that while today’s largest models are trained on tens of thousands of GPUs consuming up to tens of thousands of megawatt-hours, rapidly expanding model sizes could soon require up to hundreds of thousands of GPUs for training, perhaps requiring several gigawatts of power to train a single model https://lnkd.in/dYhQ6isx

3

1 Comment

Nice write up Bolaji ... it certainly seems that my and the SIA President John Neuffer's view on the market are diametrically out of step. Strange because the 5.7% Q1-2024 decline was SIA (WSTS) data! I chose to look at the quarter on quarter trends, whereas he looked Q1-2024 vs Q1-2023. Which is more meaningful? I'll leave that for you all to decide, but I know where my vote lies. If you want to know more, check out our recent webinar slide presentation and video at https://lnkd.in/dUVDriF8

12

9 Comments

We’re one day into DAC, and there have already been announcements that will shape the future of semiconductor design. One startup that I've followed closely for two years shared groundbreaking news that promises to revolutionize chip development and democratize accessibility to chiplets. Baya Systems emerged from stealth mode last week, and CEO and co-founder Sailesh Kumar has done a remarkable job of staying under the radar since founding the company, choosing the perfect moment to unveil updates. Baya revealed not only WHAT they’re working on but also HOW they plan to tackle some of the most pressing challenges faced by SoC architects and designers, and it’s inspiring. The complexity of system design, high costs, and long time-to-market for advanced SoCs often means that a chip can be nearly obsolete by the time it reaches customers. The rapid pace of innovation, especially in AI chips, creates an unforgiving environment where promising startups might only have one shot at success. Baya’s solutions, WeaverPro and WeaveIP, aim to accelerate the design and performance of complex single-die and multi-die SoCs and chiplets. By leveraging a software-centric architecture, Baya simplifies development and accelerates data movement between chiplets and dies. This approach reduces the overall costs associated with advanced AI and CPU acceleration and speeds up time-to-market. WeaverPro and WeaveIP have already received the best kind of acclaim, customer adoption, and Baya announced this week that chiplet technology advocate Tenstorrent will license their WeaveIP fabric to scale their AI and RISC-V chiplet solutions. Chip design icon Jim Keller, who sits on Baya’s Advisory Board, praised the company, saying: “Baya makes great, comprehensive fabric tools that start with top-level architecture and allow us to plan at a detailed level… this, coupled with their visualization tools, enables designers to build next-generation chips, chiplets, and IP.” Adding to their momentum, Baya announced a partnership with Blue Cheetah Analog Design, Inc. yesterday at DAC and introduced a combined chiplet-optimized Network on Chip and PHY interconnect IP solution that simplifies the design and architecture of chiplet-based systems. Sailesh said, “In partnership, we provide system architects with a state-of-the-art platform and powerful tools to create the most innovative and cost-effective chiplet-based products.”   I've mentioned before that enabling development is key to unlocking the door to innovation. For founders without deep pockets or connections to investors, gaining access to better tools and proven intellectual property allows them to move faster and at a lower cost, ultimately helping to bring new designs to life. I’m impressed that Baya, a small yet mighty startup, has created this accessibility and fostered partnerships to help proliferate it. My encouragement to the community - watch them closely. #semiconductorindustry #ai #chiplets

57

Check out my new blog on using our #radhard #MCU to reconfigure #Space deployed #FPGA. Please reach out if you have interest and we can provide specific details on how to deploy

13