ScyllaDB’s Post

Many distributed systems follow a leader-follower topology, where one node is designated as a leader and if it dies a new one is elected and the system continues operating. #ScyllaDB doesn't follow this model. How does distributing across multiple nodes provide scalability and fault tolerance? Find out in a excerpt from #Discord Staff Engineer Bo Ingram's book, ScyllaDB in Action > https://lnkd.in/e29FBZRe #NoSQL #database #techtips

8 months ago we launched OpenObserve - an open-source alternative to Elasticsearch, Datadog, Splunk in Rust and Vue for logs. Go self-hosted or save upto 140X on storage, with 3 year archival. Read more about our mission to help engineers understand and improve their systems, and how we are bringing simplicity and transparency to the world of observability - https://lnkd.in/djvhF2mb . . . #opensource #observability #localization #sql #openobserve

In the realm of big data processing, Apache Flink and Apache Beam emerge as two formidable frameworks, each with its unique strengths. Flink prioritizes low-latency processing and boasts an extensive feature set, whereas Beam's focus on portability and user-friendliness adds to its appeal. Grasping the distinctions between these frameworks is pivotal for making informed decisions in big data processing. Hence, assess your requirements, scrutinize the strengths of each framework, and opt for the one that best aligns with your organization's goals and objectives. ✅ Book Now with!! 15mins free call consultation from our expert. https://lnkd.in/dZfDgbPs ✅𝗜𝗳 𝘆𝗼𝘂 𝗼𝗿 𝘀𝗼𝗺𝗲𝗼𝗻𝗲 𝘆𝗼𝘂 𝗸𝗻𝗼𝘄 𝗿𝗲𝗾𝘂𝗶𝗿𝗲𝘀 our 𝘀𝗲𝗿𝘃𝗶𝗰𝗲𝘀, 𝗽𝗹𝗲𝗮𝘀𝗲 𝗱𝗼𝗻'𝘁 𝗵𝗲𝘀𝗶𝘁𝗮𝘁𝗲 𝘁𝗼 𝗿𝗲𝗮𝗰𝗵 𝗼𝘂𝘁 𝘁𝗼 𝘂𝘀 𝗮𝘁 info@genxdmcc.com 𝗮𝗻𝗱 𝘃𝗶𝘀𝗶𝘁 𝗼𝘂𝗿 𝘄𝗲𝗯𝘀𝗶𝘁𝗲 𝗮𝘁 www.genxdmcc.com ✅𝗔𝗱𝗱𝗶𝘁𝗶𝗼𝗻𝗮𝗹𝗹𝘆, 𝘄𝗲 𝗶𝗻𝘃𝗶𝘁𝗲 𝘆𝗼𝘂 𝘁𝗼 𝗰𝗼𝗻𝗻𝗲𝗰𝘁 𝘄𝗶𝘁𝗵 𝘂𝘀 𝗼𝗻 𝗼𝘂𝗿 𝗟𝗶𝗻𝗸𝗲𝗱𝗜𝗻 𝗽𝗮𝗴𝗲 𝗳𝗼𝗿 𝗿𝗲𝗴𝘂𝗹𝗮𝗿 𝘂𝗽𝗱𝗮𝘁𝗲𝘀 𝗮𝗻𝗱 𝗶𝗻𝗳𝗼𝗿𝗺𝗮𝘁𝗶𝘃𝗲 𝘁𝗲𝗰𝗵𝗻𝗼𝗹𝗼𝗴𝗶𝗰𝗮𝗹 𝗺𝗮𝘁𝘁𝗲𝗿𝘀. #bigdata #dataprocessing #ApacheFlink #ApacheBeam #streamprocessing #batchprocessing #dataanalytics #datascience #dataengineering #BigDataFrameworks

Excited to explore the potential of ScyllaDB! With its incredible speed and low-latency performance, ScyllaDB is a game-changer for handling massive datasets. Here are a few reasons why you should consider ScyllaDB: - Blazing Fast Performance: Achieve millions of transactions per second with ultra-low latencies. - Horizontal Scalability: Easily scale out by adding more nodes to handle growing workloads. - Cassandra Compatibility: Seamlessly integrates with existing Cassandra applications, making migration smooth. - Efficient Resource Utilization: Optimizes CPU and memory usage to deliver cost-effective performance. - High Availability: Ensures data redundancy and fault tolerance for mission-critical applications. If you're in need of a high-performance database, ScyllaDB is definitely worth checking out. Here it is: https://shorturl.at/i3iHe #Database #NoSQL #Fast #Scalable #ScyllaDB

Our new InfluxDB Clustered is designed for self-managed workloads at scale. It will help you reduce the total cost of ownership. https://bit.ly/3tvrfdk #InfluxDB #ApacheArrow

Hey Folks! This post explores the intricate world of key-value stores, delving into their design principles, addressing key challenges, and presenting potential solutions. You'll learn about: Functional and Non-Functional Requirements: Understand the core functionalities of a key-value store and the crucial non-functional aspects like scalability, consistency, durability, and availability. CAP Theorem and Its Impact: Explore the CAP theorem and its implications for choosing between consistency, availability, and partition tolerance in distributed systems. Single Server vs. Distributed Key-Value Stores: Compare the simplicity of single-server systems with the scalability and fault tolerance offered by distributed architectures. Building Blocks of a Distributed Key-Value Store: Discover the essential components like data replication, partitioning, consistency models, and failure handling. Cassandra as a Use Case: Learn how Cassandra, a popular key-value store, implements write and read paths, leveraging in-memory cache and persistent storage. Visit the link to read about designing key-value store in detail: https://lnkd.in/gdeaEEbQ Follow Ankit Rana for more such posts! #keyvaluestore #distributedsystems #scalability #consistency #availability #CAPtheorem #Cassandra #systemdesign

Why Apache Doris is worth taking a look at as a #log analysis solution❓ 🏠 Storage efficiency: Only requires 144GB of space to store 1TB raw log data. ✍️ Write throughput: Reaches a write speed of 500MB/s when ingesting 1TB log data with a cluster of 3 machines (16 Core, 64 GB each). (Dataset from Log and Telemetry Analytics #Benchmark by Microsoft #Azure) 📄 Text search: Provides inverted index that is fine-grained to the row, enabling efficient full-text searching. 🥪 Aggregation:  A C++-based vectorized execution engine and MPP distributed architecture to enable high performance. 🧑💼 Well-established distributed cluster management 🕸️ Seamless online scaling 🦑 High cluster availability #opensource #Elasticsearch #ClickHouse #database #bigdataanalytics

📖 Your Weekend Read: What's new and trending on the #ScyllaDB Forum this week? Here are some of our most engaging topics: - Load balancing on #Kubernetes - Materialized Views and Indexing, filtering columns by range, ALLOW FILTERING - Creating a Superuser with custom credentials, salting method and password hash generation - Running ScyllaDB on Docker with IPv6 - Upcoming free training event Check them out and weigh in here > https://ow.ly/7bL550RBX0U #database #opensource #techtips #techforum

Hello again!  Today is a day for some #ops issues out of the hood. While we are adopting Trino for our workloads, we are still using #Vertica for daily calculations. And there is something new we have faced recently. Vertica started getting killed by the OOM killer all of a sudden. Atop showed that we were not even close to the memory limit, using less than 400GB out of 500GB. And there was no comparable memory-consuming process other than Vertica itself. So where did the memory go in order for oom killer to go and sluter vertica? Graphana charts for all the hosts in a cluster showed like nothing suspicious. Too many hosts at once on a single memory chart didn't help the engineer on duty catch the glitch. He saw the common pattern of consumption for all hosts, and nothing seemed to be wrong. It turned out there was a memory leak on our hosts, and about 70GB were occupied by slab, which is not so obvious to catch an eye on on a 500GB scale. We know that Vertica sees all the memory on the host as if she owns all of it. And the OOM killer saw it in a different way with respect to slab memory. It looked like it started to grow when we set some of the policies to maintain metrics history in Vertica for a few months. Everything went back to normal after turning them off and a single reboot. You may think that nothing can take away the memory you have left for a single greedy process on a host. Well, it might not be true.

🚀 **Optimizing Airflow DAG Performance: Tips and Tricks** 🚀 Is your Apache Airflow DAG running slower than expected? Here are some tips to help you speed things up: 1. **Resource Management**: Ensure workers have sufficient CPU and memory. Scale horizontally or vertically as needed. 2. **Task Dependencies**: Review and remove unnecessary dependencies to allow parallel execution of tasks. 3. **Database Optimization**: Optimize queries, maintain the metadata database, and ensure an appropriate connection pool size. 4. **Code Efficiency**: Make sure the code within your tasks is optimized and interactions with external systems are efficient. 5. **Configuration Tuning**: Choose the right executor for your workload and adjust parallelism and concurrency settings. 6. **Monitoring**: Use tools like Grafana and Prometheus to monitor performance and identify bottlenecks. 7. **Retry Strategy**: Evaluate task retry settings and implement robust error handling to avoid overloading the system. Proactive maintenance and monitoring can make a huge difference. Keep your DAGs running smoothly and efficiently! #DataEngineering #ApacheAirflow #ETL #BigData #DataPipeline