Puneet Saini’s Post

Let's understand #kubernetes configmap 😎 #6 post on kubernetes let's understand why we need config maps 1. while performing application deployments on kubernetes cluster sometimes we need to change the application configuration file depending on environment like development quality and assurance stage or production 2. changing this application config file means we need to change source code commit the change creating a new image and then go through the complete deployment process 3. hence these configuration should be decoupled from image content in order to keep containerised application portable 4. this is where kubernetes config map comes in picture it allows as to handle configuration file much efficiently 5. conflict maps are useful storing and sharing non sensitive and encryptic config info use secrets otherwise 6. config map can be used to store fined grained info like individual properties or entire config files 7. config map are not intended to act as a replacement for a properties file config map can be accessed in following ways 1. as environment variable 2. as volumes in the pod command to create config maps kubectl create configmap <map_name> --from-file=<file_name> tagging--> Shubham Londhe Safia Khatoon Rajani Ekunde☁️ Sunil Kumar Akash Raj Mecrin Luvis Uday Kumar Ramagiri Mohammed Muqafamuddin Utkarsh Pathak Vigneshwaran P Dhruv Badaya Umar Ali Butt Sumeet Waghmode Yashraj Singh Sisodiya Shivam Tomar Divyanshi Garg Vaishali Mishra Mahesh Mahajan #kubernetes #docker #devops #python #data #amazon #email #90daysofdevops #90daysofaws #aws #cloud #like #devops #kubernetes #docker #technology #security #help #business #scalability #tensorflow #machinelearning #neuralnetworks #testing #html #javascript #frontend #job #interview

Node Exporter, Prometheus, and Grafana are tools commonly used for monitoring and visualizing system metrics. Here's a simplified explanation of how they work together: Node Exporter: Node Exporter is a small program that runs on the server you want to monitor. It collects a wide range of system-level metrics like CPU usage, memory usage, disk space, network activity, and more. Node Exporter exposes these metrics as HTTP endpoints, making them accessible for Prometheus to scrape. Prometheus: Prometheus is a monitoring and alerting system that regularly scrapes metrics from various sources, like Node Exporter. It stores the collected data in a time-series database. You configure Prometheus to pull data from Node Exporter by specifying the target servers and their respective endpoints in a configuration file. Grafana: Grafana is a visualization and dashboarding tool. It connects to Prometheus as a data source to query and retrieve metrics. You can create custom dashboards in Grafana to display the metrics in a user-friendly way. Grafana provides a graphical interface for building and customizing dashboards with charts, graphs, and tables. Here's a step-by-step overview of how you can set up this monitoring stack: Install and configure Node Exporter on the servers you want to monitor. Install and configure Prometheus, providing it with the necessary scrape targets (your servers running Node Exporter). Set up Grafana and add Prometheus as a data source. Create custom dashboards in Grafana, using the Prometheus data source, to visualize the system metrics. Once everything is set up, you'll have a powerful monitoring and visualization system. You can use Grafana to create dashboards that display real-time and historical data from your servers, helping you keep an eye on system health and performance. #kubernetes #docker #devops #python #data #amazon #email #90daysofdevops #90daysofaws #aws #cloud #like #devops #kubernetes #docker #technology #security #help #business #scalability #tensorflow #machinelearning #neuralnetworks #testing #html #javascript #frontend #job #interview

Hello LinkedIn, it's been a while since I last posted. I am excited to announce version 2 of NextRead, my author endorsement-finding project. If you are a fellow reader, and like to find new authors by who they've endorsed on their book jackets, please try out Next Read here: https://lnkd.in/dn_P7cCM Three major changes in NextRead version 2: 1. Building out my AWS infrastructure, I wanted to switch to Serverless Application Model (SAM) from Terraform. 2. I had a feeling that I could have done a better job of finding potential names in large blocks of text. AWS Comprehend API ability as a Natural Language Processing service was my solution. 3. Reduce code as much as possible for a smaller attack surface and be easier to maintain. My guiding principle for version 2 of NextRead was to reduce as much code as possible. I love this quote from Jeff Atwood, a software engineer who co-founded Stack Overflow: "...the best code is no code at all. Every new line of code you willingly bring into the world is code that has to be debugged, code that has to be read and understood, code that has to be supported. Every time you write new code, you should do so reluctantly, under duress, because you completely exhausted all your other options." -Jeff Atwood The biggest change made in version 2 was incorporating Amazon Comprehend API inside my State Machine. I switched the simplistic Python code I wrote for searching for potential author names in large blocks of text, to processing text through the Amazon Comprehend API. This in turn resulted in leaner, more secure code from 350 lines of front-end JavaScript code down to 130 lines. For more information, please read my blog post below! #aws #serverless #books #awscomprehend Disclaimer: This personal blog post is not related to my current job with NIWC Atlantic or the Department of Navy whatsoever.

Google famously introduced Bazel, a distributed build system for large code bases, gaining popularity among modern companies. However, many companies fail to do a successful migration. Here's a few things to consider while starting the Bazel voyage: - Bazel comes with a steeper learning curve, demanding time and effort for teams to adapt. - The true benefit lies in a clear dependency graph, but untangling spaghetti code can take months. - The framework is evolving rapidly, entailing continuous updates to maintain compatibility - Distributed builds and caching may demand increased resources for computation power and storage - Third party dependency management can get complex Running a Bazel POC may not yield immediate build time improvements that you may expect. Instead pick a medium-size project and improve its dependency structure before starting the Bazel migration. Remember, the goal is not just faster build times but a sustainable and efficient development pipeline that aligns with the unique needs of your project. #bazel #developerproductivity #cicd

If you are looking to integrate Amazon Bedrock from an AWS Lambda function, then look no further. Find the attached public GitHib repo that has sample code that takes cares of the integration for all popular Languages (Python 3.11, Java 17, .NET6 and NodeJS 18) with AWS SAM. Find how easy it is to build your GenAI projects using AWS Serverless. No matter if you are experimenting the art of possible or building your first PROD ready app, accelerate your development time in your preferred programming language. A supporting blogpost is still in the works and would be coming soon. https://lnkd.in/gbjErEXq #amazon-bedrock #awsserverless #java #python #dotnet #nodejs

Creating a Jenkins container using docker compose with persistent data 😎 Docker compose is a tool used to define and execute the deployment of a multi-container environment. It takes its instructions from the docker compose file. Written in YAML, the docker compose file lays out steps to build and run the containers and any attached networks or volumes. The real kicker is that once you have your docker compose file written, you can bring up or tear down entire environments using only a single command. Capable of handling many containers at once, it is the perfect tool to create, manage and tear down your microservice applications Docker Volume - Simply put, a volume in docker is where you would store any data you wish to re-use at a later time. By their nature, containers are what’s known as ephemeral. Meaning that they are stateless and any data saved to a running instance is lost once that container is stopped. The way around this is to attach a volume to your container. This will allow you to store your data outside of the container itself, allowing it to be re-used again and again by any subsequent containers. Your data “persists” through the container’s shutdown Jenkins--> Jenkins is a Continuous Integration (CI) tool used by developers to build and test their applications. However, we will not be using it in such a way. We’re only going to use it to confirm the data stored in our volume persisted. read the full blog written by Thomas puntillo on medium checkout!!! Jenkins - #kubernetes #docker #devops #python #data #amazon #email #90daysofdevops #90daysofaws #aws #cloud #like #devops #kubernetes #docker #technology #security #help #business #scalability #tensorflow #machinelearning #neuralnetworks #testing #html #javascript #frontend #job #interview

Few docker keywords we use frequently in our daily life : -------------------------------------------------------------- FROM ------ to specify the base image. MAINTAINER-----------to specify the author of docker file(deprecated now). COPY-------- to copy files from source to destination (local directory) ADD--------to copy files from source to destination but it will download the files from remote repo using http and copy to destination. RUN------ we will write some Linux commands on top of docker image. We can write multiple run instructions , they will execute from top to bottom. RUN instructions will execute in the build phase CMD--------- same as the RUN command but CMD will execute in run phase i.e while building the container. We can write multiple run instructions , they will execute only last instruction. So there is no need of writing the multiple instructions. ENTRYPOINT---------same as CMD but cmd can be overridden but NTRYPOINT can't be overridden. Entry Point should be only one command in a Docker File. WORKDIR-----used to set working directory for a image .After executing this keyword all commands execute in the working directory. ENV ----- used to set environment variables. ENV <key> <value> LABEL---- used to write meta data in docker file. Used to represent data in key-value format. ARG------used to avoid hardcoded values(docker build -t <image-name> --build-arg branch=<branch-name>) USER ---used to set user to particular image. EXPOSE----on which port number our application is running. It is just for documentation. #aws #docker #devops #keywords #awsdevopsenginner #Containerization 

Went down a very deep rabbit hole the past week investigating concurrency in FastAPI. We started off using Gunicorn to manage Uvicorn workers and started facing some odd behaviour due to Linux kernel scheduling. Task distribution wasn't even across the Uvicorn workers with some getting overloaded while others were waiting with spare capacity. Now, because we run FastAPI in Kubernetes, there's actually no need for Gunicorn. The FastAPI documentation also says the same thing. So we switched out to running 1 Uvicorn process per Kubernetes pod. But, no matter how we tried, we couldn't load the pod with more than 40 concurrent HTTP requests before it started blocking. What gives??! It turns out because FastAPI is based on Starlette, and that uses AnyIO underneath it's hood, there's a default of 40 threads that get spawned ***if you do not specify any other value***. The next logical question then is how do we change this value? You can either use a hack-ish sort of way (see https://lnkd.in/d_UQqFVr) or you can use AnyIO's to_thread interface: ---------- from anyio import to_thread to_thread.current_default_thread_limiter().total_tokens = 60 ---------- You then make sure you set the Uvicorn config parameter --limit-concurrency to have the same value. This will ensure Uvicorn starts rejecting HTTP requests with a HTTP503 error if there is more than 60 calls being serviced (calls that have not completed yet). But wait, there's more!! Turns out, you have to set --limit-concurrency to (60 + 1) i.e. 61 because of https://lnkd.in/dEaJyyEE. We're not finished yet. We also use pytest for our unit testing. Pytest would fail when you do: to_thread.current_default_thread_limiter().total_tokens = 60 with this error: "sniffio._impl.AsyncLibraryNotFoundError: unknown async library, or not in async context". I don't want to go into details of why this happens, but the fix is to wrap your call to "to_thread" in an async function like so: ----- # Needed because we use pytest. loop = asyncio.get_event_loop() if loop is not None:     # run using pytest     task = loop.create_task(set_thread_limiter()) else:     # production run without pytest     asyncio.run(set_thread_limiter) async def set_thread_limiter():     to_thread.current_default_thread_limiter().total_tokens = 60 ----- What a contrived solution!! Anyway, with this out of the way, the next thing is to figure out how to autoscale FastAPI. And for this, we turn to https://keda.sh. But that is a post for another day. :) #fastapi #python #kubernetes #autoscaling

𝗣𝗲𝗿𝘀𝗶𝘀𝘁𝗶𝗻𝗴 𝗗𝗮𝘁𝗮 𝗜𝗻 𝗗𝗼𝗰𝗸𝗲𝗿: 𝗩𝗼𝗹𝘂𝗺𝗲𝘀 & 𝗕𝗶𝗻𝗱 𝗠𝗼𝘂𝗻𝘁 Let's learn about Docker, Inc volume, this blog is for beginners who are new to Docker and want to learn about topics around docker. To learn more visit my Medium profile https://lnkd.in/dR5ieFwS Follow Muhammad Younus on LinkedIn Visit W3Schools.com To learn Web Development From Basics #docker #kubernetes #devops #deployment #volumes #development #medium #hashnode #linkedin https://lnkd.in/dR5ieFwS

Amazon CodeWhisperer is a machine learning-powered code generator that provides real-time code recommendations based on your existing code and comments. The purpose of this tool is to help developers build software more quickly. Here are some key features of Amazon CodeWhisperer : - Code suggestions: CodeWhisperer generates code suggestions ranging from snippets to full functions in real-time in the integrated development environment (IDE) based on your comments and existing code. - Security scanning: CodeWhisperer can scan your code to identify hard-to-find security vulnerabilities and provide code suggestions to help remediate the identified issues. - Reference tracking: CodeWhisperer can flag or filter code suggestions that resemble publicly available code, and provide the associated open-source project's repository URL and license. - Multi-language support: CodeWhisperer supports 15 programming languages, including Python, Java, JavaScript, TypeScript, C#, Go, Rust, Kotlin, Scala, Ruby, PHP, SQL, C, C++, and Shell Scripting. - Integration with IDEs and CLIs: CodeWhisperer fits the way that you work, and supports various IDEs, including VS Code, IntelliJ IDEA, Visual Studio (Preview), AWS Cloud9, the AWS Lambda console, JupyterLab, and Amazon SageMaker Studio, and your favorite CLIs including macOS Terminal, iTerm2, and the built-in VS Code terminal. https://lnkd.in/e78-SKab - #AI - #AICodeGenerator - #CodeGeneration - #ArtificialIntelligence - #MachineLearning - #Automation - #CodingEfficiency - #DeveloperProductivity - #AIforDevelopers - #CodeWritingAssistance - #IntelligentCoding - #FutureOfCoding - #AIpoweredDevelopment - #CodeGenerationTools - #AIassistedCoding - #CodingInnovation - #AIforSoftwareDevelopment Ameen Alam Governor Sindh Initiative for GenAI, Web3, and Metaverse Ubaid ur Rehman Nida Rizwan AMBREEN RIAZ Okasha Aijaz Zia Khan Daniyal Nagori Naeem Hussain✅ Asad Riaz Aqsa Shah Aleeza a Arzoo Pervaiz Hamzah Syed