Tuesday, 19 November 2024

How to disable Ingress in Kubernetes (K8s)

 Introduction

There are multiple ways you can disable Ingress in Kubernetes / K8s. Here are the 5 different ways you can directly or indirectly disable the Ingress



1. Deleting the Ingress Resource

If you need to completely remove ingress functionality from your Kubernetes cluster, the most straightforward approach is to delete the ingress resource. This stops all traffic from being routed through it.

Command

kubectl delete ingress <ingress-name> -n <namespace>


For example :

kubectl delete ingress my-ingress -n default


2. Scaling Down the Ingress Controller

If you only want to temporarily stop ingress traffic without removing anything permanently, you can scale down the ingress controller to zero replicas. This effectively halts ingress without deleting the controller.

    (i) First, identify the ingress controller deployment:


kubectl get deployments -n <controller-namespace>

    (ii) Then, scale the deployment down to zero replicas:


kubectl scale deployment <controller-deployment-name> --replicas=0 -n <controller-namespace>


    For example :

kubectl scale deployment <controller-deployment-name> --replicas=0 -n <controller-namespace>


3. Modifying Network Access

Another way to disable ingress traffic is by adjusting the network access or firewall settings at the external level. For example, you can modify the load balancer settings or block access via firewall rules to prevent any external traffic from reaching the ingress controller.


4. Editing the Ingress Configuration

If you want to disable certain routes or services but keep the ingress resource active, you can edit the configuration directly. Simply comment out or remove the paths you want to disable in the rules section of the ingress YAML file:

rules:
  - host: example.com
    http:
      paths:
        # - path: /old-path
        #   backend:
        #     service:
        #       name: old-service
        #       port:
        #         number: 80
        - path: /new-path
          backend:
            service:
              name: new-service
              port:
                number: 80


    Once you've updated the file, apply the changes:

kubectl apply -f <ingress-file>.yaml


5. Using Ingress Annotations

Some ingress controllers, like NGINX, allow you to control the availability of specific rules or the entire ingress resource using annotations. To disable an ingress resource entirely, you can add the following annotation:


metadata:
  annotations:
    nginx.ingress.kubernetes.io/enable: "false"


    Once you've updated the file, apply the changes:

kubectl apply -f <ingress-file>.yaml


Based on the scenario, we can choose one solution to disable ingress in Kubernetes.


Monday, 18 November 2024

Think You Know Kubernetes? Test Yourself with These 60 Questions

Kubernetes, often abbreviated as K8s, is a powerful open-source platform for automating the deployment, scaling, and management of containerized applications. Originally developed by Google, it is now maintained by the Cloud Native Computing Foundation (CNCF).

At its core, Kubernetes helps organizations run applications reliably across clusters of machines by abstracting infrastructure complexities. It enables features like automatic scaling, self-healing, load balancing, and seamless rollouts/rollbacks. With its rich ecosystem of tools and integrations, Kubernetes has become the backbone of modern DevOps and cloud-native architectures.

Whether you're managing a microservices-based application or scaling a monolithic app, Kubernetes provides the flexibility and control needed to operate in dynamic, production-grade environments.

Kubernetes Knowledge Boost: 60 Advanced Questions Explained


Here is a list of 60 advanced Kubernetes questions with concise answers. They cover architecture, concepts, troubleshooting, security, and more:


Kubernetes Architecture and Core Concepts

  1. What is a Kubernetes control plane?

    • The control plane manages the Kubernetes cluster, maintaining the desired state via components like kube-apiserver, etcd, kube-scheduler, kube-controller-manager, and cloud-controller-manager.
  2. What is etcd, and why is it important?

    • etcd is a distributed key-value store that Kubernetes uses to store all cluster data, such as configurations, secrets, and states.
  3. Explain how a Kubernetes scheduler works.

    • The scheduler assigns Pods to nodes by evaluating resource requirements, policies, and constraints, ensuring an optimal placement.
  4. What is a Pod? Why is it the smallest deployable unit in Kubernetes?

    • A Pod is a group of one or more containers sharing the same network namespace and storage. It encapsulates the application and its dependencies, making it the basic unit of deployment.
  5. How do you ensure high availability in a Kubernetes cluster?

    • By setting up a multi-master architecture, load balancers for the API server, and using tools like kubeadm, Kops, or managed Kubernetes services.

Advanced Networking

  1. What is a Kubernetes Service?

    • A Service abstracts a set of Pods, providing stable networking and load balancing. Types include ClusterIP, NodePort, LoadBalancer, and ExternalName.
  2. How does kube-proxy work?

    • kube-proxy maintains network rules to allow communication between Pods and Services using IP tables, IPVS, or user-space proxies.
  3. What is the role of CNI in Kubernetes?

    • Container Network Interface (CNI) is a standard for configuring container networking, enabling plugins like Flannel, Calico, and Weave.
  4. What are Network Policies?

    • Network Policies control traffic between Pods or Pods and external endpoints, using rules to allow or deny ingress and egress.
  5. How do you expose an application in Kubernetes to external traffic?

    • Use a Service of type LoadBalancer or NodePort, or an Ingress resource with an Ingress controller.

Storage

  1. What is Persistent Volume (PV) and Persistent Volume Claim (PVC)?

    • PV is a storage resource in the cluster, while PVC is a request for storage by a user. PVCs bind to PVs for dynamic or static provisioning.
  2. How does dynamic volume provisioning work?

    • StorageClasses define storage types, and when a PVC is created, Kubernetes dynamically provisions a matching PV.
  3. What is CSI in Kubernetes?

    • The Container Storage Interface (CSI) standardizes storage plugin management for dynamic provisioning and attaching volumes.
  4. What is ephemeral storage?

    • Temporary storage tied to the Pod lifecycle, used for caching, logs, or temporary files.
  5. How can you back up and restore etcd?

    • Use etcdctl commands to take snapshots and restore the data to another etcd cluster.

Security

  1. What is RBAC, and how does it work?

    • Role-Based Access Control (RBAC) manages access to Kubernetes resources using roles and bindings.
  2. What are Kubernetes Secrets, and how are they used?

    • Secrets store sensitive data like credentials securely, referenced by Pods via volumes or environment variables.
  3. What is Pod Security Admission (PSA)?

    • PSA enforces security standards for Pods by controlling privilege levels like baseline or restricted.
  4. How do you secure the API server?

    • By using TLS, enabling RBAC, restricting anonymous access, and enabling audit logs.
  5. What are Pod security contexts?

    • Security contexts define Pod-level permissions, such as running as a non-root user or mounting filesystems as read-only.

Troubleshooting

  1. How do you troubleshoot Pod CrashLoopBackOff?

    • Check logs using kubectl logs, inspect events using kubectl describe pod, and validate container configurations.
  2. How do you debug network issues in Kubernetes?

    • Use tools like kubectl exec, network policies, traceroute, or external tools like cilium and tcpdump.
  3. What is the significance of Node Conditions?

    • Node Conditions (e.g., MemoryPressure, DiskPressure) indicate health and resource status, guiding scheduler decisions.
  4. How do you handle etcd failure?

    • Restore from a backup and ensure quorum by adding healthy nodes.
  5. What tools can you use for Kubernetes observability?

    • Prometheus, Grafana, Fluentd, ELK stack, and Jaeger.

Scaling and Performance

  1. How does Kubernetes autoscaling work?

    • Kubernetes supports horizontal (HPA), vertical (VPA), and cluster autoscaling based on resource utilization or custom metrics.
  2. What is a ReplicaSet, and how does it differ from a Deployment?

    • A ReplicaSet ensures a specified number of Pod replicas, while Deployments manage ReplicaSets for updates and rollbacks.
  3. How do you optimize resource usage in a cluster?

    • Define resource requests/limits, monitor usage with tools like Prometheus, and right-size workloads.
  4. What is pod affinity and anti-affinity?

    • Affinity defines Pod co-location preferences, while anti-affinity enforces separation.
  5. How do you handle resource contention in a cluster?

    • Use priorities, preemption, and resource quotas.

Custom Resources and Operators

  1. What are Custom Resource Definitions (CRDs)?

    • CRDs enable users to define and manage custom Kubernetes objects.
  2. What is a Kubernetes Operator?

    • An Operator automates operational tasks for applications using custom controllers and CRDs.
  3. How do you extend Kubernetes functionality?

    • Use CRDs, admission controllers, or plugins like CNI and CSI.
  4. How do you write a custom controller?

    • Use Kubernetes client libraries (e.g., client-go) to reconcile the desired and actual states of resources.
  5. What are finalizers in Kubernetes?

    • Finalizers delay resource deletion until specific cleanup tasks complete.

Cluster Administration

  1. How do you upgrade a Kubernetes cluster?

    • Upgrade control plane components first, then worker nodes, following the Kubernetes release cycle.
  2. What is taint and toleration in Kubernetes?

    • Taints prevent Pods from scheduling on specific nodes unless they tolerate the taint.
  3. How do you perform node maintenance in Kubernetes?

    • Drain the node (kubectl drain) and cordon it to prevent scheduling.
  4. What is kubelet, and what is its role?

    • Kubelet is an agent running on each node, responsible for managing Pods and reporting to the control plane.
  5. How does Kubernetes handle rolling updates?

    • Deployments update Pods incrementally, creating new Pods while scaling down old ones.

Advanced Scenarios

  1. What is the difference between StatefulSets and Deployments?

    • StatefulSets manage stateful applications with stable network identities, while Deployments are for stateless applications.
  2. How do you ensure consistency in StatefulSets?

    • Use headless services and persistent volumes.
  3. What is a Helm chart, and how is it used?

    • Helm is a package manager for Kubernetes, and Helm charts define application configurations and resources.
  4. What are the differences between DaemonSets and Deployments?

    • DaemonSets ensure one Pod runs on each node, while Deployments scale Pods across nodes.
  5. How do you implement canary deployments?

    • Use multiple versions of Deployments with traffic splitting via Ingress or a Service mesh.

Security and Best Practices

  1. What is a Pod disruption budget (PDB)?

    • PDB defines minimum available or maximum unavailable Pods during disruptions.
  2. What is Kubernetes admission control?

    • Admission controllers validate and mutate API requests based on policies.
  3. How do you implement encryption for Secrets?

    • Enable encryption at rest using Kubernetes encryption configuration.
  4. What are the best practices for managing ConfigMaps and Secrets?

    • Use immutable ConfigMaps/Secrets, manage access with RBAC, and avoid exposing sensitive data in logs.
  5. What are namespace quotas?

    • Quotas limit resource consumption in namespaces, ensuring fair resource distribution.

Miscellaneous

  1. What is kubeadm, and how is it used?

    • Kubeadm bootstraps Kubernetes clusters by setting up the control plane and nodes.
  2. What is the difference between a job and a cronjob?

    • A job runs a task to completion, while a cronjob schedules jobs at regular intervals.
  3. What is kube-state-metrics?

    • It exposes Kubernetes resource states as Prometheus metrics for monitoring.
  4. How do you use Service mesh in Kubernetes?

    • Service meshes like Istio or Linkerd add features like traffic routing, observability, and security to microservices.
  5. What is Horizontal Pod Autoscaler (HPA)?

    • HPA scales Pods based on CPU/memory utilization or custom metrics.

Emerging Trends

  1. How does Kubernetes integrate with serverless frameworks?

    • Use Knative or OpenFaaS to run serverless workloads on Kubernetes.
  2. What is Kubernetes Federation?

    • Federation manages multiple clusters for global deployment and high availability.
  3. What is the role of policy engines like OPA/Gatekeeper?

    • Open Policy Agent (OPA) enforces policies like security or compliance in Kubernetes.
  4. What is KubeEdge?

    • KubeEdge extends Kubernetes to edge devices for IoT and edge computing.
  5. What is Multi-tenancy in Kubernetes?

    • Multi-tenancy isolates resources and workloads for different users or teams within a cluster.
That all for important advanced Kubernetes Questions and Answers that will help you in clearing DevOps Interviews.

Tuesday, 29 October 2024

Ultimate Ansible Interview Guide: 30 Advanced Questions and Solutions

 Ansible is a configuration management tools and it comes under continues delivery / deployment in DevOps Lifecycle

Here are few important Ansible questions and answers which will help you to crack interview easily. I have tried to cover all the important questions and Answers



Sunday, 1 September 2024

Understanding Sealed Class in Java and its use-case


Introduction to Sealed Class and Interfaces

Sealed Class in Java are used to allow / permit specific named Class which can extend the Sealed Class. 
With Sealed Class, we can say that Class A can only be extended by Class B and Class C and not any other class, thus limiting who can extend a particular Java Class



Detailed Explanation and use-case

Let's start with a very simple example. Consider we are creating a base class name Animal which implements basic feature of animal like walking, running etc
Now, We want few classes to extend Animal class like Cat, Dog, Lion, Tiger etc which we can do it easily and implement specific respective feature
Since this Animal Class can be extended by any class, someone can accidentally extend Animal class for Pigeon too which is not correct
 In other words, Sealed Class / Interface permits (or restricts) which all Class / Interface can use them. It prevents misuse of Java Class / Interface

Advantage and Disadvantage of using Sealed Class

Lets discuss Advantage first

Advantage of Sealed Class and Interface:

  • The developer of Sealed Class can manage who can use / or whose code can implement it
  • If we compare with access modifier, Sealed Class provide a more declarative way to restrict use of super class
  • Sealed Class and its permitted sub-class can be used as Pattern Matching using Switch statement

Disadvantage of Sealed Class and Interface

  • If not properly designed / thought about its all permitted sub-class, need to revisit code of super class to add more permitted sub-class
  • In few use-cases, unit testing would be difficult
  • Permitted class uses cannot be detected at compile time, so if it has been used and its not in list of permitted sub-class, it cannot throw compile time error
  • Limitation of usage of permitted sub-class within same module only

How to declare Sealed Class:

Sealed class can be declared with using the keyword sealed along with its permit sub-classes with keyword permits

Example of How to create Sealed Classes:

package com.tech693.java.examples.seal;

public sealed class Animal permits Cat, Dog {

    public void getIdentity(){

        System.out.println("Animal");
    }
}

Notice the sealed and permits keywords. Its mandatory to declare one sub-class using permits keyword when using sealed. 
By this we are declaring that only Cat and Dog sub-class can extend the Animal class and no other class is allowed to extend Animal class
Now, lets see how to create Cat and Dog class

Example of how to create Sealed sub-class:

package com.tech693.java.example.seal;

// Pls note its mandatory to declare sub class as either sealed or non-sealed or final
public non-sealed class Cat extends Animal { 
    
    public void getIdentity(){
        System.out.println("Cat");
    }    
}

package com.tech693.java.example.seal;

// Pls note its mandatory to declare sub class as either sealed or non-sealed or final
public final class Dog extends Animal {
    
    public void getIdentity(){
    System.out.println("Dog");
    }  
}

Also, the sub-class should directly extend the sealed class.
for eg.  Suppose, A is a sealed class which permits B to extend it and B is also a sealed class that permits C. In this scenario C cannot directly extend A

Its mandatory to declare sub class as either sealed or non-sealed or final which has different relevance
  • Sub-class as Sealed: - Can only be further extended by sub-class permitted class
  • Sub-class as final :- Cannot be further extended
  • Sub-class as non-sealed - Can be extended further by any class
    • The idea to have non-sealed is to know the developer that it extends the sealed class and is not accidently allowing further extended by its sub-class
Now lets run the above code using Main class as shown below:
package com.tech693.java.example.seal;

public class SealedClassExample {
    public static void main(String[] args) {
        Animal animal = new Cat();
        animal.getIdentity();

        Animal animal1 = new Dog();
        animal1.getIdentity();
    }
}

Output

So when Animal class is declared as cat, it will print Cat when getIdentity() is called

Here is the output of the above code :

Output:

Cat
Dog

Sealed Interface:

Sealed interface acts similarly as Sealed Class. The only difference is the difference between abstract class and interface (we have have some business logic code in abstract class but cannot have it in interface)

Sealed Interface Example:

Let's see some code example of implementing sealed as Interface
package com.tech693.java.examples.seal;

public sealed interface Animal permits Cat, Dog {

    public void getIdentity(){}
}

package com.tech693.java.example.seal;

// Pls note its mandatory to declare sub class as either sealed or non-sealed or final
public non-sealed class Cat implements Animal { 
    
    public void getIdentity(){
        System.out.println("Cat");
    }    
}

package com.tech693.java.example.seal;

// Pls note its mandatory to declare sub class as either sealed or non-sealed or final
public final class Dog implements Animal {
    
    public void getIdentity(){
    System.out.println("Dog");
    }  
}


Till now we have got the basic idea about Sealed Classes and its permitted sub-class.

Lets understand further in more depth about its use-case and how Java implemented Sealed class

Record class as permitted subclasses of a Sealed Class

Since Record class are implicitly final we can use record class in the permits clause of a Sealed Class or Interface
Here is an example of Record class as subclasses of a sealed class

package com.example.records.expressions;

sealed interface Expr permits ConstantExpr, PlusExpr, TimesExpr, NegExpr {
    public int eval();
}

record ConstantExpr(int i) implements Expr {
    public int eval() { return i(); }
}

record PlusExpr(Expr a, Expr b) implements Expr {
    public int eval() { return a.eval() + b.eval(); }
}

record TimesExpr(Expr a, Expr b) implements Expr {
    public int eval() { return a.eval() * b.eval(); }
}

record NegExpr(Expr e) implements Expr {
    public int eval() { return -e.eval(); }
}
Note that we haven't declare any modifier (final, non-sealed or sealed) to sub-class. This is because Record class are final by default.

Design considerations when using Sealed Classes and Interfaces

We should keep in mind following consideration while designing Sealed Classes or Interface
  • Design Decision - We should carefully consider which class should be declared sealed before start implementing only. Sealed class / interface consideration often results in tightly coupled design instead of loosely coupled
  • Backward Compatibility - If we plan to use sealing class, it can impact backward compatibility. Adding new permitted sub-class in future release can break existing code that depends on the sealed API
  • Package / Module Arrangement - We should carefully construct Java Module / Package structure when used sealed classes or interface as the scope of sealed class and its permitted sub-class is scoped under a single module.

Sealed Class as Pattern Matching case:

This is very important to understand. The introduction of Sealed class in Java opens up a new paradigm of using Permitted classes as Pattern Matching in switch case
Let's first understand how Sealed Class is implemented in JDK. 
Although sealed is a class modifier, there is no ACC_SEALED flag in the ClassFile structure. Instead, the class file of a sealed class has a PermittedSubclasses attribute which implicitly indicates the sealed modifier and explicitly specifies the permitted subclasses:

PermittedSubclasses_attribute {
    u2 attribute_name_index;
    u4 attribute_length;
    u2 number_of_classes;
    u2 classes[number_of_classes];

Code Example: Consider above example of Animal as sealed class and Cat and Dog as its permitted sub class. 
Here is the example where we can use sealed class in switch case

private static String checkAnimal(Animal animal){                                     
    return switch (animal) {                                                          
        case Cat c -> c.getIdentity();                                                
        case Dog d -> d.getIdentity();                                                
        default -> throw new IllegalArgumentException("Unexpected Input: " + animal); 
    };                                                                                
}


Frequently Asked Question:

Why Sealed Class is introduced in Java, what problem does it solves?
 
The purpose of sealed class is to have more control on sub-class / interface hierarchy. It provides a way to create hierarchy of classes and its sub-classes. It also solves the problem of using sealed class as Pattern Matching using switch case

What is the difference between sealed and final class?


Final class means no other class can extend it whereas sealed class means only permitted sub-class can extend it


What is the difference between sealed and abstract class?


Sealed class can be used along with abstract keyword and its a good idea because in most of the use-case sealed class is abstract and its permitted sub-class has actual implementation. for eg - Shape sealed class can have Rectangle, Triangle as its permitted sub-class and the actual implementation of its area, radius, circumference and diameter code is in its permitted sub-class


Friday, 6 January 2023

5 Best Online Course to Learn Amazon Web Services AWS in 2023

Amazon Web Services (AWS) is undoubtedly the most popular Cloud Computing Services today that provides on-demand cloud computing platform. 


Why learning AWS is worthful?

AWS has around 1/3rd of market share which indicates its popularity and indirectly relates to more jobs opening for AWS Engineers as compared to other Clouds. So learning AWS is worthful for System Administrator as well as Developers / Architects.




I have filtered out Top 5 best online courses for AWS which you can opt for learning in 2023.




  • The course instructor is Stephane Maarek an AWS Certified Cloud Practitioner, Solutions Architect,Developer
  • 4.7 Instructor Rating|533,519 Reviews|1,752,022 Students|41 Courses
  • Best Selling Instructor, 9x AWS Certified, Kafka Guru
  • Stéphane is recognized as an AWS Hero and is an AWS Certified Solutions Architect 
  • Professional & AWS Certified DevOps Professional.

Requirements:
  • Know the basics of IT
  • No AWS Cloud experience is necessary, we'll use the AWS Free Tier
  • Windows / Linux / Mac OS X Machine

Beginners welcome: No need to know anything about AWS!
34 sections • 389 lectures • 27h 10m total length

Keypoints:
  • 30-Day Money-Back Guarantee
  • This course includes:
  • 27 hours on-demand video
  • 13 articles
  • 1 downloadable resource
  • 1 practice test
  • Full lifetime access




  • This course is instructed by Neal Davis | AWS Certified Solutions Architect & Developer
  • AWS Solutions Architect & AWS Certified Instructor
  • Instructor Rating 4.6 |106,791 Reviews |507,614 Students | 16 Courses
Requirements:
  • This course is designed for Cloud Computing beginners and AWS beginners
  • Absolutely no prior experience necessary
Key points:
  • 12 sections • 88 lectures • 6h 52m total length
  • 30-Day Money-Back Guarantee
  • 7 hours on-demand video
  • 2 articles
  • Full lifetime access
  • Access on mobile and TV
  • Certificate of completion



  • This course is launched by BackSpace Academy and Paul Coady Cloud Technology Expert.
  • The fastest route to cloud certification.
  • BackSpace Academy is having 4.5 Instructor Rating 29,194 Reviews 403,020 Students 3 Courses and
  • Paul Coady is 4.5 Instructor Rating 21,868 Reviews 143,105 Students 1 course
  • BackSpace Academy is Providing accelerated learning programs taught by AWS professional level certified instructors since 2014. A unique approach that focusses on maximum results in the shortest possible time. Quality lecture videos reinforced with instructional and hands-on lab videos and, review tests.
  • Paul Coady is Highly experienced expert in cloud architecture design and deployment, not simply a certificate collector.Amazon Web Services Certified Professional. Providing a learn-by-doing approach produces students not only ready to pass an exam but also ready for employment. BackSpace Academy, the fastest route to cloud certification. Providing AWS Training since 2014 and still going strong.
What you'll learn
  • You will be fully prepared for the AWS Certified Solutions Architect Associate,
  •  AWS Certified Developer Associate and AWS Certified SysOps Administrator Associate
Requirements:
  • Basic understanding of computers and networking.
  • The student will require an AWS account to complete hands on labs sessions.
  • Windows, Linux or Mac PC to complete hands on labs sessions.

Key points:
  • 5 sections • 200 lectures • 45h 55m total length
  • 30-Day Money-Back Guarantee
  • 46 hours on-demand video
  • 7 articles
  • 41 downloadable resources
  • Full lifetime access
  • Access on mobile and TV
  • Certificate of completion



  • Very frequent batches
  • 4.5 Google Reviews |4.7 Trustpilot Reviews |4.5 G2 Reviews |4.4 Sitejabber Reviews
Key points:
  • Live Interactive Learning
  • World-Class Instructors
  • Expert-Led Mentoring Sessions
  • Instant doubt clearing
  • Lifetime Access
  • Course Access Never Expires
  • Free Access to Future Updates
  • Unlimited Access to Course Content
  • 24x7 Support
  • One-On-One Learning Assistance
  • Help Desk Support
  • Resolve Doubts in Real-time
  • Hands-On Project Based Learning
  • Industry-Relevant Projects
  • Course Demo Dataset & Files
  • Quizzes & Assignments
  • Industry Recognised Certification
  • Edureka Training Certificate
  • Graded Performance Certificate
  • Certificate of Completion

5. AWS Essentials (Udemy)


  • This course is designed by Amazon Web Services (AWS) is a subsidiary of Amazon.com
  • Amazon Web Services (AWS) 4.4 Instructor Rating | 24,888 Reviews | 123,221 Students |8 Courses
Keypoints:
  • 7 sections • 61 lectures • 3h 22m total length
  • 30-Day Money-Back Guarantee
  • 3.5 hours on-demand video
  • Full lifetime access
  • Access on mobile and TV
  • Certificate of completion
Requirements
  • Fundamental understanding of IT, storage, databases, and networking.
Disclosure: This article may contain affiliate links. When you purchase, we may earn a small commission.

Thursday, 15 August 2019

What is fishbucket in Splunk

Introduction

In this post we will learn what is fishbucket in Splunk but before that lets us understand what Splunk is and its purpose.
Splunk is used for monitoring, searching, analyzing the machine data in real time. The datasource can range from application data to user created data to sensors or devices.

Purpose of Splunk fishbucket

Before analyzing the data, Splunk index the data. The index is necessary to analyze the data. But here is the issue, what if the same data is indexed multiple times or in other words, how to avoid duplicate indexing the same chunk of data?
Splunk fishbucket keeps seek pointers and CRCs for the indexed files inside the directory. This directory is called fishbucket. Since through fishbucket we can know which data has already been indexed, so splunkd can tell if it has been read already and avoid duplicate indexing.


How fishbucket works?

File monitor processor searches the fishbucket to see if the CRC from the beginning of the file is already there or not. This is the first step of file monitor processor whenever it starts looking at a file. There can be three possible scenarios:
Scenario 1: If CRC is not present is fishbucket, the file is indexed as new. This is simple, file has never been indexed. After indexing, it stores CRC and seekpointer inside fishbucket.
Scenario 2: If CRC is present is fishbucket and seek pointer is same as current end of file , this means the file has already been indexed and has not been changed since last indexed. Seek pointer is used to check if there is change in file or not.
Scenario 3: If CRC is present is fishbucket and seek pointer is beyond the current end of file, this means something in the part of file which we have already read has been changed. Since we cannot know what has been changed, lets re-index the whole data again.
Location of fishbucket directory
All these CRCs and seek pointer is stored in location by default:
/opt/splunk/var/lib/splunk


Retention policy of fishbucket index

Via indexes.conf, we can change the retention policy of fishbucket index. This may be needed if we are indexing a lots of number of file. But we need to be careful when changing retention policy because if the file which has already been indexed but the CRCs and seek pointer got deleted due to change of retention policy, there is risk of same file getting indexed again.


Ways to track down a particular file when needed

If you need to know which file has been indexed and reindexed at which particular time, we can search all the events in the fishbucket associated with it by the file or source name. We can check seek pointer and mod time to know the required details.
We can also search fishbucket through GUI by searching for "index=_thefishbucket".

That's all for Splunk fishbucket. If you have any query, please mention in comment sections. Thanks.
Originally published at https://devopsrevisited.com

Related Articles:
You may also like:

Tuesday, 13 August 2019

Introduction to DevOps on AWS

Introduction:

Amazon Web Services(AWS) is a cloud service from Amazon, which provides services in the form of building blocks, these building blocks can be used to create and deploy any type of application in the cloud.

It is a comprehensive, easy to use computing platform. The platform is developed with a combination of infrastructure as a service (IaaS), platform as a service (PaaS) and packaged software as a service (SaaS) offerings.

Advantages of AWS for DevOps

There are many benefits of using AWS for Devops:

Get Started Fast

Each AWS service is ready to use if you have an AWS account. There is
no setup required or software to install.

Fully Managed Services

These services can help you take advantage of AWS resources
quicker. You can worry less about setting up, installing, and operating infrastructure on your own. This lets you focus on your core product.

Built for scale

You can manage a single instance or scale to thousands using AWS
services. These services help you make the most of flexible compute resources by
simplifying provisioning, configuration, and scaling.

Programmable

You have the option to use each service via the AWS Command Line
Interface or through APIs and SDKs. You can also model and provision AWS resources
and your entire AWS infrastructure using declarative AWS CloudFormation templates.

Automation

AWS helps you use automation so you can build faster and more efficiently.
Using AWS services, you can automate manual tasks or processes such as deployments,
development & test workflows, container management, and configuration management.

Secure

Use AWS Identity and Access Management (IAM) to set user permissions and
policies. This gives you granular control over who can access your resources and how they
access those resources.

Buffer In Amazon Web Services

An Elastic Load Balancer ensures that the incoming traffic is distributed optimally across
various AWS instances.

A buffer will synchronize different components and makes the arrangement additional
elastic to a burst of load or traffic.

The components are prone to work in an unstable way of receiving and processing the
requests. The buffer creates the equilibrium linking various apparatus and crafts them effort at the identical rate to supply more rapid services.

Components of Amazon Web Services

Amazon S3

With this, one can retrieve the key information which are occupied in creating
cloud structural design and amount of produced information also can be stored in this
component that is the consequence of the key specified.

Amazon EC2 instance

Helpful to run a large distributed system on the Hadoop cluster.Automatic parallelization and job scheduling can be achieved by this component.

Amazon SQS

This component acts as a mediator between different controllers. Also worn
for cushioning requirements those are obtained by the manager of Amazon.

Amazon SimpleDB

Helps in storing the transitional position log and the errands executed
by the consumers.

How Spot instance different from an On-Demand instance or Reserved Instance

Spot Instance, On-Demand instance and Reserved Instances are all models for pricing.

Moving along, spot instances provide the ability for customers to purchase compute
capacity with no upfront commitment, at hourly rates usually lower than the On-Demand
rate in each region.

Spot instances are just like bidding, the bidding price is called Spot Price. The Spot Price
fluctuates based on supply and demand for instances, but customers will never pay more
than the maximum price they have specified.

If the Spot Price moves higher than a customer’s maximum price, the customer’s EC2
instance will be shut down automatically.

But the reverse is not true, if the Spot prices come down again, your EC2 instance will not
be launched automatically, one must do that manually.

In Spot and on demand instance, there is no commitment for the duration from the user
side, however in reserved instances one must stick to the time period that he has chosen.

Amazon Elastic Container Service (ECS)

Amazon Elastic Container Service (ECS) is a highly scalable, high performance container
management service that supports Docker containers and allows us to easily run
applications on a managed cluster of Amazon EC2 instances.

AWS Lambda in AWS DevOps

AWS Lambda lets us run code without provisioning or managing servers. With Lambda,
we can run code for virtually any type of application or backend service, all with zero
administration.

Just upload your code and Lambda takes care of everything required to run and scale your
code with high availability.

Amazon EC2 security best practices:

There are several best practices to secure Amazon EC2. A few of them are given below:

  • Use AWS Identity and Access Management (IAM) to control access to your AWS resources.
  • Restrict access by only allowing trusted hosts or networks to access ports on your
  • instance.
  • Review the rules in your security groups regularly, and ensure that you apply the
  • principle of least.
  • Privilege – only open up permissions that you require.
  • Disable password-based logins for instances launched from your AMI. Passwords can be found or cracked and are a security risk.