Skip to content

Data Engineering Knowledge Base

Index

Initializing search

Data Engineering Knowledge Base

Home
Azure
Azure
- Topics
  Topics
Data Formats
Data Formats
- Index
- Topics
  Topics
  - Data Format Deep Dive Pt1
  - Parquet Format Internals
Databricks
Databricks
- Topics
  Topics
Scenarios
Scenarios
- Topics
  Topics
Spark
Spark
- Topics
  Topics
Streaming
Streaming
- Topics
  Topics
  - Architecture
    
    Architecture
    
    Topics
    Topics
    
    Use Cases Streaming
    
    Redpanda Vs Kafka Arch Differences
    
    Redpanda Architure In Depth Pt1
  - Kafka
    
    Kafka
    
    Topics
    Topics
    
    Kafka Kraft Setup
    
    Kafka Broker Properties
    
    Topic Default Properties
    
    Kafka Hardware Considerations
    
    Kafka Configuring Clusters Broker Consideration
    
    Kafka Broker Os Tuning
    
    Kafka Os Tuning Dirty Page Handling
    
    Kafka File Descriptors Overcommit Memory
    
    Kafka Production Concerns
    
    Kafka Message Types
    
    Kafka Configuring Producers Pt1
    
    Kafka Configuring Producers Pt2
    
    Kafka File Descriptors Overcommit Memory
    
    Kafka Production Concerns
    
    Kafka Message Types
    
    Kafka Configuring Producers Pt1
    
    Kafka Configuring Producers Pt2
    
    Kafka Serializers Avro Pt1
    
    Kafka Serializers Avro Pt2
    
    Kafka Partitions
    
    Kafka Headers
    
    Kafka Interceptors
    
    Kafka Quotas and Throttling
    
    Kafka Consumers Eager and Coorperative Rebalancing
    
    Kafka Consumer Static Partitioning
    
    Kafka Poll Loop
    
    Kafka Consumer Properties Part I
    
    Kafka Consumer Properties Part II
    
    Kafka Partition Assignment Strategies
    
    Kafka Commits and Offsets
    
    Kafka Types of Commits
    
    Kafka Rebalance Listeners
    
    Kafka Consuming Records with Specified Offset
    
    Kafka Exiting Consumers and Poll Loop
    
    Kafka Deserializers
    
    Kafka Standalone Consumers
    
    Kafka Internals of Zookeeper
    
    Kafka Raft Consensus Protocol
    
    Kafka Controller Quorum Concepts
    
    Kafka Replication Concepts
    
    Kafka Insync and Out Of Sync Replicas
    
    Kafka Request Processing Introduction
    
    Kafka Request Processing - Producer Requests
    
    Kafka Request Processing - Fetch Requests Part 1
    
    Kafka Request Processing - Fetch Requests Part 2
    
    Kafka Physical Storage - Introduction
    
    Kafka Tiered Storage Concepts
    
    Kafka Partition Allocation Process
    
    Kafka File Formats Introduction
    
    Kafka Message Batch Headers
    
    Kafka Indexes
    
    Kafka Compaction Concepts
    
    Kafka Tombstoning Process
    
    Kafka Reliability Guarantees
    
    Kafka Replication Procedures
    
    Kafka Broker Configuration - Replication Factor
    
    Kafka Broker Configuration - Unclean Leader Election
    
    Kafka Log Truncation and Out Of Sync Leaders
    
    Kafka Keeping Replicas In Sync
    
    Kafka Producer Failure Scenarios
Documentation Deep Dive
Documentation Deep Dive
- Topics
  Topics
  - Databricks
    Databricks
    
    Topics
    Topics
    
    What is Lakehouse?
    
    Lakehouse vs Delta Lake vs Warehouse
    
    All things Delta!
    
    High Level Databricks Architecture
    
    Databricks ACID Guarantees
    
    Medallion Design Pattern
    
    Databricks Single Source of Truth Architecture
    
    Databricks Scope of Lakehouse Architecture
    
    Databricks Architectural Guiding Principles
    
    Databricks Objects - Volumes and Tables
    
    Databricks Objects - Views
    
    Databricks Governed Tags
    
    Databricks Connecting to Cloud Object Storage
    
    Databricks Managed Storage Location Hierarchy

Streaming🔗

This is the overview page for Streaming.

Spark Dynamic Partition Pruning

© 2023-2025 Vedanth V Baliga