Apache Iceberg Fundamentals Interview Questions Answer

Intermediate Level

1. What is Apache Iceberg?

Answer:
Apache Iceberg is an open table format designed for managing large analytical datasets in data lakes. It provides features such as schema evolution, partition evolution, snapshots, time travel, ACID transactions, and efficient metadata management. Iceberg works with processing engines such as Apache Spark, Trino, Flink, and others.

2. Why is Apache Iceberg used in data lake environments?

Answer:
Iceberg improves the reliability and manageability of data lakes by introducing database-like table capabilities. It supports atomic commits, schema evolution, partition evolution, time travel, and efficient querying. These capabilities help organizations build scalable lakehouse architectures without requiring data to be moved into traditional data warehouses.

3. What are the main components of an Iceberg table?

Answer:
An Iceberg table primarily consists of:

• Data files
• Manifest files
• Manifest lists
• Metadata files
• Snapshots
• Table schema
• Partition specifications

These components work together to track table state and enable efficient data operations.

4. What is a snapshot in Apache Iceberg?

Answer:
A snapshot represents the state of an Iceberg table at a particular point in time. Every successful table commit can create a new snapshot. Snapshots allow Iceberg to provide features such as time travel, rollback, auditing, and incremental data processing.

5. What is time travel in Iceberg?

Answer:
Time travel allows users to query a previous version of an Iceberg table using an earlier snapshot or timestamp. It is useful for auditing historical data, investigating incorrect updates, reproducing previous query results, and recovering from unwanted changes.

6. What is schema evolution?

Answer:
Schema evolution allows an Iceberg table's schema to change without requiring existing data files to be rewritten. Common operations include adding, deleting, renaming, and updating columns, depending on the capabilities of the specific engine and operation.

7. What is partition evolution in Iceberg?

Answer:
Partition evolution allows the partitioning strategy of a table to change over time. Existing data does not necessarily need to be rewritten when a new partition specification is introduced. Iceberg tracks different partition specifications through table metadata.

8. What is hidden partitioning?

Answer:
Hidden partitioning means users do not need to explicitly include partition columns in queries for partition pruning. Iceberg uses partition transforms, such as days, months, years, hours, or bucket, to organize data while keeping partition implementation details separate from the logical table schema.

9. What is a manifest file?

Answer:
A manifest file contains metadata about data files belonging to an Iceberg table snapshot. It can include information such as file paths, partition values, record counts, and column-level statistics. Query engines use this information for efficient file pruning.

10. What is a manifest list?

Answer:
A manifest list identifies the manifest files associated with a particular Iceberg snapshot. It acts as an additional metadata layer between a snapshot and its manifests, helping engines determine which data files need to be considered during query planning.

11. How does Iceberg support ACID transactions?

Answer:
Iceberg uses atomic metadata commits and immutable data files to provide transactional consistency. A table update creates a new metadata state, and readers continue to see a consistent snapshot while the update is committed.

12. What is an Iceberg catalog?

Answer:
An Iceberg catalog stores and manages information about Iceberg tables, including table locations and metadata references. Iceberg can work with different catalog implementations, such as REST-based catalogs, Hive-compatible catalogs, JDBC catalogs, and cloud-oriented catalog services.

13. How does Iceberg improve query performance?

Answer:
Iceberg improves performance through metadata-based file pruning, partition pruning, manifest filtering, column statistics, and efficient table metadata. Instead of scanning every data file, compatible query engines can eliminate files that cannot contain relevant records.

14. What is compaction in Apache Iceberg?

Answer:
Compaction combines smaller data files into larger files to improve query performance and reduce file-management overhead. This is particularly useful when frequent streaming or incremental writes create many small files.

15. How does Iceberg handle concurrent writes?

Answer:
Iceberg uses optimistic concurrency control. Multiple writers can attempt changes, but commits are validated against the current table metadata. If a conflict occurs, the operation may need to be retried or resolved according to the processing engine and operation being performed.

Advanced Level

1. Explain the metadata hierarchy of an Apache Iceberg table.

Answer:
Iceberg uses a hierarchical metadata structure:

Catalog → Table Metadata → Snapshot → Manifest List → Manifest → Data Files

The catalog identifies the table's metadata location. Table metadata references snapshots, snapshots reference manifest lists, manifest lists reference manifests, and manifests track individual data files. This structure enables efficient metadata-based query planning.

2. How does Iceberg achieve atomic table updates?

Answer:
Iceberg separates immutable data files from table metadata. A write produces new or updated data files and a new metadata state. The catalog atomically updates the table's metadata pointer. Readers either see the previous valid snapshot or the newly committed snapshot, avoiding partially committed table states.

3. What is optimistic concurrency control in Iceberg?

Answer:
Optimistic concurrency control allows multiple writers to work independently without locking the entire table. During commit, Iceberg checks whether the table state has changed since the writer began. If conflicting changes are detected, the commit can fail and the writer can retry using the latest metadata.

4. How does Iceberg enable efficient partition pruning?

Answer:
Iceberg stores partition information and statistics in manifest metadata. During query planning, the engine evaluates the query predicates against this metadata and eliminates manifests or data files that cannot contain matching records. This reduces unnecessary data scanning.

5. What are partition transforms in Iceberg?

Answer:
Partition transforms determine how source column values are transformed for partitioning. Common transforms include:

• Identity
• Bucket
• Truncate
• Year
• Month
• Day
• Hour

These transformations allow tables to use efficient partition strategies without exposing physical partition details to users.

6. How does Iceberg handle schema evolution without rewriting data?

Answer:
Iceberg assigns stable field IDs to columns rather than relying solely on column positions or names. When columns are renamed or reordered, the field IDs help engines correctly associate existing data with the logical schema. Therefore, many schema changes can occur without rewriting historical files.

7. What problem does field ID-based schema management solve?

Answer:
Field IDs prevent ambiguity caused by column renaming, reordering, or schema changes. A column's identity remains associated with its field ID, allowing Iceberg to correctly interpret data files even when the logical schema changes.

8. What is snapshot expiration?

Answer:
Snapshot expiration removes older snapshots that are no longer required. This can help reduce metadata growth and allow obsolete data files to become eligible for removal. Retention policies should be designed carefully when time travel, auditing, or rollback requirements exist.

9. What is orphan file cleanup?

Answer:
Orphan files are files that exist in the table's storage location but are no longer referenced by valid Iceberg metadata. Orphan file cleanup removes these unreferenced files. However, cleanup must be performed carefully to avoid deleting files that are still required by active or recently created snapshots.

10. What is the purpose of rewrite manifests?

Answer:
Over time, frequent table modifications can create many manifest files. Rewriting manifests can consolidate and reorganize metadata, reducing metadata overhead and improving query planning efficiency.

11. How does Iceberg support incremental data processing?

Answer:
Iceberg snapshots provide a consistent representation of table changes. Processing engines can use snapshot information to identify data added or changed between table states. This enables incremental processing patterns without scanning the entire historical dataset.

12. How does Iceberg compare with traditional Hive tables?

Answer:
Traditional Hive-style tables often rely heavily on directory-based partitioning and filesystem listings. Iceberg introduces a metadata layer that tracks table state, files, snapshots, schemas, and partitions. This provides stronger transactional guarantees, better schema and partition evolution, and more reliable operations at large scale.

13. How would you optimize an Iceberg table containing millions of small files?

Answer:
A typical optimization strategy would include:

1. Identifying the source of small-file generation.
2. Running data-file compaction or file rewriting.
3. Adjusting write and distribution settings.
4. Reviewing partition strategy.
5. Optimizing manifest metadata.
6. Applying appropriate snapshot-retention policies.

The objective is to create appropriately sized files while avoiding excessive partition fragmentation.

14. What is the difference between data-file rewriting and manifest rewriting?

Answer:
Data-file rewriting reorganizes the physical data files, often combining small files or improving their layout.

Manifest rewriting reorganizes metadata files that describe those data files.

Data-file rewriting primarily affects physical data organization and query efficiency, while manifest rewriting primarily improves metadata management and query planning.

15. How would you design an Apache Iceberg table for a large-scale lakehouse?

Answer:
A good design should consider workload patterns, data volume, query predicates, update frequency, file sizes, partition transforms, catalog architecture, and retention requirements. Rather than over-partitioning, use appropriate transforms and rely on Iceberg's metadata-based pruning. Regular maintenance such as compaction, manifest optimization, and snapshot expiration should also be incorporated into the operational strategy.