Stay Ahead, Stay ONMINE

Anatomy of a Parquet File

In recent years, Parquet has become a standard format for data storage in Big Data ecosystems. Its column-oriented format offers several advantages: Faster query execution when only a subset of columns is being processed Quick calculation of statistics across all data Reduced storage volume thanks to efficient compression When combined with storage frameworks like Delta Lake or Apache Iceberg, it seamlessly integrates with query engines (e.g., Trino) and data warehouse compute clusters (e.g., Snowflake, BigQuery). In this article, the content of a Parquet file is dissected using mainly standard Python tools to better understand its structure and how it contributes to such performances. Writing Parquet file(s) To produce Parquet files, we use PyArrow, a Python binding for Apache Arrow that stores dataframes in memory in columnar format. PyArrow allows fine-grained parameter tuning when writing the file. This makes PyArrow ideal for Parquet manipulation (one can also simply use Pandas). # generator.py import pyarrow as pa import pyarrow.parquet as pq from faker import Faker fake = Faker() Faker.seed(12345) num_records = 100 # Generate fake data names = [fake.name() for _ in range(num_records)] addresses = [fake.address().replace(“n”, “, “) for _ in range(num_records)] birth_dates = [ fake.date_of_birth(minimum_age=67, maximum_age=75) for _ in range(num_records) ] cities = [addr.split(“, “)[1] for addr in addresses] birth_years = [date.year for date in birth_dates] # Cast the data to the Arrow format name_array = pa.array(names, type=pa.string()) address_array = pa.array(addresses, type=pa.string()) birth_date_array = pa.array(birth_dates, type=pa.date32()) city_array = pa.array(cities, type=pa.string()) birth_year_array = pa.array(birth_years, type=pa.int32()) # Create schema with non-nullable fields schema = pa.schema( [ pa.field(“name”, pa.string(), nullable=False), pa.field(“address”, pa.string(), nullable=False), pa.field(“date_of_birth”, pa.date32(), nullable=False), pa.field(“city”, pa.string(), nullable=False), pa.field(“birth_year”, pa.int32(), nullable=False), ] ) table = pa.Table.from_arrays( [name_array, address_array, birth_date_array, city_array, birth_year_array], schema=schema, ) print(table) pyarrow.Table name: string not null address: string not null date_of_birth: date32[day] not null city: string not null birth_year: int32 not null —- name: [[“Adam Bryan”,”Jacob Lee”,”Candice Martinez”,”Justin Thompson”,”Heather Rubio”]] address: [[“822 Jennifer Field Suite 507, Anthonyhaven, UT 98088″,”292 Garcia Mall, Lake Belindafurt, IN 69129″,”31738 Jonathan Mews Apt. 024, East Tammiestad, ND 45323″,”00716 Kristina Trail Suite 381, Howelltown, SC 64961″,”351 Christopher Expressway Suite 332, West Edward, CO 68607”]] date_of_birth: [[1955-06-03,1950-06-24,1955-01-29,1957-02-18,1956-09-04]] city: [[“Anthonyhaven”,”Lake Belindafurt”,”East Tammiestad”,”Howelltown”,”West Edward”]] birth_year: [[1955,1950,1955,1957,1956]] The output clearly reflects a columns-oriented storage, unlike Pandas, which usually displays a traditional “row-wise” table. How is a Parquet file stored? Parquet files are generally stored in cheap object storage databases like S3 (AWS) or GCS (GCP) to be easily accessible by data processing pipelines. These files are usually organized with a partitioning strategy by leveraging directory structures: # generator.py num_records = 100 # … # Writing the parquet files to disk pq.write_to_dataset( table, root_path=’dataset’, partition_cols=[‘birth_year’, ‘city’] ) If birth_year and city columns are defined as partitioning keys, PyArrow creates such a tree structure in the directory dataset: dataset/ ├─ birth_year=1949/ ├─ birth_year=1950/ │ ├─ city=Aaronbury/ │ │ ├─ 828d313a915a43559f3111ee8d8e6c1a-0.parquet │ │ ├─ 828d313a915a43559f3111ee8d8e6c1a-0.parquet │ │ ├─ … │ ├─ city=Alicialand/ │ ├─ … ├─ birth_year=1951 ├─ … The strategy enables partition pruning: when a query filters on these columns, the engine can use folder names to read only the necessary files. This is why the partitioning strategy is crucial for limiting delay, I/O, and compute resources when handling large volumes of data (as has been the case for decades with traditional relational databases). The pruning effect can be easily verified by counting the files opened by a Python script that filters the birth year: # query.py import duckdb duckdb.sql( “”” SELECT * FROM read_parquet(‘dataset/*/*/*.parquet’, hive_partitioning = true) where birth_year = 1949 “”” ).show() > strace -e trace=open,openat,read -f python query.py 2 >&1 | grep “dataset/.*.parquet” [pid 37] openat(AT_FDCWD, “dataset/birth_year=1949/city=Box%201306/e1ad1666a2144fbc94892d4ac1234c64-0.parquet”, O_RDONLY) = 3 [pid 37] openat(AT_FDCWD, “dataset/birth_year=1949/city=Box%201306/e1ad1666a2144fbc94892d4ac1234c64-0.parquet”, O_RDONLY) = 3 [pid 39] openat(AT_FDCWD, “dataset/birth_year=1949/city=Box%201306/e1ad1666a2144fbc94892d4ac1234c64-0.parquet”, O_RDONLY) = 4 [pid 39] openat(AT_FDCWD, “dataset/birth_year=1949/city=Box%203487/e1ad1666a2144fbc94892d4ac1234c64-0.parquet”, O_RDONLY) = 5 [pid 39] openat(AT_FDCWD, “dataset/birth_year=1949/city=Box%203487/e1ad1666a2144fbc94892d4ac1234c64-0.parquet”, O_RDONLY) = 3 [pid 39] openat(AT_FDCWD, “dataset/birth_year=1949/city=Clarkemouth/e1ad1666a2144fbc94892d4ac1234c64-0.parquet”, O_RDONLY) = 4 [pid 39] openat(AT_FDCWD, “dataset/birth_year=1949/city=Clarkemouth/e1ad1666a2144fbc94892d4ac1234c64-0.parquet”, O_RDONLY) = 5 [pid 39] openat(AT_FDCWD, “dataset/birth_year=1949/city=DPO%20AP%2020198/e1ad1666a2144fbc94892d4ac1234c64-0.parquet”, O_RDONLY) = 3 [pid 39] openat(AT_FDCWD, “dataset/birth_year=1949/city=DPO%20AP%2020198/e1ad1666a2144fbc94892d4ac1234c64-0.parquet”, O_RDONLY) = 4 [pid 39] openat(AT_FDCWD, “dataset/birth_year=1949/city=East%20Morgan/e1ad1666a2144fbc94892d4ac1234c64-0.parquet”, O_RDONLY) = 5 [pid 39] openat(AT_FDCWD, “dataset/birth_year=1949/city=East%20Morgan/e1ad1666a2144fbc94892d4ac1234c64-0.parquet”, O_RDONLY) = 3 [pid 39] openat(AT_FDCWD, “dataset/birth_year=1949/city=FPO%20AA%2006122/e1ad1666a2144fbc94892d4ac1234c64-0.parquet”, O_RDONLY) = 4 [pid 39] openat(AT_FDCWD, “dataset/birth_year=1949/city=FPO%20AA%2006122/e1ad1666a2144fbc94892d4ac1234c64-0.parquet”, O_RDONLY) = 5 [pid 39] openat(AT_FDCWD, “dataset/birth_year=1949/city=New%20Michelleport/e1ad1666a2144fbc94892d4ac1234c64-0.parquet”, O_RDONLY) = 3 [pid 39] openat(AT_FDCWD, “dataset/birth_year=1949/city=New%20Michelleport/e1ad1666a2144fbc94892d4ac1234c64-0.parquet”, O_RDONLY) = 4 [pid 39] openat(AT_FDCWD, “dataset/birth_year=1949/city=North%20Danielchester/e1ad1666a2144fbc94892d4ac1234c64-0.parquet”, O_RDONLY) = 5 [pid 39] openat(AT_FDCWD, “dataset/birth_year=1949/city=North%20Danielchester/e1ad1666a2144fbc94892d4ac1234c64-0.parquet”, O_RDONLY) = 3 [pid 39] openat(AT_FDCWD, “dataset/birth_year=1949/city=Port%20Chase/e1ad1666a2144fbc94892d4ac1234c64-0.parquet”, O_RDONLY) = 4 [pid 39] openat(AT_FDCWD, “dataset/birth_year=1949/city=Port%20Chase/e1ad1666a2144fbc94892d4ac1234c64-0.parquet”, O_RDONLY) = 5 [pid 39] openat(AT_FDCWD, “dataset/birth_year=1949/city=Richardmouth/e1ad1666a2144fbc94892d4ac1234c64-0.parquet”, O_RDONLY) = 3 [pid 39] openat(AT_FDCWD, “dataset/birth_year=1949/city=Richardmouth/e1ad1666a2144fbc94892d4ac1234c64-0.parquet”, O_RDONLY) = 4 [pid 39] openat(AT_FDCWD, “dataset/birth_year=1949/city=Robbinsshire/e1ad1666a2144fbc94892d4ac1234c64-0.parquet”, O_RDONLY) = 5 [pid 39] openat(AT_FDCWD, “dataset/birth_year=1949/city=Robbinsshire/e1ad1666a2144fbc94892d4ac1234c64-0.parquet”, O_RDONLY) = 3 Only 23 files are read out of 100. Reading a raw Parquet file Let’s decode a raw Parquet file without specialized libraries. For simplicity, the dataset is dumped into a single file without compression or encoding. # generator.py # … pq.write_table( table, “dataset.parquet”, use_dictionary=False, compression=”NONE”, write_statistics=True, column_encoding=None, ) The first thing to know is that the binary file is framed by 4 bytes whose ASCII representation is “PAR1”. The file is corrupted if this is not the case. # reader.py with open(“dataset.parquet”, “rb”) as file: parquet_data = file.read() assert parquet_data[:4] == b”PAR1″, “Not a valid parquet file” assert parquet_data[-4:] == b”PAR1″, “File footer is corrupted” As indicated in the documentation, the file is divided into two parts: the “row groups” containing actual data, and the footer containing metadata (schema below). The footer The size of the footer is indicated in the 4 bytes preceding the end marker as an unsigned integer written in “little endian” format (noted “ 1955 and a row group’s maximum birth year is 1954, the engine can efficiently skip that entire data section. This optimisation is called “predicate pushdown”. Parquet also stores other useful statistics like distinct value counts and null counts. # reader.py # … first_row_group = file_metadata_thrift.row_groups[0] birth_year_column = first_row_group.columns[4] min_stat_bytes = birth_year_column.meta_data.statistics.min max_stat_bytes = birth_year_column.meta_data.statistics.max min_year = struct.unpack(“

In recent years, Parquet has become a standard format for data storage in Big Data ecosystems. Its column-oriented format offers several advantages:

  • Faster query execution when only a subset of columns is being processed
  • Quick calculation of statistics across all data
  • Reduced storage volume thanks to efficient compression

When combined with storage frameworks like Delta Lake or Apache Iceberg, it seamlessly integrates with query engines (e.g., Trino) and data warehouse compute clusters (e.g., Snowflake, BigQuery). In this article, the content of a Parquet file is dissected using mainly standard Python tools to better understand its structure and how it contributes to such performances.

Writing Parquet file(s)

To produce Parquet files, we use PyArrow, a Python binding for Apache Arrow that stores dataframes in memory in columnar format. PyArrow allows fine-grained parameter tuning when writing the file. This makes PyArrow ideal for Parquet manipulation (one can also simply use Pandas).

# generator.py

import pyarrow as pa
import pyarrow.parquet as pq
from faker import Faker

fake = Faker()
Faker.seed(12345)
num_records = 100

# Generate fake data
names = [fake.name() for _ in range(num_records)]
addresses = [fake.address().replace("n", ", ") for _ in range(num_records)]
birth_dates = [
    fake.date_of_birth(minimum_age=67, maximum_age=75) for _ in range(num_records)
]
cities = [addr.split(", ")[1] for addr in addresses]
birth_years = [date.year for date in birth_dates]

# Cast the data to the Arrow format
name_array = pa.array(names, type=pa.string())
address_array = pa.array(addresses, type=pa.string())
birth_date_array = pa.array(birth_dates, type=pa.date32())
city_array = pa.array(cities, type=pa.string())
birth_year_array = pa.array(birth_years, type=pa.int32())

# Create schema with non-nullable fields
schema = pa.schema(
    [
        pa.field("name", pa.string(), nullable=False),
        pa.field("address", pa.string(), nullable=False),
        pa.field("date_of_birth", pa.date32(), nullable=False),
        pa.field("city", pa.string(), nullable=False),
        pa.field("birth_year", pa.int32(), nullable=False),
    ]
)

table = pa.Table.from_arrays(
    [name_array, address_array, birth_date_array, city_array, birth_year_array],
    schema=schema,
)

print(table)
pyarrow.Table
name: string not null
address: string not null
date_of_birth: date32[day] not null
city: string not null
birth_year: int32 not null
----
name: [["Adam Bryan","Jacob Lee","Candice Martinez","Justin Thompson","Heather Rubio"]]
address: [["822 Jennifer Field Suite 507, Anthonyhaven, UT 98088","292 Garcia Mall, Lake Belindafurt, IN 69129","31738 Jonathan Mews Apt. 024, East Tammiestad, ND 45323","00716 Kristina Trail Suite 381, Howelltown, SC 64961","351 Christopher Expressway Suite 332, West Edward, CO 68607"]]
date_of_birth: [[1955-06-03,1950-06-24,1955-01-29,1957-02-18,1956-09-04]]
city: [["Anthonyhaven","Lake Belindafurt","East Tammiestad","Howelltown","West Edward"]]
birth_year: [[1955,1950,1955,1957,1956]]

The output clearly reflects a columns-oriented storage, unlike Pandas, which usually displays a traditional “row-wise” table.

How is a Parquet file stored?

Parquet files are generally stored in cheap object storage databases like S3 (AWS) or GCS (GCP) to be easily accessible by data processing pipelines. These files are usually organized with a partitioning strategy by leveraging directory structures:

# generator.py

num_records = 100

# ...

# Writing the parquet files to disk
pq.write_to_dataset(
    table,
    root_path='dataset',
    partition_cols=['birth_year', 'city']
)

If birth_year and city columns are defined as partitioning keys, PyArrow creates such a tree structure in the directory dataset:

dataset/
├─ birth_year=1949/
├─ birth_year=1950/
│ ├─ city=Aaronbury/
│ │ ├─ 828d313a915a43559f3111ee8d8e6c1a-0.parquet
│ │ ├─ 828d313a915a43559f3111ee8d8e6c1a-0.parquet
│ │ ├─ …
│ ├─ city=Alicialand/
│ ├─ …
├─ birth_year=1951 ├─ ...

The strategy enables partition pruning: when a query filters on these columns, the engine can use folder names to read only the necessary files. This is why the partitioning strategy is crucial for limiting delay, I/O, and compute resources when handling large volumes of data (as has been the case for decades with traditional relational databases).

The pruning effect can be easily verified by counting the files opened by a Python script that filters the birth year:

# query.py
import duckdb

duckdb.sql(
    """
    SELECT * 
    FROM read_parquet('dataset/*/*/*.parquet', hive_partitioning = true)
    where birth_year = 1949
    """
).show()
> strace -e trace=open,openat,read -f python query.py 2>&1 | grep "dataset/.*.parquet"

[pid    37] openat(AT_FDCWD, "dataset/birth_year=1949/city=Box%201306/e1ad1666a2144fbc94892d4ac1234c64-0.parquet", O_RDONLY) = 3
[pid    37] openat(AT_FDCWD, "dataset/birth_year=1949/city=Box%201306/e1ad1666a2144fbc94892d4ac1234c64-0.parquet", O_RDONLY) = 3
[pid    39] openat(AT_FDCWD, "dataset/birth_year=1949/city=Box%201306/e1ad1666a2144fbc94892d4ac1234c64-0.parquet", O_RDONLY) = 4
[pid    39] openat(AT_FDCWD, "dataset/birth_year=1949/city=Box%203487/e1ad1666a2144fbc94892d4ac1234c64-0.parquet", O_RDONLY) = 5
[pid    39] openat(AT_FDCWD, "dataset/birth_year=1949/city=Box%203487/e1ad1666a2144fbc94892d4ac1234c64-0.parquet", O_RDONLY) = 3
[pid    39] openat(AT_FDCWD, "dataset/birth_year=1949/city=Clarkemouth/e1ad1666a2144fbc94892d4ac1234c64-0.parquet", O_RDONLY) = 4
[pid    39] openat(AT_FDCWD, "dataset/birth_year=1949/city=Clarkemouth/e1ad1666a2144fbc94892d4ac1234c64-0.parquet", O_RDONLY) = 5
[pid    39] openat(AT_FDCWD, "dataset/birth_year=1949/city=DPO%20AP%2020198/e1ad1666a2144fbc94892d4ac1234c64-0.parquet", O_RDONLY) = 3
[pid    39] openat(AT_FDCWD, "dataset/birth_year=1949/city=DPO%20AP%2020198/e1ad1666a2144fbc94892d4ac1234c64-0.parquet", O_RDONLY) = 4
[pid    39] openat(AT_FDCWD, "dataset/birth_year=1949/city=East%20Morgan/e1ad1666a2144fbc94892d4ac1234c64-0.parquet", O_RDONLY) = 5
[pid    39] openat(AT_FDCWD, "dataset/birth_year=1949/city=East%20Morgan/e1ad1666a2144fbc94892d4ac1234c64-0.parquet", O_RDONLY) = 3
[pid    39] openat(AT_FDCWD, "dataset/birth_year=1949/city=FPO%20AA%2006122/e1ad1666a2144fbc94892d4ac1234c64-0.parquet", O_RDONLY) = 4
[pid    39] openat(AT_FDCWD, "dataset/birth_year=1949/city=FPO%20AA%2006122/e1ad1666a2144fbc94892d4ac1234c64-0.parquet", O_RDONLY) = 5
[pid    39] openat(AT_FDCWD, "dataset/birth_year=1949/city=New%20Michelleport/e1ad1666a2144fbc94892d4ac1234c64-0.parquet", O_RDONLY) = 3
[pid    39] openat(AT_FDCWD, "dataset/birth_year=1949/city=New%20Michelleport/e1ad1666a2144fbc94892d4ac1234c64-0.parquet", O_RDONLY) = 4
[pid    39] openat(AT_FDCWD, "dataset/birth_year=1949/city=North%20Danielchester/e1ad1666a2144fbc94892d4ac1234c64-0.parquet", O_RDONLY) = 5
[pid    39] openat(AT_FDCWD, "dataset/birth_year=1949/city=North%20Danielchester/e1ad1666a2144fbc94892d4ac1234c64-0.parquet", O_RDONLY) = 3
[pid    39] openat(AT_FDCWD, "dataset/birth_year=1949/city=Port%20Chase/e1ad1666a2144fbc94892d4ac1234c64-0.parquet", O_RDONLY) = 4
[pid    39] openat(AT_FDCWD, "dataset/birth_year=1949/city=Port%20Chase/e1ad1666a2144fbc94892d4ac1234c64-0.parquet", O_RDONLY) = 5
[pid    39] openat(AT_FDCWD, "dataset/birth_year=1949/city=Richardmouth/e1ad1666a2144fbc94892d4ac1234c64-0.parquet", O_RDONLY) = 3
[pid    39] openat(AT_FDCWD, "dataset/birth_year=1949/city=Richardmouth/e1ad1666a2144fbc94892d4ac1234c64-0.parquet", O_RDONLY) = 4
[pid    39] openat(AT_FDCWD, "dataset/birth_year=1949/city=Robbinsshire/e1ad1666a2144fbc94892d4ac1234c64-0.parquet", O_RDONLY) = 5
[pid    39] openat(AT_FDCWD, "dataset/birth_year=1949/city=Robbinsshire/e1ad1666a2144fbc94892d4ac1234c64-0.parquet", O_RDONLY) = 3

Only 23 files are read out of 100.

Reading a raw Parquet file

Let’s decode a raw Parquet file without specialized libraries. For simplicity, the dataset is dumped into a single file without compression or encoding.

# generator.py

# ...

pq.write_table(
    table,
    "dataset.parquet",
    use_dictionary=False,
    compression="NONE",
    write_statistics=True,
    column_encoding=None,
)

The first thing to know is that the binary file is framed by 4 bytes whose ASCII representation is “PAR1”. The file is corrupted if this is not the case.

# reader.py

with open("dataset.parquet", "rb") as file:
    parquet_data = file.read()

assert parquet_data[:4] == b"PAR1", "Not a valid parquet file"
assert parquet_data[-4:] == b"PAR1", "File footer is corrupted"

As indicated in the documentation, the file is divided into two parts: the “row groups” containing actual data, and the footer containing metadata (schema below).

The footer

The size of the footer is indicated in the 4 bytes preceding the end marker as an unsigned integer written in “little endian” format (noted “unpack function).

# reader.py

import struct

# ...

footer_length = struct.unpack("
Footer size in bytes: 1088

The footer information is encoded in a cross-language serialization format called Apache Thrift. Using a human-readable but verbose format like JSON and then translating it into binary would be less efficient in terms of memory usage. With Thrift, one can declare data structures as follows:

struct Customer {
	1: required string name,
	2: optional i16 birthYear,
	3: optional list interests
}

On the basis of this declaration, Thrift can generate Python code to decode byte strings with such data structure (it also generates code to perform the encoding part). The thrift file containing all the data structures implemented in a Parquet file can be downloaded here. After having installed the thrift binary, let’s run:

thrift -r --gen py parquet.thrift

The generated Python code is placed in the “gen-py” folder. The footer’s data structure is represented by the FileMetaData class – a Python class automatically generated from the Thrift schema. Using Thrift’s Python utilities, binary data is parsed and populated into an instance of this FileMetaData class.

# reader.py

import sys

# ...

# Add the generated classes to the python path
sys.path.append("gen-py")
from parquet.ttypes import FileMetaData, PageHeader
from thrift.transport import TTransport
from thrift.protocol import TCompactProtocol

def read_thrift(data, thrift_instance):
    """
    Read a Thrift object from a binary buffer.
    Returns the Thrift object and the number of bytes read.
    """
    transport = TTransport.TMemoryBuffer(data)
    protocol = TCompactProtocol.TCompactProtocol(transport)
    thrift_instance.read(protocol)
    return thrift_instance, transport._buffer.tell()

# The number of bytes read is not used for now
file_metadata_thrift, _ = read_thrift(footer_data, FileMetaData())

print(f"Number of rows in the whole file: {file_metadata_thrift.num_rows}")
print(f"Number of row groups: {len(file_metadata_thrift.row_groups)}")

Number of rows in the whole file: 100
Number of row groups: 1

The footer contains extensive information about the file’s structure and content. For instance, it accurately tracks the number of rows in the generated dataframe. These rows are all contained within a single “row group.” But what is a “row group?”

Row groups

Unlike purely column-oriented formats, Parquet employs a hybrid approach. Before writing column blocks, the dataframe is first partitioned vertically into row groups (the parquet file we generated is too small to be split in multiple row groups).

This hybrid structure offers several advantages:

Parquet calculates statistics (such as min/max values) for each column within each row group. These statistics are crucial for query optimization, allowing query engines to skip entire row groups that don’t match filtering criteria. For example, if a query filters for birth_year > 1955 and a row group’s maximum birth year is 1954, the engine can efficiently skip that entire data section. This optimisation is called “predicate pushdown”. Parquet also stores other useful statistics like distinct value counts and null counts.

# reader.py
# ...

first_row_group = file_metadata_thrift.row_groups[0]
birth_year_column = first_row_group.columns[4]

min_stat_bytes = birth_year_column.meta_data.statistics.min
max_stat_bytes = birth_year_column.meta_data.statistics.max

min_year = struct.unpack("
The birth year range is between 1949 and 1958
  • Row groups enable parallel processing of data (particularly valuable for frameworks like Apache Spark). The size of these row groups can be configured based on the computing resources available (using the row_group_size property in function write_table when using PyArrow).
# generator.py

# ...

pq.write_table(
    table,
    "dataset.parquet",
    row_group_size=100,
)

# /! Keep the default value of "row_group_size" for the next parts
  • Even if this is not the primary objective of a column format, Parquet’s hybrid structure maintains reasonable performance when reconstructing complete rows. Without row groups, rebuilding an entire row might require scanning the entirety of each column which would be extremely inefficient for large files.

Data Pages

The smallest substructure of a Parquet file is the page. It contains a sequence of values from the same column and, therefore, of the same type. The choice of page size is the result of a trade-off:

  • Larger pages mean less metadata to store and read, which is optimal for queries with minimal filtering.
  • Smaller pages reduce the amount of unnecessary data read, which is better when queries target small, scattered data ranges.

Now let’s decode the contents of the first page of the column dedicated to addresses whose location can be found in the footer (given by the data_page_offset attribute of the right ColumnMetaData) . Each page is preceded by a Thrift PageHeader object containing some metadata. The offset actually points to a Thrift binary representation of the page metadata that precedes the page itself. The Thrift class is called a PageHeader and can also be found in the gen-py directory.

💡 Between the PageHeader and the actual values contained within the page, there may be a few bytes dedicated to implementing the Dremel format, which allows encoding nested data structures. Since our data has a regular tabular format and the values are not nullable, these bytes are skipped when writing the file (https://parquet.apache.org/docs/file-format/data-pages/).

# reader.py
# ...

address_column = first_row_group.columns[1]
column_start = address_column.meta_data.data_page_offset
column_end = column_start + address_column.meta_data.total_compressed_size
column_content = parquet_data[column_start:column_end]

page_thrift, page_header_size = read_thrift(column_content, PageHeader())
page_content = column_content[
    page_header_size : (page_header_size + page_thrift.compressed_page_size)
]
print(column_content[:100])
b'6x00x00x00481 Mata Squares Suite 260, Lake Rachelville, KY 874642x00x00x00671 Barker Crossing Suite 390, Mooreto'

The generated values finally appear, in plain text and not encoded (as specified when writing the Parquet file). However, to optimize the columnar format, it is recommended to use one of the following encoding algorithms: dictionary encoding, run length encoding (RLE), or delta encoding (the latter being reserved for int32 and int64 types), followed by compression using gzip or snappy (available codecs are listed here). Since encoded pages contain similar values (all addresses, all decimal numbers, etc.), compression ratios can be particularly advantageous.

As documented in the specification, when character strings (BYTE_ARRAY) are not encoded, each value is preceded by its size represented as a 4-byte integer. This can be observed in the previous output:

To read all the values (for example, the first 10), the loop is rather simple:

idx = 0
for _ in range(10):
    str_size = struct.unpack("481 Mata Squares Suite 260, Lake Rachelville, KY 87464
671 Barker Crossing Suite 390, Mooretown, MI 21488
62459 Jordan Knoll Apt. 970, Emilyfort, DC 80068
948 Victor Square Apt. 753, Braybury, RI 67113
365 Edward Place Apt. 162, Calebborough, AL 13037
894 Reed Lock, New Davidmouth, NV 84612
24082 Allison Squares Suite 345, North Sharonberg, WY 97642
00266 Johnson Drives, South Lori, MI 98513
15255 Kelly Plains, Richardmouth, GA 33438
260 Thomas Glens, Port Gabriela, OH 96758

And there we have it! We have successfully recreated, in a very simple way, how a specialized library would read a Parquet file. By understanding its building blocks including headers, footers, row groups, and data pages, we can better appreciate how features like predicate pushdown and partition pruning deliver such impressive performance benefits in data-intensive environments. I am convinced knowing how Parquet works under the hood helps making better decisions about storage strategies, compression choices, and performance optimization.

All the code used in this article is available on my GitHub repository at https://github.com/kili-mandjaro/anatomy-parquet, where you can explore more examples and experiment with different Parquet file configurations.

Whether you are building data pipelines, optimizing query performance, or simply curious about data storage formats, I hope this deep dive into Parquet’s inner structures has provided valuable insights for your Data Engineering journey.

All images are by the author.

Shape
Shape
Stay Ahead

Explore More Insights

Stay ahead with more perspectives on cutting-edge power, infrastructure, energy,  bitcoin and AI solutions. Explore these articles to uncover strategies and insights shaping the future of industries.

Shape

Intel under Tan: What enterprise IT buyers need to know

Intel’s discrete GPU ambitions — especially in enterprise AI — have often appeared reactive rather than part of a clear strategic vision. The company entered the market late, facing Nvidia’s dominant CUDA ecosystem and AMD’s aggressive push into AI GPUs. “Tan’s background suggests he is unlikely to double down on

Read More »

SUSE expands AI tools to control workloads, LLM usage

“And every few weeks we’ll continue to add to the library,” Puri says. SUSE also announced a partnership with Infosys today. The system integrator has the Topaz AI platform, which includes a set of services and solutions to help enterprises build and deploy AI applications. SUSE is also integrating the

Read More »

D-Wave uses quantum to solve real-world problem

D-Wave published its results today, peer-reviewed in the journal Science. The classical supercomputer that D-Wave benchmarked against was the Frontier supercomputer at the Department of Energy’s Oak Ridge National Laboratory. It was, until recently, the most powerful supercomputer in the world but moved to second place in November. Two different

Read More »

Realizing the Internet of Everything

“Big brother is watching you” is a catchphrase for the risk of large-scale surveillance. We could identify criminals walking on the street with widespread deployment of video, and the same technology could warn us against stepping into traffic. But the same stuff could help people stalk others, spy on people,

Read More »

Aramco Oil Sales to China Set to Fall Sharply in April

Saudi Aramco is set to supply the lowest amount of oil to China in several months, even as the OPEC+ cartel gears up to boost output. The state-owned Saudi Arabian major will send 34 million to 36 million barrels of April-loading crude to customers in China, the world’s biggest importer, according to data compiled by Bloomberg. That compares with 41 million for March, and a figure below 36 million would be the smallest since at least the first half of last year. Aramco’s monthly sales to Asia, and in particular to China, are an important source of supply for many refiners across the region. The volume of the shipments, which are sold only via long-term contracts, determines how much crude these processors will need to buy on the spot market from producers including Iraq, the United Arab Emirates and West Africa. The latest official selling prices offered by Aramco were lower than expected in a Bloomberg survey. At present, it’s unclear if the drop in sales is due to lower requests for cargoes from Chinese customers, or a reduction in supply from the kingdom. The decline in the Saudi volumes comes as the Organization of the Petroleum Exporting Countries and its allies aim to restore output from next month, kicking off the first of what could be a long series of modest supply increases. That move – coupled with wider concerns about the potential impact on energy demand from the US-led trade war – have weighed on crude futures. Three other refiners outside of China said they received all of the crude that they had requested. Aramco’s press office didn’t respond to an email seeking comment. What do you think? We’d love to hear from you, join the conversation on the Rigzone Energy Network. The Rigzone Energy Network is a new social experience

Read More »

Uniper Makes Nearly $3B Bailout-Related Payment to Germany

Uniper SE said Thursday it had remitted to Germany about EUR 2.6 billion ($2.82 billion) in aid repayments in relation to the government’s bail-out of the power and natural gas utility in 2022. Last year Uniper allotted a provisional EUR 3.4 billion to compensate the state for keeping the Düsseldorf-based company afloat during the gas crisis following Russia’s invasion of Ukraine. The amount was subject to Uniper’s 2024 performance. “In the 2024 consolidated financial statements, the amount of these repayment obligations to the Federal Republic of Germany was determined to be around EUR 2.6 billion, which were settled in full on 11 March 2025”, Uniper said in an online statement. Uniper chief financial officer Jutta Dönges commented, “Uniper has fulfilled an important condition in connection with the stabilization [bail-out] in 2022”. “It is proof that Uniper is financially stronger after the crisis and is operating profitably”, Dönges added. “We have learnt the right lessons from the crisis and are well equipped to make our contribution to a secure energy supply”. After winning arbitration against Gazprom PJSC in 2024 for undelivered gas, Uniper paid Germany EUR 530 million in September using part of claims realized from the ruling’s award of EUR 13 billion in damages, according to Uniper’s report of yearly results February 25, 2025. For 2024 it logged EUR 221 million in net profit and EUR 1.6 billion in adjusted net profit, sharply down from EUR 6.34 billion and EUR 4.43 billion, respectively, for 2023. In 2022, when Germany bailed out Uniper, it recorded a net loss of EUR 19.14 billion (-EUR 7.4 billion after adjustments). Adjusted earnings before income, taxes, depreciation and amortization (EBITDA) totaled EUR 2.61 billion for 2024, compared to EUR 7.16 billion for the prior year and -EUR 10.12 billion for 2022. Uniper attributed the huge drop in

Read More »

Shell Completes Sale of Nigerian Onshore Assets

Shell PLC said Thursday it had completed the divestment of its Niger Delta subsidiary to a Nigerian consortium, in a $1.3 billion transaction previously held back by regulators. Renaissance Africa Energy Holdings has taken over Shell Petroleum Development Company of Nigeria Ltd. (SPDC) and consequently now owns a 30 percent operating stake in the SPDC Joint Venture (JV). The SPDC JV holds 15 onshore oil mining leases (OMLs) and three shallow-water OMLs that have been plagued by oil spills, most of which the British energy giant has blamed on oil theft and sabotage. Nigerian National Petroleum Co. Ltd. is the majority owner of the SPDC JV holding 55 percent. TotalEnergies SE owns 10 percent but has entered into a deal to divest this to Chappal Energies Mauritius Ltd. for $860 million. Eni SpA owns the remaining five percent, previously held via Nigerian Agip Oil Co. Ltd. (NAOC). The Italian state-controlled company sold NAOC to local player Oando PLC for nearly $800 million last year but has decided to retain its SPDC JV stake. “The divestment of SPDC aligns with Shell’s intent to simplify its presence in Nigeria through an exit of onshore oil production in the Niger Delta and a focus of future disciplined investment in its Deepwater and Integrated Gas positions”, Shell said in an online statement Thursday. Eni and TotalEnergies also said their divestments allow them to focus on offshore assets in the West African country. “No significant impairments are expected as a result of completion of the transaction”, Shell added. The transaction, announced January 16, 2024, had been held back by regulators over concerns about the ability of the prospective new owners to manage the assets, including their associated environmental liabilities. “For the independents who are coming in onshore, we want to make sure that they align

Read More »

BP seeking partner for Lightsource BP with bids due in June

Media reports have said that oil supermajor BP aims to sell 50% of its solar unit Lightsource BP this summer. BP will divest the stake to a strategic partner for cash and a commitment of future investments, with bids due in June, Reuters said citing a sales document dated March 2025. According to the document, which calls the divestment Project Scala, BP is looking for a group with “established leaders with extensive experience” in the renewables industry, according to the document. In addition, the document stated that both companies would have joint control of the assets. With non-binding offers due in June, BP will shortlist bidders in July. BP bought into the utility-scale solar and battery storage developer in 2017 before acquiring the remaining 50.03% interest in the group in October last year for around £400m. Lightsource BP came with 5.7GW of operational assets along with a 62GW development pipeline and operations spanning 19 global markets. At the time, BP said it would look to bring in a strategic partner to join the business. BP has since confirmed it intends to bring in a partner for Lightsource BP but did not confirm the timeframe. Since making the purchase, BP has rolled back some of its renewable energy ambitions as it refocuses the company on oil and gas investments over pressure from shareholders to raise the company’s profitability. Part of its divestment programme could see it sell its historic lubricants business Castrol, which could fetch the firm up to $6-8 billion (£4.7-6.3bn), analysts have estimated. Recommended for you BP boss: ‘We went too far too fast’ as major slashes low-carbon investment by $5bn

Read More »

National Grid awards contracts under £59bn HVDC supply chain framework

National Grid has awarded deals to multiple companies under two parts of a £59bn high voltage direct current (HVDC) supply chain framework. A total of six HVDC cable suppliers received positions on the Framework Agreement, totalling around £21.3 billion, with a further four suppliers awarded places on the HVDC Converter Framework, at about £24.6bn. Both frameworks cover confirmed and anticipated projects. The successful HVDC cable suppliers are Hellenic & Jan De Nul Consortium, LS Cable & System, NKT Cables, Prysmian Group, Sumitomo Electric and Taihan Cable & Solution. The HVDC Converter Framework has been awarded to GE Vernova, Hitachi Energy, Mitsubishi Electric and Siemens Energy. Contracts have been secured for a five-year period, with the potential to extend for a further three years. President of strategic infrastructure at National Grid Carl Trowell said: “This is another exciting milestone in delivering the greatest overhaul of the grid in a generation – The Great Grid Upgrade. We are committed to building the infrastructure that will enable our country’s current and future energy needs, at pace. “This framework allows us to harness National Grid’s scale to access global supply chains, drive efficiencies, foster innovative technologies, and contribute to the UK’s economic prosperity.” National Grid launched the HVDC supply chain framework in 2023 to establish long-term, strategic, contractual relationships and secure the critical equipment needed for current and future projects. Suppliers on the framework will support the delivery of early projects including Eastern Green Link 4, Sealink, Lionlink and other projects of a similar size and nature. President of National Grid Ventures Ben Wilson said: “National Grid is already the largest operator of subsea power cables in the world, the majority of which we delivered in the last six years. “World record-breaking sites like Viking Link and pioneering projects like LionLink need an ambitious

Read More »

Impairments Hit STEP’s Results

STEP Energy Services Ltd. closed 2024 with a $1.76 million net income, well below the $50.41 million reported for 2023. However, the company said in a media release that its consolidated revenue ($955 million) and adjusted earnings before interest, taxes, depreciation, and amortization (EBITDA) ($169.1 million) were the best since 2022, when oil and natural gas prices were significantly higher than they were in 2024. The company said its net income was hampered by an impairment expense of $36.7 million. “The impairment was taken on real estate and fracturing pumps with Tier 1 and Tier 2 engines (the tiers established by the U.S. Environmental Protection Agency to control emissions) and associated ancillary fracturing equipment held in the U.S. fracturing cash generating unit”, it said. The company reported a revenue of $722.7 million for the full year, up from $580.2 million for 2023. For the fourth quarter, the company posted a net loss of $44.6 million, versus a net loss of $5.2 million for the corresponding quarter a year prior. Earnings for the fourth quarter accounted for $2.5 million in share-based compensation expense, transaction costs of $2.2 million, $2.4 million in finance costs, and impairment expense of $23.9 million. In 2024, STEP strengthened its foothold in the North American coiled tubing technology sector by acquiring the proprietary technology and intellectual property for the STE-conneCT downhole tool. The company said it has also collaborated with a major U.S. client to enhance ultra-deep coiled tubing capabilities. In December, STEP and 2659160 Alberta Ltd. mutually agreed to terminate their arrangement with ARC Energy Fund 8 due to the inability to secure necessary minority shareholder approval.  To contact the author, email [email protected] What do you think? We’d love to hear from you, join the conversation on the Rigzone Energy Network. The Rigzone Energy Network is

Read More »

VergeIO enhances VergeFabric network virtualization offering

VergeIO is not, however, using an off-the-shelf version of KVM. Rather, it is using what Crump referred to as a heavily modified KVM hypervisor base, with significant proprietary enhancements while still maintaining connections to the open-source community. VergeIO’s deployment profile is currently 70% on premises and about 30% via bare-metal service providers, with a particularly strong following among cloud service providers that host applications for their customers. The software requires direct hardware access due to its low-level integration with physical resources. “Since November of 2023, the normal number one customer we’re attracting right now is guys that have had a heart attack when they got their VMware renewal license,” Crump said. “The more of the stack you own, the better our story becomes.” A 2024 report from Data Center Intelligence Group (DCIG) identified VergeOS as one of the top 5 alternatives to VMware. “VergeIO starts by installing VergeOS on bare metal servers,” the report stated. “It then brings the servers’ hardware resources under its management, catalogs these resources, and makes them available to VMs. By directly accessing and managing the server’s hardware resources, it optimizes them in ways other hypervisors often cannot.” Advanced networking features in VergeFabric VergeFabric is the networking component within the VergeOS ecosystem, providing software-defined networking capabilities as an integrated service rather than as a separate virtual machine or application.

Read More »

Podcast: On the Frontier of Modular Edge AI Data Centers with Flexnode’s Andrew Lindsey

The modular data center industry is undergoing a seismic shift in the age of AI, and few are as deeply embedded in this transformation as Andrew Lindsey, Co-Founder and CEO of Flexnode. In a recent episode of the Data Center Frontier Show podcast, Lindsey joined Editor-in-Chief Matt Vincent and Senior Editor David Chernicoff to discuss the evolution of modular data centers, the growing demand for high-density liquid-cooled solutions, and the industry factors driving this momentum. A Background Rooted in Innovation Lindsey’s career has been defined by the intersection of technology and the built environment. Prior to launching Flexnode, he worked at Alpha Corporation, a top 100 engineering and construction management firm founded by his father in 1979. His early career involved spearheading technology adoption within the firm, with a focus on high-security infrastructure for both government and private clients. Recognizing a massive opportunity in the data center space, Lindsey saw a need for an innovative approach to infrastructure deployment. “The construction industry is relatively uninnovative,” he explained, citing a McKinsey study that ranked construction as the second least-digitized industry—just above fishing and wildlife, which remains deliberately undigitized. Given the billions of square feet of data center infrastructure required in a relatively short timeframe, Lindsey set out to streamline and modernize the process. Founded four years ago, Flexnode delivers modular data centers with a fully integrated approach, handling everything from site selection to design, engineering, manufacturing, deployment, operations, and even end-of-life decommissioning. Their core mission is to provide an “easy button” for high-density computing solutions, including cloud and dedicated GPU infrastructure, allowing faster and more efficient deployment of modular data centers. The Rising Momentum for Modular Data Centers As Vincent noted, Data Center Frontier has closely tracked the increasing traction of modular infrastructure. Lindsey has been at the forefront of this

Read More »

Last Energy to Deploy 30 Microreactors in Texas for Data Centers

As the demand for data center power surges in Texas, nuclear startup Last Energy has now announced plans to build 30 microreactors in the state’s Haskell County near the Dallas-Fort Worth Metroplex. The reactors will serve a growing customer base of data center operators in the region looking for reliable, carbon-free energy. The plan marks Last Energy’s largest project to date and a significant step in advancing modular nuclear power as a viable solution for high-density computing infrastructure. Meeting the Looming Power Demands of Texas Data Centers Texas is already home to over 340 data centers, with significant expansion underway. Google is increasing its data center footprint in Dallas, while OpenAI’s Stargate has announced plans for a new facility in Abilene, just an hour south of Last Energy’s planned site. The company notes the Dallas-Fort Worth metro area alone is projected to require an additional 43 gigawatts of power in the coming years, far surpassing current grid capacity. To help remediate, Last Energy has secured a 200+ acre site in Haskell County, approximately three and a half hours west of Dallas. The company has also filed for a grid connection with ERCOT, with plans to deliver power via a mix of private wire and grid transmission. Additionally, Last Energy has begun pre-application engagement with the U.S. Nuclear Regulatory Commission (NRC) for an Early Site Permit, a key step in securing regulatory approval. According to Last Energy CEO Bret Kugelmass, the company’s modular approach is designed to bring nuclear energy online faster than traditional projects. “Nuclear power is the most effective way to meet Texas’ growing energy demand, but it needs to be deployed faster and at scale,” Kugelmass said. “Our microreactors are designed to be plug-and-play, enabling data center operators to bypass the constraints of an overloaded grid.” Scaling Nuclear for

Read More »

Data Center Jobs: Engineering and Technician Jobs Available in Major Markets

Each month Data Center Frontier, in partnership with Pkaza, posts some of the hottest data center career opportunities in the market. Here’s a look at some of the latest data center jobs posted on the Data Center Frontier jobs board, powered by Pkaza Critical Facilities Recruiting.  Data Center Facility Engineer (Night Shift Available) Ashburn, VAThis position is also available in: Tacoma, WA (Nights), Days/Nights: Needham, MA and New York City, NY. This opportunity is working directly with a leading mission-critical data center developer / wholesaler / colo provider. This firm provides data center solutions custom-fit to the requirements of their client’s mission-critical operational facilities. They provide reliability of mission-critical facilities for many of the world’s largest organizations facilities supporting enterprise clients and hyperscale companies. This opportunity provides a career-growth minded role with exciting projects with leading-edge technology and innovation as well as competitive salaries and benefits. Electrical Commissioning Engineer New Albany, OHThis traveling position is also available in: Somerset, NJ; Boydton, VA; Richmond, VA; Ashburn, VA; Charlotte, NC; Atlanta, GA; Hampton, GA; Fayetteville, GA; Des Moines, IA; San Jose, CA; Portland, OR; St Louis, MO; Phoenix, AZ;  Dallas, TX;  Chicago, IL; or Toronto, ON. *** ALSO looking for a LEAD EE and ME CxA agents.*** Our client is an engineering design and commissioning company that has a national footprint and specializes in MEP critical facilities design. They provide design, commissioning, consulting and management expertise in the critical facilities space. They have a mindset to provide reliability, energy efficiency, sustainable design and LEED expertise when providing these consulting services for enterprise, colocation and hyperscale companies. This career-growth minded opportunity offers exciting projects with leading-edge technology and innovation as well as competitive salaries and benefits. Switchgear Field Service Technician – Critical Facilities Nationwide TravelThis position is also available in: Charlotte, NC; Atlanta, GA; Dallas,

Read More »

Amid Shifting Regional Data Center Policies, Iron Mountain and DC Blox Both Expand in Virginia’s Henrico County

The dynamic landscape of data center developments in Maryland and Virginia exemplify the intricate balance between fostering technological growth and addressing community and environmental concerns. Data center developers in this region find themselves both in the crosshairs of groups worried about the environment and other groups looking to drive economic growth. In some cases, the groups are different components of the same organizations, such as local governments. For data center development, meeting the needs of these competing interests often means walking a none-too-stable tightrope. Rapid Government Action Encourages Growth In May 2024, Maryland demonstrated its commitment to attracting data center investments by enacting the Critical Infrastructure Streamlining Act. This legislation provides a clear framework for the use of emergency backup power generation, addressing previous regulatory challenges that a few months earlier had hindered projects like Aligned Data Centers’ proposed 264-megawatt campus in Frederick County, causing Aligned to pull out of the project. However, just days after the Act was signed by the governor, Aligned reiterated its plans to move forward with development in Maryland.  With the Quantum Loop and the related data center development making Frederick County a focal point for a balanced approach, the industry is paying careful attention to the pace of development and the relations between developers, communities and the government. In September of 2024, Frederick County Executive Jessica Fitzwater revealed draft legislation that would potentially restrict where in the county data centers could be built. The legislation was based on information found in the Frederick County Data Centers Workgroup’s final report. Those bills would update existing regulations and create a floating zone for Critical Digital Infrastructure and place specific requirements on siting data centers. Statewide, a cautious approach to environmental and community impacts statewide has been deemed important. In January 2025, legislators introduced SB116,  a bill

Read More »

New Reports Show How AI, Power, and Investment Trends Are Reshaping the Data Center Landscape

Today we provide a comprehensive roundup of the latest industry analyst reports from CBRE, PwC, and Synergy Research, offering a data-driven perspective on the state of the North American data center market.  To wit, CBRE’s latest findings highlight record-breaking growth in supply, soaring colocation pricing, and mounting power constraints shaping site selection. For its part, PwC’s analysis underscores the sector’s broader economic impact, quantifying its trillion-dollar contribution to GDP, rapid job growth, and surging tax revenues.  Meanwhile, the latest industry analysis from Synergy Research details the acceleration of cloud spending, AI’s role in fueling infrastructure demand, and an unprecedented surge in data center mergers and acquisitions.  Together, these reports paint a picture of an industry at an inflection point—balancing explosive expansion with evolving challenges in power availability, cost pressures, and infrastructure investment. Let’s examine them. CBRE: Surging Demand Fuels Record Data Center Expansion CBRE says the North American data center sector is scaling at an unprecedented pace, driven by unrelenting demand from artificial intelligence (AI), hyperscale, and cloud service providers. The latest North America Data Center Trends H2 2024 report from CBRE reveals that total supply across primary markets surged by 34% year-over-year to 6,922.6 megawatts (MW), outpacing the 26% growth recorded in 2023. This accelerating expansion has triggered record-breaking construction activity and intensified competition for available capacity. Market Momentum: Scaling Amid Power Constraints According to CBRE, data center construction activity reached historic levels, with 6,350 MW under development at the close of 2024—more than doubling the 3,077.8 MW recorded a year prior. Yet, the report finds the surge in development is being met with significant hurdles, including power constraints and supply chain challenges affecting critical electrical infrastructure. As a result, the vacancy rate across primary markets has plummeted to an all-time low of 1.9%, with only a handful of sites

Read More »

Microsoft will invest $80B in AI data centers in fiscal 2025

And Microsoft isn’t the only one that is ramping up its investments into AI-enabled data centers. Rival cloud service providers are all investing in either upgrading or opening new data centers to capture a larger chunk of business from developers and users of large language models (LLMs).  In a report published in October 2024, Bloomberg Intelligence estimated that demand for generative AI would push Microsoft, AWS, Google, Oracle, Meta, and Apple would between them devote $200 billion to capex in 2025, up from $110 billion in 2023. Microsoft is one of the biggest spenders, followed closely by Google and AWS, Bloomberg Intelligence said. Its estimate of Microsoft’s capital spending on AI, at $62.4 billion for calendar 2025, is lower than Smith’s claim that the company will invest $80 billion in the fiscal year to June 30, 2025. Both figures, though, are way higher than Microsoft’s 2020 capital expenditure of “just” $17.6 billion. The majority of the increased spending is tied to cloud services and the expansion of AI infrastructure needed to provide compute capacity for OpenAI workloads. Separately, last October Amazon CEO Andy Jassy said his company planned total capex spend of $75 billion in 2024 and even more in 2025, with much of it going to AWS, its cloud computing division.

Read More »

John Deere unveils more autonomous farm machines to address skill labor shortage

Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More Self-driving tractors might be the path to self-driving cars. John Deere has revealed a new line of autonomous machines and tech across agriculture, construction and commercial landscaping. The Moline, Illinois-based John Deere has been in business for 187 years, yet it’s been a regular as a non-tech company showing off technology at the big tech trade show in Las Vegas and is back at CES 2025 with more autonomous tractors and other vehicles. This is not something we usually cover, but John Deere has a lot of data that is interesting in the big picture of tech. The message from the company is that there aren’t enough skilled farm laborers to do the work that its customers need. It’s been a challenge for most of the last two decades, said Jahmy Hindman, CTO at John Deere, in a briefing. Much of the tech will come this fall and after that. He noted that the average farmer in the U.S. is over 58 and works 12 to 18 hours a day to grow food for us. And he said the American Farm Bureau Federation estimates there are roughly 2.4 million farm jobs that need to be filled annually; and the agricultural work force continues to shrink. (This is my hint to the anti-immigration crowd). John Deere’s autonomous 9RX Tractor. Farmers can oversee it using an app. While each of these industries experiences their own set of challenges, a commonality across all is skilled labor availability. In construction, about 80% percent of contractors struggle to find skilled labor. And in commercial landscaping, 86% of landscaping business owners can’t find labor to fill open positions, he said. “They have to figure out how to do

Read More »

2025 playbook for enterprise AI success, from agents to evals

Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More 2025 is poised to be a pivotal year for enterprise AI. The past year has seen rapid innovation, and this year will see the same. This has made it more critical than ever to revisit your AI strategy to stay competitive and create value for your customers. From scaling AI agents to optimizing costs, here are the five critical areas enterprises should prioritize for their AI strategy this year. 1. Agents: the next generation of automation AI agents are no longer theoretical. In 2025, they’re indispensable tools for enterprises looking to streamline operations and enhance customer interactions. Unlike traditional software, agents powered by large language models (LLMs) can make nuanced decisions, navigate complex multi-step tasks, and integrate seamlessly with tools and APIs. At the start of 2024, agents were not ready for prime time, making frustrating mistakes like hallucinating URLs. They started getting better as frontier large language models themselves improved. “Let me put it this way,” said Sam Witteveen, cofounder of Red Dragon, a company that develops agents for companies, and that recently reviewed the 48 agents it built last year. “Interestingly, the ones that we built at the start of the year, a lot of those worked way better at the end of the year just because the models got better.” Witteveen shared this in the video podcast we filmed to discuss these five big trends in detail. Models are getting better and hallucinating less, and they’re also being trained to do agentic tasks. Another feature that the model providers are researching is a way to use the LLM as a judge, and as models get cheaper (something we’ll cover below), companies can use three or more models to

Read More »

OpenAI’s red teaming innovations define new essentials for security leaders in the AI era

Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More OpenAI has taken a more aggressive approach to red teaming than its AI competitors, demonstrating its security teams’ advanced capabilities in two areas: multi-step reinforcement and external red teaming. OpenAI recently released two papers that set a new competitive standard for improving the quality, reliability and safety of AI models in these two techniques and more. The first paper, “OpenAI’s Approach to External Red Teaming for AI Models and Systems,” reports that specialized teams outside the company have proven effective in uncovering vulnerabilities that might otherwise have made it into a released model because in-house testing techniques may have missed them. In the second paper, “Diverse and Effective Red Teaming with Auto-Generated Rewards and Multi-Step Reinforcement Learning,” OpenAI introduces an automated framework that relies on iterative reinforcement learning to generate a broad spectrum of novel, wide-ranging attacks. Going all-in on red teaming pays practical, competitive dividends It’s encouraging to see competitive intensity in red teaming growing among AI companies. When Anthropic released its AI red team guidelines in June of last year, it joined AI providers including Google, Microsoft, Nvidia, OpenAI, and even the U.S.’s National Institute of Standards and Technology (NIST), which all had released red teaming frameworks. Investing heavily in red teaming yields tangible benefits for security leaders in any organization. OpenAI’s paper on external red teaming provides a detailed analysis of how the company strives to create specialized external teams that include cybersecurity and subject matter experts. The goal is to see if knowledgeable external teams can defeat models’ security perimeters and find gaps in their security, biases and controls that prompt-based testing couldn’t find. What makes OpenAI’s recent papers noteworthy is how well they define using human-in-the-middle

Read More »

Anatomy of a Parquet File

In recent years, Parquet has become a standard format for data storage in Big Data ecosystems. Its column-oriented format offers several advantages:

Faster query execution

Read More »