Software Architectural Design -- Parquet IO Java

Introduction

Acknowledgments

This document's section structure is derived from the "arc42" architectural template by Dr. Gernot Starke, Dr. Peter Hruschka.

Terms and Abbreviations

PIOJ

Parquet IO Java

Requirement Overview

Please refer to the System Requirement Specification for user-level requirements.

Solution Strategy

Building Blocks

This section introduces the building blocks of the software. Together those building blocks make up the big picture of the software structure.

Row

dsn~row~1

A Row is class that contains list of Parquet columns.

RowParquetReader

dsn~row-parquet-reader~1

The RowParquetReader reads Parquet formatted data into a Row object.

RowParquetChunkReader

dsn~row-parquet-chunk-reader~1

The RowParquetChunkReader reads provided row group chunk's data into a Row object.

ParquetConverter

dsn~parquet-converter~1

The ParquetConverter is an abstraction of set column converters that read Parquet data into memory objects.

Runtime View

This section describes the runtime behavior of the software.

Reading Parquet File Contents

dsn~read-parquet-file-contents~1

The RowParquetReader reads contents of a Parquet file.

Covers:

• req~accessing-decoded-rows~1

Needs: impl, utest

Reading Parquet File Row Group Chunks Contents

dsn~read-parquet-file-chunks-contents~1

The RowParquetChunkReader only reads certain row group contents of a Parquet file.

Covers:

• req~reading-row-group-intervals~1

Needs: impl, utest

Converting Primitive Column Types

dsn~converting-primitive-column-types~1

The ParquetConverter allows to convert primitive Parquet types into in-memory objects.

Covers:

• req~reading-primitive-types~1

Needs: impl, utest

Converting Logical Column Types

dsn~converting-logical-column-types~1

The ParquetConverter allows to convert logical Parquet types into in-memory objects.

Covers:

• req~reading-logical-types~1

Needs: impl, utest

Converting Nanosecond Timestamp Values

dsn~converting-nanosecond-timestamp-values~1

The ParquetConverterFactory recognizes modern Parquet timestamp logical annotations and converts INT64 values with the NANOS unit into java.sql.Timestamp values, including values before the Unix epoch.

Covers:

• req~reading-nanosecond-timestamps~1

Needs: impl, utest

Converting Nested Column Types

dsn~converting-nested-column-types~1

The ParquetConverter allows to convert nested Parquet types into in-memory objects.

Covers:

• req~reading-nested-types~1

Needs: impl, utest

Quality Requirements

See Quality Requirements.