Skip to main content
Ctrl+K

Grain

Get started

  • Installing Grain
  • Choice of API

Data sources

  • Protocol
  • Bagz
  • ArrayRecord
  • Parquet
  • TFDS
  • HuggingFace
  • PyTorch
  • GCS
  • Amazon S3

Dataset

  • Basics
  • Advanced usage
  • Transformations
  • Performance debugging

DataLoader

  • Samplers
  • Transformations
  • DataLoader tutorial

Tutorials

  • JAX training tips

API reference

  • grain package
    • Dataset APIs
    • DataLoader APIs
    • grain.checkpoint module
    • grain.constants module
    • grain.experimental module
    • grain.multiprocessing module
    • grain.samplers module
    • grain.sharding module
    • grain.sources module
    • grain.transforms module
  • Change log

For contributors

  • Behind the Scenes
  • Contributing to Grain
  • Repository
  • Open issue
  • .rst

grain package

Contents

  • Subpackages
  • Flexible low-level pipelines
  • Simple high-level pipelines

grain package#

Subpackages#

  • grain.checkpoint module
  • grain.constants module
  • grain.experimental module
  • grain.multiprocessing module
  • grain.samplers module
  • grain.sharding module
  • grain.sources module
  • grain.transforms module

Flexible low-level pipelines#

Simple high-level pipelines#

previous

Plugging Grain into JAX training: batching + accelerator transfer

next

grain DataLoader

Contents
  • Subpackages
  • Flexible low-level pipelines
  • Simple high-level pipelines

By Grain team

© Copyright 2024, Grain team.