Compiler

• added Bracket accessor for GroupNodes
• asa.plot to show images in packages
• asa.torch to convert packages to PyTorch Datasets
• Enforce fragment store as read-only

Catalog

• Added source maps and CI for catalog testing

Documentation

Expands and improves documentation for working with Quilt packages.

Bug fixes and small improvements

• Load packages by hash
• Choose a custom loader for DataNodes with asa=

Registry

• Specify Ubuntu version in Dockerfiles

Catalog

• display package traffic stats in catalog

Compiler

• filter packages based on per-node metadata
• get/set metadata for package nodes
• support custom loaders in the _data method

Registry

• package commenting

Compiler

• Metadata-only package install
• Build DataFrames from existing Parquet files
• Remove HDF5 dependencies
• Code cleanup and refactoring

Registry

• Option for metadata-only package installs
• New endpoint for fetching missing fragments (e.g., from partially installed packages)
• Improved full-text search

Compiler:

• Allow building packages out of other packages and elements from other packages. A new build-file keyword, package inserts a package (or sub-package) as an element in the package being built.

Catalog:

• Upgrade router and other dependencies
• Display packages by author

Catalog Changes to support private registries

• Amin UI for controlling users and access
• Auditing views

Globbing for package builds

• Allow specifying sets of input files in build.yml

Command-line support for private registries

• Specify teams packages
• Admin commands to create and activate/deactivate users

Version 2.9.1 introduces a better progress bar for installing (downloading) Quilt packages. Quilt push now sends objects' uncompressed size to the registry. The progress bar is now based on the total bytes downloaded instead of the number of files.

Shared Local Package Storage

Import packages from shared local directories to save storage overhead and network traffic when sharing packages on the same local network.

Registry Install Stats

Log package installs in the registry to display stats on package use.

New Python API commands

• generate
• rm
• search

Drop support for Python 3.4

(BETA) Team Registries

Updates to commands and local storage to allow users to connect to different registries to support teams running private registries for internal sharing.

Fixes a bug in download that prevented retrying failed downloads.

#186 introduced an undeclared dependency on Pandas >= 0.21.0 (by catching ParserError during CSV parsing). This release removes that dependency and resolves #291.

PR #290

Quilt Catalog

Source for the Quilt data catalog is now included in this repository.

MySQL->Postgres

Ported the Quilt registry from MySQL to Postgres

Docker Compose

Improvements to the docker configuration that allows running the registry, catalog, database and authentication service from Docker compose.

Parallel Download

Data fragments can now be downloaded in parallel leading to much faster package installs for large packages.

Release Highlights

Quilt packages live in a centralized location on your machine

Quilt data packages are now available wherever you run Python. We recommend that users quilt push all local packages to the registry before upgrading. Further details on migration are here.

Faster builds with build cache

Quilt now caches build intermediates. So if you wish to update the README of a multi-gigabyte package, you can rebuild the entire package in one second. 

Group-level build parameters

You can now specify build parameters (like transform) for all children of a group in one shot. The updated syntax and docs are here.

quilt.yml is like requirments.txt but for data

You can now express dependencies on multiple packages in a single file. Docs here.

Experimental: build a package from a GitHub repo

Quilt build now accepts GitHub URLs. If you use data stored on GitHub you can turn it into a Quilt package with quilt build.

Version 2.7.1 includes several minor bug fixes and one new feature, checks. Checks allow a user to specify data integrity checks that are enforced during quilt build.

• Support installing subpackages as quilt install usr/pkg/path
• Upload fragments in parallel
• Use http sessions when accessing S3

This release adds a new command to delete a package including all versions and history from the registry.

Building a package from a directory of input files now skips generating a build file. That speeds up the build process and makes it easier to change the package contents and rebuild.