Python API

• Bug-fixes for quilt3.config and quilt3.catalog
• Performance improvements for Packages

Catalog

• Updated landing page

Catalog

• New LOCAL mode for running the catalog on localhost

Python API

• quilt3 catalog command to run the Quilt catalog on your local machine
• quilt3 verify compares the state of a directory to the contents of a package version
• Added a local file cache for installed packages
• Performance improvements for upload and download
• Support for short hashes to identify package versions

API Improvements

• Implement Package.rollback
• Drop support for object metadata (outside of packages)
• Change the number of threads used when installing and pushing from 4 to 10 (S3 default)
• Misc bug fixes

Catalog

• Fix package listing for packages with more 100 revisions
• Add stacked area charts for downloads
• 2-level file-extensions for bucket summary

Python

• Fix uploads of very large files
• Remove unnecessary copying during push

• delete_package for a specific version via top_hash=

• Bug fix: when adding python objects to a package a temporary file would be created and then deleted when the object was pushed, leading to a crash if you tried to push that package again (PR #1264)

This is the initial release of the new and improved Quilt 3 (formerly Quilt T4). For more information refer to the documentation.

Compiler

Adds a feature to allow quilt config to set a registry URL for a private Teams registry.

Compiler

• Adding a hash argument to quilt.push to allow pushing any package version to a registry.

Registry

• Make object sizes required.
• Update urllib3 version for security patch

Docs

• Improved instructions for running registries.

• Fix an ascii decoding issue related to ellipses …

Compiler

• fixes an issue with packages created on older versions of pyarrow
• improves readability for quilt inspect
• allow adding a node with metadata using sub-package build/push

Registry

• adds documentation for running a private registry in AWS

• Suppress numpy warnings under Python 2.7
• Fix subpackage build and push

Compiler

• added Bracket accessor for GroupNodes
• asa.plot to show images in packages
• asa.torch to convert packages to PyTorch Datasets
• Enforce fragment store as read-only

Catalog

• Added source maps and CI for catalog testing

Documentation

Expands and improves documentation for working with Quilt packages.

Bug fixes and small improvements

• Load packages by hash
• Choose a custom loader for DataNodes with asa=

Registry

• Specify Ubuntu version in Dockerfiles

Catalog

• display package traffic stats in catalog

Compiler

• filter packages based on per-node metadata
• get/set metadata for package nodes
• support custom loaders in the _data method

Registry

• package commenting

Compiler

• Metadata-only package install
• Build DataFrames from existing Parquet files
• Remove HDF5 dependencies
• Code cleanup and refactoring

Registry

• Option for metadata-only package installs
• New endpoint for fetching missing fragments (e.g., from partially installed packages)
• Improved full-text search

Compiler:

• Allow building packages out of other packages and elements from other packages. A new build-file keyword, package inserts a package (or sub-package) as an element in the package being built.

Catalog:

• Upgrade router and other dependencies
• Display packages by author

Catalog Changes to support private registries

• Amin UI for controlling users and access
• Auditing views

Globbing for package builds

• Allow specifying sets of input files in build.yml

Command-line support for private registries

• Specify teams packages
• Admin commands to create and activate/deactivate users

Version 2.9.1 introduces a better progress bar for installing (downloading) Quilt packages. Quilt push now sends objects' uncompressed size to the registry. The progress bar is now based on the total bytes downloaded instead of the number of files.

Shared Local Package Storage

Import packages from shared local directories to save storage overhead and network traffic when sharing packages on the same local network.

Registry Install Stats

Log package installs in the registry to display stats on package use.

New Python API commands

• generate
• rm
• search

Drop support for Python 3.4

(BETA) Team Registries

Updates to commands and local storage to allow users to connect to different registries to support teams running private registries for internal sharing.

Fixes a bug in download that prevented retrying failed downloads.

#186 introduced an undeclared dependency on Pandas >= 0.21.0 (by catching ParserError during CSV parsing). This release removes that dependency and resolves #291.

PR #290

Quilt Catalog

Source for the Quilt data catalog is now included in this repository.

MySQL->Postgres

Ported the Quilt registry from MySQL to Postgres

Docker Compose

Improvements to the docker configuration that allows running the registry, catalog, database and authentication service from Docker compose.

Parallel Download

Data fragments can now be downloaded in parallel leading to much faster package installs for large packages.