API Reference

This documents the current version of the Pypeline DB API.

pypeline.DB

class pypeline.DB(database_path, **kwargs)

The pypeline LevelDB database. This class contains collections and provides some system-level organization.

All arguments beyond the database path are passed directly to the underlying plyvel DB constructor. More details can be found at https://plyvel.readthedocs.org/en/latest/api.html#DB

Arguments: database_path – The path to the folder for database storage

collection(collection_name, reset_collection=False, create_if_missing=True, error_if_exists=False)

Returns the collection stored at collection_name, or creates it if it doesn’t exist.

Arguments:

collection_name – the name of the collection to return

Keyword arguments:

reset_collection – when True any existant data in the collection is deleted before it is returned

create_if_missing – when False a ValueError is raised if the collection doesn’t exist

error_if_exists – When True a ValueError is raised if the collection already exists

collections()

Returns a list of keys of collections contained in the database.

copy_collection(old_collection, new_collection, start=None, end=None, **kwargs)

Copies all instances in the old_collection into the new_collection

Arguments:

old_collection – The string name of the old collection

new_collection – The string name of the new collection

Keyword arguments:

start – (Optional) The index to begin copying from old_collection

end – (Optional) The index to end copying from old_collection

create_if_missing – when False a ValueError is raised if the new collection doesn’t exist (default: True)

error_if_exists – When True a ValueError is raised if the collection already exists (default: False)

delete(collection_name)

Deletes a collection.

Arguments:

collection_name – the name of the collection to delete

close()

Closes the database.

open()

Opens the database.

pypeline.Collection

class pypeline.Collection(database, items_set, name)

A collection of records stored in a database

This class should never be instantiated directly. Use the DB.collection() method instead

append(record)

Appends a single record.

Arguments:

record – Any JSON-serializable python object (dicts, lists, ints, strings, etc.)

refresh()

Reloads the collection from the database.

delete(index)

Deletes an item from the collection.

Arguments:

index – Index of the item to be deleted.

delete_all()

Deletes all items in the collection

append_all(iterable)

Appends every item in the iterable to the collection

map(function, new_collection, **kwargs)

Maps a collection to a new collection with a provided function.

Arguments:

function – The function used for mapping.

new_collection – The name of the collection to insert the new values into. Any existing values will be deleted. If None, values are mapped to the same collection.

Keyword arguments:

create_if_missing – when False a ValueError is raised if the new collection doesn’t exist

error_if_exists – When True a ValueError is raised if the new collection already exists

filter(function, new_collection, **kwargs)

Filters a collection into a new collection with a given function.

Arguments:

function – The function used for filtering.

new_collection – The name of the collection to insert the new values into. Any existing values will be deleted. If None, values are filtered in the same collection.

Keyword arguments:

create_if_missing – when False a ValueError is raised if the new collection doesn’t exist

error_if_exists – When True a ValueError is raised if the new collection already exists

reduce(function, new_collection, initializer=None, **kwargs)

Reduces a collection into a new collection with a given function.

Arguments:

function – The function used for reducing.

new_collection – The name of the collection to insert the new value into. Any existing values will be deleted. If None, the current collection is replaced with the reduction output.

Keyword arguments:

create_if_missing – when False a ValueError is raised if the new collection doesn’t exist

error_if_exists – When True a ValueError is raised if the new collection already exists

random_subset(number, new_collection, **kwargs)

Produces a random subset of a given collection and inserts it into a new collection.

Arguments:

new_collection – The name of the collection to insert the new values into. Any existing values will be deleted. If None, the subset is stored to the current collection.

Keyword arguments:

create_if_missing – when False a ValueError is raised if the new collection doesn’t exist

error_if_exists – When True a ValueError is raised if the new collection already exists

iterator(start=None, end=None)

Returns a collection iterator