s3view#

s3view is a command line tool for exploring data stored in object stores which expose their data via the S3 protocol.

It is designed to provide a way to explore and discover data stored in object stores, with a focus on exploiting the metadata stored within the files in an efficient was as possible by utilising their known structure, by caching information between calls and by exploiting parallelism to speed up access.

class cfs3.s3cmd(path=None, config_file=None)[source]#

Bases: Cmd

s3cmd is the class which implements the s3view capabilities for viewing contents in S3 repositories

The methods of this class implement the commands seen in the s3view environment. There is some limited capability to “pipe” information between selected commands (using :: for piping), and the output of all commands can be piped to the normal unix shell.

Table 1 Commands#
Command	Description
`cb`	Change to a (new) bucket.
`cd`	Change default position in bucket to expose contents as if it were a file system directory.
`cflist`	cflist a remote object
`drsview`	Extract DRS components at location
`lb`	List buckets in the current location.
`loc`	Set context to a particular S3 location as described in user minio config file.
`loglevel`	Change logging level. Usage: loglevel [debug\|info\|warning\|error\|critical]
`ls`	List the files and directories in a bucket, potentially with a wild card.
`match`	Find files which match a set of metadata expressed as key value pairs, optionally matching a
`mb`	Make bucket, return error if existing.
`mv`	Rename files within a bucket (server side).
`p5dump`	Use pyfive to approximate a ncdump -h on a remote object
`pwd`	Do the equivalent of a print working directory, that is, show whwere you are in the
`rm`	Remove a list of objects, including those which may be generated from wild card matches.
`tag`	Tag a previously existing object with a key value pair.

Commands supporting internal pipe#

Producer commands can input to consumer commands using the “::” symbol

eg: ls -1 :: p5dump

producers ls
consumers p5dump

Command Descriptions#

cb bucket -h/--help

Change to a (new) bucket.

This is not treated as just a move to another directory, as the notion of a bucket is quite different in S3 from that of a directory. It’s more like moving to the root of a different file system.

Command Arguments: Positional Arguments: - bucket: Bucket should be a valid bucket in your current location

Optional Arguments: - -h, --help: show this help message and exit

cd path -h/--help

Change default position in bucket to expose contents as if it were a file system directory.

This exploits the minio notion of prefixes to give a directory like view of some of the content in a bucket.

Command Arguments: Positional Arguments: - path: Path should be a valid path in your current bucket and location.

Optional Arguments: - -h, --help: show this help message and exit

cflist object -h/--help -c/--complete -s/--short

cflist a remote object

Command Arguments: Positional Arguments: - object: object should be a valid object in your current bucket and location.

Optional Arguments: - -h, --help: show this help message and exit - -c, --complete: Display complete descriptions of cf fields - -s, --short: Display short descriptions of cf fields

drsview path drs -h/--help -c/--collapse_list -u/--use_metadata -s/--select -o/--output

Extract DRS components at location

Command Arguments: Positional Arguments: - path: Path should be a valid path in your current bucket and location, possibly with a wildcard. - drs: Comma seperated string of DRS components to be used for contents (ignored for metadata option) default = Variable,Source,Experiment,Variant,Frequency,Period,nFields

Optional Arguments: - -h, --help: show this help message and exit - -c, --collapse_list: Comma seperated string of DRS terms where short lists are wanted - -u, --use_metadata: build drs-like view from metadata - -s, --select: Specfiy DRS component selections as key=value (multipe -s allowed) and return listing - -o, --output: Default output is a DRS view, alternative is “list” view.

lb()

List buckets in the current location.

Information about bucket contents can only be gathered after using the cb command.

loc alias -h/--help

Set context to a particular S3 location as described in user minio config file.

(We use the minio config file to allow users to keep credentials for multiple S3 locations in one place.)

Command Arguments: Positional Arguments: - alias: Where alias is a valid alias from your minio config file

Optional Arguments: - -h, --help: show this help message and exit

loglevel(): Change logging level. Usage: loglevel [debug|info|warning|error|critical]

ls path -h/--help -l/--long -s/--size -w/--width -m/--metadata -t/--tags -d/--date -o/--order -n/--max_number

List the files and directories in a bucket, potentially with a wild card.

Command Arguments: Positional Arguments: - path: Path should be a valid path in your current bucket and location, possibly with a wildcard.

Optional Arguments: - -h, --help: show this help message and exit - -l, --long: Same as -s -d -m - -s, --size: Tell us about size - -w, --width: width of display for standard output - -m, --metadata: Show user metadata - -t, --tags: Show tags - -d, --date: Show dates - -o, --order: Order by size|date - -n, --max_number: Limit the number of files returned

match keyvals -h/--help -p/--path -w/--width

Find files which match a set of metadata expressed as key value pairs, optionally matching a particular path.

This metadata match is using the DRS metadaata which has been uploaded with the file.

Command Arguments: Positional Arguments: - keyvals: Metadata key-value pairs in the format key=value which you want to match

Optional Arguments: - -h, --help: show this help message and exit - -p, --path: path prefix in which you want to find the metadata matches - -w, --width: width of display for standard output

mb bucket -h/--help

Make bucket, return error if existing.

Command Arguments: Positional Arguments: - bucket: The name of a new bucket to create in your current location

Optional Arguments: - -h, --help: show this help message and exit

mv targets -h/--help

Rename files within a bucket (server side).

This is an expensive operation as it really involves a server-side copy, not a renaming operation as you might expect in a normal file system.

Command Arguments: Positional Arguments: - targets: filenames and/or paths to be removed, e.g. mv fileA fileB

Optional Arguments: - -h, --help: show this help message and exit

p5dump object -h/--help -s/--special

Use pyfive to approximate a ncdump -h on a remote object

Accepts:

normal filename argument
piped filenames from previous command via self.__pipe_input.

Command Arguments: Positional Arguments: - object: object should be a valid HDF5 or NC4 file in your current bucket and location.

Optional Arguments: - -h, --help: show this help message and exit - -s, --special: Display special attributes of datasets in files (NOT IMPLEMENTED)

pwd(): Do the equivalent of a print working directory, that is, show whwere you are in the current bucket. Here the root (/) will be the root of a particular bucket. We do not want to encourage folks to think of different buckets as being part of the same file system.

rm targets -h/--help

Remove a list of objects, including those which may be generated from wild card matches.

Command Arguments: Positional Arguments: - targets: filenames and/or paths to be removed

Optional Arguments: - -h, --help: show this help message and exit

tag path value key -h/--help

Tag a previously existing object with a key value pair.

Allows users with object stores who support tagging to tag objects dynamically rather than utilise the user metadata option. Many object stores, including the author’s one, do not support this. This might work for you, it doesn’t work for me. Let me know if it does.

Command Arguments: Positional Arguments: - path: Path should be a valid object match (i.e. an object path, possibly with a wildcard). - value: Value for tag - key: Key for a tag

Optional Arguments: - -h, --help: show this help message and exit