cf.Data

class cf.Data(array=None, units=None, calendar=None, fill_value=None, hardmask=True, chunk=True, loadd=None, loads=None, dt=False, source=None, copy=True, dtype=None, mask=None, _use_array=True)

Bases: cfdm.data.data.Data

An N-dimensional data array with units and masked values.

• Contains an N-dimensional, indexable and broadcastable array with many similarities to a numpy array.

• Contains the units of the array elements.

• Supports masked arrays, regardless of whether or not it was initialised with a masked array.

• Stores and operates on data arrays which are larger than the available memory.

Indexing

A data array is indexable in a similar way to numpy array:

>>> d.shape
(12, 19, 73, 96)
>>> d[...].shape
(12, 19, 73, 96)
>>> d[slice(0, 9), 10:0:-2, :, :].shape
(9, 5, 73, 96)

There are three extensions to the numpy indexing functionality:

• Size 1 dimensions are never removed by indexing.

  An integer index i takes the i-th element but does not reduce the rank of the output array by one:

  >>> d.shape
  (12, 19, 73, 96)
  >>> d[0, ...].shape
  (1, 19, 73, 96)
  >>> d[:, 3, slice(10, 0, -2), 95].shape
  (12, 1, 5, 1)
  

  Size 1 dimensions may be removed with the squeeze method.

• The indices for each axis work independently.

  When more than one dimension’s slice is a 1-d boolean sequence or 1-d sequence of integers, then these indices work independently along each dimension (similar to the way vector subscripts work in Fortran), rather than by their elements:

  >>> d.shape
  (12, 19, 73, 96)
  >>> d[0, :, [0, 1], [0, 13, 27]].shape
  (1, 19, 2, 3)
  

• Boolean indices may be any object which exposes the numpy array interface.

  >>> d.shape
  (12, 19, 73, 96)
  >>> d[..., d[0, 0, 0]>d[0, 0, 0].min()]
  

Cyclic axes

Miscellaneous

A Data object is picklable.

A Data object is hashable, but note that, since it is mutable, its hash value is only valid whilst the data array is not changed in place.

Initialization

Parameters

array: optional

The array of values. May be any scalar or array-like object, including another Data instance. Ignored if the source parameter is set.

Parameter example:

array=[34.6]

Parameter example:

array=[[1, 2], [3, 4]]

Parameter example:

array=numpy.ma.arange(10).reshape(2, 1, 5)

units: str or Units, optional

The physical units of the data. if a Units object is provided then this an also set the calendar. Ignored if the source parameter is set.

The units (without the calendar) may also be set after initialisation with the set_units method.

Parameter example:

units='km hr-1'

Parameter example:

units='days since 2018-12-01'

calendar: str, optional

The calendar for reference time units. Ignored if the source parameter is set.

The calendar may also be set after initialisation with the set_calendar method.

Parameter example:

calendar='360_day'

fill_value: optional

The fill value of the data. By default, or if set to None, the numpy fill value appropriate to the array’s data-type will be used (see numpy.ma.default_fill_value). Ignored if the source parameter is set.

The fill value may also be set after initialisation with the set_fill_value method.

Parameter example:

fill_value=-999.

dtype: data-type, optional

The desired data-type for the data. By default the data-type will be inferred form the array parameter.

The data-type may also be set after initialisation with the dtype attribute.

Parameter example:

dtype=float

Parameter example:

dtype='float32'

Parameter example:

dtype=numpy.dtype('i2')

mask: optional

Apply this mask to the data given by the array parameter. By default, or if mask is None, no mask is applied. May be any scalar or array-like object (such as a list, numpy array or Data instance) that is broadcastable to the shape of array. Masking will be carried out where the mask elements evaluate to True.

This mask will applied in addition to any mask already defined by the array parameter.

source: optional

Initialize the array, units, calendar and fill value from those of source.

hardmask: bool, optional

If False then the mask is soft. By default the mask is hard.

dt: bool, optional

If True then strings (such as '1990-12-01 12:00') given by the array parameter are re-interpreted as date-time objects. By default they are not.

loadd: dict, optional

Initialise the data from a dictionary serialization of a cf.Data object. All other arguments are ignored. See the dumpd and loadd methods.

loads: str, optional

Initialise the data array from a string serialization of a Data object. All other arguments are ignored. See the dumps and loads methods.

copy: bool, optional

If False then do not deep copy input parameters prior to initialization. By default arguments are deep copied.

chunk: bool, optional

If False then the data array will be stored in a single partition. By default the data array will be partitioned if it is larger than the chunk size, as returned by the cf.chunksize function.

Examples:

>>> d = cf.Data(5)
>>> d = cf.Data([1,2,3], units='K')
>>> import numpy
>>> d = cf.Data(numpy.arange(10).reshape(2,5),
...             units=Units('m/s'), fill_value=-999)
>>> d = cf.Data(tuple('fly'))

Inspection

Attributes

array	A numpy array copy the data array.
varray	A numpy array view the data array.
dtype	The numpy data-type of the data array.
ndim	Number of dimensions in the data array.
shape	Tuple of the data array’s dimension sizes.
size	Number of elements in the data array.
nbytes	Total number of bytes consumed by the elements of the array.
dump	Return a string containing a full description of the instance.
inspect	Inspect the object for debugging.
isscalar	True if the data array is a 0-d scalar array.

Units

del_units	Delete the units.
get_units	Return the units.
has_units	Whether units have been set.
set_units	Set the units.
override_units	Override the data array units.
del_calendar	Delete the calendar.
get_calendar	Return the calendar.
has_calendar	Whether a calendar has been set.
set_calendar	Set the calendar.
override_calendar	Override the calendar of the data array elements.
change_calendar	Change the calendar of the data array elements.

Attributes

Units

The cf.Units object aining the units of the data array.

Data creation routines

Ones and zeros

empty	Create a new data array without initializing the elements.
full	Return a new data array of given shape and type, filled with the given value.
ones	TODO
zeros	TODO

From existing data

copy	Return a deep copy.
asdata	Convert the input to a Data object.
loadd	Reset the data in place from a dictionary serialization.
loads	TODO

Data manipulation routines

Changing data shape

flatten

Flatten axes of the data

Transpose-like operations

swapaxes	Interchange two axes of an array.
transpose	Permute the axes of the data array.

Changing number of dimensions

insert_dimension	Expand the shape of the data array in place.
squeeze	Remove size 1 axes from the data array.

Joining data

concatenate

Join a sequence of data arrays together.

Adding and removing elements

unique

The unique elements of the array.

Rearranging elements

flip	Reverse the direction of axes of the data array.
roll	A lot like numpy.roll

Expanding the data

halo	Expand the data by adding a halo.

Binary operations

Date-time support

Attributes

datetime_array	An independent numpy array of date-time objects.
datetime_as_string	Return an independent numpy array containing string representations of times since a reference date.
day	The day of each data array element.
hour	The hour of each data array element.
minute	The minute of each data array element.
month	The month of each data array element.
second	The second of each data array element.
year	The year of each data array element.

Indexing routines

Single value selection

datum	Return an element of the data array as a standard Python scalar.
first_element	Return the first element of the data as a scalar.
second_element	Return the second element of the data as a scalar.
last_element	Return the last element of the data as a scalar.

Iterating over data

flat	Return a flat iterator over elements of the data array.
ndindex	Return an iterator over the N-dimensional indices of the data array.

Cyclic axes

cyclic

TODO

Input and output

dumpd	Return a serialization of the data array.
dumps	Return a JSON string serialization of the data array.
tolist	Return the array as a (possibly nested) list.

Linear algebra

outerproduct

Compute the outer product with another data array.

Logic functions

Truth value testing

all	Test whether all data array elements evaluate to True.
any	Test whether any data array elements evaluate to True.

Comparison

allclose	Returns True if two broadcastable arrays have equal values, False otherwise.
isclose	Return where data are element-wise equal to other, broadcastable data.
equals	True if two data arrays are logically equal, False otherwise.

Mask support

apply_masking	Apply masking.
count	Count the non-masked elements of the data.
count_masked	Count the masked elements of the data.
compressed	Return all non-masked values in a one dimensional data array.
filled	TODO
mask_fpe	Masking of floating-point errors in the results of arithmetic operations.
mask_invalid	Mask the array where invalid values occur (NaN or inf).
del_fill_value	Delete the fill value.
get_fill_value	Return the missing data value.
has_fill_value	Whether a fill value has been set.
set_fill_value	Set the missing data value.

Attributes

binary_mask	A binary (0 and 1) mask of the data array.
hardmask	Whether the mask is hard (True) or soft (False).
ismasked	True if the data array has any masked values.
mask	The boolean missing data mask of the data array.
fill_value	The data array missing data value.

Mathematical functions

Trigonometric functions

sin	Take the trigonometric sine of the data element-wise.
cos	Take the trigonometric cosine of the data element-wise.
tan	Take the trigonometric tangent of the data element-wise.
arcsin	Take the trigonometric inverse sine of the data element-wise.
arccos	Take the trigonometric inverse cosine of the data element-wise.
arctan	Take the trigonometric inverse tangent of the data element-wise.

Hyperbolic functions

sinh	Take the hyperbolic sine of the data element-wise.
cosh	Take the hyperbolic cosine of the data element-wise.
tanh	Take the hyperbolic tangent of the data element-wise.
arcsinh	Take the inverse hyperbolic sine of the data element-wise.
arccosh	Take the inverse hyperbolic cosine of the data element-wise.
arctanh	Take the inverse hyperbolic tangent of the data element-wise.

Rounding

ceil	The ceiling of the data, element-wise.
floor	Return the floor of the data array.
rint	Round the data to the nearest integer, element-wise.
round	Evenly round elements of the data array to the given number of decimals.
trunc	Return the truncated values of the data array.

Sums, products, differences

cumsum	Return the data cumulatively summed along the given axis.
diff	Calculate the n-th discrete difference along the given axis.
sum	Collapse axes with their sum.

Convolution filters

convolution_filter

Return the data convolved along the given axis with the specified filter.

Exponents and logarithms

exp	Take the exponential of the data array.
log	TODO

Miscellaneous

clip	Clip (limit) the values in the data array in place.

Set routines

Making proper sets

unique

The unique elements of the array.

Sorting, searching, and counting

Searching

argmax	Return the indices of the maximum values along an axis.
where	Assign to data elements depending on a condition.

Counting

count	Count the non-masked elements of the data.
count_masked	Count the masked elements of the data.

Statistics

Order statistics

maximum	Collapse axes with their maximum.
maximum_absolute_value	Collapse axes with their maximum absolute value.
minimum	Collapse axes with their minimum.
minimum_absolute_value	Collapse axes with their minimum absolute value.
percentile	Compute percentiles of the data along the specified axes.

Averages and variances

mean	Collapse axes with their mean.
mean_absolute_value	Collapse axes with their mean absolute value.
mean_of_upper_decile	TODO
median	TODO
mid_range	Collapse axes with the unweighted average of their maximum and minimum values.
range	Collapse axes with the absolute difference between their maximum and minimum values.
root_mean_square	TODO Collapse axes with their weighted mean.
standard_deviation	Collapse axes by calculating their standard deviation.
variance	Collapse axes with their weighted variance.

Sums

integral	TODO
sum	Collapse axes with their sum.
sum_of_squares	Collapse axes with the sum of the squares of the values.

Histograms

digitize

Return the indices of the bins to which each value belongs.

Miscellaneous

sample_size	TODO
stats	Calculate statistics of the data.
sum_of_weights	TODO
sum_of_weights2	TODO

Error handling

seterr

Set how floating-point errors in the results of arithmetic operations are handled.

Compression by convention

get_compressed_axes	Return the dimensions that have compressed in the underlying array.
get_compressed_dimension	Return the position of the compressed dimension in the compressed array.
get_compression_type	Return the type of compression applied to the underlying array.
get_count	Return the count variable for a compressed array.
get_index	Return the index variable for a compressed array.
get_list	Return the list variable for a compressed array.
uncompress	Uncompress the underlying data.

Attributes

compressed_array

Return an independent numpy array containing the compressed data.

Miscellaneous

creation_commands	Return the commands that would create the data object.
get_filenames	Return the names of files containing parts of the data array.
get_data	TODO
source	Return the underlying array object.

Attributes

data	The data as an object identity.

Performance

nc_clear_hdf5_chunksizes	Clear the HDF5 chunksizes for the data.
nc_hdf5_chunksizes	Return the HDF5 chunksizes for the data.
nc_set_hdf5_chunksizes	Set the HDF5 chunksizes for the data.
close	Close all files referenced by the data array.
chunk	Partition the data array.
add_partitions	Add partition boundaries.
partition_boundaries	Return the partition boundaries for each partition matrix dimension.
partition_configuration	Return parameters for opening and closing array partitions.
to_disk	Store the data array on disk.
to_memory	Store each partition’s data in memory in place if the master array is smaller than the chunk size.
fits_in_memory	Return True if the master array is small enough to be retained in memory.
fits_in_one_chunk_in_memory	Return True if the master array is small enough to be retained in memory.
section	Return a dictionary of Data objects, which are the m dimensional sections of this n dimensional Data object, where m <= n.
reconstruct_sectioned_data	Expects a dictionary of Data objects with ordering information as keys, as output by the section method when called with a Data object.

Element-wise arithmetic, bit and comparison operations

Arithmetic, bit and comparison operations are defined as element-wise data array operations which yield a new cf.Data object or, for augmented assignments, modify the data in-place.

Comparison operators

__lt__	The rich comparison operator <
__le__	The rich comparison operator <=
__eq__	The rich comparison operator ==
__ne__	The rich comparison operator !=
__gt__	The rich comparison operator >
__ge__	The rich comparison operator >=

Truth value of an array

__bool__

Truth value testing and the built-in operation bool

Binary arithmetic operators

__add__	The binary arithmetic operation +
__sub__	The binary arithmetic operation -
__mul__	The binary arithmetic operation *
__div__	The binary arithmetic operation /
__truediv__	The binary arithmetic operation / (true division)
__floordiv__	The binary arithmetic operation //
__pow__	The binary arithmetic operations ** and pow
__mod__	The binary arithmetic operation %

Binary arithmetic operators with reflected (swapped) operands

__radd__	The binary arithmetic operation + with reflected operands
__rsub__	The binary arithmetic operation - with reflected operands
__rmul__	The binary arithmetic operation * with reflected operands
__rdiv__	The binary arithmetic operation / with reflected operands
__rtruediv__	The binary arithmetic operation / (true division) with reflected operands
__rfloordiv__	The binary arithmetic operation // with reflected operands
__rpow__	The binary arithmetic operations ** and pow with reflected operands
__rmod__	The binary arithmetic operation % with reflected operands

Augmented arithmetic assignments

__iadd__	The augmented arithmetic assignment +=
__isub__	The augmented arithmetic assignment -=
__imul__	The augmented arithmetic assignment *=
__idiv__	The augmented arithmetic assignment /=
__itruediv__	The augmented arithmetic assignment /= (true division)
__ifloordiv__	The augmented arithmetic assignment //=
__ipow__	The augmented arithmetic assignment **=
__imod__	The binary arithmetic operation %=

Unary arithmetic operators

__neg__	The unary arithmetic operation -
__pos__	The unary arithmetic operation +
__abs__	The unary arithmetic operation abs

Binary bitwise operators

__and__	The binary bitwise operation &
__or__	The binary bitwise operation |
__xor__	The binary bitwise operation ^
__lshift__	The binary bitwise operation <<
__rshift__	The binary bitwise operation >>

..rubric:: Binary bitwise operators with reflected (swapped) operands

__rand__	The binary bitwise operation & with reflected operands
__ror__	The binary bitwise operation | with reflected operands
__rxor__	The binary bitwise operation ^ with reflected operands
__rlshift__	The binary bitwise operation << with reflected operands
__rrshift__	The binary bitwise operation >> with reflected operands

Augmented bitwise assignments

__iand__	The augmented bitwise assignment &=
__ior__	The augmented bitwise assignment |=
__ixor__	The augmented bitwise assignment ^=
__ilshift__	The augmented bitwise assignment <<=
__irshift__	The augmented bitwise assignment >>=

Unary bitwise operators

__invert__

The unary bitwise operation ~

Special

__array__	The numpy array interface.
__contains__	Membership test operator in
__data__	Returns a new reference to self.
__deepcopy__	Called by the copy.deepcopy function.
__getitem__	Return a subspace of the data defined by indices.
__hash__	The built-in function hash
__iter__	Called when an iterator is required.
__len__	The built-in function len
__repr__	Called by the repr built-in function.
__setitem__	Implement indexed assignment.
__str__	Called by the str built-in function.
__query_set__	TODO
__query_wi__	TODO
__query_wo__	TODO