Table of Contents
Shortcuts

FileClient

class mmengine.fileio.FileClient(backend=None, prefix=None, **kwargs)[source]

A general file client to access files in different backends.

The client loads a file or text in a specified backend from its path and returns it as a binary or text file. There are two ways to choose a backend, the name of backend and the prefix of path. Although both of them can be used to choose a storage backend, backend has a higher priority that is if they are all set, the storage backend will be chosen by the backend argument. If they are all None, the disk backend will be chosen. Note that It can also register other backend accessor with a given name, prefixes, and backend class. In addition, We use the singleton pattern to avoid repeated object creation. If the arguments are the same, the same object will be returned.

Warning

FileClient will be deprecated in future. Please use io functions in https://onedl-mmengine.readthedocs.io/en/latest/api/fileio.html#file-io

Parameters:

  • backend (str, optional) – The storage backend type. Options are “disk”, “memcached”, “lmdb”, “http” and “petrel”. Defaults to None.

  • prefix (str, optional) – The prefix of the registered storage backend. Options are “s3”, “http”, “https”. Defaults to None.

Examples

>>> # only set backend
>>> file_client = FileClient(backend='petrel')
>>> # only set prefix
>>> file_client = FileClient(prefix='s3')
>>> # set both backend and prefix but use backend to choose client
>>> file_client = FileClient(backend='petrel', prefix='s3')
>>> # if the arguments are the same, the same object is returned
>>> file_client1 = FileClient(backend='petrel')
>>> file_client1 is file_client
True
client

The backend object.

Type:

BaseStorageBackend

exists(filepath)[source]

Check whether a file path exists.

Parameters:

filepath (str or Path) – Path to be checked whether exists.

Returns:

Return True if filepath exists, False otherwise.

Return type:

bool

get(filepath)[source]

Read data from a given filepath with ‘rb’ mode.

Note

There are two types of return values for get, one is bytes and the other is memoryview. The advantage of using memoryview is that you can avoid copying, and if you want to convert it to bytes, you can use .tobytes().

Parameters:

filepath (str or Path) – Path to read data.

Returns:

Expected bytes object or a memory view of the bytes object.

Return type:

bytes | memoryview

get_local_path(filepath)[source]

Download data from filepath and write the data to local path.

get_local_path is decorated by contxtlib.contextmanager(). It can be called with with statement, and when exists from the with statement, the temporary path will be released.

Note

If the filepath is a local path, just return itself.

Warning

get_local_path is an experimental interface that may change in the future.

Parameters:

filepath (str or Path) – Path to be read data.

Return type:

Generator[str | Path, None, None]

Examples

>>> file_client = FileClient(prefix='s3')
>>> with file_client.get_local_path('s3://bucket/abc.jpg') as path:
...     # do something here
Yields:

Iterable[str] – Only yield one path.

Parameters:

filepath (str | Path)

Return type:

Generator[str | Path, None, None]

get_text(filepath, encoding='utf-8')[source]

Read data from a given filepath with ‘r’ mode.

Parameters:

  • filepath (str or Path) – Path to read data.

  • encoding (str) – The encoding format used to open the filepath. Defaults to ‘utf-8’.

Returns:

Expected text reading from filepath.

Return type:

str

classmethod infer_client(file_client_args=None, uri=None)[source]

Infer a suitable file client based on the URI and arguments.

Parameters:

  • file_client_args (dict, optional) – Arguments to instantiate a FileClient. Defaults to None.

  • uri (str | Path, optional) – Uri to be parsed that contains the file prefix. Defaults to None.

Return type:

FileClient

Examples

>>> uri = 's3://path/of/your/file'
>>> file_client = FileClient.infer_client(uri=uri)
>>> file_client_args = {'backend': 'petrel'}
>>> file_client = FileClient.infer_client(file_client_args)
Returns:

Instantiated FileClient object.

Return type:

FileClient

Parameters:

  • file_client_args (dict | None)

  • uri (str | Path | None)

isdir(filepath)[source]

Check whether a file path is a directory.

Parameters:

filepath (str or Path) – Path to be checked whether it is a directory.

Returns:

Return True if filepath points to a directory, False otherwise.

Return type:

bool

isfile(filepath)[source]

Check whether a file path is a file.

Parameters:

filepath (str or Path) – Path to be checked whether it is a file.

Returns:

Return True if filepath points to a file, False otherwise.

Return type:

bool

join_path(filepath, *filepaths)[source]

Concatenate all file paths.

Join one or more filepath components intelligently. The return value is the concatenation of filepath and any members of *filepaths.

Parameters:

  • filepath (str or Path) – Path to be concatenated.

  • filepaths (str | Path)

Returns:

The result of concatenation.

Return type:

str

list_dir_or_file(dir_path, list_dir=True, list_file=True, suffix=None, recursive=False)[source]

Scan a directory to find the interested directories or files in arbitrary order.

Note

list_dir_or_file() returns the path relative to dir_path.

Parameters:

  • dir_path (str | Path) – Path of the directory.

  • list_dir (bool) – List the directories. Defaults to True.

  • list_file (bool) – List the path of files. Defaults to True.

  • suffix (str or tuple[str], optional) – File suffix that we are interested in. Defaults to None.

  • recursive (bool) – If set to True, recursively scan the directory. Defaults to False.

Yields:

Iterable[str] – A relative path to dir_path.

Return type:

Iterator[str]

static parse_uri_prefix(uri)[source]

Parse the prefix of a uri.

Parameters:

uri (str | Path) – Uri to be parsed that contains the file prefix.

Return type:

str | None

Examples

>>> FileClient.parse_uri_prefix('s3://path/of/your/file')
's3'
Returns:

Return the prefix of uri if the uri contains ‘://’ else None.

Return type:

str | None

Parameters:

uri (str | Path)

put(obj, filepath)[source]

Write data to a given filepath with ‘wb’ mode.

Note

put should create a directory if the directory of filepath does not exist.

Parameters:

  • obj (bytes) – Data to be written.

  • filepath (str or Path) – Path to write data.

Return type:

None

put_text(obj, filepath)[source]

Write data to a given filepath with ‘w’ mode.

Note

put_text should create a directory if the directory of filepath does not exist.

Parameters:

  • obj (str) – Data to be written.

  • filepath (str or Path) – Path to write data.

  • encoding (str, optional) – The encoding format used to open the filepath. Defaults to ‘utf-8’.

Return type:

None

classmethod register_backend(name, backend=None, force=False, prefixes=None)[source]

Register a backend to FileClient.

This method can be used as a normal class method or a decorator.

class NewBackend(BaseStorageBackend):

    def get(self, filepath):
        return filepath

    def get_text(self, filepath):
        return filepath

FileClient.register_backend('new', NewBackend)

or

@FileClient.register_backend('new')
class NewBackend(BaseStorageBackend):

    def get(self, filepath):
        return filepath

    def get_text(self, filepath):
        return filepath
Parameters:

  • name (str) – The name of the registered backend.

  • backend (class, optional) – The backend class to be registered, which must be a subclass of BaseStorageBackend. When this method is used as a decorator, backend is None. Defaults to None.

  • force (bool, optional) – Whether to override the backend if the name has already been registered. Defaults to False.

  • prefixes (str or list[str] or tuple[str], optional) – The prefixes of the registered storage backend. Defaults to None. New in version 1.3.15.

remove(filepath)[source]

Remove a file.

Parameters:

filepath (str, Path) – Path to be removed.

Return type:

None