API Reference¶
This document is the API reference of Remofile. All classes, functions and exceptions are accessible from a single import of the remofile package.
from remofile import *
The programming interface is essentially made of two classes which are Client and Server. They implement both side of the Remofile protocol.
Interface overview¶
The client class implements all primitive file operations and therefore any more complex file operations can, in theory, be implemented on top of it.
List of primitive file operations.
- List files
- Create a file
- Create a directory
- Upload a file
- Upload a directory
- Download a file
- Download a directory
- Delete a file (or directory)
However, the algorithm module already implements a couple of useful more advanced file operations for you. For instance, it can upload and download trees of files, understand glob patterns and handle conflicts. It also has functions to synchronize directories from client to server and server to client.
Dealing with errors is an important aspect in a code that involves file
operations. A set of exceptions are
defined and they all have the RemofileException
exception as
base class. Therefore, you can catch most of Remofile-related exceptions in
one statement.
try:
do_file_operation()
except RemofileException:
deal_with_exception()
Additionally, some helper functions like
generate_token()
, generate_keys()
and
is_file_name_valid()
are also exposed.
Two main classes¶
-
class
remofile.
Client
(hostname, port, token)¶ Remofile client.
This class implements the client side of Remofile that behaves according to the protocol.
It’s a synchronous (non-threaded) and “connectionless” interface to interact with the remote directory. Indeed, a connection is etablished behind the scene but it isn’t exposed. As a result, it simplifies the interface and you don’t have do deal directly with connections (and eventual re-connections that may happen). Instead, you use a timeout and catch the
TimeoutError
exception that is raised if the time is out. All production code should use a timeout.All native file operations are implemented such as listing files, creating files and directories, upload/download files and deleting files. Exceptions are raised whenever errors occur or if the file operation couldn’t be successfully completed. Also note that directories and symbolic links are also reffered as ‘file’.
For more complex file operations such as synchronizing directories or upload/download trees of files, while handling file conflicts and glob pattern, see the algorithm module. Otherwise implement your file operations on top of the client instance.
-
__init__
(hostname, port, token)¶ Construct a
Client
instance.The client instance is constructed from the hostname (which can either be “localhost” or any valid IP), the port and the token.
The token should be generated with the
generate_token()
to ensure validity of its value, or aValueError
exception is raised.Parameters: Raises: ValueError – If the token isn’t valid.
-
list_files
(directory, timeout=None)¶ List files in the remote directory.
It lists files of a given directory in a remote directory and returns a dictionnary associating file names with their metadata. The metadata is a tuple that includes a boolean indicating whether the file is a directory or not, the size of the file (the value is 0 in case of directory) and the last modification time of the file.
Return value example.
{ 'foo.bin' : (False, 423, 4687421324), 'bar' : (True, 0, 1654646515) }
The directory parameter must be a path-like object that refers to an existing directory in the remote directory for which it must list files for. If the directory doesn’t exist, the
FileNotFoundErrror
exception is raised, and if it’s not an actual directory, theNotADirectoryError
exception is raised. It must be an absolute path or aValueError
exception is raised.If the operation takes longer than the given timeout, a
TimeoutError
exception is raised.Parameters: - directory (path) – The given remote directory to list files for.
- timeout (int) – How many milliseconds to wait before giving up
Raises: - ValueError – If the directory is not an absolute path.
- FileNotFoundError – If the directory doesn’t exist.
- NotADirectoryError – If the directory is not a directory.
- TimeoutError – If it takes more than the timeout value to receive a response.
Returns: A dictionnary associating file name with their metadata.
Return type:
-
create_file
(name, directory, timeout=None)¶ Create a file in the remote directory.
It creates an empty file with a given name in a given directory located in the remote directory.
The name parameter must be a string of a valid file name and must not conflict with an existing file (or directory) in the given remote directory. If the name isn’t valid, a
FileNameError
is raised and if the file is conflicting, aFileExistsError
exception is raised.The directory parameter must be a path-like object that refers to an existing directory in the remote directory where the file must be created. If the directory doesn’t exist, the
FileNotFoundError
exception is raised, and if it’s not an actual directory, theNotADirectoryError
exception is raised. It must be an absolute path or aValueError
exception is raised.If the operation takes longer than the given timeout, a
TimeoutError
exception is raised.Parameters: Raises: - ValueError – If the directory is not an absolute path.
- FileNotFound – If the directory doesn’t exist.
:raises NotADirectoryError If the directory is not a directory. :raises FileNameError: If the name of the file isn’t valid. :raises FileExistsError: If the name conflicts with the name of an existing file or directory. :raises TimeoutError: If it takes more than the timeout value to receive a response.
-
make_directory
(name, directory, timeout=None)¶ Create a directory in the remote directory.
It creates an empty directory with a given name in a given directory located in the remote directory.
The name parameter must be a string of a valid file name and must not conflict with an existing file (or directory) in the given remote directory. If the name isn’t valid, a
FileNameError
is raised and if the file is conflicting, aFileExistsError
exception is raised.The directory parameter must be a path-like object that refers to an existing directory in the remote directory where the directory must be created. If the directory doesn’t exist, the
NotADirectoryError
exception is raised, and if it’s not an actual directory, theNotADirectoryError
exception is raised. It must be an absolute path or aValueError
exception is raised.If the operation takes longer than the given timeout, a
TimeoutError
exception is raised.Parameters: Raises: - ValueError – If the directory is not an absolute path.
- FileNotFound – If the directory doesn’t exist.
:raises NotADirectoryError If the directory is not a directory. :raises FileNameError: If the name isn’t valid. :raises FileExistsError: If the name conflicts with the name of an existing file or directory. :raises TimeoutError: If it takes more than the timeout value to receive a response.
-
upload_file
(source, destination, name=None, chunk_size=512, process_chunk=None, timeout=None)¶ Upload a file to the remote directory.
This method uploads a single file to a given directory in the remote directory.
The source parameter refers to the local file to be transfered to the remote directory and must to be a path-like object. If it’s a relative path, it’s treated like relative to the current working directory. If the source file can’t be found or is not a file, the
SourceNotFound
exception is raised.The destination parameter refers to the remote directory in which the file must be transfered to. It must be a path-like object of an existing directory and it must be an absolute path or the
ValueError
exception is raised. If the destination directory can’t be found or is not a directory, theDestinationNotFound
exception is raised.The name parameter can be used to rename the source file while uploading it (the content is guaranteed to be the same). It must be a string of a valid file name and must not conflict with an existing file (or directory) in the destination directory. By default, it reads the name from the source to leave it unchanged. If the name isn’t valid, a
FileNameError
is raised and if the file is conflicting, aFileExistsError
exception is raised.Additionally, you can adjust the chunk size value which defines how fragmented the file has to be sent to the server and/or pass a callback that process each fragment before it’s sent to the server. Usually, the chunk value is between 512 and 8192.
The callback is called with various parameters and in a specific order; the chunk data, the remaining bytes, the file size and the file name. The chunk data is a bytes string of the actual data about to be sent to the server. The remaining bytes is an integer indicating the number of bytes left to be sent (and this includes the current chunk of data). The file size is a fixed integer telling how large the file is, and the file name is the file name currently being processed.
For instance, it can be used to display a progress indicator. Here is an example.
def display_progress(chunk_data, remaining_bytes, file_size, file_name): chunk_size = 512 progress = (file_size - (remaining_bytes - len(chunk_data))) / file_size * 100 sys.stdout.write("
- {0:0.2f}% | {1}”.format(progress, file_name))
sys.stdout.flush()
- if remaining_bytes <= chunk_size:
- sys.stdout.write(‘
‘)
return TrueIf the operation takes longer than the given timeout, a
TimeoutError
exception is raised.param source: The (local) source file to upload. param destination: The (remote) destination directory where to upload the file. param name: The name of the file after it’s uploaded. param chunk_size: How fragmented (in bytes) the file is during the upload process. param process_chunk: Function processing chunks before they are sent out. param timeout: How many milliseconds to wait before giving up. raises ValueError: If the destination directory isn’t an absolute path. raises SourceNotFound: If the source file doesn’t exist or isn’t a file. raises DestinationNotFound: If the destination directory doesn’t exist or isn’t a directory. raises FileExistsError: If the source file conflicts with an existing file or directory. raises FileNameError: If the source file doesn’t have a valid name. raises ValueError: If the chunk size or file size is invalid. raises TimeoutError: If it takes more than the timeout value to receive a response.
-
upload_directory
(source, destination, name=None, chunk_size=512, process_chunk=None, timeout=None)¶ Upload a directory to the remote directory.
This method uploads an entire directory to a given directory in the remote directory.
The source parameter refers to the local directory to be transfered to the remote directory and must to be a path-like object. If it’s a relative path, it’s treated like relative to the current working directory. If the source directory can’t be found or is not a directory, the
SourceNotFound
exception is raised.The destination parameter refers to the remote directory in which the directory must be transfered to. It must be a path-like object of an existing directory and it must be an absolute path or the
ValueError
exception is raised. If the destination directory can’t be found or is not a directory, theDestinationNotFound
exception is raised.The name parameter can be used to rename the source directory while uploading it (the content is guaranteed to be the same). It must be a string of a valid file name and must not conflict with an existing directory (or file) in the destination directory. By default, it reads the name from the source to leave it unchanged. If the name isn’t valid, a
FileNameError
is raised and if the file is conflicting, aFileExistsError
exception is raised.Additionally, you can adjust the chunk size value which defines how fragmented files have to be sent to the server and/or pass a callback that process each fragment before it’s sent to the server. Usually, the chunk value is between 512 and 8192.
The callback is called with various parameters and in a specific order; the chunk data, the remaining bytes, the file size and the file name. The chunk data is a bytes string of the actual data about to be sent to the server. The remaining bytes is an integer indicating the number of bytes left to be sent (and this includes the current chunk of data). The file size is a fixed integer telling how large the file is, and the file name is the file name currently being processed.
For instance, it can be used to display a progress indicator. Here is an example.
def display_progress(chunk_data, remaining_bytes, file_size, file_name): chunk_size = 512 progress = (file_size - (remaining_bytes - len(chunk_data))) / file_size * 100 sys.stdout.write("
- {0:0.2f}% | {1}”.format(progress, file_name))
sys.stdout.flush()
- if remaining_bytes <= chunk_size:
- sys.stdout.write(‘
‘)
return TrueIf the operation takes longer than the given timeout, a
TimeoutError
exception is raised.param source: The (local) source directory to upload. param destination: The (remote) destination directory where to upload the directory. param name: The name of the directory after it’s uploaded. param chunk_size: How fragmented (in bytes) files are during the upload process. param process_chunk: Function processing chunks before they are sent out. param timeout: How many milliseconds to wait before giving up. raises ValueError: If the destination directory isn’t an absolute path. raises SourceNotFound: If the source directory doesn’t exist or isn’t a directory. raises DestinationNotFound: If the destination directory doesn’t exist or isn’t a directory. raises FileExistsError: If the source directory conflicts with an existing file or directory. raises FileNameError: If the source directory doesn’t have a valid name. raises ValueError: If the chunk size or file size is invalid. raises TimeoutError: If it takes more than the timeout value to receive a response.
-
download_file
(source, destination, name=None, chunk_size=512, process_chunk=None, timeout=None)¶ Download a file from the remote directory.
This method downloads a single file from a given directory in the remote directory.
The source parameter refers to the remote file to be transfered from the remote directory and must to be a path-like object. It must be an absolute path or it will raise the ValueError exception. If the source file can’t be found or is not a file, the SourceNotFound exception is raised.
The destination parameter refers to an existing local directory in which the file must be transfered to. It must be a path-like object and if it’s a relative path, it’s treated like relative to the current working directory. If the destination directory can’t be found or is not a directory, the DestinationNotFound exception is raised.
The name parameter can be used to rename the source file while downloading it (the content is guaranteed to be the same). It must be a string of a valid file name and must not conflict with an existing file (or directory) in the destination directory. By default, it reads the name from the source to leave it unchanged. If the name isn’t valid, a
FileNameError
is raised and if the file is conflicting, aFileExistsError
exception is raised.Additionally, you can adjust the chunk size value which defines how fragmented the file has to be received from the server and/or pass a callback that process each fragment before it’s written to the local file. Usually, the chunk value is between 512 and 8192.
The callback is called with various parameters and in a specific order; the chunk data, the remaining bytes, the file size and the file name. The chunk data is a bytes string of the actual data just received from the server. The remaining bytes is an integer indicating the number of bytes left to be received (and this includes the current chunk of data). The file size is a fixed integer telling how large the file is, and the file name is the file name currently being processed.
For instance, it can be used to display a progress indicator. Here is an example.
def display_progress(chunk_data, remaining_bytes, file_size, file_name): chunk_size = 512 progress = (file_size - (remaining_bytes - len(chunk_data))) / file_size * 100 sys.stdout.write("
- {0:0.2f}% | {1}”.format(progress, file_name))
sys.stdout.flush()
- if remaining_bytes <= chunk_size:
- sys.stdout.write(‘
‘)
return TrueIf the operation takes longer than the given timeout, a
TimeoutError
exception is raised.param source: The (remote) source file to download. param destination: The (local) destination directory where to download the file. param name: The name of the file after it’s downloaded. param chunk_size: Foobar. param process_chunk: Foobar. param timeout: How many milliseconds to wait before giving up. raises ValueError: If the source directory isn’t an absolute path. raises SourceNotFound: If the source file doesn’t exist or isn’t a file. raises DestinationNotFound: If the destination directory doesn’t exist or isn’t a directory. raises FileExistsError: If the source file conflicts with an existing file or directory. raises FileNameError: If the source file doesn’t have a valid name. raises TimeoutError: If it takes more than the timeout value to receive a response.
-
download_directory
(source, destination, name=None, chunk_size=512, process_chunk=None, timeout=None)¶ Download a directory from the remote directory.
This method downloads an entire directory from a given directory in the remote directory.
The source parameter refers to the remote directory to be transfered from the remote directory and must to be a path-like object. It must be an absolute path or it will raise the ValueError exception. If the source directory can’t be found or is not a directory, the SourceNotFound exception is raised.
The destination parameter refers to an existing local directory in which the directory must be transfered to. It must be a path-like object and if it’s a relative path, it’s treated like relative to the current working directory. If the destination directory can’t be found or is not a directory, the DestinationNotFound exception is raised.
The name parameter can be used to rename the source directory while downloading it (the content is guaranteed to be the same). It must be a string of a valid file name and must not conflict with an existing directory (or file) in the destination directory. By default, it reads the name from the source to leave it unchanged. If the name isn’t valid, a
FileNameError
is raised and if the file is conflicting, aFileExistsError
exception is raised.Additionally, you can adjust the chunk size value which defines how fragmented files have to be received from the server and/or pass a callback that process each fragment before it’s written to the local file. Usually, the chunk value is between 512 and 8192.
The callback is called with various parameters and in a specific order; the chunk data, the remaining bytes, the file size and the file name. The chunk data is a bytes string of the actual data just received from the server. The remaining bytes is an integer indicating the number of bytes left to be received (and this includes the current chunk of data). The file size is a fixed integer telling how large the file is, and the file name is the file name currently being processed.
For instance, it can be used to display a progress indicator. Here is an example.
def display_progress(chunk_data, remaining_bytes, file_size, file_name): chunk_size = 512 progress = (file_size - (remaining_bytes - len(chunk_data))) / file_size * 100 sys.stdout.write("
- {0:0.2f}% | {1}”.format(progress, file_name))
sys.stdout.flush()
- if remaining_bytes <= chunk_size:
- sys.stdout.write(‘
‘)
return TrueIf the operation takes longer than the given timeout, a
TimeoutError
exception is raised.param source: The (remote) source directory to download. param destination: The (local) destination directory where to download the directory. param name: The name of the directory after it’s downloaded. param chunk_size: Foobar. param process_chunk: Foobar. param timeout: How many milliseconds to wait before giving up. raises ValueError: If the source directory isn’t an absolute path. raises SourceNotFound: If the source file doesn’t exist or isn’t a file. raises DestinationNotFound: If the destination directory doesn’t exist or isn’t a directory. raises FileExistsError: If the source file conflicts with an existing file or directory. raises FileNameError: If the source file doesn’t have a valid name. raises TimeoutError: If it takes more than the timeout value to receive a response.
-
-
class
remofile.
Server
(root_directory, token, private_key=None, **kwargs)¶ Remofile server.
This class implements the server side of Remofile that behaves according to the protocol.
It’s a single-threaded server that will jail a given directory called root directory, and exposes it on Internet using a given range of IP addresses and a port. It doesn’t start listening to connections until the blocking
run()
method is called. To interrupt the loop, call theterminate()
method from a another thread. The server can also be configured with various options.The
run()
may be found within a try-except statement to catch theKeyboardInterrupt()
exception during testing.try: server.run(port) except KeyboardInterrupt: server.terminate()
The configuration options includes the file size limit and the chunk size range. The file size limit prevents clients from transfering files exceeding a given size (expressed in bytes), and the chunk size range prevents clients from
-
__init__
(root_directory, token, private_key=None, **kwargs)¶ Construct a
Server
instance.The server instance is constructed from the root directory, the token, an optional private key and the server options values.
The root directory parameter must be a path-like object of an existing directory or a
NotADirectoryError
exception is raised. The token and private key should respectively be generated with thegenerate_token()
andgenerate_keys()
to ensure validity of their values, or it will raise aValueError
.For the other parameters, check out the corresponding server options properties
file_size_limit
andchunk_size_range
.Parameters: - root_directory – The directory which will be exposed to clients.
- token – The token that clients must use to be granted access.
- private_key – The private key to use to encrypt communication with clients.
- file_size_limit – The file size limit (in bytes) files can’t exceed during upload/download.
- chunk_size_range – The minimum and maximum chunk size (in bytes) allowed during upload/download.
Raises: - NotADirectoryError – If the root directory isn’t a valid (isn’t a directory or doesn’t exist).
- ValueError – If the token or private_key aren’t valid.
-
root_directory
¶ The directory to be exposed.
The root directory must be a path-like object that refers to an existing directory. If a relative path is passed, it’s combined with the current working directory to get an absolute path. If the root directory doesn’t exist, the NotADirectoryError exception is raised.
Note that the root directory can’t be changed while the server is running, or it will raise a RuntimeError exception.
Getter: Returns the root directory.
Setter: Changes the root directory.
Type: path-like object
Raises: - NotADirectoryError – If the root directory doesn’t exist or isnt’ a directory.
- RuntimeError – If the value is changed while the server is running.
-
file_size_limit
¶ The file size limit.
The file size limit must be an integer specificying in bytes the maximum size a file can have to be accepted and transferred. The value can’t zero or negative or it will raise a ValueError exception.
Note that the file size limit can’t be changed while the server is running, or it will raise a RuntimeError exception.
Getter: Returns the file size limit in bytes.
Setter: Changes the file size limit in bytes.
Type: integer
Raises: - ValueError – If the value is less or equal to 0.
- RuntimeError – If the value is changed while the server is running.
-
chunk_size_range
¶ The chunk size range.
The chunk size range must be a tuple of two integers specifying the minimum chunk size and maximum chunk size (in bytes) allowed during negotiating upload and download transfers. The minimum chunk size value can’t zero or negative or it will raise a ValueError exception.
Note that the chunk size range can’t be changed while the server is running, or it will raise a RuntimeError exception.
Getter: Returns the chunk size range.
Setter: Changes the chunk size range.
Type: tuple of two integer
Raises: - ValueError – If the value is less or equal to 0.
- RuntimeError – If the value is changed while the server is running.
-
run
(port, address=None)¶ Start the main loop.
This method starts the main loop after it initializes the sockets which listen on a given port and optionally a specific IP address. By default, it listens on all available IP addresses. If it’s unable to listen on a specific port and/or IP address, it will raise a
RuntimeError
exception.It’s a blocking method that won’t return until the
terminate()
method is called. Usually, therun()
method is called from an external thread and the main thread calls theterminate()
method.Parameters: Raises: - RuntimeError – If the server is unable to listen on the IP address(es) and/or the port.
- RuntimeError – If the server is already running.
-
terminate
()¶ Terminate the main loop.
This method interrupt the main loop causing the server to terminate. It can safely be called from a different thread where the initial call to
run()
was made. If the server wsa transferring files, operations are all interupted and the client is disconnected.Raises: RuntimeError – If the server is not running.
-
Advanced algorithms¶
A bunch of advanced algorithms are implemented on top of the primitive file operations because in practice, we need more than that.
You will particularly want to have a look at upload_files()
and
download_files()
as they provide a more sophisticated interface to
fine-grain control what is to be uploaded or downloaded. And you may also need
synchronization features in both directions which is done with
synchronize_upload()
and synchronize_download()
.
-
remofile.
upload_files
(client, source, destination, relative_directory)¶ Upload files to the remote directory.
This method uploads one file, an entire directory or a set of files (specified by glob patterns) to a given directory in the remote directory.
The source parameter refers to the local file(s) to be transfered to the remote directory and must to be a path-like object of a relative directory that can contain glob patterns. Source files are related to the directory specified by the relative_directory parameter which itself must be a path-like object of an existing absolute directory. By default, the relative directory is the current working directory. If the source is a directory or refers to sub-directories, the recursive parameter must be set.
The destination parameter refers to the remote directory in which the file(s) must be transfered to. It must be a path-like object aof an existing absolute directory.
The exclude_files parameter is a sequence of path-like objects of relative path (that may or may not contain glob patterns) that refers to files to be excluded from the transfer.
The timeout parameter corresponds to time allowed to send each chunk.
Parameters:
-
remofile.
download_files
(client, source, destination, relative_directory)¶ Download files from the remote directory.
This method downloads one file, an entire directory or a set of files (specified by shell glob pattern) to a given directory in in the local filesystem. Additional parameters are there to refine the process.
The source parameter refers to the remote file(s) to be transfered to the local filesystem and is expected to be a path-like object, unless it’s a shell glob pattern in which case a string is expected. The path must be relative and is by default relative to the root directory. It can be altered with the relative_directory parameter which itself must be path-like object refering to an absolute directory.
The destination parameter refers to the remote directory in which the file(s) must be transfered to. It must be a path-like object and must be absolute.
Long description.
Parameters:
-
remofile.
synchronize_upload
(client, source, destination)¶ Synchronize remote files with a local directory.
Long description.
-
remofile.
synchronize_download
(client, source, destination)¶ Synchronize local files with a remote directory.
Long description.
Handling exceptions¶
This section isn’t written yet.
-
exception
remofile.
RemofileException
¶ Base exception for Remofile-related exceptions.
Remofile-related exceptions excludes filesystem-related exceptions, except for
SourceNotFound
andDestinationNotFound
which exist to simplify catching exceptions in some functions.
-
exception
remofile.
SourceNotFound
¶ Source is not found.
This exception is raised in upload/download/synchronize related functions to catch the numberous different exceptions that a bad source could raise.
It’s triggered if a source file doesn’t exist, or if it isn’t a file or a directory (according to the context).
-
exception
remofile.
DestinationNotFound
¶ Destination is not found.
This exception is raised in upload/download/synchronize related functions to catch the numberous different exceptions that a bad destination could raise.
It’s triggered if a destination directory doesn’t exist or if it’s not a directory.
-
exception
remofile.
UnexpectedError
(message)¶ Unexpected error occured.
This exception is raised whenever the server couldn’t fulfill the request because an unexpected error occured on the server side.
It’s well explained in the protocol specifications document.
Variables: message (str) – Underlying exception message that occured on the server.
-
exception
remofile.
BadRequestError
¶ Bad request error occured.
This exception is raised when the client sends a badly formatted request. For instance, it can occur when the server isn’t ready to handle the request because it’s not in a valid state.
It’s well explained in the protocol specifications document.
-
exception
remofile.
CorruptedResponse
(message, error)¶ Corrupted response was received.
This exception is raised when the client is unable to process the response returned by the server. The client stricly implements the protocol and if a response isn’t expected or doesn’t have the correct format, the response is said corrupted.
A
message
describing how the response could not be processed is available in attribute.Examples of message.
- Unable to extract response type from response
- Invalid reason type in refuse response
- Unable to extract message from error response
The
error
attribute is the underlying exception message that was raised while processing the corrupted response.Variables:
-
exception
remofile.
FileNameError
¶ File name is invalid.
This exception is raised when the file name is incorrect as stated by the Remofile protcol.
Helper functions¶
These helper functions will come in handy to generate a random valid token and generate a pair of private and public key to configure the server with. There is also a function to check validity of a filename as defined by the Remofile protocol.
-
remofile.
generate_token
()¶ Brief description.
Long description.
-
remofile.
generate_keys
()¶ Brief description.
Long description.
-
remofile.
is_file_name_valid
(name)¶ Brief description.
Long description.
Parameters: name – Description.