pstorage-overview − overview of Parallels Storage
Parallels Storage (Pstorage) is a highly−available distributed storage with built−in replication and disaster recovery. Pstorage provides a storage virtualization platform on top of commodity hardware with locally attached hard drives and enables the unification of storage into a cluster in scenarios like virtualization with the help of virtual machines (VMs) and/or Containers (CTs). Pstorage ensures a fast live migration of VMs and CTs across hardware nodes, without the need to copy VM/CT data, and high availability as the storage becomes available remotely.
The main Pstorage features are listed below:
• No special hardware requirements. Commodity hardware (SATA/SAS drives, 1Gbit+ Ethernet) can be used to create a storage.
• Strong consistency semantics. This makes Pstorage suitable for VMs and CTs running on top of it (unlike object storage such as Amazon S3 or Swift).
• Built−in replication.
• Automatic disaster recovery on hard drive or node failures.
• High availability. Data remain accessible even in case of hard drive or node failures.
• Optional SSD caching. SSD caches boost the overall performance in the cluster on write and read operations.
• Data checksumming and scrubbing. Checksumming and scrubbing greatly enhance data reliability.
• Grow on demand. More storage nodes can be added to the cluster to increase its disk space. A VM/CT image size is not limited by the size of any of the hard drives.
Pstorage
components
Any Pstorage includes three components:
• Metadata server (MDS). MDSs manage metadata, like file names, and keep control over how files are split into chunks and where the chunks are stored. They also track versions of chunks and ensure that the cluster has enough replicas. An MDS can be run in multiple instances to provide high availability. Besides, MDSs keep a global log of important events that happen in the cluster.
• Chunk server (CS). A CS is a service responsible for storing real user data chunks and providing access to these data. A Pstorage cluster must have multiple instances of CSs for high availability.
• Clients. Clients access a Pstorage cluster by communicating with MDSs and CSs. Parallels Containers and virtual machines can be run natively, i.e. directly from the Pstorage cluster. An additional Pstorage client − can be used to mount Pstorage as a conventional file system (though Pstorage is not POSIX−compliant). Besides, Pstorage files can be mounted as a block device using the "ploop" feature and formatted as ext4 file system for other needs.
A recommended cluster setup includes from 3 to 5 MDS instances (allowing you to survive the loss of 1 or 2 of MDSs, respectively) and multiple CSs providing storage capacity.
Password−Based
Authentication
Pstorage uses password−based authentication to enhance
security in clusters. Once the authentication is enabled,
you can add a new server to the cluster only after the
server passes the authentication phase. The
password−based authentication works as follows:
1. You set the authentication password when you create the first MDS server in the cluster. The password you specify is encrypted and saved into the /etc/pstorage/clusters/CLUSTER_NAME/auth_digest.key file on the server.
2. You add new MDS servers, chunk servers, or clients to the cluster and use the pstorage auth−node command to authenticate them. During authentication, you use the password you set when creating the first MDS server.
3. Pstorage compares the specified password with the one stored on the first MDS server, and if the passwords match, successfully authenticates the server.
For each physical server, authentication is a one−time process. Once a server is authenticated in the cluster (e.g., when you configure it as an MDS server), the /etc/pstorage/clusters/CLUSTER_NAME/auth_digest.key file is created on the authenticated server. When you set up this server as another cluster component (e.g., as a chunk server), the cluster checks that the auth_digest.key file is present and does not require you to authenticate the server again.
To change the current password for an existing cluster, execute the following command:
# pstorage −c CLUSTER_NAME auth−node −−change−password
This command asks for a new password (or reads the password from stdin if the −P option is specified) and recreates the /etc/pstorage/clusters/CLUSTER_NAME/auth_digest.key file.
Pstorage
setup
Select a cluster name
Every Pstorage cluster is identified by 2 IDs: a unique user−visible cluster name and an auto−generated and hidden cluster ID. The latter prevents possible conflicts when two Pstorage clusters have the same cluster name. A cluster name is used by Pstorage tools to reference the cluster and may contain only the characters a−z, A−Z, 0−9, underscore (_), and dash (−). Inside the cluster, the name is resolved to a set of MDS IP addresses Pstorage tools can communicate with. This process is called "discovery". You can use three ways to discover a Pstorage cluster: DNS records (recommended), automatic network zeroconf discovery (via multicasts), and manual bootstrap lists.
Set up Metadata services
An MDS is a critical component of any Pstorage cluster, and setting up an MDS is the first step in creating the cluster. Below is an instruction on setting up N instances of MDSs in a Pstorage cluster.
(1.1) Create the first MDS in the cluster on the host with the IP address of ADDR1 (it is recommended to use a separate storage backend network for the Pstorage):
# pstorage −c CLUSTER_NAME make−mds −I −p −a ADDR1[:PORT1] −r JOURNAL_DIR
where JOURNAL_DIR is the path to the MDS storage repository.
(1.2) Start the MDS:
# service pstorage−mdsd start
(2.1) Add more MDSs running on other nodes to the pstorage cluster:
Do authenticate new node in the cluster first:
# pstorage −c CLUSTER_NAME auth−node
# pstorage −c CLUSTER_NAME make−mds −a ADDRN[:PORTN] −r JOURNAL_DIR
Note that the commands above may fail if the cluster with the name CLUSTER_NAME is not available or not running, or the cluster name discovery does not work. In the latter case, use the −b option to specify the IP address of a running MDS.
(2.2) Start the service:
# service pstorage−mdsd start
The example below shows how to set up three MDS instances for the "testcl" pstorage cluster:
host1:~#
pstorage −c testcl make−mds −I −p
−a 192.168.0.1 −r /var/lib/pstorage/mds
host1:~# service pstorage−mdsd start
host2:~#
pstorage −c testcl auth−node
host2:~# pstorage −c testcl make−mds −a
192.168.0.2 −r /var/lib/pstorage/mds
host2:~# service pstorage−mdsd start
host3:~#
pstorage −c testcl auth−node
host3:~# pstorage −c testcl make−mds −a
192.168.0.3 −r /var/lib/pstorage/mds
host3:~# service pstorage−mdsd start
Set up CSs on storage nodes
For a Pstorage cluster to work, it must have CS services running and providing local disk capacity for the global cluster pool. It is strongly recommended to set the replication factor to 3 (default value is 1) and run the same number of CSs to store data. If the number of CSs is less than the replication factor, the cluster will not be operational: the process either stalls (until enough CSs are added to the cluster) or the ENOSPC error is displayed. You can increase the replication factor using the pstorage set−attr −p −R / replicas=norm:min command.
To set up a CS repository and start the service, execute these commands:
# pstorage −c CLUSTER_NAME make−cs −r STORAGE_DIR [−b BOOTSTRAP_LIST]
# service pstorage−csd start
As in the case with MDSs, a bootstap list is required if the MDS auto−discovery by a cluster name does not work. The bootstrap list helps a CS find the cluster on its very first start.
The example below shows how to add three CS nodes to the "testcl" cluster:
host1:~#
pstorage −c testcl make−cs −r
/var/lib/pstorage/cs
host1:~# service pstorage−csd start
host2:~#
pstorage −c testcl make−cs −r
/var/lib/pstorage/cs
host2:~# service pstorage−csd start
host3:~#
pstorage −c testcl make−cs −r
/var/lib/pstorage/cs
host3:~# service pstorage−csd start
Setup MDS discovery
See pstorage−discovery(7).
Getting access to a Pstorage
To get access to a Pstorage, mount it as a file system:
# mkdir −p MOUNTPOINT
# pstorage−mount −c CLUSTER_NAME MOUNTPOINT
Now you can check that data are accessible from multiple hosts:
host1:~# date > MOUNTPOINT/test
host2:~# cat MOUNTPOINT/test
If Parallels Virtuozzo Server was installed without Pstorage clustering features, you may need to configure the following settings to allow Containers and virtual machines to run on top of a Pstorage:
(1) Edit the /etc/vz/vz.conf file as follows:
VEFSTYPE=ext4
VELAYOUT=ploop
VE_PRIVATE=/MOUNT_POINT/VEID
(2) Set the default location for storing virtual machines:
prlsrvctl user set −−def−vm−home MOUNT_POINT
Copyright © 2011−2013, Parallels, Inc. All rights reserved.
pstorage(1), pstorage−make−cs(1), pstorage−make−mds(1), pstorage−discovery(7), pstorage−iscsi(7), csd(1), mdsd(1), pstorage−mount(1), pstorage−stat(1), pstorage−ssd(7), pstorage−config−files(7), pstorage−eventlog(1), pstorage−hwflush−check(1)