MinIO again

Now that I do have MinIO running a few days it’s time to check again…

Status

Took a while to upload my photos though finally managed to finish that. Both minio instances have been running for about 2 weeks (I restarted one today)

●  minio1.jeanbruenn.info:9000
   Status : online
   Uptime : 56 minutes 
  Version : 2019-06-19T18:24:42Z
  Storage : Used 89 GiB
   Drives : 2/2 OK
 
●  minio2.jeanbruenn.info:9000
   Status : online
   Uptime : 2 weeks 
  Version : 2019-06-19T18:24:42Z
  Storage : Used 89 GiB
   Drives : 2/2 OK

Synchronization / A few words on mc mirror

You might remember that I had some issues (no program bugs – just a usability / understanding thingy) with mc mirror in my last article and hence I did use plain rsync first. Today I’ve stumbled upon rclone. There is a nice article about rclone with minio and it does exactly what I wish mc mirror would have done.

I suggest using –size-only because without that switch rclone is super slow (ETA 16h – with –size-only just 1,5h). I also suggest using -v if you don’t want to use dry-run first. 🙂

2019/07/13 20:19:57 INFO  : import/10090706/_DSC3433.ARW: Copied (new)
2019/07/13 20:20:25 INFO  : 
Transferred:   	    2.389G / 7.650 GBytes, 31%, 993.982 kBytes/s, ETA 1h32m30s
Errors:                 0
Checks:              5808 / 5808, 100%
Transferred:           84 / 417, 20%
Elapsed time:       42m0s
Transferring:
 *                  import/10090706/_DSC3434.ARW: 100% /23.938M, 20.288k/s, 0s
 *                  import/10090706/_DSC3435.ARW: 100% /23.781M, 20.155k/s, 0s
 *                  import/10090706/_DSC3436.ARW: 100% /23.938M, 46.945k/s, 0s
 *                  import/10090706/_DSC3437.ARW: 100% /23.812M, 749.203k/s, 0s

Now that I do have a way to easily synchronize my local photos in a rsync like fashion to minio I’m pretty happy.

NGINX Stream Loadbalancing

When I first worked with MinIO in PHP I used FlySystem with a S3-driver and wondered how I might tell it to use both MinIO instances. Being an NGINX-Fanboy I made some tests:

Using minio1 to print all filenames using FlySystem

  1. 1.6490080356598s
  2. 2.1601350307465s
  3. 1.8425848484039s
  4. 1.7543859481812s
  5. 1.7964880466461s

Stripped highest and lowest, Average of the remaining: 1,79s

Using minio2 to print all filenames using FlySystem

  1. 2.6141030788422s
  2. 1.8565142154694s
  3. 1.6802151203156s
  4. 1.8362967967987s
  5. 1.8700098991394s

Stripped highest and lowest, Average of the remaining: 1,85s

Using minio1+minio2 through NGINX to print all filenames using FlySystem

  1. 2.851634979248s
  2. 1.4206428527832s
  3. 1.5750160217285s
  4. 1.5091438293457s
  5. 1.5324211120605s

Stripped highest and lowest, Average of the remaining: 1,53s

Conclusion

At a first glance just a micro-optimization. Might turn out to be helpful in large / more complex setups if you need to scale great again (couldn’t resist). Sometimes I get values around 1,3s with the NGINX variant.

Oh, if you’re interested in how I did set that up, a few notes beforehand:

  • You’ll have to disable peer certificate verification for it to work. That HAS security implications hence don’t do that
  • I’m only using this for reads – I have no idea about the reliability nor do I have any idea how this works with writes

Still interested? Here you go.

stream {
  upstream minio_backend {
    least_conn;
    server minio1.jeanbruenn.info:9000;
    server minio2.jeanbruenn.info:9000;
  }
 
  server {
    listen 9000;
    proxy_pass minio_backend;
  }
}

The S3-Client/Adapter/Flysystem will moan about certificate issues, you might disable verification of that in the s3 client configuration. Mind that this has security implications so use that only if you know what you’re doing.

'http'    => [
    'verify' => false
],

NGINX HTTP Loadbalancing

Because MinIO uses HTTP you do not need to use a stream configuration like I did above, you may as well just use normal HTTP Loadbalancing which offers a lot more features like caching. More about that in another Post.

Accessing MinIO with PHP FlySystem

First of all I did install flysystem as well as its S3-adapter and its cache:

./composer.phar require league/flysystem
./composer.phar require league/flysystem-aws-s3-v3 
./composer.phar require league/flysystem-cached-adapter

In case you do not have your own autoloader which does that for you, require the vendor/autoload.php:

require_once 'vendor/autoload.php';

A few use statements:

use Aws\S3\S3Client;
use League\Flysystem\AwsS3v3\AwsS3Adapter;
use League\Flysystem\Cached\CachedAdapter;
use League\Flysystem\Cached\Storage\Memory as MemoryStore;
use League\Flysystem\Filesystem;

And a very simple php snippet which I used to measure the time above:

$client = new S3Client($config['minio'][2]);
 
$s3adapter = new AwsS3Adapter($client, 'photos', '', [
    'StorageClass' => 'REDUCED_REDUNDANCY',
]);
$cacheStore = new MemoryStore();
$adapter = new CachedAdapter($s3adapter, $cacheStore);
 
$filesystem = new Filesystem($adapter);
 
$start = microtime(true);
$contents = $filesystem->listContents('/', true);
foreach ($contents as $object) {
    echo $object['basename'] . ' is a ' . $object['type']."\n";
}
$end = microtime(true);
$diff = $end - $start;
echo "\n".$diff.'s';

$config['minio'][0] and $config['minio'][1] contain the configuration to access my minio1 and minio2. $config['minio'][2] contains the configuration to access my nginx-stream-loadbalancer to both minios. The configuration is pretty simple, you may lookup all relevant settings in the Configuration for the AWS SDK for PHP Version 3 document. To access my minio1 I do use:

'endpoint' => 'https://minio1.jeanbruenn.info:9000',
'credentials' => [
    'key' => 'xx',
    'secret' => 'xx',
],
'region' => 'de-DE',
'version' => 'latest',
'use_path_style_endpoint' => true

For my nginx variant I do use:

'endpoint' => 'https://photos.jeanbruenn.info:9000',
'credentials' => [
    'key' => 'xx',
    'secret' => 'xx',
],
'http'    => [
    'verify' => false
],
'region' => 'de-DE',
'version' => 'latest',
'use_path_style_endpoint' => true

No Comments

Post a Comment