Coder Social home page Coder Social logo

yii2tech / file-storage Goto Github PK

View Code? Open in Web Editor NEW
119.0 14.0 30.0 96 KB

This package has been abandoned. See: https://www.patreon.com/posts/59332556

License: Other

PHP 100.00%
yii yii2 yii2-extension file files storage filesystem amazon sftp gridfs mongodb bucket

file-storage's Introduction

File Storage Extension for Yii 2


This extension provides file storage abstraction layer for Yii2.

For license information check the LICENSE-file.

Latest Stable Version Total Downloads Build Status

Installation

The preferred way to install this extension is through composer.

Either run

php composer.phar require --prefer-dist yii2tech/file-storage

or add

"yii2tech/file-storage": "*"

to the require section of your composer.json.

If you wish to use Amazon S3 storage, you should also install aws/aws-sdk-php version 2. Either run

php composer.phar require --prefer-dist aws/aws-sdk-php:~2.0

or add

"aws/aws-sdk-php": "~2.0"

to the require section of your composer.json.

If you wish to use MongoDB storage, you should also install yiisoft/yii2-mongodb version 2.1. Either run

php composer.phar require --prefer-dist yiisoft/yii2-mongodb:~2.1.0

or add

"yiisoft/yii2-mongodb": "2.1.0"

to the require section of your composer.json.

Usage

This extension provides file storage abstraction layer for Yii2. This abstraction introduces 2 main terms: 'storage' and 'bucket'. 'Storage' - is a unit, which is able to store files using some particular approach. 'Bucket' - is a logical part of the storage, which has own specific attributes and serves some logical mean. Each time you need to read/write a file you should do it via particular bucket, which is always belongs to the file storage.

Example application configuration:

return [
    'components' => [
        'fileStorage' => [
            'class' => 'yii2tech\filestorage\local\Storage',
            'basePath' => '@webroot/files',
            'baseUrl' => '@web/files',
            'dirPermission' => 0775,
            'filePermission' => 0755,
            'buckets' => [
                'tempFiles' => [
                    'baseSubPath' => 'temp',
                    'fileSubDirTemplate' => '{^name}/{^^name}',
                ],
                'imageFiles' => [
                    'baseSubPath' => 'image',
                    'fileSubDirTemplate' => '{ext}/{^name}/{^^name}',
                ],
            ]
        ],
        // ...
    ],
    // ...
];

Example usage:

$bucket = Yii::$app->fileStorage->getBucket('tempFiles');

$bucket->saveFileContent('foo.txt', 'Foo content'); // create file with content
$bucket->deleteFile('foo.txt'); // deletes file from bucket
$bucket->copyFileIn('/path/to/source/file.txt', 'file.txt'); // copy file into the bucket
$bucket->copyFileOut('file.txt', '/path/to/destination/file.txt'); // copy file from the bucket
var_dump($bucket->fileExists('file.txt')); // outputs `true`
echo $bucket->getFileUrl('file.txt'); // outputs: 'http://domain.com/files/f/i/file.txt'

Following file storages are available with this extension:

  • [[\yii2tech\filestorage\local\Storage]] - stores files on the OS local file system.
  • [[\yii2tech\filestorage\sftp\Storage]] - stores files using SSH2 SFTP.
  • [[\yii2tech\filestorage\amazon\Storage]] - stores files using Amazon simple storage service (S3).
  • [[\yii2tech\filestorage\mongodb\Storage]] - stores files using MongoDB GridFS.
  • [[\yii2tech\filestorage\hub\Storage]] - allows combination of different file storages.

Please refer to the particular storage class for more details.

Heads up! Some of the storages may require additional libraries or PHP extensions, which are not required with this package by default, to be installed. Please check particular storage class documentation for the details.

Abstraction usage

Each provided storage implements same interface for the files processing. Thus each storage can substitute another one, unless program code follows this interface. This allows you to switch between different storages without being need to adjust program source code. For example, at production server you may need to use SFTP for files storing and your application config looks like following:

return [
    'components' => [
        'fileStorage' => [
            'class' => 'yii2tech\filestorage\sftp\Storage',
            'ssh' => [
                'host' => 'file.server.com',
                'username' => 'user',
                'password' => 'some-password',
            ],
            'basePath' => '/var/www/html/files',
            'baseUrl' => 'http://file.server.com/files',
            'buckets' => [
                'temp',
                'item',
            ]
        ],
        // ...
    ],
    // ...
];

However, at development environment you may use simple local file storage instead:

return [
    'components' => [
        'fileStorage' => [
            'class' => 'yii2tech\filestorage\local\Storage',
            'basePath' => '@webroot/files',
            'baseUrl' => '@web/files',
            'buckets' => [
                'temp',
                'item',
            ]
        ],
        // ...
    ],
    // ...
];

You can also combine several different storages using [[\yii2tech\filestorage\hub\Storage]], if necessary:

return [
    'components' => [
        'fileStorage' => [
            'class' => 'yii2tech\filestorage\hub\Storage',
            'storages' => [
                [
                    'class' => 'yii2tech\filestorage\sftp\Storage',
                    'ssh' => [
                        'host' => 'file.server.com',
                        'username' => 'user',
                        'password' => 'some-password',
                    ],
                    'basePath' => '/var/www/html/files',
                    'baseUrl' => 'http://file.server.com/files',
                    'buckets' => [
                        'item',
                    ]
                ],
                [
                    'class' => 'yii2tech\filestorage\local\Storage',
                    'basePath' => '@webroot/files',
                    'baseUrl' => '@web/files',
                    'buckets' => [
                        'temp',
                    ]
                ]
            ],
        ],
        // ...
    ],
    // ...
];

Accessing files by URL

Almost all file storage implementation, implemented in this extension, provide mechanism for accessing stored files via web URL. Actual mechanism implementation may vary depending on particular storage. For example: [[\yii2tech\filestorage\local\Storage]] allows setup of the URL leading to its root folder, creating URL for particular file appending its name to base URL, while [[\yii2tech\filestorage\amazon\Storage]] uses S3 built-in object URL composition.

In order to get URL leading to the stored file, you should use [[\yii2tech\filestorage\BucketInterface::getFileUrl()]] method:

$bucket = Yii::$app->fileStorage->getBucket('tempFiles');

$fileUrl = $bucket->getFileUrl('image.jpg');

In case particular storage does not provide native URL file access, or it is not available or not desirable by some reason, you can setup composition of the file URL via Yii URL route mechanism. You need to setup baseUrl to be an array, containing route, which leads to the Yii controller action, which will return the file content. For example:

return [
    'components' => [
        'fileStorage' => [
            'class' => 'yii2tech\filestorage\local\Storage',
            'baseUrl' => ['/file/download'],
            // ...
        ],
        // ...
    ],
    // ...
];

With this configuration getFileUrl() method will use current application URL manager to create URL. Doing so it will add bucket name as bucket parameter and file name as filename parameter. For example:

use yii\helpers\Url;

$bucket = Yii::$app->fileStorage->getBucket('images');

$fileUrl = $bucket->getFileUrl('logo.png');
$manualUrl = Url::to(['/file/download', 'bucket' => 'images', 'filename' => 'logo.png']);
var_dump($fileUrl === $manualUrl); // outputs `true`

You may setup [[\yii2tech\filestorage\DownloadAction]] to handle file content web access. For example:

class FileController extends \yii\web\Controller
{
    public function actions()
    {
        return [
            'download' => [
                'class' => 'yii2tech\filestorage\DownloadAction',
            ],
        ];
    }
}

Tip: usage of the controller action for the file web access usually slower then native mechanism provided by file storage, however, you may put some extra logic into it, like allowing file access for logged in users only.

Processing of the large files

Saving or reading large files, like > 100 MB, using such methods like [[\yii2tech\filestorage\BucketInterface::saveFileContent()]] or [[\yii2tech\filestorage\BucketInterface::getFileContent()]], may easily exceed PHP memory limit, breaking the script. You should use [[\yii2tech\filestorage\BucketInterface::openFile()]] method to create a file resource similar to the one created via [[fopen()]] PHP function. Such resource can be read or written by blocks, keeping memory usage low. For example:

$bucket = Yii::$app->fileStorage->getBucket('tempFiles');

$resource = $bucket->openFile('new_file.dat', 'w');
fwrite($resource, 'content part1');
fwrite($resource, 'content part2');
// ...
fclose($resource);

$resource = $bucket->openFile('existing_file.dat', 'r');
while (!feof($resource)) {
    echo fread($resource, 1024);
}
fclose($resource);

Note: You should prefer usage of simple modes like r and w, avoiding complex ones like w+, as they may be not supported by some storages.

Logging

Each file operation performed by file storage component is logged. In order to setup a log target, which can capture all entries related to file storage, you should use category yii2tech\filestorage\*. For example:

return [
    // ...
    'components' => [
        // ...
        'log' => [
            // ...
            'targets' => [
                [
                    'class' => 'yii\log\FileTarget',
                    'logFile' => '@runtime/logs/file-storage.log',
                    'categories' => ['yii2tech\filestorage\*'],
                ],
                // ...
            ],
        ],
    ],
];

file-storage's People

Contributors

klimov-paul avatar vuongxuongminh avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

file-storage's Issues

aws version 3

Not work with aws-sdk-php ver 3

Class 'Aws\S3\Enum\CannedAcl' not found

Make this package slim

I and maybe anyone else people dont use all the storages provided here. I think storage extensions such mongodb, aws, hub, sftp should moved to its own repo, leave just the core and local storage extension instead. In fact I done the job for myself uses only and also created separated extension for aliyun oss

add minio

What steps will reproduce the problem?

What is the expected result?

What do you get instead?

Additional info

Q A
This Package Version 1.?.?
Yii Framework Version 2.0.?
File Storage Type local/sftp/amazon/mongodb
PHP version
Operating system

Suggestion for DownloadAction

Why we don't use openFile in bucket to get handler and pass it to sendStreamAsFile method in response for send file? It will flexible more option when download file we can detect send file inline or not and resolve a large file with content-range.

Support of custom bucket name

Hi, I have a need to split the bucket into subfolders, with different purposes, for match local directories.

To solve this problem, I see a good option to add support for the name field in the bucket configuration.

Or maybe there is a ready-made solution?

Example of needle config:

    'storages' => [
        's3' => [
            'class' => 'yii2tech\filestorage\amazon\Storage',
            'buckets' => [
                'bucket_a' => [
                    'name' => 'aws_bucket_name',
                    'urlName' => 'aws_bucket_name/folder',
                ],
                'bucket_b' => [
                    'name' => 'aws_bucket_name',
                    'urlName' => 'aws_bucket_name/other_folder',
                ],
            ],
        ],
        'local' => [
            'class' => 'yii2tech\filestorage\local\Storage',
            'basePath' => Yii::getAlias('@storage'),
            'filePermission' => 0777,
            'buckets' => [
                'bucket_a' => [
                    'baseSubPath' => 'folder',
                ],
                'bucket_b' => [
                    'baseSubPath' => 'other_folder',
                ],
            ],
        ],
    ],

Solution:
\yii2tech\filestorage\BaseStorage::getBucket

    public function getBucket($bucketName)
    {
        if (!array_key_exists($bucketName, $this->_buckets)) {
            throw new InvalidParamException("Bucket named '{$bucketName}' does not exists in the file storage '" . get_class($this) . "'");
        }
        $bucketData = $this->_buckets[$bucketName];
        if (is_object($bucketData)) {
            $bucketInstance = $bucketData;
        } else {
            $bucketData['name'] = isset($bucketData['name']) ? $bucketData['name'] : $bucketName;
            $bucketInstance = $this->createBucketInstance($bucketData);
            $this->_buckets[$bucketName] = $bucketInstance;
        }
        return $bucketInstance;
    }

What can you say about this?

A few questions on the use

Unfortunately, you do not have a detailed wiki, and on the README not everything is clear.

  1. By default, yii2 uses the yii\web\UploadedFile class to load files. How do I use it with this library? For example, it has a saveAs method. It already implements the storage logic. And of course he does not use this library. Should I not use it at all?

  2. There is still a method yii2tech\filestorage\DownloadAction, but I did not really understand how to use it... There's not a single example. Only in actions one line. And what's inside? And how do I make a form on the front? How to name the field to work? In general, it would be nice to have another widget for the frontend, as for example in this library. In general, for example, there are forms when some more text fields are loaded with the file. So this action does not always work.

  3. It is not shown at all whether it is possible to set a key file when connecting to ssh.

deleteFile at Mongodb not delete all chunks in database

After add file into mongo, and delete it
"files" table will delete that one record
but "chunks" table will remain n-1 chunk records which takes unreleased space.

$deleteCount = $this->getCollection()->remove(['_id' => $document['_id']], ['limit' => 1]);
I try remove ['limit' => 1] from line 187 at Bucket, and it works fine

but I'm not sure if this is right things to do, thanks!

What about different permission for directories and files?

I'm talking about yii2tech\filestorage\local\Storage::$filePermission.
As I can see, this property is used both for directory and file creation. So, if I need 755 for directories, my files will also have 755 and that's why will be executable.
Why can't we set their own permissions?

Unfortunately the best expected name for such property will be exactly filePermission, and dirPermission fits good for directories. It will break compability...

But, perhaps, we can add dirPermission with null by default, and change behavior if it's not null? Also not so good solution, of course...

AWS authorization mechanism not supported

There is the problem with authorizing AWS user on the region using V4 signature.

I'm using this configuration:

'class' => 'yii2tech\filestorage\amazon\Storage',
'awsKey' => '...',
'awsSecretKey' => '...',
'buckets' => [
    'bucket_name' => [
        'region' => Aws\Common\Enum\Region::FRANKFURT, // 'eu-central-1'
        'acl' => 'public',
        'fileSubDirTemplate' => '{ext}/{^name}/{^^name}',
    ],
],

This is the AWS response on authorization:

The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.

Based on the Aws\S3\S3Client::factory() and Aws\S3\S3Client::createSignature() methods it looks like AWS can switch to V4 based on the region provided but it can be done only on the bucket level where it's too late.

Adding region on the storage level works like a charm (yii2tech\filestorage\amazon\Storage::createAmazonS3()).

Packages versions:

  • yii2tech/file-storage: 1.1.1
  • aws/aws-sdk-php: 2.8.31

Cant download file from amazon storage using bucket's fileSubDirTemplate option

What steps will reproduce the problem?

Try to open file 08c699d1d76c228daf06f93f8e04c185.jpg using DownloadAction.

My fileStorage config is:

'fileStorage' => [
            '__class' => \common\components\filestorage\yandex\Storage::class,
            'buckets' => [
                '***' => [
                    'acl' => 'public-read',
                    'fileSubDirTemplate' => '{^name}/{^^name}/{^^^name}',
                ],
            ],
        ],

What is the expected result?

No error.

What do you get instead?

Client error: GET https://****.storage.yandexcloud.net/08c699d1d76c228daf06f93f8e04c185.jpg resulted in a 404 Not Found

Additional info

if i put '0/8/c/08c699d1d76c228daf06f93f8e04c185.jpg', then

if ($this->checkFileExistence && !$bucket->fileExists($filename)) {

tells that file not found, so that filename must be without ^name segments

i think

$streamPath = 's3://' . $this->getUrlName() . '/' . $fileName;

should be
$streamPath = 's3://' . $this->getUrlName() . '/' . $this->getFullFileName($fileName);
anyway it fixes the error

Q A
This Package Version 1.?.?
Yii Framework Version 2.0.?
File Storage Type amazon
PHP version
Operating system

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.