davenchy / alx-files_manager

ALX Files Manager Team Project

Files Manager - ALX Backend Team Project

Overview

This project is a summary of the ALX back-end trimester, covering essential concepts like authentication, NodeJS, MongoDB, Redis, pagination, and background processing. The objective is to build a simple platform to upload and view files with the following features:

• User authentication via a token

• List all files

• Upload a new file

• Change permission of a file

• View a file

• Generate thumbnails for images

This project is designed for learning purposes to assemble each piece and build a full product.

Table of Contents

• Features
• Concepts Covered
• Project Structure
• Setup
• Usage
• API Documentation
• Technologies
• License
• Authors

Features

1. User Authentication: Users can authenticate via tokens.

2. File Management:

   • List all files

   • Upload new files

   • Change file permissions

   • View files

   • Generate thumbnails for images

Concepts Covered

• Creating an API with Express: Learn how to set up and manage routes, handle requests and responses, and create a robust API using Express.js.

• User Authentication: Implement token-based authentication to secure the platform.

• Data Storage in MongoDB: Store and manage file data in MongoDB.

• Temporary Data Storage in Redis: Use Redis for caching and temporary data storage.

• Background Processing: Set up and use a background worker for tasks like generating thumbnails for images.

Project Structure

This is an initial template, probably will be changed later.

├── src
│   ├── controllers
│   ├── models
│   ├── routes
│   ├── services
│   ├── workers
│   ├── app.js
│   └── server.js
├── .env.example
├── .gitignore
├── package.json
├── README.md
└── tests

• src/controllers: Contains the controller logic for handling requests.
• src/models: Contains Mongoose schemas for MongoDB.
• src/routes: Defines the API endpoints.
• src/services: Contains business logic and utilities.
• src/workers: Contains background worker scripts.
• src/app.js: Main application setup.
• src/server.js: Server configuration and startup.
• .env.example: Example environment variables file.
• tests: Contains test cases for the application.

Setup

Prerequisites

• Node.js
• MongoDB
• Redis

Installation

1. Clone the repository:

git clone --depth=1 https://github.com/Davenchy/alx-files_manager.git
cd alx-files_manager

2. Install dependencies:

npm install
# or using bun node package manager
bun install

3. Set up environment variables:

Rename the .env.example file to .env and update the values accordingly.

4. Start MongoDB and Redis servers:

Make sure MongoDB and Redis are running on your system.

5. Start the application:

npm start

Usage

Running the Server

To start the server, use:

npm start

The server will run on http://localhost:3000 by default.

Running Tests

To run the tests, use:

npm test

API Documentation

Endpoints

• POST /api/register: Register a new user
• POST /api/login: Authenticate a user
• GET /api/data: Retrieve data from MongoDB
• POST /api/data: Store data in MongoDB
• GET /api/temp-data: Retrieve temporary data from Redis
• POST /api/temp-data: Store temporary data in Redis

Technologies

• Express.js: Web framework for Node.js
• MongoDB: NoSQL database for data storage
• Redis: In-memory data structure store for temporary data
• Node.js: JavaScript runtime environment
• Mongoose: ODM for MongoDB
• Bull: Redis-based queue for background jobs

License

This project is licensed under the MIT License. See the LICENSE file for details.

Authors

• Fadi Asaad (Davenchy): Github LinkedIn

• Mostafa Elmasry: Github LinkedIn

---

Feel free to reach out if you have any questions or need further assistance. Happy coding!

alx-files_manager's Issues

Add Feature `Image Thumbnails` - Task 9

• Update the endpoint POST /files endpoint to start a background processing for generating thumbnails for a file of type image:

  • Create a Bull queue fileQueue
  • When a new image is stored (in local and in DB), add a job to this queue with the userId and fileId

• Create a file worker.js:

  • By using the module Bull, create a queue fileQueue
  • Process this queue:
    • If fileId is not present in the job, raise an error Missing fileId
    • If userId is not present in the job, raise an error Missing userId
    • If no document is found in DB based on the fileId and userId, raise an error File not found
    • By using the module image-thumbnail, generate 3 thumbnails with width = 500, 250 and 100 - store each result on the same location of the original file by appending _<width size>

• Update the endpoint GET /files/:id/data to accept a query parameter size:

  • size can be 500, 250 or 100
  • Based on size, return the correct local file
  • If the local file doesn’t exist, return an error Not found with a status code 404

Required Paths: utils/, controllers/FilesController.js, worker.js.

A few test cases of task `2. First API` are not passing

A single requirement test case.
3 Incorrect output test cases.

I suggest renaming the functions status and stats to getStatus and getStats as described in the task description, probably this will solve the problem.

alx-files_manager/controllers/AppController.js

Line 4 in dfbb18d

function status(request, response) {

alx-files_manager/controllers/AppController.js

Line 13 in dfbb18d

async function stats(request, response) {

Task 4 - Fails

• Invalid base64 sent by client at GET /connect endpoint.

The default branch has been renamed from `dev` to `main`

Due to the ALX checker’s requirements, we no longer need the current main branch as a stable branch or the dev branch as a nightly branch. Here’s what has been done:

• The old main branch has been deleted. This change will remain until the end of the ALX project, after which we can decide on future development.
• The dev branch has been renamed to main to align with the traditional default branch naming.

Please update your local repositories with the following steps:

1. Switch to the dev branch:

   git switch dev

2. Delete the current local main branch:

   git branch -D main

3. Rename the dev branch to main:

   git branch -m main

4. Fetch the latest changes from the remote and remove any obsolete branches:

   git fetch origin --prune

5. Set the local main branch to track the remote main branch:

   git branch -u origin/main main

6. Update the remote HEAD reference to point to the new default branch:

   git remote set-head origin -a

---

This guide ensures your local repository is in sync with the updated branch structure.

Add feature `File Upload` - Task 5

Define POST /files endpoint => FilesController.postUpload.

POST /files should create a new file in DB and in disk:

• Waiting for #29

• Waiting for #31

• Retrieve the user based on the token:
  If not found, return an error Unauthorized with a status code 401.

• To create a file, you must specify:
  - name: as filename
  - type: either folder, file or image
  - parentId: (optional) as ID of the parent (default: 0 -> the root)
  - isPublic: (optional) as boolean to define if the file is public or not (default: false)
  - data: (only for type=file|image) as Base64 of the file content

• If the name is missing, return an error Missing name with a status code 400

• If the type is missing or not part of the list of accepted type, return an error Missing type with a status code 400

• If the data is missing and type != folder, return an error Missing data with a status code 400

• If the parentId is set:

  • If no file is present in DB for this parentId, return an error Parent not found with a status code 400
  • If the file present in DB for this parentId is not of type folder, return an error Parent is not a folder with a status code 400

• The user ID should be added to the document saved in DB - as owner of a file

• If the type is folder, add the new file document in the DB and return the new file with a status code 201

• If the type is not a folder:

  • All file will be stored locally in a folder (to create automatically if not present):
    • The relative path of this folder is given by the environment variable FOLDER_PATH
    • If this variable is not present or empty, use /tmp/files_manager as storing folder path
  • Create a local path in the storing folder with filename a UUID
  • Store the file in clear (reminder: data contains the Base64 of the file) in this local path
  • Add the new file document in the collection files with these attributes:
    • userId: ID of the owner document (owner from the authentication)
    • name: same as the value received
    • type: same as the value received
    • isPublic: same as the value received
    • parentId: same as the value received - if not present: 0
    • localPath: for a type=file|image, the absolute path to the file save in local
  • Return the new file with a status code 201

Required Paths: utils/, routes/index.js, controllers/FilesController.js

Feature: New user welcoming

• Update the POST /users endpoint to start a background processing for sending a “Welcome email” to the user:

  • Create a Bull queue userQueue
  • When a new user is stored (in DB), add a job to this queue with the userId

• Update the file worker.js:

  • By using the module Bull, create a queue userQueue
  • Process this queue:
    • If userId is not present in the job, raise an error Missing userId
    • If no document is found in DB based on the userId, raise an error User not found
    • Print in the console Welcome <email>!

1. MongoDB utils

Add the DBClient class in utils/db.js.

• the constructor that creates a client to MongoDB:

  • host: from the environment variable DB_HOST or default: localhost
  • port: from the environment variable DB_PORT or default: 27017
  • database: from the environment variable DB_DATABASE or default: files_manager

• a function isAlive that returns true when the connection to MongoDB is a success otherwise, false

• an asynchronous function nbUsers that returns the number of documents in the collection users

• an asynchronous function nbFiles that returns the number of documents in the collection files

Thumbnails are not generated

The files queue used to generate thumbnails has different names between the server and the worker processes.
Also, the stream used to write the generated thumbnails to the disk is incorrect, I should use stream.Readable instead.

• Update the queue name and use the utils/constants.js
• Use stream.Readable to write thumbnails to disk.

Add tasks description and information about the project

Suggestion

I suggest adding detailed descriptions of the project's tasks provided by ALX, either in a separate Markdown file or within the existing README.md. I see this addition will enhance the clarity of the tasks completed within this repository, helps viewers in understanding the purpose and scope of each task and recognizing the contributions of team members more clearly.

Proposed Structure

Table of Contents

• Task 1: Name (Contributor Name)
• Task 2: Name (Contributor Name)
• Task 3: Name (Contributor Name)

Task 1: Name (Contributor Name)

Detailed description of Task 1 provided by ALX.

Task 2: Name (Contributor Name)

Detailed description of Task 2 provided by ALX.

Task 3: Name (Contributor Name)

Detailed description of Task 3 provided by ALX.

ALX Checker can't see our dev branch

The checker have been released and only the first test case passes out of 9.
The problem is related to requirements, any ideas?

Task 3 test cases has some fails

A single requirement test case and 3 incorrect output test cases.

Probably #18 has a clue.

Task 8 Fail

Looks like task 8 is not passing because of the AuthGuard and FilesLoader middlewares.

I will remove the AuthGuard then I will tweak the FilesLoader to work with or without user authentication.

ALX Tasks checker fails

• Tasks to resolve:
  • Task 4
  • Task 5
  • Task 8

Feature `File publish/unpublish` - Task 7

Add new 2 endpoints under FilesController provided by #30:

• Waiting for #30

• PUT /files/:id/publish => FilesController.putPublish

• PUT /files/:id/publish => FilesController.putUnpublish

• PUT /files/:id/publish should set isPublic to true on the file document based on the ID:

  • Retrieve the user based on the token:
    • If not found, return an error Unauthorized with a status code 401
  • If no file document is linked to the user and the ID passed as parameter, return an error Not found with a status code 404
  • Otherwise:
    • Update the value of isPublic to true
    • And return the file document with a status code 200

• PUT /files/:id/unpublish should set isPublic to false on the file document based on the ID:

  • Retrieve the user based on the token:
    • If not found, return an error Unauthorized with a status code 401
  • If no file document is linked to the user and the ID passed as parameter, return an error Not found with a status code 404
  • Otherwise:
    • Update the value of isPublic to false
    • And return the file document with a status code 200

Required Paths: utils/, routes/index.js, controllers/FilesController.js

2. First API - No default port

At task 2. First API in server.js the server should listen on the port provided by the env var PORT or 5000 if prevented.

alx-files_manager/server.js

Line 5 in be98cee

const port = process.env.PORT;

I suggest the following:

 const PORT = process.env.PORT || 5000; 

3. Create a new user

Add users to database

• define a new endpoint route in routes/index.js:

  • POST /users => UsersController.postNew

• define the controllers/UsersController.js that contains the endpoint

• To create a user, you must specify an email and a password

• If the email is missing, return an error Missing email with a status code 400

• If the password is missing, return an error Missing password with a status code 400

• If the email already exists in DB, return an error Already exist with a status code 400

• The password must be stored after being hashed in SHA1

• The endpoint is returning the new user with only the email and the id (auto generated by MongoDB) with a status code 201

• The new user must be saved in the collection users:

  • email: same as the value received
  • password: SHA1 value of the value received

• Waiting for #12

• Waiting for #16