camptocamp / odoo-cloud-platform

C2C Odoo cloud platform addons

Odoo Cloud Addons

Camptocamp odoo addons used on our Cloud Platform.

Introduction

On the platform we want to achieve having:

• No data stored on the local filesystem so we can move an instance between hosts and even have several running front-ends
• Metrics read from the logs or sent to Prometheus to monitor the instances
• Logs sent to ElasticSearch-Kibana structured as JSON for better searching

For the storage, we store all the attachments on an object storage such as S3 or Swift, and we store the werkzeug sessions on Redis.

Setup

Python dependencies

Libraries that must be added in requirements.txt:

redis==2.10.5
python-json-logger==0.1.5
statsd==3.2.1

# For S3 object storage (Exoscale, AWS)
boto==2.42.0

# For Swift object storage (Openstack, OVH)
python-swiftclient==3.4.0
python-keystoneclient==3.13.0

Odoo Startup

The --load option of Odoo must contains the following addons:

• session_redis
• logging_json

Example:

--load=web,web_kanban,session_redis,logging_json

Server Environment

The addon cloud_platform is an addon that we use for 2 things:

• validate that we setup the required environment variables depending on the running environment
• install and configure the cloud addons

For this purpose, we use the server_environment with the following envs:

• prod
• integration
• test
• dev

The exact naming is important, because the cloud_platform addon rely on these keys to know and check the running environment.

Attachments in the Object Storage S3

• prod: stored RW in the object storage
• AWS_HOST: depends of the platform
• AWS_REGION: region's endpoint
• AWS_ACCESS_KEY_ID: depends of the platform
• AWS_SECRET_ACCESS_KEY: depends of the platform
• AWS_BUCKETNAME: <project>-odoo-prod
• integration:
• AWS_ACCESS_KEY_ID: depends of the platform
• AWS_SECRET_ACCESS_KEY: depends of the platform
• AWS_BUCKETNAME: <project>-odoo-integration
• test: attachments are stored in database

Besides, the attachment location should be set to s3 (this is automatically done by the install methods of the cloud_platform module).

• ir.config_parameter ir_attachment.location: s3

Structure of bucket name is checked against environment. It is possible to by-pass this behavior by using the following environment variable: AWS_BUCKETNAME_UNSTRUCTURED.

Attachments in the Object Storage Swift

• prod: stored RW in the object storage
• SWIFT_AUTH_URL: depends of the platform
• SWIFT_ACCOUNT: depends of the platform
• SWIFT_PASSWORD: depends of the platform
• SWIFT_WRITE_CONTAINER: <project>-odoo-prod
• integration:
• SWIFT_AUTH_URL: depends of the platform
• SWIFT_ACCOUNT: depends of the platform
• SWIFT_PASSWORD: depends of the platform
• SWIFT_WRITE_CONTAINER: <project>-odoo-integration
• test: attachments are stored in database

Besides, the attachment location should be set to swift (this is automatically done by the install methods of the cloud_platform module).

• ir.config_parameter ir_attachment.location: swift

Structure of container name is checked against environment. It is possible to by-pass this behavior by using the following environment variable: SWIFT_WRITE_CONTAINER_UNSTRUCTURED.

Sessions in Redis

• prod:
• ODOO_SESSION_REDIS: 1
• ODOO_SESSION_REDIS_HOST: depends of the platform
• ODOO_SESSION_REDIS_PASSWORD: depends of the platform
• ODOO_SESSION_REDIS_PREFIX: <project>-odoo-prod
• integration:
• ODOO_SESSION_REDIS: 1
• ODOO_SESSION_REDIS_HOST: depends of the platform
• ODOO_SESSION_REDIS_PASSWORD: depends of the platform
• ODOO_SESSION_REDIS_PREFIX: <project>-odoo-integration
• test:
• ODOO_SESSION_REDIS: 1
• ODOO_SESSION_REDIS_HOST: depends of the platform
• ODOO_SESSION_REDIS_PASSWORD: depends of the platform
• ODOO_SESSION_REDIS_PREFIX: <project>-odoo-test
• ODOO_SESSION_REDIS_EXPIRATION: 86400 (1 day)

JSON Logging

At least on production and integration, activate:

• ODOO_LOGGING_JSON: 1
• Add logging_json in the server_wide_modules option in the configuration file

Metrics (Statsd/Prometheus for Grafana)

Should be active at least on the production server

• ODOO_STATSD: 1
• STATSD_CUSTOMER: <project>
• STATSD_ENVIRONMENT: set if you want to send metrics for a special environment which does not match with the server_environment
• STATSD_HOST: depends of the platform
• STATSD_PORT: depends of the platform

Automatic Configuration

An automatic configuration can be executed from an anthem song to configure some parameters such as the ir_attachment.location and migrate the existing attachments to the object storage.

From anthem, it can be called like this: ctx.env['cloud.platform'].install()

Startup checks

At loading of the database, the addon will check if the environment variables for Redis and the object storage are set as expected for the loaded environment. It will refuse to start if anything is badly configured.

The checks can be bypassed with the environment variable ODOO_CLOUD_PLATFORM_UNSAFE set to 1.

Attachment storage disability

To prevent object storage to be accessed while failing for any kind of reason set this environment variable DISABLE_ATTACHMENT_STORAGE set to 1.

This part will be replaced when running the oca-gen-addons-table script from OCA/maintainer-tools.

Licenses

This repository is licensed under AGPL-3.0.

However, each module can have a totally different license, as long as they adhere to Camptocamp policy. Consult each module's __manifest__.py file, which contains a license key that explains its license.

---

odoo-cloud-platform's Issues

[RFC] attachment_s3: improvement to use one bucket for one customer (also multi-tenancy use?)

We would like to make some improvements to the module. Any though?

Objetive:

For one customer (project) use only one bucket for all his databases (production, training, etc)

The main advantage I see for this approach from this recently added functionality is that we can easily (and safely) rename or changed who owns an object without needing to rename the ir.attachment.filename .

How it would work:

The idea is to use one bucket for same customer/project. Then, when deleting s3 attachments, we only wants to delete them if the attachments is owned by this database.

Modifications needed:

• when s3 objects are created we add a label 'database_name = creator_db_name'
• when calling ir.attachment.unlink, if s3 file and s3 bucket = actual bucket (actual checks) and database name != s3 object.label.database_name, skip unlink
• if renaming a database we should check all s3 objects and rename the label database_name for new database name

Extra functionality:

When deleting a database thought odoo drop functionality, the s3 objects are not deleted. We could also implement to delete them on s3 checking for objects where "object.label.database_name = database_name" (this could be also done outside the module, the good thing is that we know for each object witch database is the owner with the label)

Use cases

Use case 1: duplicate production database for training

• Production database is duplicated for training (or a training database is created by restoring a pg backup)
• new files are created on new training database
• user deletes attachments on training database
• only attachments created on training database are deleted (if some of the where created by production database they will remain there)

Use case 2: rename a database, for eg, and upgraded database to new odoo version become production one)

• on the upgraded databases new files are created
• the upgraded database become the production db
• if we call the rename of the database, all objects labels are renamed so that the belongs to new database

Error message can be misleading in dev mode

Hello,

If you have a have a configuration that sets "ir_attachment.location" to s3 and your environment is not in production or integration the error message ask you to set AWS env variable.

This is quite confusing, i would be better to recommend to set the parameter to 'db', in the same fashion as here:
https://github.com/camptocamp/odoo-cloud-platform/blob/10.0/cloud_platform/models/cloud_platform.py#L106-L110

Regards

base_attachment_object_storage: performance issue with images on tree views

Replicated on v13 with attachment_s3 but should be replicable on other versions or other modules that use base_attachment_object_storage

Steps to reproduce:

1. Use a database with many products with images

2. On product.template tree view add image_128 field (for eg <field name="image_128" widget="image" string="Imagen"/>)

3. Go to product.tempalte tree view and with debuger check searc_read time:

   • tree view takes 5x more thatn kanban view (Same image_128 field)
   • using a bigger image (for eg. image_512) make it faster while should be slower because image data is fetched from s3 instead of database

VIDEO: https://drive.google.com/file/d/1hBXIN-gB4bLULsu2y14oRBJUBPVnSgfy/view

Platform check should take in account the env variable DISABLE_ATTACHMENT_STORAGE

odoo-cloud-platform/cloud_platform_azure/models/cloud_platform.py

Line 45 in 8046783

def _check_filestore(self, environment_name):

    def is_storage_disabled():
        from distutils.util import strtobool
        return bool(strtobool(os.environ.get("DISABLE_ATTACHMENT_STORAGE") or '0'))

    @api.model
    def _check_filestore(self, environment_name):
        params = self.env["ir.config_parameter"].sudo()
        use_azure = params.get_param("ir_attachment.location") == AZURE_STORE_KIND.name
        if self.is_storage_disabled():
            return False
        if environment_name in ("prod", "integration"):
        (...)       

Push modules of this repo to Pypi?

Is there any interest to push the modules of this repository to Pypi or even move the repository under the OCA?

base_attachment_object_storage refactored into OCA/storage

The base_attachment_object_storage addon is a very good addon that allows to store all the attachments from odoo into an external file system. In an effort to refactor the storage addons from OCA to make them uses the fsspec python library as an abstraction layer to any file system where the file content can be stored (see OCA/storage#250), I have started to think about the possibility to refactor storage_file and storage_image addons to uses the ir.attachment model as a storage layer and in the same time define a new kind of field specialized in the storage of file content into an external file system. (Some explanations are provided OCA/storage#252). Time after time, It appear that the way I'll extend the ir.attachment model to support the storage through the fsspec library will be very similar to the way the base_attachment_object_storage addon is implemented with the 'advantage' of being able to support a wide range of file systems without the need to implement a new addon for each file system. To keep your credits for the original implementation, I propose to extract your base_attachment_object_storage with the commit history addon into my PR and then to rename it into fs_attachment before starting its refactoring. That also means that the licence will be changed from AGPL to LGTP since my new fields could be used into addon on top of OE addons. (No concrete case at this moment)
Do you agree with this proposal?

Regards,

ping @jgrandguillaume

kwkhtmltopdf_assets not needed anymore

Hi all

Just to inform your team that a fix have been done in odoo : odoo/odoo#108261
So the module kwkhtmltopdf_assets is not needed anymore

[14.0] sessions_redis: 'werkzeug.contrib.sessions' is deprecated as of version 0.15

When I start Odoo 14 with session_redis, I get the following warning:

WARNING ? py.warnings: /odoo/addons/session_redis/session.py:7: DeprecationWarning: 'werkzeug.contrib.sessions' is deprecated as of version 0.15 and will be removed in version 1.0. It has moved to https://github.com/pallets/secure-cookie.
  from werkzeug.contrib.sessions import SessionStore

[all versions] attachment_s3: low performance for /shop

This issue is a problem between lower performance accessing s3 objects + some kind of odoo issue.

The problem is that on /shop routes, odoo is calling the method _get_combination_info. And on this method big files are read just to know if they are there (bool(product.image_1920))

For now, we've open odoo/odoo#81767 with a patch to odoo (actually is not a suitable patch)

Anyone else having same issue?

Just changing those lines in odoo we've reduce more that 70% time for /shop route

attachment_s3: Not delete from bucket

If we setup server environment with AWS_BUCKETNAME with placeholder {db} for creating bucket with db name, the files are not deleted from bucket when we delete in odoo.
When we delete file in odoo, the bucket name get from server environment will be {db} and can't find this bucket in s3.

odoo-cloud-platform/attachment_s3/models/ir_attachment.py

Line 172 in 05d111f

if bucket_name == os.environ.get('AWS_BUCKETNAME'):

attachment_swift: Keystone error on write operation

Reproduced on: 10.0

Traceback (most recent call last):
  File "/odoo/src/odoo/http.py", line 642, in _handle_exception
    return super(JsonRequest, self)._handle_exception(exception)
  File "/odoo/src/odoo/http.py", line 684, in dispatch
    result = self._call_function(**self.params)
  File "/odoo/src/odoo/http.py", line 334, in _call_function
    return checked_call(self.db, *args, **kwargs)
  File "/odoo/src/odoo/service/model.py", line 101, in wrapper
    return f(dbname, *args, **kwargs)
  File "/odoo/src/odoo/http.py", line 327, in checked_call
    result = self.endpoint(*a, **kw)
  File "/odoo/src/odoo/http.py", line 942, in __call__
    return self.method(*args, **kw)
  File "/odoo/src/odoo/http.py", line 507, in response_wrap
    response = f(*args, **kw)
  File "/odoo/src/addons/web/controllers/main.py", line 895, in call_kw
    return self._call_kw(model, method, args, kwargs)
  File "/odoo/src/addons/web/controllers/main.py", line 887, in _call_kw
    return call_kw(request.env[model], method, args, kwargs)
  File "/odoo/src/odoo/api.py", line 687, in call_kw
    return call_kw_model(method, model, args, kwargs)
  File "/odoo/src/odoo/api.py", line 672, in call_kw_model
    result = method(recs, *args, **kwargs)
  File "/odoo/src/odoo/addons/base/res/res_partner.py", line 534, in create
    partner = super(Partner, self).create(vals)
  File "/odoo/src/addons/mail/models/mail_thread.py", line 228, in create
    thread = super(MailThread, self).create(values)
  File "/odoo/src/odoo/models.py", line 3847, in create
    record = self.browse(self._create(old_vals))
  File "/odoo/src/odoo/models.py", line 3996, in _create
    field.write(self.with_context(rel_context), vals[name])
  File "/odoo/src/odoo/fields.py", line 1698, in write
    'datas': value,
  File "/odoo/src/odoo/addons/base/ir/ir_attachment.py", line 459, in create
    return super(IrAttachment, self).create(values)
  File "/odoo/src/odoo/models.py", line 3855, in create
    self._fields[key].determine_inverse(record)
  File "/odoo/src/odoo/fields.py", line 1046, in determine_inverse
    getattr(records, self.inverse)()
  File "/odoo/external-src/odoo-cloud-platform/base_attachment_object_storage/models/ir_attachment.py", line 126, in _inverse_datas
    super(IrAttachment, attach)._inverse_datas()
  File "/odoo/src/odoo/addons/base/ir/ir_attachment.py", line 208, in _inverse_datas
    vals['store_fname'] = self._file_write(value, vals['checksum'])
  File "/odoo/external-src/odoo-cloud-platform/base_attachment_object_storage/models/ir_attachment.py", line 158, in _file_write
    filename = self._store_file_write(key, bin_data)
  File "/odoo/external-src/odoo-cloud-platform/attachment_swift/models/ir_attachment.py", line 149, in _store_file_write
    conn.put_container(container)
  File "/usr/local/lib/python2.7/dist-packages/swiftclient/client.py", line 1760, in put_container
    query_string=query_string)
  File "/usr/local/lib/python2.7/dist-packages/swiftclient/client.py", line 1666, in _retry
    self.url, self.token = self.get_auth()
  File "/usr/local/lib/python2.7/dist-packages/swiftclient/client.py", line 1618, in get_auth
    timeout=self.timeout)
  File "/usr/local/lib/python2.7/dist-packages/swiftclient/client.py", line 635, in get_auth
    interface=interface)
  File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 1225, in get_endpoint
    return auth.get_endpoint(self, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 380, in get_endpoint
    allow_version_hack=allow_version_hack, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 271, in get_endpoint_data
    service_catalog = self.get_access(session).service_catalog
  File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 134, in get_access
    self.auth_ref = self.get_auth_ref(session)
  File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/v3/base.py", line 184, in get_auth_ref
    authenticated=False, log=False, **rkwargs)
  File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 1131, in post
    return self.request(url, 'POST', **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 968, in request
    raise exceptions.from_response(resp, method, url)
BadRequest: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-82d6744f-c891-4fcc-a13b-88e6fa8107f8

V15 dump the attachments to the minio, can not access these attachments in odoo.

In the odoo15 version, the attachments originally stored in the database are transferred out to the filtere, and then the env['ir.attachment'] .force_storage() command is executed to dump the attachments to the minio, and then the folder directory is removed (no longer needed). When entering the system, odoo has an error with the following message: Traceback (most recent call last):
File "/home/vcs/vcs/vcs/addons/base/models/ir_attachment.py", line 119, in _file_read
with open(full_path, 'rb') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/vcs/vcs/share/filestore/RCB/74/74fb750898f19dc963c14b85fab3b9c734a5b655'。 Since we imported the attachment information into the minio, of course we want to be able to access these attachments in odoo.

Does monitoring_prometheus need dependency to server_environment?

Thank you for really nice odoo apps. I wanted to try out monitoring_prometheus for an odoo application I am working on. I noticed that it has dependency on server_environment.

odoo-cloud-platform/monitoring_prometheus/__manifest__.py

Line 14 in 1731912

"server_environment",

But this dependecy does not seem to be leveraged in the module. So I was wondering, if I should remove this dependency before? If it is intentional I would like to know why and I can also contribute that to ReadMe once I know.

Thank you.

Attachments with mimetype application/octet-stream

Not sure why but on almost every database we have files with mimetype = 'application/octet-stream'.
By default (by default ir_attachment.storage.force.database value), those files are stored on the object storage.

Some of those files seems to be broken images, some assets (js, css) (for eg /web/js/website.assets_editor/dbe8624, /web/css/website.assets_frontend/ac45671).

I try to verify if those files where actually used, as far as I've tested, at least these are used:

• user_values.scss
• user_color_palette.scss
• user_theme_color_palette.scss
• user_custom_rules.scss

Someone else with same behavior? It would be a good idea to store application/octet-stream on database? Or improve detection of mimetypes?

Any suggestion?

attachment_s3 : Module 'odoo.modules.registry' has no attribute 'RegistryManager'

Here comes the output (https://caas.camptocamp.com/env/1a233618/apps/stacks/1st38499/services/1s211023/containers):

24/08/2020 11:23:07{"asctime": "2020-08-24 09:23:07,621", "pid": 222, "levelname": "WARNING", "dbname": "winter_sunset_3601_integration_13", "name": "odoo.modules.loading", "message": "Transient module states were reset", "perf_info": ""}
24/08/2020 11:23:07{"asctime": "2020-08-24 09:23:07,625", "pid": 222, "levelname": "ERROR", "dbname": "winter_sunset_3601_integration_13", "name": "odoo.modules.registry", "message": "Failed to load registry", "exc_info": "Traceback (most recent call last):\n  File \"/odoo/src/odoo/modules/registry.py\", line 86, in new\n    odoo.modules.load_modules(registry._db, force_demo, status, update_module)\n  File \"/odoo/src/odoo/modules/loading.py\", line 419, in load_modules\n    force, status, report, loaded_modules, update_module, models_to_check)\n  File \"/odoo/src/odoo/modules/loading.py\", line 315, in load_marked_modules\n    perform_checks=perform_checks, models_to_check=models_to_check\n  File \"/odoo/src/odoo/modules/loading.py\", line 230, in load_module_graph\n    migrations.migrate_module(package, 'post')\n  File \"/odoo/src/odoo/modules/migration.py\", line 180, in migrate_module\n    migrate(self.cr, installed_version)\n  File \"/odoo/external-src/odoo-cloud-platform/attachment_s3/migrations/13.0.0.0.1/post-migration.py\", line 26, in migrate\n    registry = odoo.modules.registry.RegistryManager.get(cr.dbname)\nAttributeError: module 'odoo.modules.registry' has no attribute 'RegistryManager'", "perf_info": ""}
24/08/2020 11:23:07{"asctime": "2020-08-24 09:23:07,627", "pid": 222, "levelname": "CRITICAL", "dbname": "winter_sunset_3601_integration_13", "name": "odoo.service.server", "message": "Failed to initialize database `winter_sunset_3601_integration_13`.", "exc_info": "Traceback (most recent call last):\n  File \"/odoo/src/odoo/service/server.py\", line 1190, in preload_registries\n    registry = Registry.new(dbname, update_module=update_module)\n  File \"/odoo/src/odoo/modules/registry.py\", line 86, in new\n    odoo.modules.load_modules(registry._db, force_demo, status, update_module)\n  File \"/odoo/src/odoo/modules/loading.py\", line 419, in load_modules\n    force, status, report, loaded_modules, update_module, models_to_check)\n  File \"/odoo/src/odoo/modules/loading.py\", line 315, in load_marked_modules\n    perform_checks=perform_checks, models_to_check=models_to_check\n  File \"/odoo/src/odoo/modules/loading.py\", line 230, in load_module_graph\n    migrations.migrate_module(package, 'post')\n  File \"/odoo/src/odoo/modules/migration.py\", line 180, in migrate_module\n    migrate(self.cr, installed_version)\n  File \"/odoo/external-src/odoo-cloud-platform/attachment_s3/migrations/13.0.0.0.1/post-migration.py\", line 26, in migrate\n    registry = odoo.modules.registry.RegistryManager.get(cr.dbname)\nAttributeError: module 'odoo.modules.registry' has no attribute 'RegistryManager'", "perf_info": ""}
24/08/2020 11:23:07{"asctime": "2020-08-24 09:23:07,633", "pid": 222, "levelname": "INFO", "dbname": "winter_sunset_3601_integration_13", "name": "odoo.service.server", "message": "Initiating shutdown", "perf_info": ""}
24/08/2020 11:23:07{"asctime": "2020-08-24 09:23:07,634", "pid": 222, "levelname": "INFO", "dbname": "winter_sunset_3601_integration_13", "name": "odoo.service.server", "message": "Hit CTRL-C again or send a second signal to force the shutdown.", "perf_info": ""}
24/08/2020 11:23:08Traceback (most recent call last):
24/08/2020 11:23:08  File "/usr/local/bin/marabunta", line 8, in <module>
24/08/2020 11:23:08    sys.exit(main())
24/08/2020 11:23:08  File "/usr/local/lib/python3.7/dist-packages/marabunta/core.py", line 154, in main
24/08/2020 11:23:08    migrate(config)
24/08/2020 11:23:08  File "/usr/local/lib/python3.7/dist-packages/marabunta/core.py", line 143, in migrate
24/08/2020 11:23:08    runner.perform()
24/08/2020 11:23:08  File "/usr/local/lib/python3.7/dist-packages/marabunta/runner.py", line 115, in perform
24/08/2020 11:23:08    VersionRunner(self, version).perform()
24/08/2020 11:23:08  File "/usr/local/lib/python3.7/dist-packages/marabunta/runner.py", line 169, in perform
24/08/2020 11:23:08    self._perform_version(version)
24/08/2020 11:23:08  File "/usr/local/lib/python3.7/dist-packages/marabunta/runner.py", line 200, in _perform_version
24/08/2020 11:23:08    self.perform_addons()
24/08/2020 11:23:08  File "/usr/local/lib/python3.7/dist-packages/marabunta/runner.py", line 226, in perform_addons
24/08/2020 11:23:08    operation.execute(self.log)
24/08/2020 11:23:08  File "/usr/local/lib/python3.7/dist-packages/marabunta/model.py", line 363, in execute
24/08/2020 11:23:08    self._execute(log, interactive=sys.stdout.isatty())
24/08/2020 11:23:08  File "/usr/local/lib/python3.7/dist-packages/marabunta/model.py", line 352, in _execute
24/08/2020 11:23:08    child.exitstatus
24/08/2020 11:23:08marabunta.exception.OperationError: command 'odoo --workers=0 --stop-after-init --no-xmlrpc -i account_plaid,account_invoice_extract_purchase,account_yodlee,account_ponto,camptocamp_website_tools,mass_editing,account_bank_statement_import_csv,base_import_async,barcodes_mobile,l10n_generic_auto_transfer_demo,website_sale_delivery,account_online_sync,account_bank_statement_import_camt -u analytic,website_google_map,web,website_forum,hr_contract,portal,quality_control,database_cleanup,http_routing,website_event_track,mass_mailing_resend,website_event_questions,hr_contract_sign,website,website_event,account,website_blog,website_mail_channel,monitoring_status,report_xlsx,base_technical_features,im_livechat_enterprise,website_helpdesk_form,monitoring_statsd,website_helpdesk_livechat,hr_recruitment_survey,website_sale_slides,base_location,google_account,auditlog,base_geolocalize,hr_expense,crm_livechat,account_financial_report,server_environment_files,cloud_platform,base_automation,website_sale_product_configurator,account_intrastat,website_hr_recruitment,helpdesk,base_iban,l10n_multilang,quality_mrp,barcodes,mail_environment,crm,stock_split_picking,attachment_s3,web_environment_ribbon,product,odoo_referral_portal,hr_attendance,contacts,website_partner,stock_barcode,mass_mailing_partner,hr_work_entry,date_range,logging_json,quality_mrp_workorder,purchase_mrp_workorder_quality,server_environment_ir_config_parameter,website_crm_partner_assign,website_livechat,event,sale_order_type,rating,project,mis_builder,purchase,account_budget,sales_team,link_tracker,utm,website_crm_livechat,l10n_generic_coa,bus,website_payment,hr_holidays,quality,gamification,calendar,phone_validation,website_helpdesk,fleet,sale_product_configurator,partner_firstname,website_sale,im_livechat_mail_bot,hr_timesheet,website_sale_product_attribute_value_filter_existing,mail,website_sale_stock_product_configurator,hr,mass_mailing,mis_builder_budget,resource,note,auth_ldap,website_form,im_livechat,sale,stock_dropshipping,l10n_ch_states,website_slides,hr_recruitment,account_analytic_default,account_asset,helpdesk_timesheet,partner_fax,stock,website_customer,base_address_city,sale_management,project_task_default_stage,board,hr_appraisal,l10n_ch,theme_default,product_state,base,product_dimension,mrp,survey,website_sale_comparison,cloud_platform_exoscale,sign,hr_org_chart,ocn_client,account_accountant,stock_intrastat,session_redis,delivery,server_environment,mrp_workorder,base_sparse_field' returned 255```

[monitoring_log_requests] v13 leaks passwords!

In v14 and v15 only three specific fields of "params" get logged

https://github.com/camptocamp/odoo-cloud-platform/blob/14.0/monitoring_log_requests/models/ir_http.py#L73
https://github.com/camptocamp/odoo-cloud-platform/blob/15.0/monitoring_log_requests/models/ir_http.py#L73

        info.update({
            'model': request.params.get('model'),
            'model_method': request.params.get('method'),
            'workflow_signal': request.params.get('signal'),
        })

but in v13 the entire "params" just gets logged as one big dictionary

https://github.com/camptocamp/odoo-cloud-platform/blob/13.0/monitoring_log_requests/models/ir_http.py#L96

    if hasattr(request, "params"):
        info["params"] = dict(request.params)

which will include "login" and "password" from the login page (or API call), in cleartext.

Migration to version 13.0

TODO:

https://github.com/OCA/maintainer-tools/wiki/Migration-to-version-13.0

GLOBAL:

• #81 ✔️

MODULES TO MIGRATE:

• attachment_s3 #82 ✔️
• attachment_swift #85 ✔️
• base_attachment_object_storage #83 ✔️
• base_fileurl_field #86 ✔️
• cloud_platform #87 ✔️
• cloud_platform_exoscale #88 ✔️
• cloud_platform_ovh #93 ✔️
• logging_json #89 ✔️
• monitoring_log_requests #94 ✔️
• monitoring_statsd #90 ✔️
• monitoring_status #91 ✔️
• session_redis #92 ✔️
• test_base_fileurl_field

session_redis: Sessions leaking to data_dir and crashing odoo

I've found an odd behavior and it's that from time to time sessions leak to the data_dir causing the sessions folder to increase in size.

When there are thousands of sessions in the Folder, Odoo fails to start as this module tries to delete all of them first; failing to do so (or timing out in the process) hangs the server.

On environments like Kubernetes this causes that the container restarts infinitely (because of the healthchecks) until crashing.

attachment_swift: AttributeError: 'module' object has no attribute 'identity'

When accessing a attachment form, this error raises:

  File "/odoo/src/openerp/addons/base/ir/ir_attachment.py", line 132, in _data_get
    result[attach.id] = self._file_read(cr, uid, location, attach.store_fname, bin_size)
  File "/odoo/external-src/odoo-cloud-platform/base_attachment_object_storage/models/ir_attachment.py", line 191, in _file_read
    return self._store_file_read(fname, bin_size=bin_size)
  File "/odoo/external-src/odoo-cloud-platform/attachment_swift/models/ir_attachment.py", line 125, in _store_file_read
    conn = self._get_swift_connection()
  File "/odoo/external-src/odoo-cloud-platform/attachment_swift/models/ir_attachment.py", line 108, in _get_swift_connection
    auth_url=host,
  File "/odoo/external-src/odoo-cloud-platform/attachment_swift/models/ir_attachment.py", line 58, in get_session
    auth = keystoneauth1.identity.v3.Password(
AttributeError: 'module' object has no attribute 'identity'

file is copied to s3 but stays in the local filesystem with the same name

I'm trying to migrate existing filestore setup to google cloud s3 (few issues there solved, will report later)

but for some reason the migration tool is copying files to the s3 but never updates the stored filename so the system keeps looking for the attachment on the local file storage.

that's version 12.0

anybody else saw that?

11.0 AttributeError

Got this in the log,

Not sure how to reproduce, might be when you have an old session and your browser tries to connect with it.

werkzeug
Error on request:
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/werkzeug/serving.py", line 193, in run_wsgi
    execute(self.server.app)
  File "/usr/local/lib/python3.5/dist-packages/werkzeug/serving.py", line 181, in execute
    application_iter = app(environ, start_response)
  File "/opt/odoo/src/odoo/service/wsgi_server.py", line 164, in application
    return werkzeug.contrib.fixers.ProxyFix(application_unproxied)(environ, start_response)
  File "/usr/local/lib/python3.5/dist-packages/werkzeug/contrib/fixers.py", line 152, in __call__
    return self.app(environ, start_response)
  File "/opt/odoo/src/odoo/service/wsgi_server.py", line 154, in application_unproxied
    result = handler(environ, start_response)
  File "/opt/odoo/src/odoo/http.py", line 1298, in __call__
    return self.dispatch(environ, start_response)
  File "/opt/odoo/src/odoo/http.py", line 1272, in __call__
    return self.app(environ, start_wrapped)
  File "/usr/local/lib/python3.5/dist-packages/werkzeug/wsgi.py", line 599, in __call__
    return self.app(environ, start_response)
  File "/opt/odoo/src/odoo/http.py", line 1437, in dispatch
    explicit_session = self.setup_session(httprequest)
  File "/opt/odoo/src/odoo/http.py", line 1345, in setup_session
    httprequest.session = self.session_store.get(sid)
  File "/opt/odoo/external-src/odoo-cloud-platform/session_redis/session.py", line 66, in get
    saved = self.redis.get(key).decode('utf-8')
AttributeError: 'NoneType' object has no attribute 'decode'

[RFC] attachment_s3: Support for database operations

We would like to add the support for the following database operations with the Odoo DB manager:

• creation: When the module is installed, check the environment variables, set the config parameters, create the bucket and run force_storage().
• deletion: When the database is dropped, delete the bucket.
• duplication: When the database is duplicated, duplicate the bucket and update the store_fname in the ir_attachment table:

UPDATE ir_attachment AS t 
SET store_fname = s.store_fname 
FROM (
    SELECT id,REPLACE(store_fname, 'old', 'new') AS store_fname 
    FROM ir_attachment 
    WHERE db_datas is NULL)
AS s(id,store_fname)
WHERE t.id = s.id;

In version 12 and 14.

monitoring_log_requests: TypeError: _dispatch() missing 1 required positional argument: 'self'

Hello guys,

I wanted to give a try to the module monitoring_log_requests but I have an error when I launch the instance.
Did you already see this error? I have the feeling I miss a setup. Where should I take a look?

odoo_1  | Traceback (most recent call last):
odoo_1  |   File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 205, in run_wsgi
odoo_1  |     execute(self.server.app)
odoo_1  |   File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 193, in execute
odoo_1  |     application_iter = app(environ, start_response)
odoo_1  |   File "/usr/lib/python3/dist-packages/odoo/service/wsgi_server.py", line 166, in application
odoo_1  |     return application_unproxied(environ, start_response)
odoo_1  |   File "/usr/lib/python3/dist-packages/odoo/service/wsgi_server.py", line 154, in application_unproxied
odoo_1  |     result = handler(environ, start_response)
odoo_1  |   File "/usr/lib/python3/dist-packages/odoo/http.py", line 1318, in __call__
odoo_1  |     return self.dispatch(environ, start_response)
odoo_1  |   File "/usr/lib/python3/dist-packages/odoo/http.py", line 1292, in __call__
odoo_1  |     return self.app(environ, start_wrapped)
odoo_1  |   File "/usr/lib/python3/dist-packages/werkzeug/wsgi.py", line 599, in __call__
odoo_1  |     return self.app(environ, start_response)
odoo_1  |   File "/usr/lib/python3/dist-packages/odoo/http.py", line 1490, in dispatch
odoo_1  |     result = ir_http._dispatch()
odoo_1  |   File "/usr/lib/python3/dist-packages/odoo/addons/auth_signup/models/ir_http.py", line 19, in _dispatch
odoo_1  |     return super(Http, cls)._dispatch()
odoo_1  |   File "/usr/lib/python3/dist-packages/odoo/addons/web_editor/models/ir_http.py", line 22, in _dispatch
odoo_1  |     return super(IrHttp, cls)._dispatch()
odoo_1  | TypeError: _dispatch() missing 1 required positional argument: 'self'

attachment_swift: Keystone error when data are downloaded

It was working well with the previous version of the module.

File "/odoo/external-src/odoo-cloud-platform/attachment_swift/models/ir_attachment.py", line 136, in _store_file_read
swifturi.item())
File "/usr/local/lib/python2.7/dist-packages/swiftclient/client.py", line 1786, in get_object
headers=headers)
File "/usr/local/lib/python2.7/dist-packages/swiftclient/client.py", line 1666, in _retry
self.url, self.token = self.get_auth()
File "/usr/local/lib/python2.7/dist-packages/swiftclient/client.py", line 1618, in get_auth
timeout=self.timeout)
File "/usr/local/lib/python2.7/dist-packages/swiftclient/client.py", line 635, in get_auth
interface=interface)
File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 1225, in get_endpoint
return auth.get_endpoint(self, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 380, in get_endpoint
allow_version_hack=allow_version_hack, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 271, in get_endpoint_data
service_catalog = self.get_access(session).service_catalog
File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 134, in get_access
self.auth_ref = self.get_auth_ref(session)
File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/v3/base.py", line 184, in get_auth_ref
authenticated=False, log=False, **rkwargs)
File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 1131, in post
return self.request(url, 'POST', **kwargs)
File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 968, in request
raise exceptions.from_response(resp, method, url)
NotFound: The resource could not be found. (HTTP 404) (Request-ID: req-37f7e6fb-7e90-4627-b3fc-4d2153cd76bb)

s3 bucket download performance / direct download

Hi,

So the current download of the S3 files is quite slow as to consider switching back to PVC.

However as the most of the time the content is used to display in the client browser it seems quite logical to just serve a signed url to the storage directly instead of proxying the download via the odoo instance?

Is any work is going on in that direction?

odoo15 dump the attachments to the minio，can not be access attachments in odoo.

In the odoo15 version, the attachments originally stored in the database are transferred out to the filtere, and then the env['ir.attachment'] .force_storage() command is executed to dump the attachments to the minio, and then the folder directory is removed (no longer needed). When entering the system, odoo has an error with the following message: Traceback (most recent call last):
File "/home/vcs/vcs/vcs/addons/base/models/ir_attachment.py", line 119, in _file_read
with open(full_path, 'rb') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/vcs/vcs/share/filestore/RCB/74/74fb750898f19dc963c14b85fab3b9c734a5b655'。 Since we imported the attachment information into the minio, of course we want to be able to access these attachments in odoo.

V12 attachment_s3 : SVG wrong content-type on S3 --> upgrade extra arg boto3

We have an issue when we upload some picture SVG to the S3 in odoo V12.

Context :
we need to upload SVG file to the S3 and display the image in an image_URL field on the view

error case :
When we uploaded SVG files in the S3 bucket, the content-type is not "image/svg+xml" but "octet-stream". when we display the the file from the url, odoo return an error " image can't be displayed".

Possible solution
the problem is that the content type seems to be wrong or not passed with the file to the bucket S3. As discussed with @gurneyalex , @yvaucher this can be update by adapting boto3 request to pass extra parameters ( metadata) : https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-uploading-files.html

It's done with this PR : https://github.com/camptocamp/odoo-cloud-platform/pull/404/files

session_redis: change in the management of anonymous sessions in odoo 16

In Odoo 16 anonymous sessions are no longer stored (unless they change a default setting).

TODO: check if we can simplify the code of session_redis regarding the expiration of anonymous sessions to take advantage of this.

how used for Enterprise network deployment.

Unable to connect to the Internet

Is there an extension of FTP, SFTP, this kind of private storage?

It is used for Enterprise network deployment.

Use nginx to provide front-end use.

attachment_s3: reverting back to regular filestore storage

Hi,

can you please recommend a method for reverting back to regular filestore storage. Does the env['ir.attachment'].force_storage() apply to this use case too? If so, what happens to the files that were copied to the database storage? Will they also return to the filestore?

Thank you!

[14.0] sessions_redis: inactivity timer

issue
When setting ODOO_SESSION_REDIS_EXPIRATION to 3600. This seems to timeout the user after 1 hour active or not.

Is there a way to start this timer after 1 hour of inactivity?

[ADD] attachment_azure

Hello,

We have a customer who will run Odoo in Azure using Azure Blob Storage (https://azure.microsoft.com/en-us/services/storage/blobs/) for the Odoo filestore.

So we are planning to create a new module attachment_azure, similar to attachment_s3.

Do you mind hosting the module in this repo?

Thank you.

attachment_s3: One directory per database

Context

Using version 12.0 with Digital Ocean Spaces

Steps to reproduce

• Setup the AWS environment variables
• Create a database "MASTER"

Result

• Files are stored directly in /BUCKETNAME/

Expected result

• Files are stored in /BUCKETNAME/MASTER/

Question about the License

Hi Guys

I have a question about the licensing of these modules. They are currently licensed under the AGPL.

We are likely to have customers that want to use Odoo's web portal for the general public to access their products and services. It is also likely that they will pay us to build some custom code for their system that they may not want to have to make available to their competitors.

If we use these modules, do we have to make all our customer's custom modules open source, since technically the general public can access and use their system? Or do we just have to make sure our direct customer can access the source code of there system if they want it?

Appreciate any advice on this!

[logging_json / monitoring_log_requests] monitoring.http.requests message logged as escaped string, not JSON

When I run both logging_json and monitoring_log_requests, the output of the latter comes out as an escaped string, rather than nested JSON (and thus parsed as a string by logstash):

odooapp        | {"asctime": "2021-08-30 00:35:49,662", "pid": 1, "levelname": "INFO", "dbname": "DannyDEV", "name": "monitoring.http.requests", "message": "{\"start_time\": \"2021-08-30 00:35:49\", \"duration\": 0.008325576782226562, \"method\": \"POST\", \"url\": \"http://localhost:8069/web/dataset/call_kw/res.users/review_user_count\", \"status_code\": 200, \"headers\": {\"wsgi.version\": [1, 0], \"wsgi.url_scheme\": \"http\", \"wsgi.input\": \"<non-serializable: BufferedReader>\", \"wsgi.errors\": \"<non-serializable: TextIOWrapper>\", \"wsgi.multithread\": true, \"wsgi.multiprocess\": false, \"wsgi.run_once\": false, \"werkzeug.server.shutdown\": \"<non-serializable: function>\", \"SERVER_SOFTWARE\": \"Werkzeug/0.16.1\", \"REQUEST_METHOD\": \"POST\", \"SCRIPT_NAME\": \"\", \"PATH_INFO\": \"/web/dataset/call_kw/res.users/review_user_count\", \"QUERY_STRING\": \"\", \"REQUEST_URI\": \"/web/dataset/call_kw/res.users/review_user_count\", \"RAW_URI\": \"/web/dataset/call_kw/res.users/review_user_count\", \"REMOTE_ADDR\": \"172.26.0.1\", \"REMOTE_PORT\": 51594, \"SERVER_NAME\": \"0.0.0.0\", \"SERVER_PORT\": \"8069\", \"SERVER_PROTOCOL\": \"HTTP/1.1\", \"HTTP_HOST\": \"localhost:8069\", \"HTTP_CONNECTION\": \"keep-alive\", \"CONTENT_LENGTH\": \"219\", \"HTTP_ACCEPT\": \"application/json, text/javascript, */*; q=0.01\", \"HTTP_X_REQUESTED_WITH\": \"XMLHttpRequest\", \"HTTP_USER_AGENT\": \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36\", \"CONTENT_TYPE\": \"application/json\", \"HTTP_SEC_GPC\": \"1\", \"HTTP_ORIGIN\": \"http://localhost:8069\", \"HTTP_SEC_FETCH_SITE\": \"same-origin\", \"HTTP_SEC_FETCH_MODE\": \"cors\", \"HTTP_SEC_FETCH_DEST\": \"empty\", \"HTTP_REFERER\": \"http://localhost:8069/web\", \"HTTP_ACCEPT_ENCODING\": \"gzip, deflate, br\", \"HTTP_ACCEPT_LANGUAGE\": \"en\", \"HTTP_DNT\": \"1\", \"werkzeug.request\": \"<non-serializable: Request>\"}, \"uid\": 2, \"server_environment\": \"test\", \"session\": {\"db\": \"DannyDEV\", \"uid\": 2, \"login\": \"admin\", \"context\": {\"lang\": \"en_GB\", \"tz\": \"Pacific/Auckland\", \"uid\": 2}, \"debug\": \"assets\", \"geoip\": {}}, \"params\": {\"args\": [], \"model\": \"res.users\", \"method\": \"review_user_count\", \"kwargs\": {}}}", "perf_info": "", "request_id": "9e83f62b-de18-4e7d-bd74-d49cfe3d5257", "uid": 2}
odooapp        | {"asctime": "2021-08-30 00:35:49,673", "pid": 1, "levelname": "INFO", "dbname": "DannyDEV", "name": "werkzeug", "message": "172.26.0.1 - - [30/Aug/2021 00:35:49] \"POST /web/dataset/call_kw/res.users/review_user_count HTTP/1.1\" 200 -", "perf_info": "2 0.002 0.025", "request_id": "9e83f62b-de18-4e7d-bd74-d49cfe3d5257", "uid": 2}

❯ grep LOG_HANDLER odooapp/odoo.conf
LOG_HANDLER=":WARNING,monitoring.http.requests:INFO"
❯ 

[Question] How to reduce database migration time when attachment_s3 is used

Hello, I installed attachment_s3 in v12 environment and I have 100000 products. I want to migrate to v13 and I used openupgrade from OCA for migration. So, when I run the migration script yesterday and still not finish migrating product module. I think the loading time is taking time when pass datas value to image_1920 from attachment https://github.com/OCA/OpenUpgrade/blob/b3bed799c2c73b50af62eb6796385ce88ffe9fb7/addons/product/migrations/13.0.1.2/post-migration.py#L12. It fetch data from s3 and pass that's why taking time too much. How can I reduce time? If someone has the experience about version migration with attachment_s3, please guide me.

V15 session redis

How to set the parameters on Odoo sh when i have no access in the odoo.conf file?

_inverse_datas not callled

base_attachment_object_storage in v9 defines a function called _inverse_datas. The default inverse function for datas (https://github.com/odoo/odoo/blob/9.0/openerp/addons/base/ir/ir_attachment.py#L279) is _data_set so it will not be called.

We define a datas field in attachment_s3 which uses _inverse_datas a inverse. Can we move it? currently there is no dependency between the two afterwards attachment_s3 would have to depend on base_attachment_object_storage

Migration to v14

TODO:

https://github.com/OCA/maintainer-tools/wiki/Migration-to-version-14.0

GLOBAL:

• #209

MODULES TO MIGRATE:

• attachment_s3 #210
• attachment_swift #210
• base_attachment_object_storage #210
• base_fileurl_field #210
• cloud_platform #211
• cloud_platform_exoscale #211
• cloud_platform_ovh #211
• logging_json #205
• monitoring_log_requests #212
• monitoring_statsd #207
• monitoring_status #208
• session_redis #206
• test_base_fileurl_field

[IMP] extend odoo cdn to use s3 storages

I believe it would be more-less easy to extend odoo CDN functionality so that s3 objects are used instead.

Any thoughts ?

[12.0] session_redis: Odoo server error when accessing a shared record

Hello,
I’m getting internal server error after using this module to store my odoo sessions in Redis. When sharing a document for public access (https://mydomain.com/my/orders/24?access_token=982cfbfb-0b4e-4929-97c3-2f428ea25731&). If the user is logged in, the URL works fine, otherwise I get the following error:

Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/werkzeug/serving.py", line 303, in run_wsgi
execute(self.server.app)
File "/usr/local/lib/python3.7/site-packages/werkzeug/serving.py", line 291, in execute
application_iter = app(environ, start_response)
File "/usr/local/lib/python3.7/site-packages/odoo/service/server.py", line 434, in app
return self.app(e, s)
File "/usr/local/lib/python3.7/site-packages/odoo/service/wsgi_server.py", line 142, in application
return application_unproxied(environ, start_response)
File "/usr/local/lib/python3.7/site-packages/odoo/service/wsgi_server.py", line 117, in application_unproxied
result = odoo.http.root(environ, start_response)
File "/usr/local/lib/python3.7/site-packages/odoo/http.py", line 1320, in call
return self.dispatch(environ, start_response)
File "/usr/local/lib/python3.7/site-packages/odoo/http.py", line 1293, in call
return self.app(environ, start_wrapped)
File "/usr/local/lib/python3.7/site-packages/werkzeug/middleware/shared_data.py", line 220, in call
return self.app(environ, start_response)
File "/usr/local/lib/python3.7/site-packages/odoo/http.py", line 1492, in dispatch
response = self.get_response(httprequest, result, explicit_session)
File "/usr/local/lib/python3.7/site-packages/odoo/http.py", line 1426, in get_response
self.session_store.save(httprequest.session)
File "/var/lib/odoo/extra-addons/camptocamp/odoo-cloud-platform/session_redis/session.py", line 60, in save
data = json.dumps(dict(session)).encode('utf-8')
File "/usr/local/lib/python3.7/json/init.py", line 231, in dumps
return _default_encoder.encode(obj)
File "/usr/local/lib/python3.7/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/local/lib/python3.7/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/usr/local/lib/python3.7/site-packages/odoo/tools/func.py", line 251, in default
return json_encoder_default(self, o)
File "/usr/local/lib/python3.7/json/encoder.py", line 179, in default
raise TypeError(f'Object of type {o.class.name} '
TypeError: Object of type date is not JSON serializable 84 0.087 0.072

helm charts?

This repository is really awesome! We've been trying to run Odoo on GKE and it's been a battle - will definitely try this now.

I've seen your docker repository is there also a helm chart and is there a plan to make a terraform module?

[14.0] sessions_redis: "Lost Connection", "Reconnected" messages

Issue:
When the session gets killed, the user sees "Lost Connection", "Reconnected" messages that pop up in the top right of the screen. This just happens over and over as you navigate until you go to logout and log back in.

Expected outcome:
The user should see a message saying "Your session has expired".

Question: Redis for website/ecommerce

Hello

Does your REDIS module also support caching parts of website to gain faster website/ecommerce?
I couldn't find any specific documentation on this in the module.

Thanks.

Feature request: Ability to connect to Redis Cluster

The session_redis module works great with Redis servers using the 'Generic' connection method, and I see there is (undocumented) support for the Sentinel method.

https://redis.readthedocs.io/en/stable/connections.html

However, I am trying to switch to a Cluster method which requires calling a different constructor, and including a username.

https://github.com/camptocamp/odoo-cloud-platform/blob/64dc2fc8fdb046c654e8033d24003c58b1f07cbd/session_redis/http.py#L53C1-L53C76

[14.0] attachment_s3: Attachment migration not happening

After configuring ir_attachment.location to s3, setting the environment variables and installing the attachment_s3 module, I created a server action with env['ir.attachment'].force_storage() which shows "migrating files to the object storage" but nothings happens.

I can also confirm that uploading new PDFs does not use the S3. How can I confirm if I'm missing something?

V15 deprecation warning

2022-01-24 13:22:49,187 180 WARNING odoodb py.warnings: /odoo/external-src/odoo-cloud-platform/session_redis/session.py:7: DeprecationWarning: 'werkzeug.contrib.sessions' is deprecated as of version 0.15 and will be removed in version 1.0. It has moved to https://github.com/pallets/secure-cookie.
File "/usr/local/bin/anthem", line 33, in
sys.exit(load_entry_point('anthem', 'console_scripts', 'anthem')())
File "/src/anthem/anthem/cli.py", line 76, in main
run(odoo_args, args.target, options)
File "/src/anthem/anthem/cli.py", line 103, in run
with Context(odoo_args, options) as ctx:
File "/src/anthem/anthem/cli.py", line 120, in init
self.env = self._build_odoo_env(odoo_args)
File "/src/anthem/anthem/cli.py", line 154, in _build_odoo_env
registry = odoo.modules.registry.Registry(dbname)
File "/odoo/src/odoo/modules/registry.py", line 63, in new
return cls.new(db_name)
File "/odoo/src/odoo/modules/registry.py", line 87, in new
odoo.modules.load_modules(registry, force_demo, status, update_module)
File "/odoo/src/odoo/modules/loading.py", line 470, in load_modules
processed_modules += load_marked_modules(cr, graph,
File "/odoo/src/odoo/modules/loading.py", line 363, in load_marked_modules
loaded, processed = load_module_graph(
File "/odoo/src/odoo/modules/loading.py", line 179, in load_module_graph
load_openerp_module(package.name)
File "/odoo/src/odoo/modules/module.py", line 396, in load_openerp_module
import('odoo.addons.' + module_name)
File "/odoo/external-src/odoo-cloud-platform/session_redis/init.py", line 2, in
from . import http
File "/odoo/external-src/odoo-cloud-platform/session_redis/http.py", line 13, in
from .session import RedisSessionStore
File "/odoo/external-src/odoo-cloud-platform/session_redis/session.py", line 7, in
from werkzeug.contrib.sessions import SessionStore

[RFC] attachment_s3: improvements to avoid s3 backup need

We would like to make some improvements to the module. Any though?
This improvement could work together with this other idea #234

Objetive:

Avoid needing to backup attachments and only need to deal with odoo postgres backup.

How it would work:

The idea is that in production databases, when an attachment is deleted, it would not be actually deleted but It will mark to be deleted later (let's said 6 months later).

Modifications needed:

• new optional parameter: attachment_s3.retention
• on ir.attachment new boolean field to_delete
• when calling ir.attachment.unlink, if s3 file and s3 bucket = actual bucket (actual checks) and attachment_s3.retention = True, then:
  a) a new ir.attachment with same values (linked to same s3 objetc) and with to_delete = True
  b) the attachments that was being deleted is deleted but the actual s3 object is not deleted
• new cron parametrizable that will unlink ir.attachments with to_delete = True older than some period

Use cases

Use case 1: check backup for deleted files

• attachment_s3.retention = True
• Production database creates attachments
• postgres backups are created
• someone delete attachments (they're actually marked to be deleted)
• We can restore a postgres backup and access attachments normally (because they still exists on the bucket), user can do whatever he wants to be restored on production databases

Use case 2: restore as production database a previous backup

• attachment_s3.retention = True
• Production database creates attachments
• postgres backups are created
• someone delete attachments (they're actually marked to be deleted)
• We can restore a postgres backup and use it as production database, deleted attachments will be accesible and are not going to be deleted because on this restored backup they're not marked to be deleted

Use case 4: a production database is deleted (for eg. customer leave us) (actually this already work without needing the proposed change)

• attachment_s3.retention = True
• we make a final backup (only postgres)
• we can schedule to delete s3 bucket in a while
• if needed, we can restore postgres backup and all attachments should be there

Use case 5: a training database, in this case if attachments are deleted we wont to delete them

• attachment_s3.retention = True or not configured
• user creates s3 attachments
• user delete s3 attachments
• the s3 attachments objects are actually deleted (actual behavior without this proposed improvement)

ModuleNotFoundError: No module named 'werkzeug.contrib' on session_redis install

Hi. I tried to install the redis python package and only the session_redis(v14) module on odoo 14 enterprise version, but I get the following error:

File "/usr/lib/python3/dist-packages/odoo/modules/registry.py", line 89, in new
  odoo.modules.load_modules(registry._db, force_demo, status, update_module)
File "/usr/lib/python3/dist-packages/odoo/modules/loading.py", line 459, in load_modules
  processed_modules += load_marked_modules(cr, graph,
File "/usr/lib/python3/dist-packages/odoo/modules/loading.py", line 347, in load_marked_modules
  loaded, processed = load_module_graph(
File "/usr/lib/python3/dist-packages/odoo/modules/loading.py", line 179, in load_module_graph
  load_openerp_module(package.name)
File "/usr/lib/python3/dist-packages/odoo/modules/module.py", line 385, in load_openerp_module
  __import__('odoo.addons.' + module_name)
File "/mnt/extra-addons/external_addons/session_redis/__init__.py", line 2, in <module>
  from . import http
File "/mnt/extra-addons/external_addons/session_redis/http.py", line 13, in <module>
  from .session import RedisSessionStore
File "/mnt/extra-addons/external_addons/session_redis/session.py", line 7, in <module>
  from werkzeug.contrib.sessions import SessionStore

ModuleNotFoundError: No module named 'werkzeug.contrib'

I have Werkzeug==0.16.1 installed.
Do you have any suggestion why this might happen?
Thank you kindly in advance!

[DISCUSSION] How to check if cron job fails

odoo-cloud-platform/monitoring_status/controllers/main.py

Line 34 in 185458f

# queue job, cron, database, ...

I've seen that you have module to check the status of the Odoo db. Is there a way to check for the status of the cron thread?

While running several Odoo docker instances I got a runtime error on the cron loop.

odoo/odoo#79823

Not sure why I am the only one out there having this issue.

Now I am looking for a way to check the status of the cron thread across all instances. Maybe you got some input.