giolekva / pcloud Goto Github PK

https://github.com/itaysk/regogo#performance

Is there a downside to use https://github.com/tidwall/gjson ?

Just thinking about possible improvements, and tradeoffs

Provide scripts to easily and quickly set up development environment.
Maybe using kind

Right now Event Processor processes one event at a time. This was sufficient for proof of concept but will need to change before launching the product.

It must be able work on multiple events concurrently while not exhausting infrastructure resources such as CPU, RAM, ... We could artificially limit number of maximum events processed at a time but too simplistic and will most probably not utilize resources to maximum capacity. Would be better if EP, or potentially AM, was aware of infrastructure utilization and make smart decisions when to trigger new actions and when to back off.

This is an umbrella issue about creating full-fledged Photo Gallery app. It can and should be split into smaller tasks. See bellow how it should look like at the end.

Right now prototype photo gallery app is very simple. It just lists all pictures from Knowledge Graph and once picture is opened detected faces are highlighted.

Once finished Photo Gallery app should support:

• Albums:
  • Individual pictures can be grouped into albums
  • Single picture can be in multiple albums at the same time
  • Supporting albums will require adding new ImageAlbum type into Knowledge Graph
• Expand Image node type in GraphQL:
  • Images imported into knowledge graph should be annotated with EXIF metadata such as:
    • When was the picture taken
    • Location where picture was taken
    • Using what camera/app was picture taken
    • ...
• Implement UI:
  • Something like this would be really cool: https://www.npmjs.com/package/react-photo-gallery Maybe we can just reuse this frontend and avoid writing one from scratch
  • Pictures/Albums must be searchable by their metadata

Possibly using HashiCorp Vault: https://www.hashicorp.com/products/vault

Right now Event Processor just launches triggers via Application Manager and forgets about them. Once all triggers are activated EP moves event into done state.

This is not enough as activate triggers must be monitored and optionally restarted in case of failure, we need to be careful though as actions might not be idempotent.
This will require extending Event type in KG and individually storing statuses of each activated trigger.

EP must use AM for launching and monitoring trigger actions. This way Kubernetes API will only be exposed into AM and EP will be isolated from details of the infrastructure it is running on.

When trying ./dev/install_core_services.sh there is errors in bazel build

Error log

ERROR: /root/pcloud/core/events/BUILD:19:8: Writing: bazel-out/k8-fastbuild/bin/core/events/chart.tar.gz failed (Exit 127): build_tar failed: error executing command bazel-out/k8-opt-exec-2B5CBBC6/bin/external/rules_pkg/build_tar @bazel-out/k8-fastbuild/bin/core/events/chart.args

Use --sandbox_debug to see verbose messages from the sandbox build_tar failed: error executing command bazel-out/k8-opt-exec-2B5CBBC6/bin/external/rules_pkg/build_tar @bazel-out/k8-fastbuild/bin/core/events/chart.args

Use --sandbox_debug to see verbose messages from the sandbox
/usr/bin/env: 'python': No such file or directory
Target //core/events:install failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.172s, Critical Path: 0.03s
INFO: 2 processes: 2 internal.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully

Right now KG only servers information stored within graph database. But individual applications might store extra information locally such as Object Store storing individual files while KG having only their path.
To retrieve such resources provided by one application from another, applications have to talk to each other directly. There are multiple problems with such approach:

• There is no way of enforcing ACLs globally.
• Makes integrating multiple applications with each other hard as each application will have different ways of exposing resources.

We will need to come up with unified way for applications to register resources into KG so it knows how to retrieve resources upon request from other applications. With such approach KG will also be able to enforce ACLs and expose auditing tools to the user.

Note that this issue is not about enforcing ACLs itself but rather coming up with API to register and retrieve resources from apps. Implementing ACL infrastructure will be covered in a separate issue.

Not to confuse with the Application Manager:

• Application Registry (AR) will be a service which indexes and makes application charts registered on different Helm Chart Repositories available installable on local PCloud cluster
• While Application Manager (AM) is responsible for actually installing and managing applications

Users should be able to:

1. Add new Helm Repository location into AR
2. AR should monitor registered repositories for new applications, index them locally expose this information via REST API
3. AM should use AR API to allow users to search and install applications

Helm Repository is described here in detail: https://helm.sh/docs/topics/chart_repository/
In short it is a web server where one can find

1. index.yaml file with the list of all charts and their metadata
2. Actual chart archive files

As a proof of concept AM stores list of installed applications in a local file. This will not scale as more and more applications are added to the system. Plus is not easily queryable by other parts of the infrastructure.

My idea is to store information about installed applications and their APIs into KG itself so it can be queried same way as any other information stored in KG.
This will require extending KG schema with new Application type encompassing everything about the app so it can be used in other parts of the infrastructure such as when enforcing ACLs.

Users should be able to override application chart values before installation.
Configuration properties can be found in values.yaml inside chart archive.
We should be able to parse at and present it on UI in a structured form where users can see their default values and override them if necessary.

Besides on-prem PCloud instances users might want to install them on public cloud providers such as GCE, AWS, Azure, ...
Easy to use installer package will be very helpful, especially when wanting to just try out the system without fully committing to it.

By introducing PCloud specific bazel flags and config settings we can make rules such as :push_to_dev to deploy to environment of developers choosing. Something like:
bazel run //core/api:push --pcloud_env=dev # this value being default
bazel run //core/api:push --pcloud_env=personal

https://docs.bazel.build/versions/master/configurable-attributes.html
https://docs.bazel.build/versions/master/be/general.html#config_setting

Right now it is hardcoded in the installation and build scripts to use ports 8080 for accessing PCloud services and 30500 for container registry on host machine.
Such configuration will cause an issue if host machine is already using same ports for some other reason.
Ideally such configuration will be handled by #26 but that's a bigger task, in the meantime we should probably default to some less common ports to reduce chance of collision with other applications.

PCloud needs some kind of discovery service so applications installed on the cluster can be found and reached from outside the cluster.
We should probably use a DNS server plus some custom logic to create DNS records for Ingress routes.
This DNS service then can be used on client devices to resolve application hostnames to cluster IP.

We can use PiHole for that but not sure if that is the best way to go. I can see Pihole as a standalone app and we need such DNS service regardless Pihole is installed or not.

https://github.com/spf13/cobra