giolekva / pcloud Goto Github PK
View Code? Open in Web Editor NEWPrivate Cloud Infrastructure
Private Cloud Infrastructure
https://github.com/itaysk/regogo#performance
Is there a downside to use https://github.com/tidwall/gjson ?
Just thinking about possible improvements, and tradeoffs
Provide scripts to easily and quickly set up development environment.
Maybe using kind
Right now Event Processor processes one event at a time. This was sufficient for proof of concept but will need to change before launching the product.
It must be able work on multiple events concurrently while not exhausting infrastructure resources such as CPU, RAM, ... We could artificially limit number of maximum events processed at a time but too simplistic and will most probably not utilize resources to maximum capacity. Would be better if EP, or potentially AM, was aware of infrastructure utilization and make smart decisions when to trigger new actions and when to back off.
This is an umbrella issue about creating full-fledged Photo Gallery app. It can and should be split into smaller tasks. See bellow how it should look like at the end.
Right now prototype photo gallery app is very simple. It just lists all pictures from Knowledge Graph and once picture is opened detected faces are highlighted.
Once finished Photo Gallery app should support:
Possibly using HashiCorp Vault: https://www.hashicorp.com/products/vault
Right now Event Processor just launches triggers via Application Manager and forgets about them. Once all triggers are activated EP moves event into done state.
This is not enough as activate triggers must be monitored and optionally restarted in case of failure, we need to be careful though as actions might not be idempotent.
This will require extending Event type in KG and individually storing statuses of each activated trigger.
EP must use AM for launching and monitoring trigger actions. This way Kubernetes API will only be exposed into AM and EP will be isolated from details of the infrastructure it is running on.
When trying ./dev/install_core_services.sh
there is errors in bazel build
ERROR: /root/pcloud/core/events/BUILD:19:8: Writing: bazel-out/k8-fastbuild/bin/core/events/chart.tar.gz failed (Exit 127): build_tar failed: error executing command bazel-out/k8-opt-exec-2B5CBBC6/bin/external/rules_pkg/build_tar @bazel-out/k8-fastbuild/bin/core/events/chart.args
Use --sandbox_debug to see verbose messages from the sandbox build_tar failed: error executing command bazel-out/k8-opt-exec-2B5CBBC6/bin/external/rules_pkg/build_tar @bazel-out/k8-fastbuild/bin/core/events/chart.args
Use --sandbox_debug to see verbose messages from the sandbox
/usr/bin/env: 'python': No such file or directory
Target //core/events:install failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.172s, Critical Path: 0.03s
INFO: 2 processes: 2 internal.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully
Right now KG only servers information stored within graph database. But individual applications might store extra information locally such as Object Store storing individual files while KG having only their path.
To retrieve such resources provided by one application from another, applications have to talk to each other directly. There are multiple problems with such approach:
We will need to come up with unified way for applications to register resources into KG so it knows how to retrieve resources upon request from other applications. With such approach KG will also be able to enforce ACLs and expose auditing tools to the user.
Note that this issue is not about enforcing ACLs itself but rather coming up with API to register and retrieve resources from apps. Implementing ACL infrastructure will be covered in a separate issue.
Not to confuse with the Application Manager:
Users should be able to:
Helm Repository is described here in detail: https://helm.sh/docs/topics/chart_repository/
In short it is a web server where one can find
As a proof of concept AM stores list of installed applications in a local file. This will not scale as more and more applications are added to the system. Plus is not easily queryable by other parts of the infrastructure.
My idea is to store information about installed applications and their APIs into KG itself so it can be queried same way as any other information stored in KG.
This will require extending KG schema with new Application type encompassing everything about the app so it can be used in other parts of the infrastructure such as when enforcing ACLs.
Users should be able to override application chart values before installation.
Configuration properties can be found in values.yaml inside chart archive.
We should be able to parse at and present it on UI in a structured form where users can see their default values and override them if necessary.
Besides on-prem PCloud instances users might want to install them on public cloud providers such as GCE, AWS, Azure, ...
Easy to use installer package will be very helpful, especially when wanting to just try out the system without fully committing to it.
By introducing PCloud specific bazel flags and config settings we can make rules such as :push_to_dev to deploy to environment of developers choosing. Something like:
bazel run //core/api:push --pcloud_env=dev # this value being default
bazel run //core/api:push --pcloud_env=personal
https://docs.bazel.build/versions/master/configurable-attributes.html
https://docs.bazel.build/versions/master/be/general.html#config_setting
Right now it is hardcoded in the installation and build scripts to use ports 8080 for accessing PCloud services and 30500 for container registry on host machine.
Such configuration will cause an issue if host machine is already using same ports for some other reason.
Ideally such configuration will be handled by #26 but that's a bigger task, in the meantime we should probably default to some less common ports to reduce chance of collision with other applications.
PCloud needs some kind of discovery service so applications installed on the cluster can be found and reached from outside the cluster.
We should probably use a DNS server plus some custom logic to create DNS records for Ingress routes.
This DNS service then can be used on client devices to resolve application hostnames to cluster IP.
We can use PiHole for that but not sure if that is the best way to go. I can see Pihole as a standalone app and we need such DNS service regardless Pihole is installed or not.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.