Please read the AppBuilder State documentation.
Contributions are welcomed! Read the Contributing Guide for more information.
This project is licensed under the Apache V2 License. See LICENSE for more information.
Adobe App Builder State storage library
Home Page: https://developer.adobe.com/app-builder/docs/guides/application_state/#state
License: Apache License 2.0
Please read the AppBuilder State documentation.
Contributions are welcomed! Read the Contributing Guide for more information.
This project is licensed under the Apache V2 License. See LICENSE for more information.
429s from CosmosDB might arise if RU/s are consumed. State lib should wrap the error to hide the internal CosmosDB error and let the user catch the specific error based on a documented error code.
An optional retry mechanism could be directly implemented in state lib.
The error is unknown to state lib, hence it is not wrapped and the internal cosmosDB error is shown
"_internal": {
"stack": "Error: Message: {\"Errors\":[\"Request rate is large. More Request Units may be needed, so no changes were made. Please retry this request later. Learn more: http://aka.ms/cosmosdb-error-429\"]}\r\nActivityId:
[..]
If the user wants to implement an error mechanism he has to parse the private _internal
field of the error
using lib in typescript application it will build without error
Action Cleaning Done
node_modules/@adobe/aio-lib-state/types.d.ts:79:25 - error TS2552: Cannot find name 'StateLibError'. Did you mean 'StateLibErrors'?
79 ERROR_BAD_ARGUMENT: StateLibError;
~~~~~~~~~~~~~
node_modules/@adobe/aio-lib-state/types.d.ts:80:24 - error TS2552: Cannot find name 'StateLibError'. Did you mean 'StateLibErrors'?
80 ERROR_BAD_REQUEST: StateLibError;
~~~~~~~~~~~~~
node_modules/@adobe/aio-lib-state/types.d.ts:81:28 - error TS2552: Cannot find name 'StateLibError'. Did you mean 'StateLibErrors'?
81 ERROR_NOT_IMPLEMENTED: StateLibError;
~~~~~~~~~~~~~
node_modules/@adobe/aio-lib-state/types.d.ts:82:30 - error TS2552: Cannot find name 'StateLibError'. Did you mean 'StateLibErrors'?
82 ERROR_PAYLOAD_TOO_LARGE: StateLibError;
~~~~~~~~~~~~~
node_modules/@adobe/aio-lib-state/types.d.ts:83:28 - error TS2552: Cannot find name 'StateLibError'. Did you mean 'StateLibErrors'?
83 ERROR_BAD_CREDENTIALS: StateLibError;
~~~~~~~~~~~~~
node_modules/@adobe/aio-lib-state/types.d.ts:84:21 - error TS2552: Cannot find name 'StateLibError'. Did you mean 'StateLibErrors'?
84 ERROR_INTERNAL: StateLibError;
Build fails
typescript app and try to use the state lib in a function.
"typescript": "^4.6.4"
For the record this worked error free over a year ago.
System:
OS: Windows 10 10.0.22000
CPU: (20) x64 Intel(R) Xeon(R) W-2255 CPU @ 3.70GHz
Memory: 90.87 GB / 127.69 GB
Binaries:
Node: 14.19.1
Yarn: 1.22.18
npm: 6.14.16
Virtualization:
Docker: Not Found
npmGlobalPackages:
@adobe/aio-cli: Not Found
Proxies:
http: (not set)
https: (not set)
CLI plugins:
core:
@adobe/aio-cli 8.3.0
@adobe/aio-cli-plugin-app 8.6.1
@adobe/aio-cli-plugin-auth 2.6.0
@adobe/aio-cli-plugin-certificate 0.3.1
@adobe/aio-cli-plugin-config 3.0.1
@adobe/aio-cli-plugin-console 3.4.2
@adobe/aio-cli-plugin-events 1.1.6
@adobe/aio-cli-plugin-info 2.1.0
@adobe/aio-cli-plugin-runtime 5.4.0
@oclif/plugin-autocomplete 0.3.0
@oclif/plugin-help 3.3.1
@oclif/plugin-not-found 2.3.1
@oclif/plugin-plugins 1.10.11
@oclif/plugin-warn-if-update-available 1.7.3
user:
link:
tsconfig.json
`{
"compilerOptions": {
/* Basic Options */
"module": "commonjs", /Specify module code generation: "None", "CommonJS", "AMD", "System", "UMD", "ES6", "ES2015" or "ESNext". Only "AMD" and "System" can be used in conjunction with --outFile. "ES6" and "ES2015" values may be used when targeting "ES5" or lower./
"target": "es5", /Specify ECMAScript target version: "ES3" (default),"ES5","ES6"/"ES2015","ES2016","ES2017","ES2018","ES2019","ES2020","ESNext"/
//"lib": [ "es2015", "dom" ], /*List of library files to be included in the compilation. see https://www.typescriptlang.org/docs/handbook/compiler-options.html /
//"removeComments": true,
//"resolveJsonModule": true,
//"esModuleInterop": true,
//"typeRoots" : [
// "./node_modules/@types/",
// "./typings/"
//],
// "lib": [], / Specify library files to be included in the compilation. /
"allowJs": false, / Allow javascript files to be compiled. /
"checkJs": false, / Report errors in .js files. /
// "jsx": "preserve", / Specify JSX code generation: 'preserve', 'react-native', or 'react'. /
"declaration": false, / Generates corresponding '.d.ts' file. /
// "declarationMap": true, / Generates a sourcemap for each corresponding '.d.ts' file. /
"sourceMap": false, / Generates corresponding '.map' file. /
// "outFile": "./", / Concatenate and emit output to single file. /
"outDir": "actions", / Redirect output structure to the directory. /
// "rootDir": "./", / Specify the root directory of input files. Use to control the output directory structure with --outDir. /
// "composite": true, / Enable project compilation /
// "removeComments": true, / Do not emit comments to output. /
// "noEmit": true, / Do not emit outputs. /
//"importHelpers": true, / Import emit helpers from 'tslib'. /
// "downlevelIteration": true, / Provide full support for iterables in 'for-of', spread, and destructuring when targeting 'ES5' or 'ES3'. /
// "isolatedModules": true, / Transpile each file as a separate module (similar to 'ts.transpileModule'). */
/* Strict Type-Checking Options */
//"strict": true, /* Enable all strict type-checking options. */
"noImplicitAny": false, /* Raise error on expressions and declarations with an implied 'any' type. */
//"strictNullChecks": false, /* Enable strict null checks. */
// "strictFunctionTypes": true, /* Enable strict checking of function types. */
// "strictPropertyInitialization": true, /* Enable strict checking of property initialization in classes. */
// "noImplicitThis": true, /* Raise error on 'this' expressions with an implied 'any' type. */
// "alwaysStrict": true, /* Parse in strict mode and emit "use strict" for each source file. */
/* Additional Checks */
// "noUnusedLocals": true, /* Report errors on unused locals. */
// "noUnusedParameters": true, /* Report errors on unused parameters. */
// "noImplicitReturns": true, /* Report error when not all code paths in function return a value. */
// "noFallthroughCasesInSwitch": true, /* Report errors for fallthrough cases in switch statement. */
/* Module Resolution Options */
"moduleResolution": "node", /* Specify module resolution strategy: 'node' (Node.js) or 'classic' (TypeScript pre-1.6). */
// "baseUrl": "./", /* Base directory to resolve non-absolute module names. */
// "paths": {}, /* A series of entries which re-map imports to lookup locations relative to the 'baseUrl'. */
// "rootDirs": [], /* List of root folders whose combined content represents the structure of the project at runtime. */
// "typeRoots": [], /* List of folders to include type definitions from. */
// "types": ["jest", "node"], /* Type declaration files to be included in compilation. */
// "allowSyntheticDefaultImports": true, /* Allow default imports from modules with no default export. This does not affect code emit, just typechecking. */
"esModuleInterop": true /* Enables emit interoperability between CommonJS and ES Modules via creation of namespace objects for all imports. Implies 'allowSyntheticDefaultImports'. */
// "preserveSymlinks": true, /* Do not resolve the real path of symlinks. */
/* Source Map Options */
// "sourceRoot": "", /* Specify the location where debugger should locate TypeScript files instead of source locations. */
// "mapRoot": "", /* Specify the location where debugger should locate map files instead of generated locations. */
// "inlineSourceMap": true, /* Emit a single file with source maps instead of having a separate file. */
// "inlineSources": true, /* Emit the source alongside the sourcemaps within a single file; requires '--inlineSourceMap' or '--sourceMap' to be set. */
/* Experimental Options */
// "experimentalDecorators": true, /* Enables experimental support for ES7 decorators. */
// "emitDecoratorMetadata": true, /* Enables experimental support for emitting type metadata for decorators. */
},
"include": [
"./actions-src/**/*"
],
"exclude": [
"dev-keys"
]
}`
Is this a problem or do I not fully understand some type of definition rule or do i have a TypeScript check on I dont need.
To reduce costs and enhance performance use an in-memory cache in front of the provider.
Too many requests to aio-tvm/tvm/azure-cosmos
to the tune of 5k requests per sec hit metadata rate limit of cosmos db.
[@adobe/aio-tvm] error: Message: {"Errors":["The request did not complete due to a high rate of metadata requests. Increasing request units will have no impact and is not recommended. Please retry the request. https:\/\/aka.ms\/CosmosDB\/sql\/errors\/metadata-429"]}
We should explore reusing the cosmos client and hence reducing number of initializations which could be one of the causes for metadata rate limit.
Ref https://docs.microsoft.com/en-us/azure/cosmos-db/sql/troubleshoot-request-rate-too-large#rate-limiting-on-metadata-requests
The solution should take into account the concurrency of TVM actions.
Investigate support for multi-arch in distroless, and how we can package a multi-arch image.
Currently strong consistency is only supported for operations executed within one state instance (i.e. result of stateLib.init()
). This behavior is guaranteed by Cosmos DB session consistency model.
This can be easily extended to support strong consistency across instances within the same process by storing the cosmos session id into a static variable when initializing the first instance and reusing the same id for subsequent instances.
Currently VSCode intellisense parses the jsconfig.json
file and generates suggestions for files in lib/
.
While auto-completion works right now, the types.d.ts
file is not used
see adobe/aio-lib-customer-profile#5 (comment) on how to fix it
3.4.0
to 3.4.1
.π¨ View failing branch.
This version is covered by your current version range and after updating it in your project the build failed.
@azure/cosmos is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.
There is a collection of frequently asked questions. If those donβt help, you can always ask the humans behind Greenkeeper.
Your Greenkeeper Bot π΄
await state.get("?test", "test")
{
"error": "[StateLib:ERROR_FIREWALL] cannot access underlying DB provider because your IP is blocked by a firewall, please make sure to run in an Adobe I/O Runtime action"
}
await state.get("?test", "test")
The library doesn't throw an error on await state.put("?test")
, should it?
Works as described.
Works as described, with an extra Promise
in the chain.
N/A
N/A
N/A
N/A
N/A
In I/O Runtime, the first activation of an action is a cold start (which means the action code is downloaded to the docker container, npm modules are loaded into memory and library initialization are done).
In my test, the normal cold start time of a bare minimal action (only return a json with some texts) is approx. 1.5 to 2 seconds.
However, the exact same code containing state.init
to initialize the State SDK and state.get
to retrieve some value from State would double this overhead during cold start, i.e. 3 to 4 seconds.
Once the action container is warm, subsequent activations take less than 200ms, which means the actual retrieval state.get
does not take much time. That leads to state.init
being the biggest suspect of the huge overhead of up to 2s as described above.
Action code that resulted in approx. 4s cold start:
const { State } = require('@adobe/aio-sdk')
async function main (params) {
const state = await State.init()
const valueObj = await state.get(params.key)
let value = null
if (valueObj){
value = valueObj.value
}
const response = {
statusCode: 200,
body: {
key: params.key,
value
}
}
return response
}
exports.main = main
Same action code, without the use of State, resulted in approx. 2s cold start:
const { State } = require('@adobe/aio-sdk')
async function main (params) {
const value = 'Hello'
const response = {
statusCode: 200,
body: {
key: params.key,
value
}
}
return response
}
exports.main = main
If an SDK user wants to use his/her own cosmosDB account, the SDK is initiated as below:
const state = await stateLib.init({ cosmos: { endpoint, masterKey, databaseId, containerId, partitionKey } })
The cosmos DB, container and partition have to be set up correctly. Specifically, the key path of the partition must be /partitionKey
so that getting value would work. As a to-do task, we should either:
Related discussion: https://experienceleaguecommunities.adobe.com/t5/project-firefly-questions/access-to-own-storage-using-aio-lib-state-and-aio-lib-action/qaq-p/378249/
const state = await stateLib.init() // => autocompletes
state. // => doesn't autocomplete
vscode's intellisense uses typescript types which seem to be compatible/inferred from our JSDoc. But there might be some sort of conflict.
The following code can be used to get a list of all keys, but it exposes the inner workings of the state-lib. It would be nice to build this into the api so it is just an opaque-box.
async state.listKeys(continuationToken:string = '')
returning {
keys: [],
continuationToken: ''
}
const stateLib = require('@adobe/aio-lib-state')
const state = await stateLib.init()
const queryPlan = state._cosmos.container.items.query(`SELECT c.id,c.ttl,c._ts from c where c.partitionKey='${state._cosmos.partitionKey}'`, {
initialHeaders: {
'x-ms-documentdb-partitionkey': `["${state._cosmos.partitionKey}"]`
},
continuationToken: params.continuationToken
})
const queryRes = await queryPlan.fetchNext()
const res = queryRes.resources.map(x => ({ key: x.id, expiration: getExpiration(x.ttl, x._ts) }))
const response = {
statusCode: 200,
// todo check the continuation token
body: {
message: 'success',
keys: res,
hasMoreResults: queryRes.hasMoreResults,
continuationToken: queryPlan.continuationToken
} // default is 24hours
}
See investigations under #63
Under a cold start, a state.get will take approx less than a second.
Under a cold start, a state.get will take approx 1800ms.
On a warm start, a state.get will still take approx 450ms
The @azure/cosmos
promise that is resolved here, takes up 99.9% of the time for a state.get
call:
aio-lib-state/lib/impl/CosmosStateStore.js
Line 128 in e296e3e
I don't think there can be any more code optimizations possible here since it seems the bottleneck is the CosmosDB read call. The only possible solutions I can see are:
During a load test against one of our actions where we're using State
from aio-lib-state to cache a token for reuse, we observed that on very rare occasions, the state.get
call takes just a little over 60s to respond with the requested value. This leads to developer errors due to the action exceeding the default timeout
of 60s.
Here are a few activation IDs for which this was observed:
ad65d2b56866448aa5d2b56866848ab4
ea2d8cee79ca4841ad8cee79cad841c2
61d28c62fad0444a928c62fad0d44a1f
While using state in an action to cache a token for reuse, I noticed that initializing the state lib via State.init()
sometimes takes a very long time on cold starts.
The time it takes to init ranges from a few seconds up until 30s worst case so far! Following an excerpt of cold start activations that show long init times. Of course, the time it takes for the cold start and some internal operations need to be discounted (~2-3s) but the major part is taken up by State.init()
.
When I use aio-lib-state
from outside my company's firewall, Cosmos DB reports an error Request originated from client IP x.x.x.x This is blocked by your Cosmos DB account firewall settings
. This circumstance should be somehow reflected in the exception returned from aio-lib-state
.
I'm getting an exception with message [StateLib:ERROR_BAD_CREDENTIALS] cannot access underlying DB provider
. This gives the impression that the credentials are bad and leads me up the wrong track.
Outside company firewall that has access rules.
Run code locally that uses aio-lib-state
.
macOS 10.15.3, node v10.18.0
const stateLib = require('@adobe/aio-lib-state');
function main(params) {
const state = await stateLib.init();
const obj = await state.get('foo');
}
Error: Illegal characters ['/', '\\', '?', '#'] cannot be used in resourceId
is reported in the logs.
[StateLib:ERROR_INTERNAL] unknown error response from provider with status: unknown
is reported in the logs.
try to get/put using a key such as "abc/def
"
2021-06-30T12:27:31.615Z stdout: 2021-06-30T12:27:31.615Z @adobe/aio-lib-state:debug got internal error with status undefined: Illegal characters ['/', '\', '?', '#'] cannot be used in resourceId
2021-06-30T12:27:31.632Z stdout: 2021-06-30T12:27:31.632Z @adobe/aio-lib-state:error {
"sdk": "StateLib",
"sdkDetails": {
"key": "titan/task/update/adoberm",
"_internal": {
"stack": "Error: Illegal characters ['/', '\\', '?', '#'] cannot be used in resourceId\n at validateResourceId (/nodejsAction/4tfYwswP/index.js:56177:15)\n at createDocumentUri (/nodejsAction/4tfYwswP/index.js:56325:5)\n at Item.get url [as url] (/nodejsAction/4tfYwswP/index.js:59961:16)\n at Item.<anonymous> (/nodejsAction/4tfYwswP/index.js:59993:47)\n at Generator.next (<anonymous>)\n at /nodejsAction/4tfYwswP/index.js:178946:71\n at new Promise (<anonymous>)\n at Module.__awaiter (/nodejsAction/4tfYwswP/index.js:178942:12)\n at Item.read (/nodejsAction/4tfYwswP/index.js:59988:22)\n at CosmosStateStore._get (/nodejsAction/4tfYwswP/index.js:38789:94)",
"message": "Illegal characters ['/', '\\', '?', '#'] cannot be used in resourceId"
}
},
"code": "ERROR_INTERNAL",
"message": "[StateLib:ERROR_INTERNAL] unknown error response from provider with status: unknown",
"stacktrace": "StateLibError: [StateLib:ERROR_INTERNAL] unknown error response from provider with status: unknown\n at new <anonymous> (/nodejsAction/4tfYwswP/index.js:20660:9)\n at _wrap (/nodejsAction/4tfYwswP/index.js:38705:17)"
}
how does the state sdk behave in case of rd after write?
what happens in case of wr after wr?
When trying to put
new data with a ttl
being a number as a string, I'd expect a validation error to be thrown.
No validation error is thrown, and a generic error message with no further details is thrown.
According to @shazron, javascript coerces the string to a number but only for validation purposes.
ttl
of '300'
(string)aio-lib-state 1.1.1
nodeJS 14
const State = require('@adobe/aio-lib-state');
const state = await State.init();
await state.put('key', 'value', { ttl: '300' });
StateLibError: [StateLib:ERROR_INTERNAL] unknown error response from provider with status: 400
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.