Coder Social home page Coder Social logo

sdk-python's Introduction

Temporal Python SDK

Python 3.8+ PyPI MIT

Temporal is a distributed, scalable, durable, and highly available orchestration engine used to execute asynchronous, long-running business logic in a scalable and resilient way.

"Temporal Python SDK" is the framework for authoring workflows and activities using the Python programming language.

Also see:

In addition to features common across all Temporal SDKs, the Python SDK also has the following interesting features:

Type Safe

This library uses the latest typing and MyPy support with generics to ensure all calls can be typed. For example, starting a workflow with an int parameter when it accepts a str parameter would cause MyPy to fail.

Different Activity Types

The activity worker has been developed to work with async def, threaded, and multiprocess activities. While async def activities are the easiest and recommended, care has been taken to make heartbeating and cancellation also work across threads/processes.

Custom asyncio Event Loop

The workflow implementation basically turns async def functions into workflows backed by a distributed, fault-tolerant event loop. This means task management, sleep, cancellation, etc have all been developed to seamlessly integrate with asyncio concepts.

See the blog post introducing the Python SDK for an informal introduction to the features and their implementation.


Contents

Quick Start

We will guide you through the Temporal basics to create a "hello, world!" script on your machine. It is not intended as one of the ways to use Temporal, but in reality it is very simplified and decidedly not "the only way" to use Temporal. For more information, check out the docs references in "Next Steps" below the quick start.

Installation

Install the temporalio package from PyPI.

These steps can be followed to use with a virtual environment and pip:

  • Create a virtual environment
  • Update pip - python -m pip install -U pip
    • Needed because older versions of pip may not pick the right wheel
  • Install Temporal SDK - python -m pip install temporalio

The SDK is now ready for use. To build from source, see "Building" near the end of this documentation.

NOTE: This README is for the current branch and not necessarily what's released on PyPI.

Implementing a Workflow

Create the following in activities.py:

from temporalio import activity

@activity.defn
def say_hello(name: str) -> str:
    return f"Hello, {name}!"

Create the following in workflows.py:

from datetime import timedelta
from temporalio import workflow

# Import our activity, passing it through the sandbox
with workflow.unsafe.imports_passed_through():
    from .activities import say_hello

@workflow.defn
class SayHello:
    @workflow.run
    async def run(self, name: str) -> str:
        return await workflow.execute_activity(
            say_hello, name, schedule_to_close_timeout=timedelta(seconds=5)
        )

Create the following in run_worker.py:

import asyncio
import concurrent.futures
from temporalio.client import Client
from temporalio.worker import Worker

# Import the activity and workflow from our other files
from .activities import say_hello
from .workflows import SayHello

async def main():
    # Create client connected to server at the given address
    client = await Client.connect("localhost:7233")

    # Run the worker
    with concurrent.futures.ThreadPoolExecutor(max_workers=100) as activity_executor:
        worker = Worker(
          client,
          task_queue="my-task-queue",
          workflows=[SayHello],
          activities=[say_hello],
          activity_executor=activity_executor,
        )
        await worker.run()

if __name__ == "__main__":
    asyncio.run(main())

Assuming you have a Temporal server running on localhost, this will run the worker:

python run_worker.py

Running a Workflow

Create the following script at run_workflow.py:

import asyncio
from temporalio.client import Client

# Import the workflow from the previous code
from .workflows import SayHello

async def main():
    # Create client connected to server at the given address
    client = await Client.connect("localhost:7233")

    # Execute a workflow
    result = await client.execute_workflow(SayHello.run, "my name", id="my-workflow-id", task_queue="my-task-queue")

    print(f"Result: {result}")

if __name__ == "__main__":
    asyncio.run(main())

Assuming you have run_worker.py running from before, this will run the workflow:

python run_workflow.py

The output will be:

Result: Hello, my-name!

Next Steps

Temporal can be implemented in your code in many different ways, to suit your application's needs. The links below will give you much more information about how Temporal works with Python:

  • Code Samples - If you want to start with some code, we have provided some pre-built samples.
  • Application Development Guide Our Python specific Developer's Guide will give you much more information on how to build with Temporal in your Python applications than our SDK README ever could (or should).
  • API Documentation - Full Temporal Python SDK package documentation.

Usage

From here, you will find reference documentation about specific pieces of the Temporal Python SDK that were built around Temporal concepts. This section is not intended as a how-to guide -- For more how-to oriented information, check out the links in the Next Steps section above.

Client

A client can be created and used to start a workflow like so:

from temporalio.client import Client

async def main():
    # Create client connected to server at the given address and namespace
    client = await Client.connect("localhost:7233", namespace="my-namespace")

    # Start a workflow
    handle = await client.start_workflow(MyWorkflow.run, "some arg", id="my-workflow-id", task_queue="my-task-queue")

    # Wait for result
    result = await handle.result()
    print(f"Result: {result}")

Some things to note about the above code:

  • A Client does not have an explicit "close"
  • To enable TLS, the tls argument to connect can be set to True or a TLSConfig object
  • A single positional argument can be passed to start_workflow. If there are multiple arguments, only the non-type-safe form of start_workflow can be used (i.e. the one accepting a string workflow name) and it must be in the args keyword argument.
  • The handle represents the workflow that was started and can be used for more than just getting the result
  • Since we are just getting the handle and waiting on the result, we could have called client.execute_workflow which does the same thing
  • Clients can have many more options not shown here (e.g. data converters and interceptors)
  • A string can be used instead of the method reference to call a workflow by name (e.g. if defined in another language)
  • Clients do not work across forks

Clients also provide a shallow copy of their config for use in making slightly different clients backed by the same connection. For instance, given the client above, this is how to have a client in another namespace:

config = client.config()
config["namespace"] = "my-other-namespace"
other_ns_client = Client(**config)

Data Conversion

Data converters are used to convert raw Temporal payloads to/from actual Python types. A custom data converter of type temporalio.converter.DataConverter can be set via the data_converter client parameter. Data converters are a combination of payload converters, payload codecs, and failure converters. Payload converters convert Python values to/from serialized bytes. Payload codecs convert bytes to bytes (e.g. for compression or encryption). Failure converters convert exceptions to/from serialized failures.

The default data converter supports converting multiple types including:

  • None
  • bytes
  • google.protobuf.message.Message - As JSON when encoding, but has ability to decode binary proto from other languages
  • Anything that can be converted to JSON including:
    • Anything that json.dump supports natively
    • dataclasses
    • Iterables including ones JSON dump may not support by default, e.g. set
    • Any class with a dict() method and a static parse_obj() method, e.g. Pydantic models
      • The default data converter is deprecated for Pydantic models and will warn if used since not all fields work. See this sample for the recommended approach.
    • IntEnum, StrEnum based enumerates
    • UUID

This notably doesn't include any date, time, or datetime objects as they may not work across SDKs.

Users are strongly encouraged to use a single dataclass for parameter and return types so fields with defaults can be easily added without breaking compatibility.

Classes with generics may not have the generics properly resolved. The current implementation does not have generic type resolution. Users should use concrete types.

Custom Type Data Conversion

For converting from JSON, the workflow/activity type hint is taken into account to convert to the proper type. Care has been taken to support all common typings including Optional, Union, all forms of iterables and mappings, NewType, etc in addition to the regular JSON values mentioned before.

Data converters contain a reference to a payload converter class that is used to convert to/from payloads/values. This is a class and not an instance because it is instantiated on every workflow run inside the sandbox. The payload converter is usually a CompositePayloadConverter which contains a multiple EncodingPayloadConverters it uses to try to serialize/deserialize payloads. Upon serialization, each EncodingPayloadConverter is tried until one succeeds. The EncodingPayloadConverter provides an "encoding" string serialized onto the payload so that, upon deserialization, the specific EncodingPayloadConverter for the given "encoding" is used.

The default data converter uses the DefaultPayloadConverter which is simply a CompositePayloadConverter with a known set of default EncodingPayloadConverters. To implement a custom encoding for a custom type, a new EncodingPayloadConverter can be created for the new type. For example, to support IPv4Address types:

class IPv4AddressEncodingPayloadConverter(EncodingPayloadConverter):
    @property
    def encoding(self) -> str:
        return "text/ipv4-address"

    def to_payload(self, value: Any) -> Optional[Payload]:
        if isinstance(value, ipaddress.IPv4Address):
            return Payload(
                metadata={"encoding": self.encoding.encode()},
                data=str(value).encode(),
            )
        else:
            return None

    def from_payload(self, payload: Payload, type_hint: Optional[Type] = None) -> Any:
        assert not type_hint or type_hint is ipaddress.IPv4Address
        return ipaddress.IPv4Address(payload.data.decode())

class IPv4AddressPayloadConverter(CompositePayloadConverter):
    def __init__(self) -> None:
        # Just add ours as first before the defaults
        super().__init__(
            IPv4AddressEncodingPayloadConverter(),
            *DefaultPayloadConverter.default_encoding_payload_converters,
        )

my_data_converter = dataclasses.replace(
    DataConverter.default,
    payload_converter_class=IPv4AddressPayloadConverter,
)

Imports are left off for brevity.

This is good for many custom types. However, sometimes you want to override the behavior of the just the existing JSON encoding payload converter to support a new type. It is already the last encoding data converter in the list, so it's the fall-through behavior for any otherwise unknown type. Customizing the existing JSON converter has the benefit of making the type work in lists, unions, etc.

The JSONPlainPayloadConverter uses the Python json library with an advanced JSON encoder by default and a custom value conversion method to turn json.loaded values to their type hints. The conversion can be customized for serialization with a custom json.JSONEncoder and deserialization with a custom JSONTypeConverter. For example, to support IPv4Address types in existing JSON conversion:

class IPv4AddressJSONEncoder(AdvancedJSONEncoder):
    def default(self, o: Any) -> Any:
        if isinstance(o, ipaddress.IPv4Address):
            return str(o)
        return super().default(o)
class IPv4AddressJSONTypeConverter(JSONTypeConverter):
    def to_typed_value(
        self, hint: Type, value: Any
    ) -> Union[Optional[Any], _JSONTypeConverterUnhandled]:
        if issubclass(hint, ipaddress.IPv4Address):
            return ipaddress.IPv4Address(value)
        return JSONTypeConverter.Unhandled

class IPv4AddressPayloadConverter(CompositePayloadConverter):
    def __init__(self) -> None:
        # Replace default JSON plain with our own that has our encoder and type
        # converter
        json_converter = JSONPlainPayloadConverter(
            encoder=IPv4AddressJSONEncoder,
            custom_type_converters=[IPv4AddressJSONTypeConverter()],
        )
        super().__init__(
            *[
                c if not isinstance(c, JSONPlainPayloadConverter) else json_converter
                for c in DefaultPayloadConverter.default_encoding_payload_converters
            ]
        )

my_data_converter = dataclasses.replace(
    DataConverter.default,
    payload_converter_class=IPv4AddressPayloadConverter,
)

Now IPv4Address can be used in type hints including collections, optionals, etc.

Workers

Workers host workflows and/or activities. Here's how to run a worker:

import asyncio
import logging
from temporalio.client import Client
from temporalio.worker import Worker
# Import your own workflows and activities
from my_workflow_package import MyWorkflow, my_activity

async def run_worker(stop_event: asyncio.Event):
    # Create client connected to server at the given address
    client = await Client.connect("localhost:7233", namespace="my-namespace")

    # Run the worker until the event is set
    worker = Worker(client, task_queue="my-task-queue", workflows=[MyWorkflow], activities=[my_activity])
    async with worker:
        await stop_event.wait()

Some things to note about the above code:

  • This creates/uses the same client that is used for starting workflows
  • While this example accepts a stop event and uses async with, run() and shutdown() may be used instead
  • Workers can have many more options not shown here (e.g. data converters and interceptors)

Workflows

Definition

Workflows are defined as classes decorated with @workflow.defn. The method invoked for the workflow is decorated with @workflow.run. Methods for signals, queries, and updates are decorated with @workflow.signal, @workflow.query and @workflow.update respectively. Here's an example of a workflow:

import asyncio
from datetime import timedelta
from temporalio import workflow

# Pass the activities through the sandbox
with workflow.unsafe.imports_passed_through():
    from .my_activities import GreetingInfo, create_greeting_activity

@workflow.defn
class GreetingWorkflow:
    def __init__() -> None:
        self._current_greeting = "<unset>"
        self._greeting_info = GreetingInfo()
        self._greeting_info_update = asyncio.Event()
        self._complete = asyncio.Event()

    @workflow.run
    async def run(self, name: str) -> str:
        self._greeting_info.name = name
        while True:
            # Store greeting
            self._current_greeting = await workflow.execute_activity(
                create_greeting_activity,
                self._greeting_info,
                start_to_close_timeout=timedelta(seconds=5),
            )
            workflow.logger.debug("Greeting set to %s", self._current_greeting)
            
            # Wait for salutation update or complete signal (this can be
            # cancelled)
            await asyncio.wait(
                [
                    asyncio.create_task(self._greeting_info_update.wait()),
                    asyncio.create_task(self._complete.wait()),
                ],
                return_when=asyncio.FIRST_COMPLETED,
            )
            if self._complete.is_set():
                return self._current_greeting
            self._greeting_info_update.clear()

    @workflow.signal
    async def update_salutation(self, salutation: str) -> None:
        self._greeting_info.salutation = salutation
        self._greeting_info_update.set()

    @workflow.signal
    async def complete_with_greeting(self) -> None:
        self._complete.set()

    @workflow.query
    def current_greeting(self) -> str:
        return self._current_greeting
    
    @workflow.update
    def set_and_get_greeting(self, greeting: str) -> str:
      old = self._current_greeting
      self._current_greeting = greeting
      return old

This assumes there's an activity in my_activities.py like:

from dataclasses import dataclass
from temporalio import workflow

@dataclass
class GreetingInfo:
    salutation: str = "Hello"
    name: str = "<unknown>"

@activity.defn
def create_greeting_activity(info: GreetingInfo) -> str:
    return f"{info.salutation}, {info.name}!"

Some things to note about the above workflow code:

  • Workflows run in a sandbox by default.
    • Users are encouraged to define workflows in files with no side effects or other complicated code or unnecessary imports to other third party libraries.
    • Non-standard-library, non-temporalio imports should usually be "passed through" the sandbox. See the Workflow Sandbox section for more details.
  • This workflow continually updates the queryable current greeting when signalled and can complete with the greeting on a different signal
  • Workflows are always classes and must have a single @workflow.run which is an async def function
  • Workflow code must be deterministic. This means no threading, no randomness, no external calls to processes, no network IO, and no global state mutation. All code must run in the implicit asyncio event loop and be deterministic.
  • @activity.defn is explained in a later section. For normal simple string concatenation, this would just be done in the workflow. The activity is for demonstration purposes only.
  • workflow.execute_activity(create_greeting_activity, ... is actually a typed signature, and MyPy will fail if the self._greeting_info parameter is not a GreetingInfo

Here are the decorators that can be applied:

  • @workflow.defn - Defines a workflow class
    • Must be defined on the class given to the worker (ignored if present on a base class)
    • Can have a name param to customize the workflow name, otherwise it defaults to the unqualified class name
    • Can have dynamic=True which means all otherwise unhandled workflows fall through to this. If present, cannot have name argument, and run method must accept a single parameter of Sequence[temporalio.common.RawValue] type. The payload of the raw value can be converted via workflow.payload_converter().from_payload.
  • @workflow.run - Defines the primary workflow run method
    • Must be defined on the same class as @workflow.defn, not a base class (but can also be defined on the same method of a base class)
    • Exactly one method name must have this decorator, no more or less
    • Must be defined on an async def method
    • The method's arguments are the workflow's arguments
    • The first parameter must be self, followed by positional arguments. Best practice is to only take a single argument that is an object/dataclass of fields that can be added to as needed.
  • @workflow.signal - Defines a method as a signal
    • Can be defined on an async or non-async function at any hierarchy depth, but if decorated method is overridden, the override must also be decorated
    • The method's arguments are the signal's arguments
    • Can have a name param to customize the signal name, otherwise it defaults to the unqualified method name
    • Can have dynamic=True which means all otherwise unhandled signals fall through to this. If present, cannot have name argument, and method parameters must be self, a string signal name, and a Sequence[temporalio.common.RawValue].
    • Non-dynamic method can only have positional arguments. Best practice is to only take a single argument that is an object/dataclass of fields that can be added to as needed.
    • Return value is ignored
  • @workflow.query - Defines a method as a query
    • All the same constraints as @workflow.signal but should return a value
    • Should not be async
    • Temporal queries should never mutate anything in the workflow or call any calls that would mutate the workflow
  • @workflow.update - Defines a method as an update
    • May both accept as input and return a value
    • May be async or non-async
    • May mutate workflow state, and make calls to other workflow APIs like starting activities, etc.
    • Also accepts the name and dynamic parameters like signals and queries, with the same semantics.
    • Update handlers may optionally define a validator method by decorating it with @update_handler_method.validator. To reject an update before any events are written to history, throw an exception in a validator. Validators cannot be async, cannot mutate workflow state, and return nothing.

Running

To start a locally-defined workflow from a client, you can simply reference its method like so:

from temporalio.client import Client
from my_workflow_package import GreetingWorkflow

async def create_greeting(client: Client) -> str:
    # Start the workflow
    handle = await client.start_workflow(GreetingWorkflow.run, "my name", id="my-workflow-id", task_queue="my-task-queue")
    # Change the salutation
    await handle.signal(GreetingWorkflow.update_salutation, "Aloha")
    # Tell it to complete
    await handle.signal(GreetingWorkflow.complete_with_greeting)
    # Wait and return result
    return await handle.result()

Some things to note about the above code:

  • This uses the GreetingWorkflow from the previous section
  • The result of calling this function is "Aloha, my name!"
  • id and task_queue are required for running a workflow
  • client.start_workflow is typed, so MyPy would fail if "my name" were something besides a string
  • handle.signal is typed, so MyPy would fail if "Aloha" were something besides a string or if we provided a parameter to the parameterless complete_with_greeting
  • handle.result is typed to the workflow itself, so MyPy would fail if we said this create_greeting returned something besides a string

Invoking Activities

  • Activities are started with non-async workflow.start_activity() which accepts either an activity function reference or a string name.
  • A single argument to the activity is positional. Multiple arguments are not supported in the type-safe form of start/execute activity and must be supplied via the args keyword argument.
  • Activity options are set as keyword arguments after the activity arguments. At least one of start_to_close_timeout or schedule_to_close_timeout must be provided.
  • The result is an activity handle which is an asyncio.Task and supports basic task features
  • An async workflow.execute_activity() helper is provided which takes the same arguments as workflow.start_activity() and awaits on the result. This should be used in most cases unless advanced task capabilities are needed.
  • Local activities work very similarly except the functions are workflow.start_local_activity() and workflow.execute_local_activity()
    • ⚠️Local activities are currently experimental
  • Activities can be methods of a class. Invokers should use workflow.start_activity_method(), workflow.execute_activity_method(), workflow.start_local_activity_method(), and workflow.execute_local_activity_method() instead.
  • Activities can callable classes (i.e. that define __call__). Invokers should use workflow.start_activity_class(), workflow.execute_activity_class(), workflow.start_local_activity_class(), and workflow.execute_local_activity_class() instead.

Invoking Child Workflows

  • Child workflows are started with async workflow.start_child_workflow() which accepts either a workflow run method reference or a string name. The arguments to the workflow are positional.
  • A single argument to the child workflow is positional. Multiple arguments are not supported in the type-safe form of start/execute child workflow and must be supplied via the args keyword argument.
  • Child workflow options are set as keyword arguments after the arguments. At least id must be provided.
  • The await of the start does not complete until the start has been accepted by the server
  • The result is a child workflow handle which is an asyncio.Task and supports basic task features. The handle also has some child info and supports signalling the child workflow
  • An async workflow.execute_child_workflow() helper is provided which takes the same arguments as workflow.start_child_workflow() and awaits on the result. This should be used in most cases unless advanced task capabilities are needed.

Timers

  • A timer is represented by normal asyncio.sleep()
  • Timers are also implicitly started on any asyncio calls with timeouts (e.g. asyncio.wait_for)
  • Timers are Temporal server timers, not local ones, so sub-second resolution rarely has value
  • Calls that use a specific point in time, e.g. call_at or timeout_at, should be based on the current loop time (i.e. workflow.time()) and not an actual point in time. This is because fixed times are translated to relative ones by subtracting the current loop time which may not be the actual current time.

Conditions

  • workflow.wait_condition is an async function that doesn't return until a provided callback returns true
  • A timeout can optionally be provided which will throw a asyncio.TimeoutError if reached (internally backed by asyncio.wait_for which uses a timer)

Asyncio and Cancellation

Workflows are backed by a custom asyncio event loop. This means many of the common asyncio calls work as normal. Some asyncio features are disabled such as:

  • Thread related calls such as to_thread(), run_coroutine_threadsafe(), loop.run_in_executor(), etc
  • Calls that alter the event loop such as loop.close(), loop.stop(), loop.run_forever(), loop.set_task_factory(), etc
  • Calls that use anything external such as networking, subprocesses, disk IO, etc

Cancellation is done the same way as asyncio. Specifically, a task can be requested to be cancelled but does not necessarily have to respect that cancellation immediately. This also means that asyncio.shield() can be used to protect against cancellation. The following tasks, when cancelled, perform a Temporal cancellation:

  • Activities - when the task executing an activity is cancelled, a cancellation request is sent to the activity
  • Child workflows - when the task starting or executing a child workflow is cancelled, a cancellation request is sent to cancel the child workflow
  • Timers - when the task executing a timer is cancelled (whether started via sleep or timeout), the timer is cancelled

When the workflow itself is requested to cancel, Task.cancel is called on the main workflow task. Therefore, asyncio.CancelledError can be caught in order to handle the cancel gracefully.

Workflows follow asyncio cancellation rules exactly which can cause confusion among Python developers. Cancelling a task doesn't always cancel the thing it created. For example, given task = asyncio.create_task(workflow.start_child_workflow(..., calling task.cancel does not cancel the child workflow, it only cancels the starting of it, which has no effect if it has already started. However, cancelling the result of handle = await workflow.start_child_workflow(... or task = asyncio.create_task(workflow.execute_child_workflow(... does cancel the child workflow.

Also, due to Temporal rules, a cancellation request is a state not an event. Therefore, repeated cancellation requests are not delivered, only the first. If the workflow chooses swallow a cancellation, it cannot be requested again.

Workflow Utilities

While running in a workflow, in addition to features documented elsewhere, the following items are available from the temporalio.workflow package:

  • continue_as_new() - Async function to stop the workflow immediately and continue as new
  • info() - Returns information about the current workflow
  • logger - A logger for use in a workflow (properly skips logging on replay)
  • now() - Returns the "current time" from the workflow's perspective

Exceptions

  • Workflows can raise exceptions to fail the workflow or the "workflow task" (i.e. suspend the workflow retrying).
  • Exceptions that are instances of temporalio.exceptions.FailureError will fail the workflow with that exception
    • For failing the workflow explicitly with a user exception, use temporalio.exceptions.ApplicationError. This can be marked non-retryable or include details as needed.
    • Other exceptions that come from activity execution, child execution, cancellation, etc are already instances of FailureError and will fail the workflow when uncaught.
  • All other exceptions fail the "workflow task" which means the workflow will continually retry until the workflow is fixed. This is helpful for bad code or other non-predictable exceptions. To actually fail the workflow, use an ApplicationError as mentioned above.

External Workflows

  • workflow.get_external_workflow_handle() inside a workflow returns a handle to interact with another workflow
  • workflow.get_external_workflow_handle_for() can be used instead for a type safe handle
  • await handle.signal() can be called on the handle to signal the external workflow
  • await handle.cancel() can be called on the handle to send a cancel to the external workflow

Testing

Workflow testing can be done in an integration-test fashion against a real server, however it is hard to simulate timeouts and other long time-based code. Using the time-skipping workflow test environment can help there.

The time-skipping temporalio.testing.WorkflowEnvironment can be created via the static async start_time_skipping(). This internally downloads the Temporal time-skipping test server to a temporary directory if it doesn't already exist, then starts the test server which has special APIs for skipping time.

NOTE: The time-skipping test environment does not work on ARM. The SDK will try to download the x64 binary on macOS for use with the Intel emulator, but for Linux or Windows ARM there is no proper time-skipping test server at this time.

Automatic Time Skipping

Anytime a workflow result is waited on, the time-skipping server automatically advances to the next event it can. To manually advance time before waiting on the result of a workflow, the WorkflowEnvironment.sleep method can be used.

Here's a simple example of a workflow that sleeps for 24 hours:

import asyncio
from temporalio import workflow

@workflow.defn
class WaitADayWorkflow:
    @workflow.run
    async def run(self) -> str:
        await asyncio.sleep(24 * 60 * 60)
        return "all done"

An integration test of this workflow would be way too slow. However the time-skipping server automatically skips to the next event when we wait on the result. Here's a test for that workflow:

from temporalio.testing import WorkflowEnvironment
from temporalio.worker import Worker

async def test_wait_a_day_workflow():
    async with await WorkflowEnvironment.start_time_skipping() as env:
        async with Worker(env.client, task_queue="tq1", workflows=[WaitADayWorkflow]):
            assert "all done" == await env.client.execute_workflow(WaitADayWorkflow.run, id="wf1", task_queue="tq1")

That test will run almost instantly. This is because by calling execute_workflow on our client, we have asked the environment to automatically skip time as much as it can (basically until the end of the workflow or until an activity is run).

To disable automatic time-skipping while waiting for a workflow result, run code inside a with env.auto_time_skipping_disabled(): block.

Manual Time Skipping

Until a workflow is waited on, all time skipping in the time-skipping environment is done manually via WorkflowEnvironment.sleep.

Here's workflow that waits for a signal or times out:

import asyncio
from temporalio import workflow

@workflow.defn
class SignalWorkflow:
    def __init__(self) -> None:
        self.signal_received = False

    @workflow.run
    async def run(self) -> str:
        # Wait for signal or timeout in 45 seconds
        try:
            await workflow.wait_condition(lambda: self.signal_received, timeout=45)
            return "got signal"
        except asyncio.TimeoutError:
            return "got timeout"

    @workflow.signal
    def some_signal(self) -> None:
        self.signal_received = True

To test a normal signal, you might:

from temporalio.testing import WorkflowEnvironment
from temporalio.worker import Worker

async def test_signal_workflow():
    async with await WorkflowEnvironment.start_time_skipping() as env:
        async with Worker(env.client, task_queue="tq1", workflows=[SignalWorkflow]):
            # Start workflow, send signal, check result
            handle = await env.client.start_workflow(SignalWorkflow.run, id="wf1", task_queue="tq1")
            await handle.signal(SignalWorkflow.some_signal)
            assert "got signal" == await handle.result()

But how would you test the timeout part? Like so:

from temporalio.testing import WorkflowEnvironment
from temporalio.worker import Worker

async def test_signal_workflow_timeout():
    async with await WorkflowEnvironment.start_time_skipping() as env:
        async with Worker(env.client, task_queue="tq1", workflows=[SignalWorkflow]):
            # Start workflow, advance time past timeout, check result
            handle = await env.client.start_workflow(SignalWorkflow.run, id="wf1", task_queue="tq1")
            await env.sleep(50)
            assert "got timeout" == await handle.result()

Also, the current time of the workflow environment can be obtained via the async WorkflowEnvironment.get_current_time method.

Mocking Activities

Activities are just functions decorated with @activity.defn. Simply write different ones and pass those to the worker to have different activities called during the test.

Workflow Sandbox

By default workflows are run in a sandbox to help avoid non-deterministic code. If a call that is known to be non-deterministic is performed, an exception will be thrown in the workflow which will "fail the task" which means the workflow will not progress until fixed.

The sandbox is not foolproof and non-determinism can still occur. It is simply a best-effort way to catch bad code early. Users are encouraged to define their workflows in files with no other side effects.

The sandbox offers a mechanism to pass through modules from outside the sandbox. By default this already includes all standard library modules and Temporal modules. For performance and behavior reasons, users are encouraged to pass through all third party modules whose calls will be deterministic. This includes modules containing the activities to be referenced in workflows. See "Passthrough Modules" below on how to do this.

If you are getting an error like:

temporalio.worker.workflow_sandbox._restrictions.RestrictedWorkflowAccessError: Cannot access http.client.IncompleteRead.__mro_entries__ from inside a workflow. If this is code from a module not used in a workflow or known to only be used deterministically from a workflow, mark the import as pass through.

Then you are either using an invalid construct from the workflow, this is a known limitation of the sandbox, or most commonly this is from a module that is safe to pass through (see "Passthrough Modules" section below).

How the Sandbox Works

The sandbox is made up of two components that work closely together:

  • Global state isolation
  • Restrictions preventing known non-deterministic library calls

Global state isolation is performed by using exec. Upon workflow start, the file that the workflow is defined in is imported into a new sandbox created for that workflow run. In order to keep the sandbox performant a known set of "passthrough modules" are passed through from outside of the sandbox when they are imported. These are expected to be side-effect free on import and have their non-deterministic aspects restricted. By default the entire Python standard library, temporalio, and a couple of other modules are passed through from outside of the sandbox. To update this list, see "Customizing the Sandbox".

Restrictions preventing known non-deterministic library calls are achieved using proxy objects on modules wrapped around the custom importer set in the sandbox. Many restrictions apply at workflow import time and workflow run time, while some restrictions only apply at workflow run time. A default set of restrictions is included that prevents most dangerous standard library calls. However it is known in Python that some otherwise-non-deterministic invocations, like reading a file from disk via open or using os.environ, are done as part of importing modules. To customize what is and isn't restricted, see "Customizing the Sandbox".

Avoiding the Sandbox

There are three increasingly-scoped ways to avoid the sandbox. Users are discouraged from avoiding the sandbox if possible.

To remove restrictions around a particular block of code, use with temporalio.workflow.unsafe.sandbox_unrestricted():. The workflow will still be running in the sandbox, but no restrictions for invalid library calls will be applied.

To run an entire workflow outside of a sandbox, set sandboxed=False on the @workflow.defn decorator when defining it. This will run the entire workflow outside of the workflow which means it can share global state and other bad things.

To disable the sandbox entirely for a worker, set the Worker init's workflow_runner keyword argument to temporalio.worker.UnsandboxedWorkflowRunner(). This value is defaulted to temporalio.worker.workflow_sandbox.SandboxedWorkflowRunner() so by changing it to the unsandboxed runner, the sandbox will not be used at all.

Customizing the Sandbox

⚠️ WARNING: APIs in the temporalio.worker.workflow_sandbox module are not yet considered stable and may change in future releases.

When creating the Worker, the workflow_runner is defaulted to temporalio.worker.workflow_sandbox.SandboxedWorkflowRunner(). The SandboxedWorkflowRunner's init accepts a restrictions keyword argument that is defaulted to SandboxRestrictions.default. The SandboxRestrictions dataclass is immutable and contains three fields that can be customized, but only two have notable value. See below.

Passthrough Modules

By default the sandbox completely reloads non-standard-library and non-Temporal modules for every workflow run. To make the sandbox quicker and use less memory when importing known-side-effect-free third party modules, they can be marked as passthrough modules.

For performance and behavior reasons, users are encouraged to pass through all third party modules whose calls will be deterministic.

One way to pass through a module is at import time in the workflow file using the imports_passed_through context manager like so:

# my_workflow_file.py

from temporalio import workflow

with workflow.unsafe.imports_passed_through():
    import pydantic

@workflow.defn
class MyWorkflow:
    ...

Alternatively, this can be done at worker creation time by customizing the runner's restrictions. For example:

my_worker = Worker(
  ...,
  workflow_runner=SandboxedWorkflowRunner(
    restrictions=SandboxRestrictions.default.with_passthrough_modules("pydantic")
  )
)

In both of these cases, now the pydantic module will be passed through from outside of the sandbox instead of being reloaded for every workflow run.

Invalid Module Members

SandboxRestrictions.invalid_module_members contains a root matcher that applies to all module members. This already has a default set which includes things like datetime.date.today() which should never be called from a workflow. To remove this restriction:

my_restrictions = dataclasses.replace(
    SandboxRestrictions.default,
    invalid_module_members=SandboxRestrictions.invalid_module_members_default.with_child_unrestricted(
      "datetime", "date", "today",
    ),
)
my_worker = Worker(..., workflow_runner=SandboxedWorkflowRunner(restrictions=my_restrictions))

Restrictions can also be added by |'ing together matchers, for example to restrict the datetime.date class from being used altogether:

my_restrictions = dataclasses.replace(
    SandboxRestrictions.default,
    invalid_module_members=SandboxRestrictions.invalid_module_members_default | SandboxMatcher(
      children={"datetime": SandboxMatcher(use={"date"})},
    ),
)
my_worker = Worker(..., workflow_runner=SandboxedWorkflowRunner(restrictions=my_restrictions))

See the API for more details on exact fields and their meaning.

Known Sandbox Issues

Below are known sandbox issues. As the sandbox is developed and matures, some may be resolved.

Global Import/Builtins

Currently the sandbox references/alters the global sys.modules and builtins fields while running workflow code. In order to prevent affecting other sandboxed code, thread locals are leveraged to only intercept these values during the workflow thread running. Therefore, technically if top-level import code starts a thread, it may lose sandbox protection.

Sandbox is not Secure

The sandbox is built to catch many non-deterministic and state sharing issues, but it is not secure. Some known bad calls are intercepted, but for performance reasons, every single attribute get/set cannot be checked. Therefore a simple call like setattr(temporalio.common, "__my_key", "my value") will leak across sandbox runs.

The sandbox is only a helper, it does not provide full protection.

Sandbox Performance

The sandbox does not add significant CPU or memory overhead for workflows that are in files which only import standard library modules. This is because they are passed through from outside of the sandbox. However, every non-standard-library import that is performed at the top of the same file the workflow is in will add CPU overhead (the module is re-imported every workflow run) and memory overhead (each module independently cached as part of the workflow run for isolation reasons). This becomes more apparent for large numbers of workflow runs.

To mitigate this, users should:

  • Define workflows in files that have as few non-standard-library imports as possible
  • Alter the max workflow cache and/or max concurrent workflows settings if memory grows too large
  • Set third-party libraries as passthrough modules if they are known to be side-effect free
Extending Restricted Classes

Extending a restricted class causes Python to instantiate the restricted metaclass which is unsupported. Therefore if you attempt to use a class in the sandbox that extends a restricted class, it will fail. For example, if you have a class MyZipFile(zipfile.ZipFile) and try to use that class inside a workflow, it will fail.

Classes used inside the workflow should not extend restricted classes. For situations where third-party modules need to at import time, they should be marked as pass through modules.

Certain Standard Library Calls on Restricted Objects

If an object is restricted, internal C Python validation may fail in some cases. For example, running dict.items(os.__dict__) will fail with:

descriptor 'items' for 'dict' objects doesn't apply to a '_RestrictedProxy' object

This is a low-level check that cannot be subverted. The solution is to not use restricted objects inside the sandbox. For situations where third-party modules need to at import time, they should be marked as pass through modules.

is_subclass of ABC-based Restricted Classes

Due to https://bugs.python.org/issue44847, classes that are wrapped and then checked to see if they are subclasses of another via is_subclass may fail (see also this wrapt issue).

Compiled Pydantic Sometimes Using Wrong Types

If the Pydantic dependency is in compiled form (the default) and you are using a Pydantic model inside a workflow sandbox that uses a datetime type, it will grab the wrong validator and use date instead. This is because our patched form of issubclass is bypassed by compiled Pydantic.

To work around, either don't use datetime-based Pydantic model fields in workflows, or mark datetime library as passthrough (means you lose protection against calling the non-deterministic now()), or use non-compiled Pydantic dependency.

Activities

Definition

Activities are decorated with @activity.defn like so:

from temporalio import activity

@activity.defn
def say_hello_activity(name: str) -> str:
    return f"Hello, {name}!"

Some things to note about activity definitions:

  • The say_hello_activity is synchronous which is the recommended activity type (see "Types of Activities" below), but it can be async
  • A custom name for the activity can be set with a decorator argument, e.g. @activity.defn(name="my activity")
  • Long running activities should regularly heartbeat and handle cancellation
  • Activities can only have positional arguments. Best practice is to only take a single argument that is an object/dataclass of fields that can be added to as needed.
  • Activities can be defined on methods instead of top-level functions. This allows the instance to carry state that an activity may need (e.g. a DB connection). The instance method should be what is registered with the worker.
  • Activities can also be defined on callable classes (i.e. classes with __call__). An instance of the class should be what is registered with the worker.
  • The @activity.defn can have dynamic=True set which means all otherwise unhandled activities fall through to this. If present, cannot have name argument, and the activity function must accept a single parameter of Sequence[temporalio.common.RawValue]. The payload of the raw value can be converted via activity.payload_converter().from_payload.

Types of Activities

There are 3 types of activity callables accepted and described below: synchronous multithreaded, synchronous multiprocess/other, and asynchronous. Only positional parameters are allowed in activity callables.

Synchronous Activities

Synchronous activities, i.e. functions that do not have async def, can be used with workers, but the activity_executor worker parameter must be set with a concurrent.futures.Executor instance to use for executing the activities.

All long running, non-local activities should heartbeat so they can be cancelled. Cancellation in threaded activities throws but multiprocess/other activities does not. The sections below on each synchronous type explain further. There are also calls on the context that can check for cancellation. For more information, see "Activity Context" and "Heartbeating and Cancellation" sections later.

Note, all calls from an activity to functions in the temporalio.activity package are powered by contextvars. Therefore, new threads starting inside of activities must copy_context() and then .run() manually to ensure temporalio.activity calls like heartbeat still function in the new threads.

If any activity ever throws a concurrent.futures.BrokenExecutor, the failure is consisted unrecoverable and the worker will fail and shutdown.

Synchronous Multithreaded Activities

If activity_executor is set to an instance of concurrent.futures.ThreadPoolExecutor then the synchronous activities are considered multithreaded activities. If max_workers is not set to at least the worker's max_concurrent_activities setting a warning will be issued. Besides activity_executor, no other worker parameters are required for synchronous multithreaded activities.

By default, cancellation of a synchronous multithreaded activity is done via a temporalio.exceptions.CancelledError thrown into the activity thread. Activities that do not wish to have cancellation thrown can set no_thread_cancel_exception=True in the @activity.defn decorator.

Code that wishes to be temporarily shielded from the cancellation exception can run inside with activity.shield_thread_cancel_exception():. But once the last nested form of that block is finished, even if there is a return statement within, it will throw the cancellation if there was one. A try + except temporalio.exceptions.CancelledError would have to surround the with to handle the cancellation explicitly.

Synchronous Multiprocess/Other Activities

If activity_executor is set to an instance of concurrent.futures.Executor that is not concurrent.futures.ThreadPoolExecutor, then the synchronous activities are considered multiprocess/other activities. Users should prefer threaded activities over multiprocess ones since, among other reasons, threaded activities can raise on cancellation.

These require special primitives for heartbeating and cancellation. The shared_state_manager worker parameter must be set to an instance of temporalio.worker.SharedStateManager. The most common implementation can be created by passing a multiprocessing.managers.SyncManager (i.e. result of multiprocessing.managers.Manager()) to temporalio.worker.SharedStateManager.create_from_multiprocessing().

Also, all of these activity functions must be "picklable".

Asynchronous Activities

Asynchronous activities are functions defined with async def. Asynchronous activities are often much more performant than synchronous ones. When using asynchronous activities no special worker parameters are needed.

⚠️ WARNING: Do not block the thread in async def Python functions. This can stop the processing of the rest of the Temporal.

Cancellation for asynchronous activities is done via asyncio.Task.cancel. This means that asyncio.CancelledError will be raised (and can be caught, but it is not recommended). A non-local activity must heartbeat to receive cancellation and there are other ways to be notified about cancellation (see "Activity Context" and "Heartbeating and Cancellation" later).

Activity Context

During activity execution, an implicit activity context is set as a context variable. The context variable itself is not visible, but calls in the temporalio.activity package make use of it. Specifically:

  • in_activity() - Whether an activity context is present
  • info() - Returns the immutable info of the currently running activity
  • heartbeat(*details) - Record a heartbeat
  • is_cancelled() - Whether a cancellation has been requested on this activity
  • wait_for_cancelled() - async call to wait for cancellation request
  • wait_for_cancelled_sync(timeout) - Synchronous blocking call to wait for cancellation request
  • shield_thread_cancel_exception() - Context manager for use in with clauses by synchronous multithreaded activities to prevent cancel exception from being thrown during the block of code
  • is_worker_shutdown() - Whether the worker has started graceful shutdown
  • wait_for_worker_shutdown() - async call to wait for start of graceful worker shutdown
  • wait_for_worker_shutdown_sync(timeout) - Synchronous blocking call to wait for start of graceful worker shutdown
  • raise_complete_async() - Raise an error that this activity will be completed asynchronously (i.e. after return of the activity function in a separate client call)

With the exception of in_activity(), if any of the functions are called outside of an activity context, an error occurs. Synchronous activities cannot call any of the async functions.

Heartbeating and Cancellation

In order for a non-local activity to be notified of cancellation requests, it must be given a heartbeat_timeout at invocation time and invoke temporalio.activity.heartbeat() inside the activity. It is strongly recommended that all but the fastest executing activities call this function regularly. "Types of Activities" has specifics on cancellation for synchronous and asynchronous activities.

In addition to obtaining cancellation information, heartbeats also support detail data that is persisted on the server for retrieval during activity retry. If an activity calls temporalio.activity.heartbeat(123, 456) and then fails and is retried, temporalio.activity.info().heartbeat_details will return an iterable containing 123 and 456 on the next run.

Heartbeating has no effect on local activities.

Worker Shutdown

An activity can react to a worker shutdown. Using is_worker_shutdown or one of the wait_for_worker_shutdown functions an activity can react to a shutdown.

When the graceful_shutdown_timeout worker parameter is given a datetime.timedelta, on shutdown the worker will notify activities of the graceful shutdown. Once that timeout has passed (or if wasn't set), the worker will perform cancellation of all outstanding activities.

The shutdown() invocation will wait on all activities to complete, so if a long-running activity does not at least respect cancellation, the shutdown may never complete.

Testing

Unit testing an activity or any code that could run in an activity is done via the temporalio.testing.ActivityEnvironment class. Simply instantiate this and any callable + params passed to run will be invoked inside the activity context. The following are attributes/methods on the environment that can be used to affect calls activity code might make to functions on the temporalio.activity package.

  • info property can be set to customize what is returned from activity.info()
  • on_heartbeat property can be set to handle activity.heartbeat() calls
  • cancel() can be invoked to simulate a cancellation of the activity
  • worker_shutdown() can be invoked to simulate a worker shutdown during execution of the activity

Workflow Replay

Given a workflow's history, it can be replayed locally to check for things like non-determinism errors. For example, assuming history_str is populated with a JSON string history either exported from the web UI or from tctl, the following function will replay it:

from temporalio.client import WorkflowHistory
from temporalio.worker import Replayer

async def run_replayer(history_str: str):
  replayer = Replayer(workflows=[SayHello])
  await replayer.replay_workflow(WorkflowHistory.from_json(history_str))

This will throw an error if any non-determinism is detected.

Replaying from workflow history is a powerful concept that many use to test that workflow alterations won't cause non-determinisms with past-complete workflows. The following code will make sure that all workflow histories for a certain workflow type (i.e. workflow class) are safe with the current code.

from temporalio.client import Client, WorkflowHistory
from temporalio.worker import Replayer

async def check_past_histories(my_client: Client):
  replayer = Replayer(workflows=[SayHello])
  await replayer.replay_workflows(
    await my_client.list_workflows("WorkflowType = 'SayHello'").map_histories(),
  )

OpenTelemetry Support

OpenTelemetry support requires the optional opentelemetry dependencies which are part of the opentelemetry extra. When using pip, running

pip install temporalio[opentelemetry]

will install needed dependencies. Then the temporalio.contrib.opentelemetry.TracingInterceptor can be created and set as an interceptor on the interceptors argument of Client.connect. When set, spans will be created for all client calls and for all activity and workflow invocations on the worker, spans will be created and properly serialized through the server to give one proper trace for a workflow execution.

Protobuf 3.x vs 4.x

Python currently has two somewhat-incompatible protobuf library versions - the 3.x series and the 4.x series. Python currently recommends 4.x and that is the primary supported version. Some libraries like Pulumi require 4.x. Other libraries such as ONNX and Streamlit, for one reason or another, have/will not leave 3.x.

To support these, Temporal Python SDK allows any protobuf library >= 3.19. However, the C extension in older Python versions can cause issues with the sandbox due to global state sharing. Temporal strongly recommends using the latest protobuf 4.x library unless you absolutely cannot at which point some proto libraries may have to be marked as Passthrough Modules.

Known Compatibility Issues

Below are known compatibility issues with the Python SDK.

gevent Patching

When using gevent.monkey.patch_all(), asyncio event loops can get messed up, especially those using custom event loops like Temporal. See this gevent issue. This is a known incompatibility and users are encouraged to not use gevent in asyncio applications (including Temporal). But if you must, there is a sample showing how it is possible.

Development

The Python SDK is built to work with Python 3.8 and newer. It is built using SDK Core which is written in Rust.

Building

Prepare

To build the SDK from source for use as a dependency, the following prerequisites are required:

  • Python >= 3.8
  • Rust
  • poetry (e.g. python -m pip install poetry)
  • poe (e.g. python -m pip install poethepoet)

macOS note: If errors are encountered, it may be better to install Python and Rust as recommended from their websites instead of via brew.

With the prerequisites installed, first clone the SDK repository recursively:

git clone --recursive https://github.com/temporalio/sdk-python.git
cd sdk-python

Use poetry to install the dependencies with --no-root to not install this package (because we still need to build it):

poetry install --no-root

Build

Now perform the release build:

This will take a while because Rust will compile the core project in release mode (see Local SDK development environment for the quicker approach to local development).

poetry build

The compiled wheel doesn't have the exact right tags yet for use, so run this script to fix it:

poe fix-wheel

The whl wheel file in dist/ is now ready to use.

Use

The wheel can now be installed into any virtual environment.

For example, create a virtual environment somewhere and then run the following inside the virtual environment:

pip install wheel
pip install /path/to/cloned/sdk-python/dist/*.whl

Create this Python file at example.py:

import asyncio
from temporalio import workflow, activity
from temporalio.client import Client
from temporalio.worker import Worker

@workflow.defn
class SayHello:
    @workflow.run
    async def run(self, name: str) -> str:
        return f"Hello, {name}!"

async def main():
    client = await Client.connect("localhost:7233")
    async with Worker(client, task_queue="my-task-queue", workflows=[SayHello]):
        result = await client.execute_workflow(SayHello.run, "Temporal",
            id="my-workflow-id", task_queue="my-task-queue")
        print(f"Result: {result}")

if __name__ == "__main__":
    asyncio.run(main())

Assuming there is a local Temporal server running, execute the file with python (or python3 if necessary):

python example.py

It should output:

Result: Hello, Temporal!

Local SDK development environment

For local development, it is often quicker to use debug builds and a local virtual environment.

While not required, it often helps IDEs if we put the virtual environment .venv directory in the project itself. This can be configured system-wide via:

poetry config virtualenvs.in-project true

Now perform the same steps as the "Prepare" section above by installing the prerequisites, cloning the project, installing dependencies, and generating the protobuf code:

git clone --recursive https://github.com/temporalio/sdk-python.git
cd sdk-python
poetry install --no-root

Now compile the Rust extension in develop mode which is quicker than release mode:

poe build-develop

That step can be repeated for any Rust changes made.

The environment is now ready to develop in.

Testing

To execute tests:

poe test

This runs against Temporalite. To run against the time-skipping test server, pass --workflow-environment time-skipping. To run against the default namespace of an already-running server, pass the host:port to --workflow-environment. Can also use regular pytest arguments. For example, here's how to run a single test with debug logs on the console:

poe test -s --log-cli-level=DEBUG -k test_sync_activity_thread_cancel_caught

Proto Generation and Testing

To allow for backwards compatibility, protobuf code is generated on the 3.x series of the protobuf library. To generate protobuf code, you must be on Python <= 3.10, and then run poetry add "protobuf<4". Then the protobuf files can be generated via poe gen-protos. Tests can be run for protobuf version 3 by setting the TEMPORAL_TEST_PROTO3 env var to 1 prior to running tests.

Do not commit poetry.lock or pyproject.toml changes. To go back from this downgrade, restore pyproject.toml and run poetry update protobuf grpcio-tools.

For a less system-intrusive approach, you can:

docker build -f scripts/_proto/Dockerfile .
docker run -v "${PWD}/temporalio/api:/api_new" -v "${PWD}/temporalio/bridge/proto:/bridge_new" <just built image sha>
poe format

Style

  • Mostly Google Style Guide. Notable exceptions:
    • We use Black for formatting, so that takes precedence
    • In tests and example code, can import individual classes/functions to make it more readable. Can also do this for rarely in library code for some Python common items (e.g. dataclass or partial), but not allowed to do this for any temporalio packages (except temporalio.types) or any classes/functions that aren't clear when unqualified.
    • We allow relative imports for private packages
    • We allow @staticmethod

sdk-python's People

Contributors

adrien-f avatar aezomz avatar afitz0 avatar antlai-temporal avatar bergundy avatar cretz avatar dandavison avatar dnnsthnnr avatar jackdawm avatar jmdacruz avatar josh-berry avatar kmilhan avatar lorensr avatar mbernier avatar nathanielobrown avatar ndtretyak avatar northpowered avatar oracen avatar osobo avatar pvcnt avatar rachfop avatar rgehan avatar sushisource avatar tomwheeler avatar william-almy-skydio avatar ytaben avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

sdk-python's Issues

Multiprocess heartbeat and cancel improvement research

Is your feature request related to a problem? Please describe.

Right now we're using multiprocessing.Manager()'s queue for heartbeating and event for cancellation. Then we move from the sync queue onto the sync heartbeat which schedules an async heartbeat task that we try to ensure runs when the activity completes.

Describe the solution you'd like

This is potentially flaky and/or confusing. Try to streamline the threaded/multiprocess heartbeat queuing.

Per-call gRPC options

Is your feature request related to a problem? Please describe.

Need to be able to set timeouts and headers per gRPC call

Describe the solution you'd like

  • From the Rust side, change the macro to alter the tonic request as needed
  • From the Python side, maybe have a RpcConfig option that could have a timeout: Optional[timedelta] and a metadata: Mapping[str, str] field (technically Mapping[str, Iterable[str]] matches HTTP spec, but we can add that functionality when dupe values are needed later).

[Feature Request] Support non-function callables as activities

Is your feature request related to a problem? Please describe.

Currently the activity definition implementation here requires __code__ on a callable. From what I can tell it seems like it's only using that to verify that there are no kwonly arguments.

Describe the solution you'd like

We can instead use the inspect module to analyze the callable's signature, allowing a broader set of non-function callables to be used as activities.

[Feature Request] Ability to install without having Rust installed

Is your feature request related to a problem?

Requiring rust to be installed to pip install this package complicates deployment

Describe the solution you'd like

Publish a version that does not require Rust to install with Pip

Additional context

Trying to use temporal sdk at my company, but we don’t have Rust on our base images. Requiring it complicates things.

Improve dataclass type hinting

Is your feature request related to a problem? Please describe.

Our use of https://github.com/konradhalas/dacite is too primitive. It only supports to/from the object itself, not collections, optional, etc.

Describe the solution you'd like

All of the following should be supported via type hints:

  • MyDataClass
  • List[MyDataClass], Set[MyDataClass], other collections, etc
  • Mapping[str, MyDataClass], Dict[str, MyDataClass], other dicts, etc
  • Union[MyDataClass1, MyDataClass2], Union[List[int], Dict[str, MyDataClass]], etc
  • Optional[MyDataClass1], Optional[List[Dict[str, MyDataClass]]], etc

Here are the libraries found so far:

We should probably support customization via instance.to_json()/instance.to_dict() and Class.to_json()/Class.to_dict(). This means those using some of those other libraries will work automatically.

Needs investigation...

temporal-sdk-core-0.1.0: 1 vulnerabilities (highest severity is: 7.5) - autoclosed

Vulnerable Library - temporal-sdk-core-0.1.0

Found in HEAD commit: 0392468f21d9604c828d50d34daf8c9be03b31cd

Vulnerabilities

CVE Severity CVSS Dependency Type Fixed in (temporal-sdk-core version) Remediation Available
CVE-2023-22895 High 7.5 bzip2-0.4.3.crate Transitive N/A*

*For some transitive vulnerabilities, there is no version of direct dependency with a fix. Check the section "Details" below to see if there is a version of transitive dependency where vulnerability is fixed.

Details

CVE-2023-22895

Vulnerable Library - bzip2-0.4.3.crate

Bindings to libbzip2 for bzip2 compression and decompression exposed as Reader/Writer streams.

Library home page: https://crates.io/api/v1/crates/bzip2/0.4.3/download

Dependency Hierarchy:

  • temporal-sdk-core-0.1.0 (Root Library)
    • zip-0.6.3.crate
      • bzip2-0.4.3.crate (Vulnerable Library)

Found in HEAD commit: 0392468f21d9604c828d50d34daf8c9be03b31cd

Found in base branch: main

Vulnerability Details

The bzip2 crate before 0.4.4 for Rust allow attackers to cause a denial of service via a large file that triggers an integer overflow in mem.rs. NOTE: this is unrelated to the https://crates.io/crates/bzip2-rs product.

Publish Date: 2023-01-10

URL: CVE-2023-22895

CVSS 3 Score Details (7.5)

Base Score Metrics:

  • Exploitability Metrics:
    • Attack Vector: Network
    • Attack Complexity: Low
    • Privileges Required: None
    • User Interaction: None
    • Scope: Unchanged
  • Impact Metrics:
    • Confidentiality Impact: None
    • Integrity Impact: None
    • Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-22895

Release Date: 2023-01-10

Fix Resolution: bzip2 - 0.4.4

Remove gRPC dependency

Is your feature request related to a problem? Please describe.

We generate pure Python gRPC unnecessarily since we have Core client.

Describe the solution you'd like

Remove gRPC from the regular dependencies, and we can still use it as a dev dependency during tests

Investigate issues with gevent.monkey.patch_all()

Is your feature request related to a problem? Please describe.

Users on an older version of gevent (1.5.0) reported that running gevent.monkey.patch_all() breaks workflow asyncio.

Describe the solution you'd like

  • Replicate the error on that gevent version (a simple workflow is probably fine)
  • Upgrade gevent to see if it still occurs
    • If not, essentially bisect to find offending version minimum and maybe document in README
    • If so, find workaround and document in README

[Feature Request] Get workflow handle/send signals from non-async code

Is your feature request related to a problem? Please describe.

I have a non-async activity and I want to send signals to the parent workflow from this activity. I want to do this because I have existing code that scrapes websites and yields those results in a generator and I want to notify the parent workflow of those results as they are fetched. I would prefer not to convert the activity to async because:

  1. The code the activity calls is not async and I don't want to convert it completely to async (it would ruin my tests requests mocking for one thing)
  2. I'm nervous about executing my non-async code in a thread pool. I know there is already a threadpool, but spawning threadpools all over the place feels like it could get out of control. Better to have one thread pool and be able to control it's size is my thought
  3. If I convert to async code and run my sync code in a thread pool, I'd have to communicate generator results through a queue and it would be a complicated solution

Describe the solution you'd like

Would it be possible to add synchronous versions of workflow.get_external_workflow_handle_for (not defined as async, but fails if called outside of async function) and workflow_handle.signal? In general I guess the question is, are the functions for interacting with workflows going to support use from non-async code?

I can imagine lots of future cases where I'll want to interact form non-async code as my database interactions are not async so lots of my code is not async.

Additional context

Maybe there are some async patterns I can use to mitigate these issues (for example asyncer) but I have not had much luck so far. I'm not an async expert, so any advice would be appreciated.

[Bug] retry_policy not availably in activity info

What are you really trying to do?

From withing an activity I want to get the attempt number

Describe the bug

I pass in a retry_policy to workflow.execute_activity and expect to be able to get that policy from within the execute activity with activity.iinfo().retry_policy but I get None instead.

Minimal Reproduction

Not entirely self contained example, but I hope you get the idea.

I tried to make a failing test in this repo, but I'm still unable to run the development environment. I guess this is a separate issue, but running the tests fails with ImportError: dlopen(/Users/nob/repos/sdk-python/temporalio/bridge/temporal_sdk_bridge.abi3.so, 0x0002): tried: '/Users/nob/repos/sdk-python/temporalio/bridge/temporal_sdk_bridge.abi3.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')). My guess is that rust is cross compiling for x86_64 rather than the arm64e I need for my M1 Mac for some reason 🤷‍♂️.

from datetime import timedelta
import uuid
from temporalio import activity, workflow
from temporalio.client import Client
from temporalio.common import RetryPolicy
from bots_interface.temporal.server import temporal_worker_for_test


@activity.defn
async def return_retry_policy() -> RetryPolicy:
    activity_info = activity.info()
    # This is None!!!
    return activity_info.retry_policy


@workflow.defn
class Workflow:
    @workflow.run
    async def run(self) -> RetryPolicy:
        policy = await workflow.execute_activity(
            return_retry_policy,
            retry_policy=RetryPolicy(maximum_attempts=3),
            schedule_to_close_timeout=timedelta(seconds=5),
        )
        return policy


async def test_activity_info_retry_policy(temporal_client: Client):
    async with temporal_worker_for_test(
        temporal_client, Workflow, activities=[return_retry_policy]
    ) as worker:
        result = await temporal_client.execute_workflow(
            Workflow.run,
            id=str(uuid.uuid4()),
            task_queue=worker.task_queue,
            execution_timeout=timedelta(seconds=10),
            retry_policy=RetryPolicy(maximum_attempts=1),
        )
    assert result.maximum_attempts == 3  # This fails

Environment/Versions

  • OS and processor: M1 Mac, MacOS 12.4
  • Temporal Version: temporalio==0.1a2
  • Are you using Docker or Kubernetes or building Temporal from source?: Using embedded golang temporal server as used in the repo. So compiling myself I guess

Linux aarch64 wheel

Is your feature request related to a problem? Please describe.

The Linux aarch64 wheel on the builds needs to be tested and confirm it works. Unfortunately it takes a really long time to build in CI (disabled except on main merge).

Investigate more generic manylinux support

Is your feature request related to a problem? Please describe.

Right now our Linux wheel is named temporalio-0.1a1.dev1-cp37-abi3-manylinux_2_31_x86_64.whl. This means it only works on glibc 2.31+ per https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/.

Describe the solution you'd like

We can use manylinux container to get Linux builds that work with a wider range of linuxes (e.g. manylinux2010 or manylinux2014). We need to investigate the effort/value of this.

Publish to pip

Hi, congrats on the progress, this is amazing. I've seen you published this package to pypi once in the past. Would you mind publishing it once more ? We need it currently in my team to experiment, but having to build everything from scratch is not ideal to integrate in our main repo.

Test client/worker/core behavior across fork

Is your feature request related to a problem? Please describe.

Right now, it is basically undefined how core will work across forks. We do know it fails. For example, the client var, when copied across the fork, has Rust file descriptors copied for the gRPC stuff that fail during Rust drop when the child process's version of the var is GC'd.

Describe the solution you'd like

We need to, without leak, make sure that fork does not fail unless you explicitly use something from across the fork, at which point it should fail. No surprise failures like on implicit Python GC failure during drop.

I don't know how yet to do this.

Create workflow sandbox

Describe the solution you'd like

We need to run workflows in a sandbox with some things disabled to prevent non-determinism. Research has already started on this. More will continue.

Deserialize traceback from stack trace string in Temporal failures if able

Is your feature request related to a problem? Please describe.

We chain errors when converting from failures by setting __cause__, but there is a report that the chained errors are not logged like normally chained errors

EDIT: We don't rehydrate the traceback from the stack trace string

Describe the solution you'd like

Make sure we log chained errors normally and write a test to ensure it

EDIT: Java parses their string stack trace back to stack trace elements, so we should too. See https://github.com/ionelmc/python-tblib/blob/dd926c1e5dc5bbe5e1fc494443bbac8970c7d3ee/src/tblib/__init__.py#L200 for an example of how to do this.

More workflow tests

Describe the solution you'd like

See the list of TODO's at the bottom of test files

Support macOS arm64

Is your feature request related to a problem? Please describe.

Need a prebuilt macOS aarch64 wheel. Don't currently have it because cross-compilation is a bit of a pain with pyo3 and we may get a mac m1 runner soon.

Describe the solution you'd like

Update CI build build and test on this platform

Stress tests

Is your feature request related to a problem? Please describe.

We need to know reasonable default maximums and resource utilization of heavy workflow use

Describe the solution you'd like

Make a repeatable and parameterized stress test. While it should run in CI to confirm it always works, the parameters for CI should be much lower than one might want when actually stressing a system.

go.temporal.io/server-v1.17.1: 6 vulnerabilities (highest severity is: 7.5) - autoclosed

Vulnerable Library - go.temporal.io/server-v1.17.1

Found in HEAD commit: efc1c7babb9a81afdd5a8915f64787c5c5ea4f98

Vulnerabilities

CVE Severity CVSS Dependency Type Fixed in (go.temporal.io/server-v1.17.1 version) Remediation Available
CVE-2019-0205 High 7.5 detected in multiple dependencies Transitive N/A*
CVE-2022-27664 High 7.5 golang.org/x/net-v0.0.0-20220708220712-1185a9018129 Transitive N/A*
CVE-2022-32149 High 7.5 golang.org/x/text-v0.3.7 Transitive N/A*
CVE-2019-0210 High 7.5 detected in multiple dependencies Transitive N/A*
CVE-2018-11798 Medium 6.5 github.com/apache/thrift-0.10.0 Transitive N/A*
CVE-2020-28928 Medium 5.5 modernc.org/libc-v1.16.10 Transitive N/A*

*For some transitive vulnerabilities, there is no version of direct dependency with a fix. Check the section "Details" below to see if there is a version of transitive dependency where vulnerability is fixed.

Details

CVE-2019-0205

Vulnerable Libraries - github.com/apache/thrift-0.10.0, github.com/uber-go/tally/v4-v4.1.2

github.com/apache/thrift-0.10.0

Apache Thrift

Dependency Hierarchy:

  • go.temporal.io/server-v1.17.1 (Root Library)
    • github.com/uber/tchannel-go-v1.22.3
      • github.com/apache/thrift-0.10.0 (Vulnerable Library)

github.com/uber-go/tally/v4-v4.1.2

A Go metrics interface with fast buffered metrics and third party reporters

Library home page: https://proxy.golang.org/github.com/uber-go/tally/v4/@v/v4.1.2.zip

Dependency Hierarchy:

  • go.temporal.io/server-v1.17.1 (Root Library)
    • github.com/uber-go/tally/v4-v4.1.2 (Vulnerable Library)

Found in HEAD commit: efc1c7babb9a81afdd5a8915f64787c5c5ea4f98

Found in base branch: main

Vulnerability Details

In Apache Thrift all versions up to and including 0.12.0, a server or client may run into an endless loop when feed with specific input data. Because the issue had already been partially fixed in version 0.11.0, depending on the installed version it affects only certain language bindings.

Publish Date: 2019-10-29

URL: CVE-2019-0205

CVSS 3 Score Details (7.5)

Base Score Metrics:

  • Exploitability Metrics:
    • Attack Vector: Network
    • Attack Complexity: Low
    • Privileges Required: None
    • User Interaction: None
    • Scope: Unchanged
  • Impact Metrics:
    • Confidentiality Impact: None
    • Integrity Impact: None
    • Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-0205

Release Date: 2019-10-29

Fix Resolution: org.apache.thrift:libthrift:0.13.0

CVE-2022-27664

Vulnerable Library - golang.org/x/net-v0.0.0-20220708220712-1185a9018129

Library home page: https://proxy.golang.org/golang.org/x/net/@v/v0.0.0-20220708220712-1185a9018129.zip

Dependency Hierarchy:

  • go.temporal.io/server-v1.17.1 (Root Library)
    • golang.org/x/oauth2-v0.0.0-20220622183110-fd043fe589d2
      • golang.org/x/net-v0.0.0-20220708220712-1185a9018129 (Vulnerable Library)

Found in HEAD commit: efc1c7babb9a81afdd5a8915f64787c5c5ea4f98

Found in base branch: main

Vulnerability Details

In net/http in Go before 1.18.6 and 1.19.x before 1.19.1, attackers can cause a denial of service because an HTTP/2 connection can hang during closing if shutdown were preempted by a fatal error.

Publish Date: 2022-09-06

URL: CVE-2022-27664

CVSS 3 Score Details (7.5)

Base Score Metrics:

  • Exploitability Metrics:
    • Attack Vector: Network
    • Attack Complexity: Low
    • Privileges Required: None
    • User Interaction: None
    • Scope: Unchanged
  • Impact Metrics:
    • Confidentiality Impact: None
    • Integrity Impact: None
    • Availability Impact: High

For more information on CVSS3 Scores, click here.

CVE-2022-32149

Vulnerable Library - golang.org/x/text-v0.3.7

Library home page: https://proxy.golang.org/golang.org/x/text/@v/v0.3.7.zip

Dependency Hierarchy:

  • go.temporal.io/server-v1.17.1 (Root Library)
    • golang.org/x/oauth2-v0.0.0-20220622183110-fd043fe589d2
      • golang.org/x/net-v0.0.0-20220708220712-1185a9018129
        • golang.org/x/text-v0.3.7 (Vulnerable Library)

Found in HEAD commit: efc1c7babb9a81afdd5a8915f64787c5c5ea4f98

Found in base branch: main

Vulnerability Details

An attacker may cause a denial of service by crafting an Accept-Language header which ParseAcceptLanguage will take significant time to parse.

Publish Date: 2022-10-14

URL: CVE-2022-32149

CVSS 3 Score Details (7.5)

Base Score Metrics:

  • Exploitability Metrics:
    • Attack Vector: Network
    • Attack Complexity: Low
    • Privileges Required: None
    • User Interaction: None
    • Scope: Unchanged
  • Impact Metrics:
    • Confidentiality Impact: None
    • Integrity Impact: None
    • Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://www.cve.org/CVERecord?id=CVE-2022-32149

Release Date: 2022-10-14

Fix Resolution: v0.3.8

CVE-2019-0210

Vulnerable Libraries - github.com/uber-go/tally/v4-v4.1.2, github.com/apache/thrift-0.10.0

github.com/uber-go/tally/v4-v4.1.2

A Go metrics interface with fast buffered metrics and third party reporters

Library home page: https://proxy.golang.org/github.com/uber-go/tally/v4/@v/v4.1.2.zip

Dependency Hierarchy:

  • go.temporal.io/server-v1.17.1 (Root Library)
    • github.com/uber-go/tally/v4-v4.1.2 (Vulnerable Library)

github.com/apache/thrift-0.10.0

Apache Thrift

Dependency Hierarchy:

  • go.temporal.io/server-v1.17.1 (Root Library)
    • github.com/uber/tchannel-go-v1.22.3
      • github.com/apache/thrift-0.10.0 (Vulnerable Library)

Found in HEAD commit: efc1c7babb9a81afdd5a8915f64787c5c5ea4f98

Found in base branch: main

Vulnerability Details

In Apache Thrift 0.9.3 to 0.12.0, a server implemented in Go using TJSONProtocol or TSimpleJSONProtocol may panic when feed with invalid input data.

Publish Date: 2019-10-29

URL: CVE-2019-0210

CVSS 3 Score Details (7.5)

Base Score Metrics:

  • Exploitability Metrics:
    • Attack Vector: Network
    • Attack Complexity: Low
    • Privileges Required: None
    • User Interaction: None
    • Scope: Unchanged
  • Impact Metrics:
    • Confidentiality Impact: None
    • Integrity Impact: None
    • Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: http://mail-archives.apache.org/mod_mbox/thrift-dev/201910.mbox/%3C277A46CA87494176B1BBCF5D72624A2A%40HAGGIS%3E

Release Date: 2019-10-29

Fix Resolution: 0.13.0

CVE-2018-11798

Vulnerable Library - github.com/apache/thrift-0.10.0

Apache Thrift

Dependency Hierarchy:

  • go.temporal.io/server-v1.17.1 (Root Library)
    • github.com/uber/tchannel-go-v1.22.3
      • github.com/apache/thrift-0.10.0 (Vulnerable Library)

Found in HEAD commit: efc1c7babb9a81afdd5a8915f64787c5c5ea4f98

Found in base branch: main

Vulnerability Details

The Apache Thrift Node.js static web server in versions 0.9.2 through 0.11.0 have been determined to contain a security vulnerability in which a remote user has the ability to access files outside the set webservers docroot path.

Publish Date: 2019-01-07

URL: CVE-2018-11798

CVSS 3 Score Details (6.5)

Base Score Metrics:

  • Exploitability Metrics:
    • Attack Vector: Network
    • Attack Complexity: Low
    • Privileges Required: Low
    • User Interaction: None
    • Scope: Unchanged
  • Impact Metrics:
    • Confidentiality Impact: High
    • Integrity Impact: None
    • Availability Impact: None

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-11798

Release Date: 2019-01-07

Fix Resolution: v0.12.0

CVE-2020-28928

Vulnerable Library - modernc.org/libc-v1.16.10

Library home page: https://proxy.golang.org/modernc.org/libc/@v/v1.16.10.zip

Dependency Hierarchy:

  • go.temporal.io/server-v1.17.1 (Root Library)
    • modernc.org/sqlite-v1.17.3
      • modernc.org/libc-v1.16.10 (Vulnerable Library)

Found in HEAD commit: efc1c7babb9a81afdd5a8915f64787c5c5ea4f98

Found in base branch: main

Vulnerability Details

In musl libc through 1.2.1, wcsnrtombs mishandles particular combinations of destination buffer size and source character limit, as demonstrated by an invalid write access (buffer overflow).

Publish Date: 2020-11-24

URL: CVE-2020-28928

CVSS 3 Score Details (5.5)

Base Score Metrics:

  • Exploitability Metrics:
    • Attack Vector: Local
    • Attack Complexity: Low
    • Privileges Required: Low
    • User Interaction: None
    • Scope: Unchanged
  • Impact Metrics:
    • Confidentiality Impact: None
    • Integrity Impact: None
    • Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://nvd.nist.gov/vuln/detail/CVE-2020-28928

Release Date: 2020-11-24

Fix Resolution: musl - 1.2.2-1,1.2.2-1,1.1.16-3+deb9u1

Search attributes

Describe the solution you'd like

Full search attribute support on client and inside workflows. Make sure to type the search attributes properly.

temporal-client-0.1.0: 1 vulnerabilities (highest severity is: 7.5) - autoclosed

Vulnerable Library - temporal-client-0.1.0

Vulnerabilities

CVE Severity CVSS Dependency Type Fixed in Remediation Available
CVE-2022-24713 High 7.5 regex-1.5.4.crate Transitive N/A

Details

CVE-2022-24713

Vulnerable Library - regex-1.5.4.crate

An implementation of regular expressions for Rust. This implementation uses finite automata and guarantees linear time matching on all inputs.

Library home page: https://crates.io/api/v1/crates/regex/1.5.4/download

Dependency Hierarchy:

  • temporal-client-0.1.0 (Root Library)
    • temporal-sdk-core-protos-0.1.0
      • tonic-build-0.6.2.crate
        • prost-build-0.9.0.crate
          • regex-1.5.4.crate (Vulnerable Library)

Found in base branch: main

Vulnerability Details

regex is an implementation of regular expressions for the Rust language. The regex crate features built-in mitigations to prevent denial of service attacks caused by untrusted regexes, or untrusted input matched by trusted regexes. Those (tunable) mitigations already provide sane defaults to prevent attacks. This guarantee is documented and it's considered part of the crate's API. Unfortunately a bug was discovered in the mitigations designed to prevent untrusted regexes to take an arbitrary amount of time during parsing, and it's possible to craft regexes that bypass such mitigations. This makes it possible to perform denial of service attacks by sending specially crafted regexes to services accepting user-controlled, untrusted regexes. All versions of the regex crate before or equal to 1.5.4 are affected by this issue. The fix is include starting from regex 1.5.5. All users accepting user-controlled regexes are recommended to upgrade immediately to the latest version of the regex crate. Unfortunately there is no fixed set of problematic regexes, as there are practically infinite regexes that could be crafted to exploit this vulnerability. Because of this, it us not recommend to deny known problematic regexes.

Publish Date: 2022-03-08

URL: CVE-2022-24713

CVSS 3 Score Details (7.5)

Base Score Metrics:

  • Exploitability Metrics:
    • Attack Vector: Network
    • Attack Complexity: Low
    • Privileges Required: None
    • User Interaction: None
    • Scope: Unchanged
  • Impact Metrics:
    • Confidentiality Impact: None
    • Integrity Impact: None
    • Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: GHSA-m5pq-gvj9-9vr8

Release Date: 2022-03-08

Fix Resolution: regex - 1.5.5

temporal-sdk-core-api-0.1.0: 1 vulnerabilities (highest severity is: 5.5) - autoclosed

Vulnerable Library - temporal-sdk-core-api-0.1.0

Path to dependency file: /temporalio/bridge/Cargo.toml

Path to vulnerable library: /temporalio/bridge/Cargo.toml

Vulnerabilities

CVE Severity CVSS Dependency Type Fixed in (temporal-sdk-core-api version) Remediation Possible**
WS-2023-0366 Medium 5.5 rustix-0.38.8.crate Transitive N/A*

*For some transitive vulnerabilities, there is no version of direct dependency with a fix. Check the "Details" section below to see if there is a version of transitive dependency where vulnerability is fixed.

**In some cases, Remediation PR cannot be created automatically for a vulnerability despite the availability of remediation

Details

WS-2023-0366

Vulnerable Library - rustix-0.38.8.crate

Safe Rust bindings to POSIX/Unix/Linux/Winsock2-like syscalls

Library home page: https://static.crates.io/crates/rustix/rustix-0.38.8.crate

Path to dependency file: /temporalio/bridge/Cargo.toml

Path to vulnerable library: /temporalio/bridge/Cargo.toml

Dependency Hierarchy:

  • temporal-sdk-core-api-0.1.0 (Root Library)
    • temporal-sdk-core-protos-0.1.0
      • prost-wkt-types-0.4.2.crate
        • prost-build-0.11.9.crate
          • tempfile-3.8.0.crate
            • rustix-0.38.8.crate (Vulnerable Library)

Found in base branch: main

Vulnerability Details

rustix's rustix::fs::Dir iterator with the linux_raw backend can cause memory explosion

Publish Date: 2023-10-18

URL: WS-2023-0366

CVSS 3 Score Details (5.5)

Base Score Metrics:

  • Exploitability Metrics:
    • Attack Vector: Local
    • Attack Complexity: Low
    • Privileges Required: None
    • User Interaction: Required
    • Scope: Unchanged
  • Impact Metrics:
    • Confidentiality Impact: None
    • Integrity Impact: None
    • Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: GHSA-c827-hfw6-qwvm

Release Date: 2023-10-18

Fix Resolution: rustix - 0.35.15,0.36.16,0.37.25,0.38.19

Improve workflow stack trace query result

Is your feature request related to a problem? Please describe.

Right now the stack traces are not very good

Describe the solution you'd like

Make them better hopefully without unnecessary performance penalty

Accept `host:port` for client instead of only URL

Is your feature request related to a problem? Please describe.

All other SDKs use host:port, we should too

Describe the solution you'd like

  • Change tls_config: Optional[TLSConfig] = None to tls_config: Union[bool, TLSConfig] = False
  • If a scheme is not present (meaning we're backwards compatible here), before sending to core, set it as http if tls_config is false or https otherwise

github.com/temporalio/temporal-v1.17.1: 3 vulnerabilities (highest severity is: 7.5) - autoclosed

Vulnerable Library - github.com/temporalio/temporal-v1.17.1

Found in HEAD commit: 2288f4128e7e1dfd80ecbde2df89d8e81810860e

Vulnerabilities

CVE Severity CVSS Dependency Type Fixed in Remediation Available
CVE-2019-0205 High 7.5 github.com/apache/thrift-0.10.0 Transitive N/A
CVE-2019-0210 High 7.5 github.com/apache/thrift-0.10.0 Transitive N/A
CVE-2018-11798 Medium 6.5 github.com/apache/thrift-0.10.0 Transitive N/A

Details

CVE-2019-0205

Vulnerable Library - github.com/apache/thrift-0.10.0

Apache Thrift

Dependency Hierarchy:

  • github.com/temporalio/temporal-v1.17.1 (Root Library)
    • github.com/temporalio/ringpop-go-6f91b5915e95e7b08817aae78fea3a38ea1b5bd2
      • github.com/apache/thrift-0.10.0 (Vulnerable Library)

Found in HEAD commit: 2288f4128e7e1dfd80ecbde2df89d8e81810860e

Found in base branch: main

Vulnerability Details

In Apache Thrift all versions up to and including 0.12.0, a server or client may run into an endless loop when feed with specific input data. Because the issue had already been partially fixed in version 0.11.0, depending on the installed version it affects only certain language bindings.

Publish Date: 2019-10-29

URL: CVE-2019-0205

CVSS 3 Score Details (7.5)

Base Score Metrics:

  • Exploitability Metrics:
    • Attack Vector: Network
    • Attack Complexity: Low
    • Privileges Required: None
    • User Interaction: None
    • Scope: Unchanged
  • Impact Metrics:
    • Confidentiality Impact: None
    • Integrity Impact: None
    • Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-0205

Release Date: 2019-10-29

Fix Resolution: org.apache.thrift:libthrift:0.13.0

CVE-2019-0210

Vulnerable Library - github.com/apache/thrift-0.10.0

Apache Thrift

Dependency Hierarchy:

  • github.com/temporalio/temporal-v1.17.1 (Root Library)
    • github.com/temporalio/ringpop-go-6f91b5915e95e7b08817aae78fea3a38ea1b5bd2
      • github.com/apache/thrift-0.10.0 (Vulnerable Library)

Found in HEAD commit: 2288f4128e7e1dfd80ecbde2df89d8e81810860e

Found in base branch: main

Vulnerability Details

In Apache Thrift 0.9.3 to 0.12.0, a server implemented in Go using TJSONProtocol or TSimpleJSONProtocol may panic when feed with invalid input data.

Publish Date: 2019-10-29

URL: CVE-2019-0210

CVSS 3 Score Details (7.5)

Base Score Metrics:

  • Exploitability Metrics:
    • Attack Vector: Network
    • Attack Complexity: Low
    • Privileges Required: None
    • User Interaction: None
    • Scope: Unchanged
  • Impact Metrics:
    • Confidentiality Impact: None
    • Integrity Impact: None
    • Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: http://mail-archives.apache.org/mod_mbox/thrift-dev/201910.mbox/%3C277A46CA87494176B1BBCF5D72624A2A%40HAGGIS%3E

Release Date: 2019-10-29

Fix Resolution: 0.13.0

CVE-2018-11798

Vulnerable Library - github.com/apache/thrift-0.10.0

Apache Thrift

Dependency Hierarchy:

  • github.com/temporalio/temporal-v1.17.1 (Root Library)
    • github.com/temporalio/ringpop-go-6f91b5915e95e7b08817aae78fea3a38ea1b5bd2
      • github.com/apache/thrift-0.10.0 (Vulnerable Library)

Found in HEAD commit: 2288f4128e7e1dfd80ecbde2df89d8e81810860e

Found in base branch: main

Vulnerability Details

The Apache Thrift Node.js static web server in versions 0.9.2 through 0.11.0 have been determined to contain a security vulnerability in which a remote user has the ability to access files outside the set webservers docroot path.

Publish Date: 2019-01-07

URL: CVE-2018-11798

CVSS 3 Score Details (6.5)

Base Score Metrics:

  • Exploitability Metrics:
    • Attack Vector: Network
    • Attack Complexity: Low
    • Privileges Required: Low
    • User Interaction: None
    • Scope: Unchanged
  • Impact Metrics:
    • Confidentiality Impact: High
    • Integrity Impact: None
    • Availability Impact: None

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-11798

Release Date: 2019-01-07

Fix Resolution: v0.12.0

Move from Sphinx API docs to pydoctor

Describe the solution you'd like

We already have gen-docs-pydoctor alternative, we just need to make that the default and remove Sphinx. This requires upstreaming support for @overload to pydoctor.

[Feature Request] Inject dependencies into activities

Is your feature request related to a problem? Please describe.

Hi,

This may admittedly be more of a question than a feature request as there might already be a simple solution I missed. But nonetheless, this may be a topic worth documenting for users new to async/temporal.

I'm looking for a way to pass request/job-specific dependencies into the activity functions, such as the DB session. One idea described below uses a factory to create the worker, like you'd see in some other frameworks. The problem is creating a closure around the activities makes it impossible to import those activities into the workflows throwing away type-safety.

Describe the solution you'd like

Here's an example of how I initially tried to solve the issue:

# main.py
def create_worker() -> Worker:
    my_app: MyApp = bootstrap()

    @activity.defn
    async def task1(user_id: uuid.UUID) -> str:
        return my_app.task1(
            user_id=user_id
        )

    @activity.defn
    async def task2(user_id: uuid.UUID) -> str:
        return my_app.task2(
            user_id=user_id
        )

    @activity.defn
    async def task3(user_id: uuid.UUID) -> str:
        return my_app.task3(
            user_id=user_id
        )
    
    client = await Client.connect("http://localhost:7233")
    return Worker(
        client,
        task_queue="my-task-queue",
        workflows=[MyWorkflow],
        activities=[
            task1,
            task2,
            task3,
        ],
    )


async def main():
    worker = create_worker()
    await worker.run()


if __name__ == "__main__":
    asyncio.run(main())

This is similar to how you might write a Flask API's create_app factory. Another framework, FastAPI, supports async and has full DI around its route handlers.

I don't have a great deal of experience building async apps in Python, so it isn't completely clear to me whether the above solution would even be safe. The MyApp object returned by bootstrap handles transaction management and all the business logic (follows a Hexagonal approach), so it definitely shouldn't be used for more than one activity at a time.

I'm going to feel silly now after writing all of this if the simplest solution is just to call bootstrap within the activity like below:

@activity.defn
async def task1(user_id: uuid.UUID) -> str:
    my_app = bootstrap()
    return my_app.task1(
        user_id=user_id
    )

I've always thought this would be bad to repeatedly throw away the internal DB session and various HTTP/SDK clients after each "request". But I guess that only works in sync frameworks because it processes one request at a time.

Additional context

I think I may have just rubber-ducked the problem to myself, but it would be good to see some guidance on the best practice here.

Thanks!

go.temporal.io/server-v1.14.4: 4 vulnerabilities (highest severity is: 7.5) - autoclosed

Vulnerable Library - go.temporal.io/server-v1.14.4

Vulnerabilities

CVE Severity CVSS Dependency Type Fixed in Remediation Available
CVE-2019-0205 High 7.5 github.com/apache/thrift-0.10.0 Transitive N/A
CVE-2022-21698 High 7.5 github.com/prometheus/client_golang-v1.11.0 Transitive N/A
CVE-2019-0210 High 7.5 github.com/apache/thrift-0.10.0 Transitive N/A
CVE-2018-11798 Medium 6.5 github.com/apache/thrift-0.10.0 Transitive N/A

Details

CVE-2019-0205

Vulnerable Library - github.com/apache/thrift-0.10.0

Apache Thrift

Dependency Hierarchy:

  • go.temporal.io/server-v1.14.4 (Root Library)
    • github.com/uber/tchannel-go-v1.21.0
      • github.com/apache/thrift-0.10.0 (Vulnerable Library)

Found in base branch: main

Vulnerability Details

In Apache Thrift all versions up to and including 0.12.0, a server or client may run into an endless loop when feed with specific input data. Because the issue had already been partially fixed in version 0.11.0, depending on the installed version it affects only certain language bindings.

Publish Date: 2019-10-29

URL: CVE-2019-0205

CVSS 3 Score Details (7.5)

Base Score Metrics:

  • Exploitability Metrics:
    • Attack Vector: Network
    • Attack Complexity: Low
    • Privileges Required: None
    • User Interaction: None
    • Scope: Unchanged
  • Impact Metrics:
    • Confidentiality Impact: None
    • Integrity Impact: None
    • Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-0205

Release Date: 2019-10-29

Fix Resolution: org.apache.thrift:libthrift:0.13.0

CVE-2022-21698

Vulnerable Library - github.com/prometheus/client_golang-v1.11.0

Prometheus instrumentation library for Go applications

Dependency Hierarchy:

  • go.temporal.io/server-v1.14.4 (Root Library)
    • github.com/prometheus/client_golang-v1.11.0 (Vulnerable Library)

Found in base branch: main

Vulnerability Details

client_golang is the instrumentation library for Go applications in Prometheus, and the promhttp package in client_golang provides tooling around HTTP servers and clients. In client_golang prior to version 1.11.1, HTTP server is susceptible to a Denial of Service through unbounded cardinality, and potential memory exhaustion, when handling requests with non-standard HTTP methods. In order to be affected, an instrumented software must use any of promhttp.InstrumentHandler* middleware except RequestsInFlight; not filter any specific methods (e.g GET) before middleware; pass metric with method label name to our middleware; and not have any firewall/LB/proxy that filters away requests with unknown method. client_golang version 1.11.1 contains a patch for this issue. Several workarounds are available, including removing the method label name from counter/gauge used in the InstrumentHandler; turning off affected promhttp handlers; adding custom middleware before promhttp handler that will sanitize the request method given by Go http.Request; and using a reverse proxy or web application firewall, configured to only allow a limited set of methods.

Publish Date: 2022-02-15

URL: CVE-2022-21698

CVSS 3 Score Details (7.5)

Base Score Metrics:

  • Exploitability Metrics:
    • Attack Vector: Network
    • Attack Complexity: Low
    • Privileges Required: None
    • User Interaction: None
    • Scope: Unchanged
  • Impact Metrics:
    • Confidentiality Impact: None
    • Integrity Impact: None
    • Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: GHSA-cg3q-j54f-5p7p

Release Date: 2022-02-15

Fix Resolution: v1.11.1

CVE-2019-0210

Vulnerable Library - github.com/apache/thrift-0.10.0

Apache Thrift

Dependency Hierarchy:

  • go.temporal.io/server-v1.14.4 (Root Library)
    • github.com/uber/tchannel-go-v1.21.0
      • github.com/apache/thrift-0.10.0 (Vulnerable Library)

Found in base branch: main

Vulnerability Details

In Apache Thrift 0.9.3 to 0.12.0, a server implemented in Go using TJSONProtocol or TSimpleJSONProtocol may panic when feed with invalid input data.

Publish Date: 2019-10-29

URL: CVE-2019-0210

CVSS 3 Score Details (7.5)

Base Score Metrics:

  • Exploitability Metrics:
    • Attack Vector: Network
    • Attack Complexity: Low
    • Privileges Required: None
    • User Interaction: None
    • Scope: Unchanged
  • Impact Metrics:
    • Confidentiality Impact: None
    • Integrity Impact: None
    • Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: http://mail-archives.apache.org/mod_mbox/thrift-dev/201910.mbox/%3C277A46CA87494176B1BBCF5D72624A2A%40HAGGIS%3E

Release Date: 2019-10-29

Fix Resolution: 0.13.0

CVE-2018-11798

Vulnerable Library - github.com/apache/thrift-0.10.0

Apache Thrift

Dependency Hierarchy:

  • go.temporal.io/server-v1.14.4 (Root Library)
    • github.com/uber/tchannel-go-v1.21.0
      • github.com/apache/thrift-0.10.0 (Vulnerable Library)

Found in base branch: main

Vulnerability Details

The Apache Thrift Node.js static web server in versions 0.9.2 through 0.11.0 have been determined to contain a security vulnerability in which a remote user has the ability to access files outside the set webservers docroot path.

Publish Date: 2019-01-07

URL: CVE-2018-11798

CVSS 3 Score Details (6.5)

Base Score Metrics:

  • Exploitability Metrics:
    • Attack Vector: Network
    • Attack Complexity: Low
    • Privileges Required: Low
    • User Interaction: None
    • Scope: Unchanged
  • Impact Metrics:
    • Confidentiality Impact: High
    • Integrity Impact: None
    • Availability Impact: None

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-11798

Release Date: 2019-01-07

Fix Resolution: v0.12.0

Propagate fatal worker errors

Describe the solution you'd like

Make sure that fatal worker errors bubble out of run() and there is some way for people to add a "fatal error handler" for async with which has no way to otherwise send an error.

Roadmap

Hello there!

We'll be looking into using Temporal in the upcoming quarter. We have quite a few Python services running in our backend. So I was wondering, can you maybe share a bit on the roadmap for this Python SDK? Among other things, I'm wondering how soon it will be mature enough to put on production, or at least leave the alpha stage.

Thanks!

[Feature Request] `retry_policy` on `continue_as_new`

Is your feature request related to a problem? Please describe.

From temporalio/sdk-java#1200 and temporalio/sdk-java#1201:

Right now continueAsNew is implemented in a way that assumes that most parameters are carried over by the server if not specified on the continueAsNew request. While in fact, it's the opposite. Server doesn't carry over most parameters.

If in the following code:

  • raise on iteration == 0: The workflow is run (retried) 3 time before ending with FAILD (retryState: MaximumAttemptsReached)
  • raise on iteration == 1: The workflow runs one time successfully Continued as New. The next run it ends with Failed (retryState: RetryPolicyNotSet).

I'd expect the workflow to retry 3 time before giving up.

import asyncio
import logging
from dataclasses import dataclass
from typing import NoReturn
from datetime import timedelta

from temporalio import workflow, activity
from temporalio.client import Client
from temporalio.worker import Worker
from temporalio.common import RetryPolicy

@dataclass
class ComposeGreetingInput:
    greeting: str
    name: str

@activity.defn
async def compose_greeting(input: ComposeGreetingInput) -> NoReturn:
    # Always raise exception
    raise RuntimeError(f"Greeting exception: {input.greeting}, {input.name}!")


@workflow.defn(name="LoopingWorkflow")
class LoopingWorkflow:
    @workflow.run
    async def run(self, iteration: int) -> None:
        if iteration == 40:
            return
        workflow.logger.info("Running workflow iteration %s", iteration)
        await asyncio.sleep(1)
        
        # on which iteration to raise
        if (iteration == 0):
            await workflow.execute_activity(
                compose_greeting,
                ComposeGreetingInput("Hello", f"{iteration}"),
                start_to_close_timeout=timedelta(seconds=10),
                retry_policy=RetryPolicy(maximum_attempts=1),
            )

        workflow.continue_as_new(
            iteration + 1,
            #TypeError: continue_as_new() got an unexpected keyword argument 'retry_policy'
            #retry_policy=RetryPolicy(maximum_attempts=3),
        )



async def main():
    # Enable logging for this sample
    logging.basicConfig(level=logging.INFO)

    # Start client
    client = await Client.connect("localhost:7233")

    # Run a worker for the workflow
    async with Worker(
        client,
        task_queue="hello-continue-as-new-task-queue",
        workflows=[LoopingWorkflow],
        activities=[compose_greeting],
    ):

        await client.start_workflow(
            "LoopingWorkflow",
            0,
            id="hello-continue-as-new-workflow-id",
            task_queue="hello-continue-as-new-task-queue",
            retry_policy=RetryPolicy(maximum_attempts=3),
        )

        await asyncio.Future()


if __name__ == "__main__":
    asyncio.run(main())

Describe the solution you'd like

The real solution might be for temporal server to carry over workflow options on continue_as_new. For the time being, we could work around it by adding retry_policy to continue_as_new (similar to search_attributes).

[Bug] M1 Mac grpcio error

What are you really trying to do?

Running temporal python workers on M1 Mac

Describe the bug

import cygrpc error when trying to run sample python code

Screenshot 2022-07-22 at 6 29 45 PM

Minimal Reproduction

  1. create conda env for temporal with python 3.82.
  2. pip install --no-binary :all: grpcio --ignore-installed3.
  3. pip install --no-binary :all: grpcio-tools --ignore-installed4.
  4. pip install temporalio 5.
    Run the sample run_worker.py
  • OS and processor: M1 Mac OS 12.4
  • Temporal Version: latest and/or SDK version 0.1a2
  • Are you using Docker or Kubernetes or building Temporal from source? Docker Compose to run Temporal

Additional context

Properly treat activity/child cancellation errors bubbled as workflow cancel

Describe the bug

Workflow cancellation not reported as such when bubbling the activity cancellation. TypeScript properly converts activity error and child workflow error that are caused by cancellation into workflow cancellation when a cancellation is requested. We need to do the same.

Technically it means that if a user wraps the error the workflow will show as failed not cancelled, but there's not much we can do about that.

Async activity support

Describe the solution you'd like

Need async activity support in client and activity itself

Immediate start-then-cancel not working properly

Describe the bug

When you start-then-cancel, i.e. this event set:
image

then the primary task is getting cancelled before even running which means the workflow code never runs which means the cancelled event is never caught. We need to defer to self._primary_task.cancel() so it is put on the event loop after the start so the user's code still runs and can react to cancellation.

github.com/temporalio/temporal-v1.14.4: 4 vulnerabilities (highest severity is: 7.5) - autoclosed

Vulnerable Library - github.com/temporalio/temporal-v1.14.4

Found in HEAD commit: f1aa1c91b664fedd68e65b2f6ce9727e24e5f2af

Vulnerabilities

CVE Severity CVSS Dependency Type Fixed in Remediation Available
CVE-2019-0205 High 7.5 github.com/apache/thrift-0.10.0 Transitive N/A
CVE-2022-21698 High 7.5 github.com/prometheus/client_golang-v1.11.0 Transitive N/A
CVE-2019-0210 High 7.5 github.com/apache/thrift-0.10.0 Transitive N/A
CVE-2018-11798 Medium 6.5 github.com/apache/thrift-0.10.0 Transitive N/A

Details

CVE-2019-0205

Vulnerable Library - github.com/apache/thrift-0.10.0

Apache Thrift

Dependency Hierarchy:

  • github.com/temporalio/temporal-v1.14.4 (Root Library)
    • github.com/temporalio/ringpop-go-6f91b5915e95e7b08817aae78fea3a38ea1b5bd2
      • github.com/apache/thrift-0.10.0 (Vulnerable Library)

Found in HEAD commit: f1aa1c91b664fedd68e65b2f6ce9727e24e5f2af

Found in base branch: main

Vulnerability Details

In Apache Thrift all versions up to and including 0.12.0, a server or client may run into an endless loop when feed with specific input data. Because the issue had already been partially fixed in version 0.11.0, depending on the installed version it affects only certain language bindings.

Publish Date: 2019-10-29

URL: CVE-2019-0205

CVSS 3 Score Details (7.5)

Base Score Metrics:

  • Exploitability Metrics:
    • Attack Vector: Network
    • Attack Complexity: Low
    • Privileges Required: None
    • User Interaction: None
    • Scope: Unchanged
  • Impact Metrics:
    • Confidentiality Impact: None
    • Integrity Impact: None
    • Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-0205

Release Date: 2019-10-29

Fix Resolution: org.apache.thrift:libthrift:0.13.0

CVE-2022-21698

Vulnerable Library - github.com/prometheus/client_golang-v1.11.0

Prometheus instrumentation library for Go applications

Dependency Hierarchy:

  • github.com/temporalio/temporal-v1.14.4 (Root Library)
    • github.com/prometheus/client_golang-v1.11.0 (Vulnerable Library)

Found in HEAD commit: f1aa1c91b664fedd68e65b2f6ce9727e24e5f2af

Found in base branch: main

Vulnerability Details

client_golang is the instrumentation library for Go applications in Prometheus, and the promhttp package in client_golang provides tooling around HTTP servers and clients. In client_golang prior to version 1.11.1, HTTP server is susceptible to a Denial of Service through unbounded cardinality, and potential memory exhaustion, when handling requests with non-standard HTTP methods. In order to be affected, an instrumented software must use any of promhttp.InstrumentHandler* middleware except RequestsInFlight; not filter any specific methods (e.g GET) before middleware; pass metric with method label name to our middleware; and not have any firewall/LB/proxy that filters away requests with unknown method. client_golang version 1.11.1 contains a patch for this issue. Several workarounds are available, including removing the method label name from counter/gauge used in the InstrumentHandler; turning off affected promhttp handlers; adding custom middleware before promhttp handler that will sanitize the request method given by Go http.Request; and using a reverse proxy or web application firewall, configured to only allow a limited set of methods.

Publish Date: 2022-02-15

URL: CVE-2022-21698

CVSS 3 Score Details (7.5)

Base Score Metrics:

  • Exploitability Metrics:
    • Attack Vector: Network
    • Attack Complexity: Low
    • Privileges Required: None
    • User Interaction: None
    • Scope: Unchanged
  • Impact Metrics:
    • Confidentiality Impact: None
    • Integrity Impact: None
    • Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: GHSA-cg3q-j54f-5p7p

Release Date: 2022-02-15

Fix Resolution: v1.11.1

CVE-2019-0210

Vulnerable Library - github.com/apache/thrift-0.10.0

Apache Thrift

Dependency Hierarchy:

  • github.com/temporalio/temporal-v1.14.4 (Root Library)
    • github.com/temporalio/ringpop-go-6f91b5915e95e7b08817aae78fea3a38ea1b5bd2
      • github.com/apache/thrift-0.10.0 (Vulnerable Library)

Found in HEAD commit: f1aa1c91b664fedd68e65b2f6ce9727e24e5f2af

Found in base branch: main

Vulnerability Details

In Apache Thrift 0.9.3 to 0.12.0, a server implemented in Go using TJSONProtocol or TSimpleJSONProtocol may panic when feed with invalid input data.

Publish Date: 2019-10-29

URL: CVE-2019-0210

CVSS 3 Score Details (7.5)

Base Score Metrics:

  • Exploitability Metrics:
    • Attack Vector: Network
    • Attack Complexity: Low
    • Privileges Required: None
    • User Interaction: None
    • Scope: Unchanged
  • Impact Metrics:
    • Confidentiality Impact: None
    • Integrity Impact: None
    • Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: http://mail-archives.apache.org/mod_mbox/thrift-dev/201910.mbox/%3C277A46CA87494176B1BBCF5D72624A2A%40HAGGIS%3E

Release Date: 2019-10-29

Fix Resolution: 0.13.0

CVE-2018-11798

Vulnerable Library - github.com/apache/thrift-0.10.0

Apache Thrift

Dependency Hierarchy:

  • github.com/temporalio/temporal-v1.14.4 (Root Library)
    • github.com/temporalio/ringpop-go-6f91b5915e95e7b08817aae78fea3a38ea1b5bd2
      • github.com/apache/thrift-0.10.0 (Vulnerable Library)

Found in HEAD commit: f1aa1c91b664fedd68e65b2f6ce9727e24e5f2af

Found in base branch: main

Vulnerability Details

The Apache Thrift Node.js static web server in versions 0.9.2 through 0.11.0 have been determined to contain a security vulnerability in which a remote user has the ability to access files outside the set webservers docroot path.

Publish Date: 2019-01-07

URL: CVE-2018-11798

CVSS 3 Score Details (6.5)

Base Score Metrics:

  • Exploitability Metrics:
    • Attack Vector: Network
    • Attack Complexity: Low
    • Privileges Required: Low
    • User Interaction: None
    • Scope: Unchanged
  • Impact Metrics:
    • Confidentiality Impact: High
    • Integrity Impact: None
    • Availability Impact: None

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-11798

Release Date: 2019-01-07

Fix Resolution: v0.12.0

Replayer

Describe the solution you'd like

Need workflow replayer (also includes history JSON parser/fixer)

Add more robust docs to docstrings

Describe the solution you'd like

  • Module/package level docs should have more docs in the docstrings explaining usage of that package
  • Maybe even examples, but don't go overboard

Add build docs to README

Is your feature request related to a problem? Please describe.

It is not clear from the README how to build the repo from source for use as a dependency. We only have local build instructions.

Describe the solution you'd like

Add a section for building from source and even an example for using after built from source. Also make it clear that users must clone recursively because temporalio/bridge/sdk-core is a git submodule.

Timed out wait_condition futures still attempt setting result

Describe the bug

When using workflow.wait_condition, if a wait condition times out then gets a bool success, we attempt to set the future which causes an `InvalidStateError.

Minimal Reproduction

Just timeout a wait_condition and let its callback become true

Test framework

Describe the solution you'd like

Need test framework that leverages the test server

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.