equinor / oneseismic

oneseismic

Next-generation seismic in the cloud.

Please note that oneseismic is under heavy development, and parts of this document may be outdated.

What is oneseismic?

Technically, oneseismic is an API for reading seismic data in an easy and scalable manner. The biggest challenge with seismic is the size of it - single surveys span from a few hundred megabytes to tens or even hundreds of gigabytes. The standard format for storing seismic, SEG-Y, is unfit for efficient data extraction and querying.

The guiding design principle and focus of oneseismic is programs first - it is the idea that if you can build a solid foundation then imagining new applications on top is fast and easy. The aim is to provide a powerful feature that empowers developers and geoscientists so they can develop new and novel applications, and get results faster and easier.

The best way to illustrate this is with a motivating example:

import oneseismic
import oneseismic.simple as simple

cubeid = '...'
client = simple.client('https://oneseismic.url')
inline24    = cube.sliceByLineno(cubeid, dim = 0, lineno = 24 )().numpy()
crossline13 = cube.sliceByLineno(cubeid, dim = 1, lineno = 13 )().numpy()
time220     = cube.sliceByLineno(cubeid, dim = 2, lineno = 220)().numpy()

This Python program gets three slices - an inline slice, a crossline slice, and a time slice, and makes them immediately available. This simple example only demonstrates the fetching of arbitrary data, but we can also do something useful with it:

vintage1 = '...'
vintage2 = '...'
client = simple.client('https://oneseismic.url')
proc1 = client.sliceByLineno(vintage1, dim = 1, lineno = 13)()
proc2 = client.sliceByLineno(vintage2, dim = 1, lineno = 13)()
slicev1 = proc1.numpy()
slicev2 = proc2.numpy()

diff = slicev2 - slicev1

This program computes the difference between the samples between two vintages of the same field.

Notice that instead of immediately realising the data as a numpy array, this program uses the temporaries proc1 and proc2. Creating a process will schedule the fetch, but not actually start serve the data right away. This makes the oneseismic server process both queries in parallel.

Is oneseismic a database?

That depends on the definition of database. The goal of oneseismic is not to be a universal storage solution for seismic data, but rather an efficient way to work with and requests bits and pieces of seismic data. In that sense, it is a database.

Why not SEG-Y?

SEG-Y was designed for data exchange, which means density, single-file and in-band metadata are useful properties because it allows for space-efficient and lossless transfer between parties. SEG-Y works well for this (with the exception of rampant SEG-Y standards violations). However, SEG-Y is unfit for modern computer programs:

1. Meta-data is interleaved with the data. That means the file can be split multiple places (good for tape!), but also means that the data cannot be contiguously copied.
2. SEG-Y is very trace-oriented, but there is no requirement that traces are laid out for efficient access of 3D shapes. Well-organised files are laid out for efficient inline access, but without an index it requires extra information, or a linear scan.
3. Even if the file is well organised, reading a single time/depth slice or horizon is very time consuming.

Installation

Oneseismic is primarily an API, so there is no installation - the system is up and running and be queried at any time. The Python package is a user-friendly way to use the API, and can be installed with:

pip install oneseismic

However, the API is perfectly usable without going through the Python package. Please note that it is still under heavy development, and may change with little notice.

Examples

Developer's corner

This section is for the developers of oneseismic, and describes the architecture and design choices that power oneseismic.

Offline partitioning

When uploaded, the volume is partitioned into equally-sized chunks, which are then addressed by its coordinates in this coarser grid. This process is time consuming, but is only performed once. With a unique address <volume>/<resolution>/<partioning>/<chunk>, which can be easily computed from any coordinate, oneseismic can efficiently get arbitrary shapes from large volumes.

Terminology

A lot of the familiar terminology in oneseismic is lifted from the unix family of operating systems, since the concepts in use in oneseismic map pretty well onto the concepts in unix. This is an incomplete list of terms used throughout code and documentation:

process : The process is the high-level procedure from a user request until data is delivered.

PID : The PID, process identifier, is the key used to identify a single process across the sub systems. Please note that unlike traditional unix systems, this is represented as a string, and not a single integer.

A day in the life (of a request)

License

The server is licensed under AGPL v3+, while the connector and python libraries are licensed under the LGPL v3+.

oneseismic's Issues

API: Generate default config file with command

Add CI

Set up continuous integration on seismic-cloud

closed by #9

Http server should be configurable with injection of needed services.

API: Log to "central repository"

• Write logs to database (pgresql for mobility?)
• Write an error handler / severity handler for logging
• Reqrite/ refactor out existing log/ logger calls

API: Handle issuer iss other than AuthServer

API: Deploy to benchmark env

API: Create glossary for seismic terms in use in the API

Benchmark suite

There are several techniques we can try to improve performance, but without a simple and correct way of comparing results its impossible to experiment.

So what we need is a benchmark suite - a set of test queries (lines, horizontal slices, curtains, horizons and horizons with window) with timing reports.

The timing reports from the driver program can be a part of the response itself, at least with a parameter, as well as a wallclock timing start-to-finish from the client.

Obviously, on top of that, the response should be tested for correctness, i.e. that the correct data is returned.

Persist profiling numbers server side

• Provide file/pipe for stitch so the wrapper API can read profiling numbers from stitch
• Write profile to persistence layer
• Expose endpoint that provides access to profiling data for sessions, i.e (/profile/{sessionID})

Maifests should be read in first from stream to SC

Move seismic cloud to a cheaper cluster setup.

The /core functionality should be "containerified".
The api can already be deployed as a standalone container. But it should be able to stream to a core program over tcp.

API: API to SC integration

As a minimum we should discuss the elements below with regards to integration between the two components:

• Manifest
• Byte stream
• Error handling (Including what should be propagated to the client calling the API)

API: Integration test in CI

Add an integration test with docker containers. Use the python client and some setup to run a little battery of test surfaces.

Integration tests:
Read manifests from Path

Ingestion: Add test for segmenter with primary/secondary key as 2-byte words

Add tests for primary and secondary keys as 2-byte values.

• key1 2-byte, key2 2-byte
• key1 2-byte, key2 4-byte
• key1 4-byte, key2 2-byte

Create client for the seismic cloud CI integration testing

API: API definition

So we need an API definition and a swagger file. Arguably, the API is the real product here, and needs a lot of love.

The API definition work can be done in parallel with the benchmark suite (at least for the most part). Both in terms of implementation, testing, and design.

API: Manifests should be stored in a DB

The manifest can be read and stored in memory on startup of the API

• Create CosmosDB in Azure
• Access CosmosDB with MongoDB driver for Golang
• Read Manifest from DB

Add Web API, with jwt token authorization

Deploy to radix

API: Simplify Azure devops yaml file

API & SC: Manifest in payload when API calls SC

API: Set up parallel stage build CI

API: Add profiling middleware

We need a profiling middleware thatt times the request. And sets a header for the client so it knows how long the server is working on the request.

API: The service should read config from a file/ or environment

Shatter Client: Create client towards Shatter service

InMemory ProfileStore should remove items 1 hour after last append

Realistic (very large) synthetic SEGY dataset prep

@maxschub

During our investigation efforts, in order to undertand the demands of processing, querying a SEGY dataset on any Azure data service(s), it is crucial that we work with a realistic (very large) synthetic SEGY dataset.

• How big should we go here (time, surface coverage area, approx. file size)?
• When can you provide it?

Thanks!

API: New stitch controller using surface store

New endpoint for stitch using the surface store

API: Integrate with core grpc server

Core is going to be a grpc server implemented in python. We will have to implement a layer that interfaces with this new direction. The core will have the opportunity of running as a independent server, but we will be doing the deployment/management of this.

API: enable https

API: Increase test coverage

Set test coverage requirement to pass build

Sample data prep and queries

• @jokva Export the Valve (SEGY) sample dataset as CSV. Place it under a new container/folder in the segyngrmdfdev Azure storage account.
• @jokva Provide sample cube queries (code) for the several different cases: Axis parallel slicing, Curtain Section, Amplitude/Attribute map and the most complex Generalization/random points in cube.

SC: shatter of incoming cubes

Add Python client

Ingestion: Test fun variations of segments

Phenomena such as:

• Decreasing line numbers
• Missing traces
• Detecting duplicate line numbers outside segment

Investigate, evaluate Azure data services

• @CarlosSardo Investigate, evaluate what Azure data services fit best, several options:
• Azure Data Explorer: fast and highly scalable data exploration service; ideal for analyzing large volumes of diverse data; Scales quickly to terabytes of data, in minutes, allowing rapid iterations of data exploration to discover relevant insights; Supports analysis of high volumes of heterogeneous data (structured and unstructured); Offers an innovative query language, optimized for high performance data analytics.
• Azure HDInsight: popular open source frameworks—including Apache Hadoop, Hive, Spark, and Kafka—using Azure HDInsight, a cost-effective, enterprise-grade service for open source analytics. Effortlessly process massive amounts of data and get all the benefits of the broad open source ecosystem with the global scale of Azure.
• Azure Databricks: Apache Spark-based analytics platform optimized for the Microsoft Azure cloud services platform; Dynamically autoscale clusters up and down;

Ingestion: Add test for segments with skipped line numbers

E.g. key1/key2 pairs of [(1, 1), (1, 2), (3, 1), (3, 2), (4, 1), (4, 2)]

Containerify core programs and wrap with netcat.

parent #59

Clarify what we should log

API: Create monitoring dashboard

• Configure Prometheus sample equinor/radix-platform#102
• Test integration with Azure Monitor/Dashboard
  Azure does not allow anonymous export of dashboard
• Dashboard with Grafana equinor/radix-platform#115
• Deploy to Radix
• Docker composer for local testing
• Deploy dashboard configuration from git
• Serve metrics to nonpublic port
• Grafana AD Authentication (Later)
• Persisting storage of metrics (Separate Issue) https://github.com/equinor/radix/issues/8
• Deploy monitoring to separate deploy repo https://github.com/equinor/oneSeismic-deploy

API: Create stubbed response in API

• Create surface
• Put surfaces in blob storage (we are skipping this part in the first place)
• RPC from API to core stub
• Refactor stitch.go

Framework evaluation

API framework evaluation

SC: Create deployment pipeline for the whole system

Add stdin/out link to seismic cloud

Run the seismic cloud program inside the api, stream body to stdin and return stdout

connects #16

Simple manifest service should read manifests from folder.

Provisioning of Azure resources

• @maxschub Create new Azure resource group under the same region where the SEGY data should be placed.
• @maxschub After creating the new Azure resource group, assign the users ([email protected], jokva (Guest) and yourself) as co-Owners (I ask this high-access level, to avoid any resource permissions constraints).

API: Upload surfaces to azure blobstorage

Get API spec from Bluware

We are planning to implement Bluware's API spec, and hence to get our hands on it ..

API: Introduce abstract factory to generate services

Ingestion: Test segmentation outputs segments with the right name for primary/secondary keys

        {
            "binInfoStart": {
                "crosslineNumber": 20,
                "ensembleXCoordinate": 0.0,
                "ensembleYCoordinate": 0.0,
                "inlineNumber": 1
            },
            "binInfoStop": {
                "crosslineNumber": 24,
                "ensembleXCoordinate": 0.0,
                "ensembleYCoordinate": 0.0,
                "inlineNumber": 1
            },
            "primaryKey": 1,
            "traceStart": 0,
            "traceStop": 4
        },

This is fine when inline/crossline are 189/193, but check with what SEGYScan outputs for other keys, and verify they're the same.