Coder Social home page Coder Social logo

conda-forge-nightly-controller's Introduction

Centralized nightly CI builds for TileDB conda feedstocks

Name status azure version last updated downloads
TileDB (feedstock) tiledb tiledb-azure version last updated downloads
TileDB-Py (feedstock) tiledb-py tiledb-py-azure version last updated downloads

How it works

  • The GitHub Actions workflows in the repository are scheduled run each night (they can also be manually triggered)

  • The job clones both the TileDB-Inc fork of the feedstock repo and also the source repo

  • The job updates the recipe (meta.yaml) to use the version string (X.X.X.YYYY_MM_DD), where X.X.X are derived from the source repo

  • The job also updates the upload channels so that the conda binaries are uploaded to the tiledb channel on anaconda.org with the label "nightlies".

  • The job force pushes to the feedstock branch "nightly-build" to trigger Azure builds and uploads (this is made possible by manually configured SSH keys; see below)

Run and test locally

To semi-reproduce the GitHub Actions workflows, run the following:

bash run-local-tiledb.sh
bash run-local-tiledb-py.sh

# To push changes to feedstock repos
# (not recommended unless GitHub Actions is broken)
bash run-local-tiledb.sh TRUE
# Wait for tiledb-feedstock runs to finish
bash run-local-tiledb-py.sh TRUE

To locally install a nightly version:

mamba create --yes -n test-nightlies \
  -c conda-forge -c "tiledb/label/nightlies" \
  --override-channels tiledb-py="*2022*"

SSH keys

For each feedstock, generate a new SSH key pair:

  1. Generate SSH keys on local machine. Hit enter twice to omit a password:

    mkdir /tmp/ssh-temp/
    ssh-keygen -t rsa -b 4096 -C "GitHub Actions for tiledb-nightlies" -f /tmp/ssh-temp/key
    head -n 1 /tmp/ssh-temp/key
    ## -----BEGIN RSA PRIVATE KEY-----
  2. Add SSH private key (/tmp/ssh-temp/key) to tiledb-nightlies as a repository secret named SSH_PRIVATE_KEY_<software>:

    • Settings -> Secrets -> Actions -> New repository secret
    • Note: the name of the secret cannot include dashes (GitHub restriction)
  3. Add SSH public key (/tmp/ssh-temp/key.pub) to TileDB-Inc fork of feedstock repository as a deploy key with write access:

    • Settings -> Deploy keys -> Add deploy key
    • Recommended to name it "tiledb-nightlies" to make the purpose of the key more obvious, but the name has no effect on functionality
    • Make sure to tick the box "Allow write access"!
  4. Delete the keys locally. It's best practice to limit each key pair to only allow push access to a single repository, and regardless GitHub won't let you re-use them anyways

    rm -r /tmp/ssh-temp/

Anaconda.org upload token (BINSTAR_TOKEN)

The instructions below are based off of the conda-smithy instructions for making a new feedstock.

  • (Once per GitHub org/user) Create an account on Azure DevOps by authenticating with the corresponding GitHub user or org account. Importantly, you don't need to sign-up for the full Azure cloud experience to run Azure pipelines (I made this mistake, and now my inbox is flooded with Azure how-to emails)

  • (Once per GitHub org/user) Either pay for builds or apply for free builds for open source projects at https://aka.ms/azpipelines-parallelism-request. By default new Azure DevOps organizations are granted zero parallel builds, which means you can't run anything, even serially

  • (Once per GitHub org/user) Create a new project named "feedstock-builds". This will be used to run the CI for all the feedstocks in your GitHub org/user. By default it is private. If you haven't already, you have to first allow public projects in your Azure DevOps account before you can make it public

  • (Once per GitHub org/user) Connect the project "feedstock-builds" to your GitHub org/user via a "service connection". Go to "Project Settings" (bottom left in UI) -> "Service Connections" -> "Create Service Connection" -> GitHub. Grant it authorization via OAuth using the GitHub OAuth App AzurePipelines (to avoid having to generate a PAT). Name the service connection the same as your GitHub user/org

  • Fork the conda-forge feedstock repo to your org/user account

  • Clone the fork to your local machine

  • (Once per machine) Install conda-smithy

    mamba install -c conda-forge conda-smithy
  • Create an Azure token and save it in ~/.conda-smithy/azure.token

  • Run conda smithy register-ci in the local feedstock repo to activate builds on Azure DevOps. Note that you don't need to bother with the Anaconda token. Azure requires you to upload it manually later, so there's no point in saving it locally

    export AZURE_ORG_OR_USER="<your GitHub org/user>"
    # Switch --organization with --user for a GitHub user account
    conda smithy register-ci \
      --organization "<your GitHub org/user>" \
      --feedstock_directory . \
      --without-travis \
      --without-circle \
      --without-appveyor \
      --without-drone \
      --without-webservice \
      --without-anaconda-token

    If you were successful, you'll now see the new feedstock listed at https://dev.azure.com/<account>/feedstock-builds/_build?view=folders

  • Create a token for anaconda.org

    • Login to your account at anaconda.org
    • Settings -> Access
    • Choose scope "Allow all API operations". I couldn't find any documentation that linked the scopes to allowable actions. I tried "Allow all operations on Conda repositories", but that failed due to insufficient permissions.
  • Add the anaconda.org token as a pipeline variable on Azure. From the pipeline page: Edit -> Variables -> New variable -> Name it BINSTAR_TOKEN -> Copy-paste token -> Check "Keep this value secret" -> OK -> Save

conda-forge-nightly-controller's People

Contributors

dudoslav avatar ihnorton avatar jdblischak avatar runneradmin avatar shaunrd0 avatar teo-tsirpanis avatar

Stargazers

 avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

conda-forge-nightly-controller's Issues

Nightly libtiledb builds have been failing for over a month

I noticed today that the libtiledb nightlies builds have been broken for over a month (feedstock build logs for branch nightly-build). The oldest build log is for May 17. The error message is:

-- Found AWS lib: aws-cpp-sdk-identity-management ($SRC_DIR/build/externals/install/lib64/libaws-cpp-sdk-identity-management.a)
-- The TileDB library is compiled with Azure support.
CMake Error at tiledb/CMakeLists.txt:529 (find_package):
  Could not find a package configuration file provided by
  "azure-storage-blobs-cpp" with any of the following names:

    azure-storage-blobs-cppConfig.cmake
    azure-storage-blobs-cpp-config.cmake

  Add the installation prefix of "azure-storage-blobs-cpp" to
  CMAKE_PREFIX_PATH or set "azure-storage-blobs-cpp_DIR" to a directory
  containing one of the above files.  If "azure-storage-blobs-cpp" provides a
  separate development package or SDK, be sure it has been installed.


-- Configuring incomplete, errors occurred!

Thus I think the problem is related to the recently added support for the new Azure SDK for C++, which was merged on April 23rd

https://github.com/search?q=repo%3ATileDB-Inc%2FTileDB+azure-storage-blobs-cpp&type=code
TileDB-Inc/TileDB#3910
TileDB-Inc/TileDB@969dd1c

But has not yet been included in a release

https://github.com/TileDB-Inc/TileDB/releases
https://github.com/conda-forge/tiledb-feedstock/pulls?q=is%3Apr+is%3Aclosed

Is this simply a new dependency we will need to add to the conda recipe dependencies? I see that conda-forge provides azure-storage-blob

cc: @ihnorton, @Shelnutt2

Fix libtiledb pin to nightly version for tiledb-py

I need to update the following sed pattern:

s/"- tiledb [0-9].\+"/"- tiledb \*.$date"/ \

because the pin no longer starts with an integer, but is instead the range tiledb >=2.15.1,<2.16, which was implemented about 2 months ago in conda-forge/tiledb-py-feedstock@f153315

This is related to the failing libtiledb builds (#7). The only reason that some tiledb-py nightly binaries have been able to be produced without the accompanying nightly libtiledb is because this pin was no longer making it into the meta.yaml

Nightly builds started failing on 2023-01-16

A few things:

  • Both the tiledb and tiledb-py jobs failed back on Jan 16th. It was during the rerendering step, which was unexpected. I'm hoping it's spurious, but I have to investigate further to diagnose the problem

  • Both workflows have been stopped completely due to the GitHub Actions policy of automatically disabling scheduled workflows after 60 days of inactivity in the repo. I loathe this policy, as this affects many of my other projects as well where I create a scheduled job to notify me if an external dependency breaks my software

    This scheduled workflow is disabled because there hasn't been activity in this repository for at least 60 days. Enable this workflow to resume scheduled runs.

    • I'm not sure what we can do to prevent this besides manually re-renabling every 60 days. One alternative is we could have the workflow push a dummy commit. GitHub Actions knows when a commit is made by a workflow, but based on anecdotal evidence from one of my other repos, I think that may be sufficient to qualify as "activity"
  • This is a good reminder that I should update the workflows to automatically open an Issue on failure. I already watch all the events on this repo, but since I don't have write access, there is no way for me to get notified about a failed build. Note that this is simpler than my related to-do item of opening an Issue for a failed feedstock build on Azure (since that happens on a separate repo on a separate CI platform)

Action items:

  • John: Diagnose rerendering error
  • @shaunrd0 or @ihnorton: Go here and here, and click each button to re-enable both builds
  • John: update pipeline to open Issue for failed nightly build #2
  • John: update pipeline to regularly push a dummy commit at some interval #3

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.