Release Procedures

Annual

Pre-Release (e.g., v1.3rc1)

Release (e.g., v1.3)

Post-Release (e.g., v1.3.1)

Update samples

  • Run make sphinxman at least once by hand

  • Check in resulting psifiles.py and all the updated and new samples/ files and dirs

  • Make a lone PR and warn reviewers not to read it, since autogenerated

Collect new authors

  • Survey contributions to current Milestone. Add new contribs and PR lists to release notes GitHub issue

  • Figure out any new “Additional Contributors” authors since last release

  • Edit psi4/header.py accordingly, make PR

  • Get permission of new authors and their particulars for codemeta.json

  • Invite any contributors with at least 3 PRs to join GitHub Organization

Anticipate next release

Build Conda ecosystem stack

By “ecosystem stack”, mean packages that are upstream, downstream, required, and optional for a fully featured Psi4 build and which we can’t get from “defaults” or “conda-forge” channels.

  • Main directions are in [cbcy](https://github.com/psi4/psi4meta/blob/master/conda-recipes/conda_build_config.yaml) and [poodle](https://github.com/psi4/psi4meta/blob/master/psinet-nightly/kitandkapoodle.py)

  • A couple weeks before the first “rc” is planned, start going through L/LT in poodle, checking with upstream to see if new versions have been released. If good changes present, rebuild the packages, changing the version numbers in the respective recipes

  • When L/LT all built and passed, edit the individual package version numbers in cbcy and increment to a new ltrtver with updated version numbers and/or build numbers (only if code changes)

  • Build L/PSI4. If any trouble, edit psi4 code. Iterate until builds and passes. This stage is the only full ctest & pytest on Psi4+upstream

  • Build L/RT-MP. If any trouble, edit code in L/RT and rebuild those package(s). Iterate until builds and passes. This stage is the only full ctest & pytest of Psi4+downstream

  • Build L/DEV. If any trouble, edit psi4 build system, plugin system, or OpenMP setup. Iterate until builds and passes

  • Build L/DOCS. If any trouble, edit the docs or the tests. Iterate until builds and passes

  • Results of last should upload to psicode.org (docs) and codecov.io (coverage)

  • Changes to targets’ “source” and “version” in individual recipes should be edited in psi4 external/*/*/CMakeLists.txt files

  • Once everything’s working on Linux, repeat on Mac

  • At this point, ready to fine-tune builds of “Psi4 stack”

Assemble postrelease changes

  • Collect PRs with “backport” label, and request other backport suggestions on upcoming RN issue

  • Cherry-pick backport PRs and commits, apply other changes manually, not forgetting CI files

  • Do the draft parts of Publish GitHub postrelease

  • Test thoroughly locally, including psi4-rt (may have to step back for dependency versions or builds)

  • Note that while can’t change maintenance branch’s history, can push to it directly on upstream

  • If want to do trial conda builds, requires source/git_tag: 1.3.x and fake package/version: v1.3.1rc1

Tweak Conda for postrelease

  • In psi4-* recipes, find the best combination of master meta.yaml/build.sh and the v1N(x-1)-labeled ones

    • comment out any dependencies in master but not yet in postrelease

    • add {{ dep_bld }} entries if needed to step back build. make sure normal operation is ''

  • In cbcy,

    • create a postrelease new line and record anything not in continuous order with <> or blanking

    • step back any dependencies versions or build numbers to ones compatible with the postrelease

Do final pass before release tag

Tag (pre)release

  • Thorough version bump directions at master http://psicode.org/psi4manual/master/manage_git.html#how-to-bump-a-version

  • Below is tl;dr

    # be on clean master up-to-date with upstream in both commits and tags
# * mind which version strings get "v" and which don't
# * if not fork, replace "upstream" with "origin"

>>> vi psi4/metadata.py
>>> git diff
diff --git a/psi4/metadata.py b/psi4/metadata.py
...
-__version__ = '1.3rc1'
-__version_long = '1.3rc1+5a7522a'
-__version_upcoming_annotated_v_tag = '1.3rc2'
+__version__ = '1.3rc2'
+__version_long = '1.3rc2+zzzzzzz'
+__version_upcoming_annotated_v_tag = '1.3rc3'

>>> git add psi4/metadata.py
>>> git commit -m "v1.3rc2"
[master bc8d7f5] v1.3rc2

>>> git log --oneline | head -1
bc8d7f5 v1.3rc2
>>> git tag -a v1.3rc2 bc8d7f5 -m "v1.3rc2"

# goto GH:psi4/psi4 > Settings > Branches > master > Edit
#      https://github.com/psi4/psi4/settings/branch_protection_rules/424295
# uncheck "Include administrators" and Save changes

>>> git push --atomic upstream master v1.3rc2

# pause here and push to upstream and let Azure complete for an
#       on-tag Windows conda package and docs, not tag+1.dev1 .
#       the atomic flag below pushes commit and tag together so only one CI
#       which is necessary for Windows conda package to compute the right version.
#       After push, can temporarily re-engage "Include administrators" protections.
#       also, grab the docs build from GHA artifacts

>>> vi psi4/metadata.py
>>> git diff
diff --git a/psi4/metadata.py b/psi4/metadata.py
...
-__version_long = '1.3rc2+zzzzzzz'
+__version_long = '1.3rc2+bc8d7f5'

>>> git add psi4/metadata.py
>>> git commit -m "Records tag for v1.3rc2"
[master 16dbd3e] Records tag for v1.3rc2

# goto GH:psi4/psi4 > Settings > Branches > master > Edit
#      https://github.com/psi4/psi4/settings/branch_protection_rules/424295
# uncheck "Include administrators" and Save changes

>>> git push upstream master

# re-engage "Include administrators" protections

Tag postrelease

# be on clean maintenance branch up-to-date with upstream in both commits and tags
# * mind which version strings get "v" and which don't
# * if not fork, replace "upstream" with "origin"

>>> git checkout 1.3.x
Switched to branch '1.3.x'

>>> vi psi4/metadata.py
>>> git diff
diff --git a/psi4/metadata.py b/psi4/metadata.py
...
-__version__ = '1.3'
-__version_long = '1.3+zzzzzzz'
+__version__ = '1.3.1'
+__version_long = '1.3.1+zzzzzzz'

>>> git add psi4/metadata.py
>>> git commit -m "v1.3.1"
[1.3.x 2ce1c29] v1.3.1

>>> git log --oneline | head -1
786fb2b v1.3.1
>>> git tag -a v1.3.1 2ce1c29 -m "v1.3.1"

# skipping the hash recording and "upcoming" step b/c only tags matter on maintenance branch

# free pushing to maintenance branches at present so GitHub interface steps not needed

# see note at "Tag postrelease" for why atomic commit needed. Collect docs from GHA artifacts.

>>> git push --atomic upstream 1.3.x v1.3.1

Initialize release branch

  • follow tagging procedure

  • before re-engaing the “include admin” button, push a branch at the tag commit (not the records commit)

    >>> git log --online | head -2
45315cb Records tag for v1.3
20e5c7e v1.3

>>> git checkout 20e5c7e
>>> git checkout -b 1.3.x
Switched to a new branch '1.3.x'
>>> git push upstream 1.3.x

  • set up new branch as protected branch through GitHub psi4 org Settings. Should be already covered under 1.*.x rule.

Build Conda Psi4 stack at specific commit

By “Psi4 stack”, mean packages psi4, psi4-rt, psi4-dev. Package psi4-docs used to be in “Psi4 stack”, but it’s handled by GHA and netlify now, not Conda, so skip directions below. Other packages, the “ecosystem stack” (e.g., libint, v2rdm_casscf) should be already built.

  • Check poodle for stray channels that may have crept in for dependencies (like c-f for ACS season). Copy over new dependencies if needed to psi4 channel

  • Particularly before release (not prerelease), consider max pinnings on dependencies, particularly any fast-moving dependencies (e.g., qcel) and whether they need version space to grow compatibly and grow incompatibly.

  • Nightly conda-builds work from master psi4. (Postrelease “practice” works from 1.N.x psi4.)

  • For releases and postreleases (not prereleases), in conda_build_config.yaml, edit ltrtver to a new non-dev label (probably a ditto) matching the release (e.g., “1.3”)

  • Set kitandkapoodle.py to the normal *** stack. Should be (psi4, psi4-rt, psi4-dev) * python_versions for Linux & Mac. Also single psi4-docs from Linux

  • Run kitandkapoodle.py and allow stack to upload to anaconda.org to psi4/label/dev. Poodle emits with --label dev so will go to the subchannel. May need to delete packages to clear out space on anaconda.org

  • Copy meta.yaml and build.sh of at least psi4-multiout and psi4-rt to e.g., v121-build.sh files for easy reference or rebuilding as dependencies in master change.

Publish to main conda label

  • Go through each active conda package off https://anaconda.org/psi4/repo

    • Find the most recent build set (Linux/Mac, active py versions) that psi4/psi4-rt/psi4-dev is using

    • _add_ (not replace) the main label.

  • This makes a conda install psi4 -c psi4 get everything psi4 needs. For the moment conda install psi4 -c psi4/label/dev will get the same set, until package psi4-1.4a1.dev1 gets uploaded. May help to check versions and build versions against ltrtver in conda_build_config.yaml.

  • This step is manual, so takes a while. (It gets checked when Psi4conda installers are built b/c that pulls from “main”, not “dev”)

Build Psi4conda set (pre-Spring 2021)

Installers are build using the project constructor and build binary bash scripts, one per OS per Python version. In analogy to Miniconda, they’re called Psi4Conda. They can be built anywhere (Mac can be built on Linux) and get served from vergil (cdsgroup webserver).

  • Need a conda env with constructor and cookiecutter. This env presently accessed through conda activate cookie.

  • Enter “constructor-cutter-unified” in the psi4meta repo. There’s a good README there, https://github.com/psi4/psi4meta/blob/master/conda-recipes/constructor-cutter-unified/README.md

  • Edit cookiecutter/cookiecutter.json for control

    • Edit which python versions, if necessary

    • Edit release field

    • Edit hash field. This is the 7-char hash that’s on every psi4 conda pkg as part of version

    • Edit ltrtver field. This matches the current setting in conda_build_config.yaml

    • For prereleases, "channel_tag": "/label/dev", while for (post)releases, it should be the empty string

    • Leave this file set to a “rc” with Git, as that has more details

  • For (post)releases (not prereleases), copy cookiecutter.json to cookiecutter.json-vXXX

  • Edit cookiecutter/{{.../construct.yaml for templating. This is rarely needed

  • If it’s been a while or you need the space, clear out ~/.conda/constructor, where the downloaded packages are cached

  • Note that installers get written to build/ and this gets regenerated each time. Clear out between runs.

  • python run.py

  • [Outdated] Watch out for py_ in buildstring as this means a noarch package has been pulled. It must be eliminated. Constructors can’t handle “noarch” packages and will fail at runtime. If see a “noarch” package, must find the recipe and rebuild for all OS & Python combinations. Then run constructor again.

  • [Replacement] After adding a post_install to reposition the noarch packages, they are safe to include.

  • If fetching times out, may have to run run.py several times. Clear out build/ in between. It’s the fetching that takes a long time, not constucting

  • In the end, should have several installers

    >>> ls -l build/psi4conda-1.3-py3.*/*64.sh
-rwxr-xr-x. 516M Feb 28 20:30 build/psi4conda-1.3-py3.6-linux-64/psi4conda-1.3-py36-Linux-x86_64.sh
-rwxr-xr-x. 299M Feb 28 20:31 build/psi4conda-1.3-py3.6-osx-64/psi4conda-1.3-py36-MacOSX-x86_64.sh
-rwxr-xr-x. 518M Feb 28 20:30 build/psi4conda-1.3-py3.7-linux-64/psi4conda-1.3-py37-Linux-x86_64.sh
-rwxr-xr-x. 299M Feb 28 20:31 build/psi4conda-1.3-py3.7-osx-64/psi4conda-1.3-py37-MacOSX-x86_64.sh

  • Upload installer files to vergil, scp -r build/Psi4*/Psi4*sh root@vergil.chemistry.gatech.edu:/var/www/html/psicode-download/

  • Log in to vergil root and make WindowsWSL symlinks

Build Psi4conda set

Installers are build using the project constructor and build binary bash or exe scripts, one per OS per Python version. In analogy to Miniconda, they’re called Psi4Conda. They are built through GHA on the psi4meta repo and get served from vergil (cdsgroup webserver).

  • Edit recipe https://github.com/psi4/psi4meta/blob/master/installers/construct.yaml

    • Edit the top matter for Configuration. See snapshots in directory for examples.

      • Edit release fields. (Windows is often dev1 unless separate steps.)

      • Edit ltrtver field. This matches the current setting in conda_build_config.yaml

      • For prereleases, "channel_tag": "/label/dev", while for (post)releases, it should be the empty string

    • Edit the packages and channels info if necessary. Probably long-term stable.

  • Edit the GHA control file https://github.com/psi4/psi4meta/blob/master/.github/workflows/Installers.yml matrix.cfg

  • All conda packages must already have been built and in the right channel_tag channel.

  • Commit construct.yaml to trigger installer builds.

  • When all build successfully, hover over the artifacts, and note the consecutive numbers GH has assigned them. These artifacts only linger for a day.

  • Log in to vergil root and move to /var/www/html/psicode-download.

  • Use the pull_gha_installers script to download the installers from GH to vergil. First two arguments are first and last of the artifact numbers, and third argument is an auth token. bash pull_gha_installers.sh 47226565 47226573 715...4f3.

  • Make WindowsWSL and any other symlinks the script head matter advises.

Generate download page for psicode.org

Collect documentation snapshot

  • Documentation is built automatically by GHA from the latest psi4 master commit. It gets pushed to the psi4/psi4docs repository and thence served by netlify to a site independent of psicode.org. The netlify psicode.org site has a redirect so that psicode.org/psi4manual/master presents the psi4docs netlify content.

  • GHA controller is psi4/.github/workflows/docs.yml

  • This setup works great for “latest” docs, but it won’t build a nice copy on the tag because the tag commit is pushed before the tag itself, so the version shows up “undefined”.

  • So, anytime after “Tag (pre)release” is over, navigate on psi4 GH to the tag commit (not the record commit) and retrigger the docs GHA. Download the artifact (zipped docs dir) at the end to a local computer.

  • In your hugo site clone, create a new directory under static/psi4manual. Copy the zipped docs there, unpack, rearrange so that static/psi4manual/<new-tag>/index.html is present. Check in.

Publish GitHub release

  • On GitHub site “Draft a New Release” with newly minted tag

  • Fill in frontmatter style and links from previous GitHub release

  • Fill in RN from hopefully existing RN issue

  • Fill in RN by going through the top posts from all PRs from this milestone

  • “publish” release. This establishes release date for GitHub API

  • Close the RN issue.

  • Close the milestone (should be 100% complete).

Publish GitHub postrelease

  • On GitHub site “Draft a New Release” for anticipated or newly minted tag

  • Fill in frontmatter style and links from previous GitHub release

  • Fill in RN as cherry-pick to or edit on branch

  • “publish” release. This establishes release date for GitHub API

Publish psicode release

Finalize release

  • Make new PR with * edits to main README.md badges, python versions, etc. * edits to CMakeLists.txt find_package(PythonLibsNew 3.6 REQUIRED)

  • Tweet about release

Reset psi4meta for nightly operation

On both Linux and Mac:

  • After (post)release (not prerelease), in conda_build_config.yaml, edit ltrtver to a new “release.dev” label

  • After postrelease, unpin any dependencies in cbcy that needed older either ver or bld (e.g., v2rdm_casscf_bld)

  • Edit psi4ver back to '' in cbcy

  • Edit source/git_tag back to master for psi4-multiout, psi4-docs

  • Edit build string back to 0 if psi4-multiout needed multiple passes

  • Edit kitandkapoodle.py back to *** stack

  • Check in all release, construct, recipe changes on Linux and Mac. Synchronize both to GitHub psi4meta

  • Copy meta.yaml and build.sh files to vMmp-prefixed files for the record.

  • Edit crontab back to 2am “norm”. Comment out “anom”