Release Procedures

Annual

Pre-Release (e.g., v1.3rc1)

Release (e.g., v1.3)

Post-Release (e.g., v1.8.2)

On January first

Update counter scripts

  • Log onto vergil as cdsgroup.

  • Add new years, Python versions, and Psi4 versions as needed.

  • check vergil /home/cdsgroup/psi4meta/download-analysis/installer: vi downloads_updater.py any new patterns to add?

  • check vergil if changing any download patterns

Update samples

  • Run make sphinxman at least once by hand

  • Check in resulting psifiles.py and all the updated and new samples/ files and dirs

  • Make a lone PR and warn reviewers not to read it, since autogenerated

Collect new authors

  • Survey contributions to current Milestone. Add new contributors to the release notes GitHub issue.

  • Figure out any new “Additional Contributors” authors since last release.

  • Get permission of new authors and their particulars for codemeta.json.

  • Edit psi4/header.py accordingly and make PR.

Run long tests

  • Run the full test suite with threading parallelism on. There’s one CC test that’s a known fail in parallel b/c it sets memory very low, iirc.

    • CTest (CLI):

      +++ b/tests/runtest.py
@@ -104,7 +104,7 @@ def backtick(exelist):

 # run psi4 and collect testing status from any compare_* in input file
 if os.path.isfile(infile):
-    exelist = [psi, infile, outfile, '-l', psidatadir]
+    exelist = [psi, infile, outfile, '-l', psidatadir, '-n2']
     # On Windows set Python interpreter explicitly as the shebang is ignored
     if sys.platform.startswith('win'):

    • Pytest (API):

      +++ b/tests/pytests/conftest.py
@@ -22,6 +22,7 @@ def pytest_collection_modifyitems(config, items):
 def set_up_overall(request, tmp_path_factory):
     import psi4

+    psi4.core.set_num_threads(2)
     psi4.set_output_file("pytest_output.dat", False)
     os.chdir(tmp_path_factory.getbasetemp())
     request.addfinalizer(tear_down)

+++ b/tests/pytests/conftest.py
@@ -34,6 +34,7 @@ def set_up():
     psi4.core.clean()
     psi4.core.clean_timers()
     psi4.core.clean_options()
+    psi4.set_num_threads(2)
     psi4.set_output_file("pytest_output.dat", True)

    • Pytest (CLI):

      +++ b/tests/pytests/addons.py
@@ -223,8 +223,8 @@ def ctest_runner(inputdatloc, *, extra_infiles: List = None, outfiles: List = No
     if Path(psi4.executable).suffix == ".exe":
         command = [psi4.executable, inputdat]
     else:
-        command = [sys.executable, psi4.executable, inputdat]
-    _, output = execute(command, infiles_with_contents, outfiles, environment=env, scratch_messy=False)
+        command = [sys.executable, psi4.executable, inputdat, "-n2"]
+    _, output = execute(command, infiles_with_contents, outfiles, environment=env, scratch_messy=True)

     success = output["proc"].poll() == 0
     assert success, output["stdout"] + output["stderr"]

  • Run a full stdsuite tests to generate new capabilities tables for docs. Detailed instructions are in psi4/psi4/share/psi4/scripts/merge_stdsuite.py .

Anticipate next release

Build Conda ecosystem stack

  • By “ecosystem stack”, we mean packages that are upstream, downstream, required, and optional for a fully featured Psi4 build and which we have some role in packaging.

  • These packages (e.g., libint, gdma) should already be updated and built on conda-forge. Survey them to check version tick and other key PRs have been merged.

  • Changes to targets’ “source” and “version” in individual recipes should be edited in psi4 external/*/*/CMakeLists.txt files

  • Edit any added or dropped dependencies in main psi4/CMakeLists.txt and docs psi4/doc/sphinxman/source/build_planning.rst .

Assemble postrelease changes

  • Collect PRs with “backport” label, and request other backport suggestions through slack.

  • Be on the maintenance branch (e.g., 1.8.x). git fetch upstream or any other remotes you’re going to be cherry-picking from.

  • Cherry-pick backport PRs and commits (git cherry-pick sha), apply other changes manually, not forgetting CI files or samples.

  • Possibly apply other changes manually, including:

  • Tag it (Tag postrelease for details). This is needed to compute a version on a maintenance branch to even run. Until the release is published on GH (final step of Publish GitHub postrelease), it’s ok to revise a tag (force push to maintenance branch).

    • Add or increment patch number in psi4/metadata.py; leave the “z”s. Commit file.

    • Tag with git tag -a v1.8.2 -m "v1.8.2", then push git push --atomic upstream 1.8.x v1.8.2.

  • Test core PSI4 thoroughly locally (ecosystem will get tested by c-f) by running pytest ../tests/ -n auto (psithon and psiapi tests). Possibly you may have to step back for dependency versions from what master needs.

  • Start Tweak Conda for postrelease PR at conda-forge. This will thoroughly test the ecosystem.

  • If more changes are needed, git tag -d v1.8.2 to delete the tag, then make more commits, retag, push, and repeat. Try to finalize the postrelease tag within a session or a day, so tentative tags don’t linger.

  • Start the draft parts of Publish GitHub postrelease from the backported PRs assembled here.

Tweak Conda for postrelease

  • Start a PR to https://github.com/conda-forge/psi4-feedstock . Always store the PR branch on your fork, never on the conda-forge feedstock (or your branch itself will get publically packaged). https://github.com/conda-forge/psi4-feedstock/pull/15 is an example.

    • Always: edit version and commit jinja variables.

    • Possibly: edit source/url field for trial locations.

    • Always: edit sha256 jinja variable from e.g., curl -sL https://github.com/psi4/psi4/archive/v1.8.2.tar.gz | openssl sha256

    • Always: edit build/number. Either reset to 0 if version increments or bump if version doesn’t increment.

    • Possibly: uncomment skip: true  # [py != 310] if you want to test one Python version on all architectures before the whole build matrix (currently 16 builds). After editing this (and opening the PR), you’ll have to issue a comment @conda-forge-admin, please rerender for the matrix slimming to take effect.

    • Possibly: remove any old patches that are now in the main codebase.

    • Possibly: add in or remove any dependency or ecosystem packages.

    • Possibly: for any relevant addition, make sure -D CMAKE_INSIST_FIND_PACKAGE_<project>=ON or -D ENABLE_<project>=ON is set in build.sh and bld.bat.

    • Possibly: add or release version constraints or architecture constraints (e.g., [not win]) on packages.

  • Submit the PR and rerender (cmd above). Monitor the CI.

  • When all CI lanes are passing and the tag is final on the maintenance branch, rerender (may be no-op) and merge the PR.

  • After all the packages are built on main and show up at https://anaconda.org/conda-forge/psi4/files with a couple downloads, this means they’re been mirrored and are generally installable. Announce on slack general channel.

Do final pass before release tag

Tag (pre)release

  • Thorough version bump directions at master http://psicode.org/psi4manual/master/manage_git.html#how-to-bump-a-version

  • Below is tl;dr

    # be on clean master up-to-date with upstream in both commits and tags
# * mind which version strings get "v" and which don't
# * if not fork, replace "upstream" with "origin"

>>> vi psi4/metadata.py
>>> git diff
diff --git a/psi4/metadata.py b/psi4/metadata.py
...
-__version__ = '1.3rc1'
-__version_long = '1.3rc1+5a7522a'
-__version_upcoming_annotated_v_tag = '1.3rc2'
+__version__ = '1.3rc2'
+__version_long = '1.3rc2+zzzzzzz'
+__version_upcoming_annotated_v_tag = '1.3rc3'

>>> git add psi4/metadata.py
>>> git commit -m "v1.3rc2"
[master bc8d7f5] v1.3rc2

>>> git log --oneline | head -1
bc8d7f5 v1.3rc2
>>> git tag -a v1.3rc2 bc8d7f5 -m "v1.3rc2"

# goto GH:psi4/psi4 > Settings > Branches > master > Edit
#      https://github.com/psi4/psi4/settings/branch_protection_rules/424295
# uncheck "Do not allow bypassing the above settings" for admins and Save changes

>>> git push --atomic upstream master v1.3rc2

# pause here and push to upstream and let Azure complete for an
#       on-tag Windows conda package and docs, not tag+1.dev1 .
#       the atomic flag below pushes commit and tag together so only one CI
#       which is necessary for Windows conda package to compute the right version.
#       After push, can temporarily re-engage admins "Do not allow ..." protections.
#       also, grab the docs build from GHA artifacts

>>> vi psi4/metadata.py
>>> git diff
diff --git a/psi4/metadata.py b/psi4/metadata.py
...
-__version_long = '1.3rc2+zzzzzzz'
+__version_long = '1.3rc2+bc8d7f5'

>>> git add psi4/metadata.py
>>> git commit -m "Records tag for v1.3rc2"
[master 16dbd3e] Records tag for v1.3rc2

# goto GH:psi4/psi4 > Settings > Branches > master > Edit
#      https://github.com/psi4/psi4/settings/branch_protection_rules/424295
# uncheck admins "Do not allow ..." and Save changes

>>> git push upstream master

# re-engage admins "Do not allow ..." protections

Tag postrelease

# be on clean maintenance branch up-to-date with upstream in both commits and tags
# * mind which version strings get "v" and which don't
# * if not fork, replace "upstream" with "origin"

>>> git checkout 1.3.x
Switched to branch '1.3.x'

>>> vi psi4/metadata.py
>>> git diff
diff --git a/psi4/metadata.py b/psi4/metadata.py
...
-__version__ = '1.3'
-__version_long = '1.3+zzzzzzz'
+__version__ = '1.3.1'
+__version_long = '1.3.1+zzzzzzz'

>>> git add psi4/metadata.py
>>> git commit -m "v1.3.1"
[1.3.x 2ce1c29] v1.3.1

>>> git log --oneline | head -1
786fb2b v1.3.1
>>> git tag -a v1.3.1 2ce1c29 -m "v1.3.1"

# skipping the hash recording and "upcoming" step b/c only tags matter on maintenance branch

# free pushing to maintenance branches at present so GitHub interface steps not needed

# see note at "Tag (pre)release" for why atomic commit needed. Collect docs from GHA artifacts.

>>> git push --atomic upstream 1.3.x v1.3.1

Initialize release branch

  • follow tagging procedure

  • before re-engaging the admins “Do not allow …” button, push a branch at the tag commit (not the records commit)

    >>> git log --online | head -2
45315cb Records tag for v1.3
20e5c7e v1.3

>>> git checkout 20e5c7e
>>> git checkout -b 1.3.x
Switched to a new branch '1.3.x'
>>> git push upstream 1.3.x

  • set up new branch as protected branch through GitHub psi4 org Settings. Should be already covered under 1.*.x rule.

Build extra Conda packages for Psi4 channel

Once upon a time, “Psi4 stack”, meant packages psi4, psi4-rt, psi4-dev, and psi4-docs. Package psi4-docs used to be in “Psi4 stack”, but it’s handled by GHA and netlify now, not Conda. Package psi4-rt used to be in “Psi4 stack”, but a maximum ecosystem package isn’t provided now, only a customizable env spec. Package psi4-dev used to be in “Psi4 stack”, but now build environment and guidance is in-repo with psi4-path-advisor.py. Other packages in the “ecosystem stack” (e.g., libint, gdma) should already be updated and built on conda-forge. Survey them to check version tick PRs have been merged.

Conda-forge overwhelmingly handles the psi4 package itself, with a full architecture and Python version matrix. What remains are specialty or development builds for the psi4 channel.

  • High AM and multiarch psi4 builds for Linux

    • Especially at tagged releases, update and reconcile c-f psi4/feedstock recipe with psinet psi4meta/conda-recipes/psi4-cf recipe. Differences include:

      • restricted to only even python versions

      • c-f libint vs. psi4 libint2 packages (latter with high AM)

      • smoke vs. full tests

      • no git rev-parse lines

      • load Intel compilers and specify them in compilers and flags CMake arguments

  • Prepare recipe, make sure psi4-cf is the only target uncommented in kitandkapoodle.py, set crontab, view in kpd-anom.log.

  • Files will upload to psi4/label/dev. For releases and postreleases, on the anaconda.org site (logged in as psi4), add, not replace, main label, so accessible from psi4/label/main.

Build Psi4conda set

Installers are build using the project constructor to build binary bash or exe scripts, one per OS per Python version. For example, there’s 16 installers when OSes are linux-64, win-64, osx-64, osx-arm64 and pythons are 38, 39, 310, 311. In analogy to Miniconda, they’re called Psi4Conda. They are built through GHA on the https://github.com/psi4/psi4meta repository and get served from vergil (the cdsgroup webserver).

  • If the previous release hasn’t had a snapshot saved, copy construct.yaml into a version-labeled file and check it in.

  • Edit recipe https://github.com/psi4/psi4meta/blob/master/installers/construct.yaml

    • Edit the top matter for Configuration, mainly the release field. See snapshots in directory for examples.

    • Edit the packages and channels info if necessary. Probably long-term stable.

  • Edit the GHA control file https://github.com/psi4/psi4meta/blob/master/.github/workflows/Installers.yml matrix.cfg list if Python versions or target architectures have changed.

  • All conda packages must already have been built and present in the right channels on https://anaconda.org/conda-forge/psi4/files .

  • Commit construct.yaml to trigger installer builds. (Even workflow edits need a dummy commit to construct.yaml to retrigger.)

  • When all build successfully, hover over the artifacts, and note the smallest and largest of the near-consecutive numbers GH has assigned them. These artifacts only linger for a day.

  • Log in to vergil root and cd to /var/www/html/psicode-download.

  • Use the pull_gha_installers.sh script to download the installers from GH to vergil. First two arguments are first and last of the artifact numbers, and third argument is an auth token. bash pull_gha_installers.sh 47226565 47226573 715...4f3.

  • Make WindowsWSL and any other symlinks the script frontmatter advises.

Build Docker images

Docker images are built through GHA on the https://github.com/psi4/psi4meta repository from a Conda environment specification and get served from DockerHub, https://hub.docker.com/r/psi4/psi4/tags .

Generate download page for psicode.org

  • Be in local clone of repository https://github.com/psi4/psicode-hugo-website .

  • Copy and edit a new file akin to content/installs/v182.md. Add it to the git index.

    • Note the edition string v182 in frontmatter for this and future filenames.

    • Don’t postdate the date string in frontmatter or it won’t render.

    • Ultimately, make sure the aliases:\n  - /installs/latest/ lines are added to this new file and removed from the previous latest file, but this can wait until the installer page has been tested.

  • Copy and edit a new file akin to data/installs/v182.yaml. Add it to the git index.

    • Glance through the menu and notes content to make sure they’re up-to-date. This file determines the structure of the install page.

    • Add or remove python versions and architectures if necessary.

    • Every couple years, update the default python version in datakey: python/selected and in optsHandler at the end.

    • Always adjust the datakey: branch/stable block.

    • For releases, adjust the datakey: branch/previous and nightly blocks.

  • Enter the scripts/ directory. If the previous release hasn’t had a snapshot saved, copy install-generator.py into a version-labeled file, and add it to the git index.

  • Edit scripts/install-generator.py.

    • Primarily, edit edition at the top.

    • Also, edit other arrays (stuff above ## Outputs) or messages (logic below ## Outputs) that should change.

  • Run the install-generator.py in place. It will dump two new files, e.g., data/installs/cmd/{edition}.json and data/installs/dlbtn/{edition}.json. Add these to the index (no need to inspect them).

  • The installer page is now ready for inspection. Run hugo server --watch=false and view in browser at http://localhost:1313/ . Click around the options to make sure the buttons and instructions all look right.

  • Iterate on the data/installs/{edition}.yaml and the install-generator.py until correct. It’s fine to push to psicode.org to see it in place. But wait until it’s final (and all the packages and installers are ready) to shift “latest” alias in frontmatter from whichever page is currently active to the new page, content/installs/{edition}.md. This makes sure “Downloads” on the navigation bar points to the new page.

  • Commit the new files, PR, and deploy psicode site.

Collect documentation snapshot

  • Documentation is built automatically by GHA from the latest psi4 master commit. It gets pushed to a special “master” folder on the https://github.com/psi4/psi4docs repository. From there, it and other docs snapshots are built and served to https://psi4.github.io/psi4docs/master/ (independent of psicode.org). The netlify psicode.org site has a redirect so that https://psicode.org/psi4manual/master/index.html presents the psi4docs repo content.

  • This setup works great for “latest” docs, but it won’t necessarily build a nice copy on the tag commit itself for release and postrelease snapshots. Get a snapshot on the tag by some means:

    • For releases and postreleases, any commit to the maintenance branch will build docs and upload to the branch name in psi4docs. This should be pre-positioned by GHA, so check that docs with the right version are deployed and then no further action required (can skip ahead to README.md and netlify.toml steps).

    • For releases, you can do the atomic push of the tag commit, wait for the docs build to complete, download the GHA artifact (zipped docs dir), then continue by pushing the record commit.

    • For releases beyond the atomic push, navigate on psi4 GH to the tag commit (not the record commit) and retrigger the docs GHA, then download the GHA artifact (zipped docs dir).

    • For postreleases, build the docs locally at the tag and collect the docs dir.

  • In your local clone of https://github.com/psi4/psi4docs, find the appropriate folder and unpack your docs snapshot into it.

    • For releases, you’ll need to make a new folder, e.g. sphinxman/1.8.x.

    • For postreleases, you’ll overwrite the contents of the existing folder.

    • Unpack and rearrange so that in the end, e.g., sphinxman/1.8.x/index.html is present.

  • Commit all your docs files and push to master.

  • For releases, add a line to the top-level table https://github.com/psi4/psi4docs/edit/master/README.md .

  • For releases, add a new redirects block to https://github.com/psi4/psicode-hugo-website/edit/master/netlify.toml .

  • Details:

    • If you sequentially push the tag commit, push the tag, push the record commit, GHA will build the docs at v1.{Y+1}.dev1, not at v1.Y .

    • If you sequentially push the tag commit, then push the tag, GHA will build the docs at the tag commit, but the version will show up as “undefined”.

    • If you atomic push the tag commit and tag together and wait, GHA will build the docs at v1.Y, as desired.

Publish GitHub release

  • With an anticipated or newly minted tag, go to https://github.com/psi4/psi4/releases/new (or “Draft a new release” button on GitHub site).

  • Release title takes the form: v1.8, 2023-05-11

  • Fill in frontmatter style and links from previous GitHub release

  • Fill in RN from hopefully existing RN issue.

  • Fill in RN by going through the frontmatter from all PRs from this milestone, particularly the “User API & Changelog headlines” section.

  • Save the draft release until tag is finalized.

  • “publish” release. This establishes the release date for the GitHub API.

  • Close the RN issue.

  • Close the milestone (should be 100% complete).

  • Open a milestone for the release that’s a year out.

Publish GitHub postrelease

  • With an anticipated or newly minted tag, go to https://github.com/psi4/psi4/releases/new (or “Draft a new release” button on GitHub site).

  • Release title takes the form: v1.8.2, 2023-10-03

  • Fill in frontmatter style and links from previous GitHub release.

  • Fill in RN bullets for changes cherry-picked or edited to the maintenance branch.

  • Save the draft release until tag is finalized.

  • “publish” release. This establishes the release date for the GitHub API.

Publish psicode.org release

  • Be in local clone of repository https://github.com/psi4/psicode-hugo-website .

  • Execute https://api.github.com/repos/psi4/psi4/releases/latest or https://api.github.com/repos/psi4/psi4/releases/tags/v1.8.2 (substituting tag) and note the id field value.

  • Copy and edit a new file akin to content/posts/v182.md. Add it to the git index.

    • e.g., v1.8.2 is used for Title and Release Notes.

    • e.g., 1.8.x is used for Documentation and Source.

    • e.g., v182 (edition string) is used for Image and Installers.

    • Use the id value in the shortcode call at the bottom near ghRN.

  • Add a new release page to the psi4_release_fireworks.key Keynote presentation. Run the slide transition and screenshot the fireworks. Open the PNG file in Preview and save as JPEG while downsampling to ~400kB. Place the file at e.g., static/images/portfolio/fireworks_slide_v182.jpg. Add it to the git index.

  • Edit data/portfolio.yml to add a new block for the release (order matters).

  • Include these changes in a PR. Check the generated preview if needed. Merge the PR yourself or ask for it to be merged.

Finalize release

  • Make new PR with * edits to main README.md badges, python versions, etc.

  • Tweet about release

Misc.

  • Consider rebuilding the PSI4 binder image.

  • If you want to do trial conda builds from a maintenance branch w/o pushing the tag, requires source/git_tag: 1.3.x and fake package/version: v1.3.1rc1