Upgrade builder-datastore to use core/postgresql17 #1892

jasonheath · 2025-05-12T19:18:32Z

Adds components/builder-datastore/hooks/install
Removes components/builder-datastore/hooks/init
Updates components/builder-datastore/hooks/run
Uses core/postgresql17 as the package ident everywhere
Removes unused keys from components/builder-api/habitat/default.toml
Reworks components/builder-api/habitat/hooks/init
Updates or removes postgres conf items from postgresql.conf and default.toml files as appropriate
Updates changed items from postgresql.conf and applicabale default.toml files
Removes ext_semver_version from datastore plan

mwrock

I'm still going through the install hook but thought I would put these comments out now.

.expeditor/scripts/post_habitat_release/shared.sh

components/builder-api/habitat/hooks/init

components/builder-api/habitat/plan.sh

mwrock

I still feel like I need to go over this a little more but I'm running out of steam. One thing to consider is if we want to keep the dbinit and createdb in the init hooks. The advantage is that sometimes it's nice to blow away the data directory of builder-db and restart the instance to get a fresh clean instance. The init hooks will just recreate the empty database.

components/builder-datastore/hooks/install

mwrock

Went over some of the install hook some more. My gut feeling is that we are trying to d oa little too much here or maybe just making it overly complex. Roughly it seems like it should be doing this:

Look at PG_VERSION
does it exist?
no? do nothing
is it less than our version?
no? - do nothing.
yes?
install older postgress
dump
initialize new postgres
load dump

.expeditor/verify.pipeline.yml

components/builder-api/habitat/hooks/init

components/builder-datastore/hooks/install

mwrock

Went over some of the install hook some more. My gut feeling is that we are trying to d oa little too much here or maybe just making it overly complex. Roughly it seems like it should be doing this:

Look at PG_VERSION
does it exist?
no? do nothing
is it less than our version?
no? - do nothing.
yes?
install older postgress
dump
initialize new postgres
load dump

.expeditor/templates/studiorc

components/builder-api/habitat/default.toml

components/builder-api/habitat/hooks/init

components/builder-api/habitat/hooks/run

components/builder-datastore/hooks/install

components/builder-datastore/hooks/run

mwrock

I only saw one thing that jumped out at me.

components/builder-api/habitat/hooks/init

jasonheath · 2025-06-13T19:07:35Z

@jasonheath what is pending for this?

It's currently pending review and/or approval.

.studiorc

mwrock · 2025-06-16T15:29:24Z

components/builder-api/habitat/plan.sh

@@ -8,6 +8,9 @@ pkg_origin=habitat
 pkg_maintainer="The Habitat Maintainers <[email protected]>"
 pkg_license=('Apache-2.0')
 pkg_bin_dirs=(bin)
+pkg_include_dirs=(include)
+pkg_lib_dirs=(lib lib64)
+pkg_pconfig_dirs=(lib/pkgconfig lib64/pkgconfig)


did we find these lines necessary? I'd think these would only be needed if we were building other binaries that had a dep on builder-api, which we don't.

Removing in a future commit. Also removing the pkg_bin_dirs because that's the default value.

I removed those. I did go extra and remove the pkg_bin_dirs=(bin) too because that's the default value.

components/builder-api/habitat/plan.sh

support/ci/shared_build_environment.sh

mwrock · 2025-06-17T14:19:16Z

support/ci/shared_build_environment.sh

+install_hab_pkg core/sed
+install_hab_pkg core/sudo
+install_hab_pkg core/zeromq
+install_hab_pkg core/zlib


This seems like it has quite a few more packages than we were using in the past (eg libb2, sed, etc). Are we sure we need all of these?

With these I feel I can confidently say yes. Because of the way I was locking everything down I would occasionally hit something new a script that was trying to run. For instance I remember being surprised by the two you've happened to call out.

- Adds components/builder-datastore/hooks/install - Removes components/builder-datastore/hooks/init - Updates components/builder-datastore/hooks/run - Uses core/postgresql17 as the package ident everywhere - Removes unused keys from components/builder-api/habitat/default.toml - Reworks components/builder-api/habitat/hooks/init - Updates or removes postgres conf items from postgresql.conf and default.toml files as appropriate - Updates changed items from postgresql.conf and applicabale default.toml files - Removes ext_semver_version from datastore plan Signed-off-by: Jason Heath <[email protected]>

Signed-off-by: Jason Heath <[email protected]>

…hooks/run Signed-off-by: Jason Heath <[email protected]>

Signed-off-by: Jason Heath <[email protected]>

…ared.sh Signed-off-by: Jason Heath <[email protected]>

I keep noticing this $(run) so I've removed it because it keeps catching my attention and I've learned that it should have been removed several years ago as part of a commit that removed the targets that built out and used $(run). Signed-off-by: Jason Heath <[email protected]>

Signed-off-by: Jason Heath <[email protected]>

- Fixes the environment for test/run_clippy restoring the lint step in the verify pipeline - Fixes the environment for test/run_cargo_test restoring our unit test focused steps in the verify pipeline - Fixes environment for .expeditor/scripts/verify/builder-api-functional.sh restoring our functional testing focused step in the verify pipeline - The hardcoding of core/rust/1.79.0/20250606210134 is done as part of this. When we upgrade rust we will restore our established 'core/rust/"$(tail -n 1 "../../../rust-toolchain" | cut -d'"' -f 2)"' - Uses "dynamic_shared_memory_type = 'mmap'" for testing - Updates .studiorc to make it rely on the environment variable that have been set - Updates .studiorc such that the psql function can use core/postgresql17 or core/postgresql17-client depending on what's installed - Updates .expeditor/scripts/post_habitat_release/cargo_update.sh to use shared build environment - Adds a check for already running sup in .expeditor/templates/studiorc - Updates ./build.sh to use shared build environment Signed-off-by: Jason Heath <[email protected]>

Signed-off-by: Jason Heath <[email protected]>

…20250606210134 Signed-off-by: Jason Heath <[email protected]>

Signed-off-by: Jason Heath <[email protected]>

mwrock · 2025-06-18T21:55:43Z

components/builder-datastore/hooks/install

+  # https://www.postgresql.org/support/versioning/
+  # https://endoflife.date/postgresql
+
+  read -ra installed_version <<<"$(sudo -u hab hab pkg exec "$INSTALLED_PG_IDENT" pg_config -- --version | awk '{print $2}')"


While upgrading (and this would probably be the case on a fresh install too), I get this error:

hab : user NOT in sudoers ; PWD=/hab/svc/builder-datastore ; USER=hab ; COMMAND=/usr/bin/hab pkg exec core/postgresql96 pg_config -- --version

This is going to fail on all sudo calls in the hook. This is run by the hab user that the supervisor uses for services which is not in sudoers so we need t ofigure out how to get this to work outside of sudo.

jasonheath requested review from mwrock and sajjaphani as code owners May 12, 2025 19:18

jasonheath changed the title ~~Upgrade builder-datasore to use core/postgresql17~~ Upgrade builder-datastore to use core/postgresql17 May 14, 2025

jasonheath force-pushed the jah/postgresql@17-upgrade branch 2 times, most recently from 106f8fe to 70c5a7f Compare May 14, 2025 16:48

mwrock reviewed May 14, 2025

View reviewed changes

.expeditor/scripts/post_habitat_release/shared.sh Outdated Show resolved Hide resolved

components/builder-api/habitat/hooks/init Show resolved Hide resolved

components/builder-api/habitat/plan.sh Outdated Show resolved Hide resolved

mwrock reviewed May 15, 2025

View reviewed changes

jasonheath force-pushed the jah/postgresql@17-upgrade branch 4 times, most recently from b3c4a65 to 2ac2e18 Compare May 16, 2025 17:42

mwrock reviewed May 16, 2025

View reviewed changes

.expeditor/verify.pipeline.yml Outdated Show resolved Hide resolved

components/builder-api/habitat/hooks/init Show resolved Hide resolved

components/builder-datastore/hooks/install Outdated Show resolved Hide resolved

jasonheath force-pushed the jah/postgresql@17-upgrade branch 2 times, most recently from 32c9588 to 26f83d4 Compare May 19, 2025 19:23

mwrock reviewed May 19, 2025

View reviewed changes

jasonheath force-pushed the jah/postgresql@17-upgrade branch 5 times, most recently from 6572f75 to 31d5687 Compare May 21, 2025 23:52

mwrock reviewed May 22, 2025

View reviewed changes

components/builder-api/habitat/hooks/init Show resolved Hide resolved

jasonheath force-pushed the jah/postgresql@17-upgrade branch 5 times, most recently from 6a2f95d to dc09dda Compare June 3, 2025 14:36

jasonheath force-pushed the jah/postgresql@17-upgrade branch 4 times, most recently from f75800b to 1b235d6 Compare June 9, 2025 20:08

mwrock reviewed Jun 17, 2025

View reviewed changes

jasonheath force-pushed the jah/postgresql@17-upgrade branch from 0e80638 to c04f629 Compare June 18, 2025 19:42

jasonheath and others added 19 commits June 18, 2025 15:44

Replaces hardcoded instances of 30 with PG_TIMEOUT

211a5ca

Signed-off-by: Jason Heath <[email protected]>

Updates builder-api init hook for PGPASSWORD={{cfg.datastore.password}}

abb0b9e

Signed-off-by: Jason Heath <[email protected]>

Restores postgresql-clent usage in plan.sh

544b542

Signed-off-by: Jason Heath <[email protected]>

Removes DEBUG from builder-datastore/hooks/install

079ce66

Signed-off-by: Jason Heath <[email protected]>

Adds password = "" back to builder-api/habitat/default.toml

6f303c9

Signed-off-by: Jason Heath <[email protected]>

Removes "sudo hab svc load "$ACTIVE_IDENT" --force"

3b9dca0

Signed-off-by: Jason Heath <[email protected]>

Moves hab pkg install "$INSTALLED_PG_IDENT" into upgrade_postgres

a08dd1c

Signed-off-by: Jason Heath <[email protected]>

Add a whitespace back to builder-api/habitat/hooks/run

486a28c

Signed-off-by: Jason Heath <[email protected]>

Removes export PGDATA="{{pkg.svc_data_path}}" from builder-datastore/…

cdae126

…hooks/run Signed-off-by: Jason Heath <[email protected]>

Removes pwfile check from init hook

d4d25b1

Signed-off-by: Jason Heath <[email protected]>

Removes duped function, addresses shellcheck warning in support/ci/sh…

288bac8

…ared.sh Signed-off-by: Jason Heath <[email protected]>

Captures platform-inspection.sh

b7318fc

Signed-off-by: Jason Heath <[email protected]>

Captures shared_build_environment.sh

cf9c0eb

Signed-off-by: Jason Heath <[email protected]>

Rewrites .envrc to eliminated duplication and clean up HAB_ORIGIN_KEYS

ffb3a4e

Signed-off-by: Jason Heath <[email protected]>

Adds a comment explaining an error that heals itself

4942ee2

Signed-off-by: Jason Heath <[email protected]>

Updates cleanup-integration-tests.sh

897e932

Signed-off-by: Jason Heath <[email protected]>

jasonheath force-pushed the jah/postgresql@17-upgrade branch from c04f629 to feac6ca Compare June 18, 2025 19:45

Jason Heath added 4 commits June 18, 2025 16:04

Removes core/zlib from builder-api pkg_deps

fd77e88

Signed-off-by: Jason Heath <[email protected]>

Removes commented out lines in shared_build_environment.sh

4e51106

Signed-off-by: Jason Heath <[email protected]>

Removes pkg_{bin,include,lib,pconfig}_dirs from builder-api plan.sh

7c9cbd5

Signed-off-by: Jason Heath <[email protected]>

Restores core/rust/"$toolchain" removing "unstable" core/rust/1.79.0/…

9b8a89e

…20250606210134 Signed-off-by: Jason Heath <[email protected]>

jasonheath force-pushed the jah/postgresql@17-upgrade branch from feac6ca to 9b8a89e Compare June 18, 2025 20:12

Restores RUST_LOG, HAB_NONINTERACTIVE, HAB_NOCOLORING in .studiorc

878d097

Signed-off-by: Jason Heath <[email protected]>

mwrock reviewed Jun 18, 2025

View reviewed changes

Upgrade builder-datastore to use core/postgresql17 #1892

Are you sure you want to change the base?

Upgrade builder-datastore to use core/postgresql17 #1892

Conversation

jasonheath commented May 12, 2025

Uh oh!

mwrock left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mwrock left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mwrock left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mwrock left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mwrock left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jasonheath commented Jun 13, 2025

Uh oh!

Uh oh!

mwrock Jun 16, 2025

Choose a reason for hiding this comment

Uh oh!

jasonheath Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

jasonheath Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mwrock Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

jasonheath Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

mwrock Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!