Add INI sections to config parser.

[Zelaf]

I was wondering if it's possible to add sections to the INI parser so you can have a clearer config structure.

I'm currently working on a script that has a few config options available for the user so I want to be able to have sections for the config file and if they want to update file output file locations and such they can do that, but instead of having long strings like

command_config_option = setting
othercommand_output_location = folder

It would be nice to be able to do

[command]
config_option = setting

[othercommand]
output_location = folder

and if you were to call to the config file like $(config_get command_config_option) it would return setting.

If you know any better approach to this it would also be welcomed. At first I tried to use the yaml addon but realised that it can't really set the options so if I were to want a reset option or more expandability it wouldn't really work in the bigger picture.

[DannyBen]

I agree it is a good idea. It will most likely make the library code much more complex, especially for setting values.

But I will take a closer look at it, see if I can implement.

[Zelaf]

I was able to make a very bad mockup for the config_get (With a little bit of help of AI) but I didn't save it since it barely worked. I was trying to see what the possibilities were for the config_set but I couldn't make heads or tails of it.

I know the yaml library uses awk and sed, maybe it could potentially be easier with that since sed can modify the files.

[DannyBen]

I will try in the next few days to make something.
I would like to make use dot syntax config_get section.key

[Zelaf]

That would be super nice.

[generate]
folder = /here
name = output_

[options]
colours = true
global = false

And just get the specific one with config_get options.colours. I'm excited for the possibilities of having more clean and user friendly config files!

[DannyBen]

This seems to be a working config_get with section support:

## Get a value from the config.
## Usage: result=$(config_get KEY) -or- result=$(config_get SECTION.KEY)
config_get() {
  local key=$1

  # Determine if we received SECTION.KEY or just KEY
  if [[ $key == *.* ]]; then
    IFS='.' read -r section key <<< "$key"
  else
    section=""
  fi

  local regex="^$key *= *(.+)$"
  local section_regex="^\[$section\]"
  local other_section_regex="^\["
  local value=""

  config_init

  while IFS= read -r line || [ -n "$line" ]; do
    # If SECTION was given, find it in the INI first
    if [[ $section && $found_section != true ]]; then
      [[ $line =~ $section_regex ]] && found_section=true
      continue
    fi

    # If SECTION was given and found, any other [SECTION] means we did not
    # find the requested KEY in the requested SECTION - abort.
    [[ $section && $line =~ $other_section_regex ]] && break

    # Found the value
    if [[ $line =~ $regex ]]; then
      value="${BASH_REMATCH[1]}"
      break
    fi
  done <"$CONFIG_FILE"

  echo "$value"
}

The config_set will take me more time.

[Zelaf]

A bit off topic, but do you know what the best way to approach to check if a config file exists? I want to make sure one exists with some default values before running any commands.

[DannyBen]

do you know what the best way to approach to check if a config file exists?

I would do this:

1. bashly add hooks - and delete src/before.sh and src/after.sh - leaving just initialize.sh
2. In this file, do something like:

CONFIG_FILE=path_to_config.ini   
[[ -f $CONFIG_FILE ]] || create_default_config

and then of course, add src/lib/create_default_config.sh function that writes the content.

Another option, is to avoid checking/creating anything - just when you use config_get, assign a default value if it is empty.
Then, users can run without any config file, and still have defaults.

[Zelaf]

Ooh, default values is smart, that would even make so if someone would to break the config file it'll still work! Should I add the default values in the initialize script too?

[Zelaf]

Alright so currently I'm doing this i my init script:

generate_file_name=uptime_kuma_service_push_
generate_folder_location=push-scripts

## Load config file
CONFIG_FILE=config

## Checks for config file
[[ -f $CONFIG_FILE ]] || echo "$generate_folder_location and $generate_file_name" && exit 1

The echo and exit is just for testing for now, but is there a way to load all the collected variables like you can with the $eval in the yaml addon?

[DannyBen]

The below config_get accepts a second argument which is the default value:

## Get a value from the config.
## Usage: result=$(config_get KEY [DEFAULT_VALUE])
##        result=$(config_get SECTION.KEY [DEFAULT_VALUE])
config_get() {
  local key="$1"
  local value="$2"

  # Determine if we received SECTION.KEY or just KEY
  if [[ $key == *.* ]]; then
    IFS='.' read -r section key <<< "$key"
  else
    section=""
  fi

  local regex="^$key *= *(.+)$"
  local section_regex="^\[$section\]"
  local other_section_regex="^\["

  config_init

  while IFS= read -r line || [ -n "$line" ]; do
    # If SECTION was given, find it in the INI first
    if [[ $section && $found_section != true ]]; then
      [[ $line =~ $section_regex ]] && found_section=true
      continue
    fi

    # If SECTION was given and found, any other [SECTION] means we did not
    # find the requested KEY in the requested SECTION - abort.
    [[ $section && $line =~ $other_section_regex ]] && break

    # Found the value
    if [[ $line =~ $regex ]]; then
      value="${BASH_REMATCH[1]}"
      break
    fi
  done <"$CONFIG_FILE"

  echo "$value"
}

as for changing the approach so that it is all one big associative array - it is an option. It could be a good option, but will require a complete rewrite of this code.

[DannyBen]

How about this prototype?

#!/usr/bin/env bash

config_load() {
  declare -gA config

  local section=""
  local key=""
  local value=""
  local section_regex="^\[(.+)\]"
  local key_regex="^([^ =]+) *= *(.*) *$"
  local comment_regex="^;"
  
  while IFS= read -r line; do
    if [[ $line =~ $comment_regex ]]; then
      continue
    elif [[ $line =~ $section_regex ]]; then
      section="${BASH_REMATCH[1]}."
    elif [[ $line =~ $key_regex ]]; then
      key="${BASH_REMATCH[1]}"
      value="${BASH_REMATCH[2]}"
      config["${section}${key}"]="$value"
    fi
  done < config.ini
}

config_show() {
  for key in "${!config[@]}"; do 
    echo "$key=${config[$key]}"
  done
}

config_save() {
  local filename="out.ini"
  local current_section=""

  rm -f "$filename"

  for key in "${!config[@]}"; do
    IFS="." read -r section_name key_name <<< "$key"
    value="${config[$key]}"
    
    if [ "$current_section" != "$section_name" ]; then
      [[ $current_section ]] && echo >> "$filename"
      echo "[$section_name]" >> "$filename"
      current_section="$section_name"
    fi
    
    echo "$key_name = $value" >> "$filename"
  done
}

# read file to the config var
config_load

# use the values
echo "${config[section1.key]}"
echo "${config[section1.key2]}"
echo "${config[section2.key]}"
echo "${config[section2.key2]}"
echo "${config[section1.under_key]}"

# show the entire loaded config (associative array)
config_show

# update a value
config[section1.under_key]="updated"

# delete a value
unset config[section1.key2]
unset config[section2.key2]

# save the entire array back to the file
config_save

[Zelaf]

These are bash arrays - a default value can be obtained with ${config[section.key]:-"default value"}.

By design. The reference model is loading a file, or a YAML file, or any other type of file with any other programming language. It is the developer's responsibility to load a file that exists.

Contrary to the previous implementation - where the filename was implicit (via config var, or config.ini) - the filename here is explicit.
I don't think there is a need to invent a new construct for this. Do you?

I'm sadly not versed in bash enough to understand it all sadly, but from what I gather, if you can set a default value using ${config[section.key]:-"default value"} then that's great. If the filename is now explicit I think that'll work out too for better implementations since if you're using configs anyways, the options should either be put in place already through a generated config file with default values.

Currently I think I will implement a check for a config file in the same manner to what you've done here; #405 (reply in thread) and then if the config file doesn't exist, it'll use the new config implementation to write to it. In the future I'm probably going to add to my script a validation that some config options exist and add only those if they don't.

I'm gonna continue to explore the usage and implementation to my current script and report back how it works!

[DannyBen]

if you can set a default value using ${config[section.key]:-"default value"}

Yes. Not only for arrays - any variable.
See https://stackoverflow.com/a/2013589/413924

Also - get the new config.sh library from the master branch - I think I fixed the issues.

[Zelaf]

What am I missing? Trying to use what set_command.sh is using.

write_config() {

## Default vaules
config[generate.file_name]=uptime-kuma-service-push
config[generate.folder_location]=push-scripts
## Make config file
touch $script_path/config.ini
## Save config
config_save $script_path/config.ini

}

Error:

uptime-kuma-service-push: line 222: generate.file_name: syntax error: invalid arithmetic operator (error token is ".file_name")
uptime-kuma-service-push: line 217: generate.folder_location: syntax error: invalid arithmetic operator (error token is ".folder_location")

Tried with double quotes and $ before config too, can't seem to get it.

[DannyBen]

You are missing many thing:

1. bash does not know what is config, since you did not load it from anywhere. Since you are creating your own config, you need to declare it as associative array.
2. I do not know what is $script_path and where it is coming from
3. There is no need to touch it first

write_config() {
  declare -A config

  ## Default vaules
  config[generate.file_name]=uptime-kuma-service-push
  config[generate.folder_location]=push-scripts
  
  ## Save config
  config_save "config.ini"
}

[Zelaf]

Yeah, should've given more context, sorry.
$script_path comes from the initialize.sh to be able to save the script to any location and still use the scripts folder.

After your explanation I noticed that I did work the scripts a bit wrong. I can't load the config the normal way because the config doesn't exist. I also can't write the config because I can't load the config because the file still doesn't exist.

## Variable for script location
script_path=$(dirname "$(readlink -f "$0")")

## Generate config file if it doesn't exists
[[ -f "config.ini" ]] || write_config

## Set config file
config_load $script_path/config.ini

Setting declare -A config solved that perfectly!

[DannyBen]

See #406 - with a new config library and updated example.

Since this is a breaking change, I need to consider the release plan more carefully, but you can already use this example by simply copying the config.sh file from that PR.

Let me know how it works for you.

[Zelaf]

Just added this to my current project and it's working wonderfully!

Thank you so much for this, it'll help me a lot!

[DannyBen]

Thanks for the suggestion and for confirming it is working.
I will release a version with it soon.

[DannyBen]

@Zelaf - note:

1. I have decided to keep the old config library as is
2. I have added a new library called ini which is more or less what you are using now - with all config_* functions renamed to ini_*

You can either:

1. Continue using what you have - in this case, remove the [@bashly-upgrade config] magic comment from your config.sh file (or avoid using bashly generate --upgrade)
2. Take the new ini library, and update your code accordingly.

[DannyBen]

One more thing.

There may be a way for me to merge the two libraries, if we accept this feature reduction:
If we remove the explicit filename from the load/save functions - it might be easier to have both array-level access, and function access (e.g. config_set key value which will set it in the array and save the file).

Meaning - we use it like the old way, where the filename is dictated by CONFIG_FILE.

Thoughts?

[Zelaf]

Can you rephrase? I don't understand the question.

I'm trying to say that I don't really understand what an array really is lol

Meaning - we use it like the old way, where the filename is dictated by CONFIG_FILE.

I wouldn't mind this. Currently I load the filename in the initialize.sh in a way that would probably be more suited for the old config system.

initialize.sh

## Variable for script location
script_path=$(dirname "$(readlink -f "$0")")

## Generate config file if it doesn't exists
[[ -f "config.ini" ]] || write_config

## Set config file
config_load $script_path/config.ini

write_config.sh

write_config() {

    declare -A config

    ## Default vaules
    config[generate.file_name]=uptime-kuma-service-push
    config["generate.folder_location"]=push-scripts

    ## Save config
    config_save "$script_path/config.ini"

}

For my use case it probably wouldn't matter too much and since it's a new feature anyway why not try and focus on backwards compatibility? In the end the important feature for me are the sections, how they get implemented would be to what's best for Bashly as a whole.

[DannyBen]

I'm trying to say that I don't really understand what an array really is lol

Oh - array is a list of variables (or literal values) that is treated as a whole.
Sample array in bash:

fruits=("Apple" "Banana" "Orange" "Grapes" "Mango")
echo "First fruit: ${fruits[0]}"

The new INI library uses associative array - which is also known as hash or dictionary in other languages.
In this case, the index of each item is a string (normally) and not a number:

config[options.username]

script_path=$(dirname "$(readlink -f "$0")")

A shorter way: SCRIPT_DIR="$(dirname "$0")"
And I would also consider using uppercase for "super globals" like this to distinguish them from local vars, or from dynamically generated global vars (like the ini array).

I wouldn't mind this

I will give it a shot.

[Zelaf]

A shorter way: SCRIPT_DIR="$(dirname "$0")"

Will this work with symlinks too? My script is going to be installable to /usr/local/bin/ so need to read where the symlink points too as well!

[DannyBen]

Put configs in ~/.config/.
dirname "$0" will pring the dir of the symlink.

[DannyBen]

Ok - another attempt. Do you want to try it out?

Highlights

1. No more direct access through the array. All access is done via dedicated functions.
2. File is auto-loaded and auto-saved, with the ability to force reload if someone needs to (e.g. the rush use case).
3. Does not load the file unless it is necessary (minimum possible IO operations).
4. Backwards compatible with the old config library

src/lib/config.sh

Show Code

## Config (INI) functions [@bashly-upgrade config]
## This file is a part of Bashly standard library
##
## Usage:
##
## - Set the global variable CONFIG_FILE to the path of your INI file somewhere
##   in your script (for example, in src/initialize.sh)
## - Use any of the following functions to access and manipulate the values.
## - INI sections are optional (i.e., sectionless key=value pairs are allowed).
##

## Show all the key-value pairs from your config file
config_show() {
  config_load

  for key in $(config_keys); do 
    echo "$key = ${config[$key]}"
  done
}

## Get a single value from the config file:
##
##   config_get login.email
##
## Get the value or a default one if it is not set:
##
##   config_get login.user guest
##
## Assign the result to a variable:
##
##   theme="$(config_get interface.theme)"
##
config_get() {
  local key="$1"
  local default_value="$2"
  
  config_load  
  echo "${config["$key"]:-$default_value}"
}

## Create/update a key=value pair:
##
##   config_set cloud.provider aws
##
config_set() {
  local key="$1"
  shift
  local value="$*"
  
  config_load
  config["$key"]="$value"
  config_save
}

## Delete a key=value pair:
##
##   config_del login.email
##
config_del() {
  local key="$1"
  
  config_load
  unset "config[$key]"
  config_save
}

## Get an array of all keys:
##
##   for k in $(config_keys); do
##     echo "- $k = $(config_get "$k")";
##   done
##
config_keys() {
  config_load

  local keys=("${!config[@]}")
  for a in "${keys[@]}"; do echo "$a"; done | sort
}

## Check if a key exists:
##
##   if config_has_key "key"; then
##     echo "key exists"
##   fi
##
config_has_key() {
  [[ $(config_get "$1") ]]
}

## Force-load from file
## This should normally not be called, unless you suspect that the INI file
## was modified by external means during the run of your script.
config_reload() {
  declare -g config_loaded=false
  config_load
}

## === PRIVATE - FOR INTERNAL USE ===

## Load an INI file (unless loaded) and populate the associative array
config_load() {
  [[ "$config_loaded" == "true" ]] && return

  declare -g CONFIG_FILE=${CONFIG_FILE:=config.ini}
  declare -g config_loaded=true
  declare -gA config

  local section=""
  local key=""
  local value=""
  local section_regex="^\[(.+)\]"
  local key_regex="^([^ =]+) *= *(.*) *$"
  local comment_regex="^;"

  [[ -f "$CONFIG_FILE" ]] || touch "$CONFIG_FILE"

  while IFS= read -r line; do
    if [[ $line =~ $comment_regex ]]; then
      continue
    elif [[ $line =~ $section_regex ]]; then
      section="${BASH_REMATCH[1]}."
    elif [[ $line =~ $key_regex ]]; then
      key="${BASH_REMATCH[1]}"
      value="${BASH_REMATCH[2]}"
      config["${section}${key}"]="$value"
    fi
  done <"$CONFIG_FILE"
}

## Save the associative array back to a file
config_save() {
  declare -gA config

  local current_section=""
  local has_free_keys=false

  rm -f "$CONFIG_FILE"

  for key in $(config_keys); do
    [[ $key == *.* ]] && continue
    has_free_keys=true
    value="${config[$key]}"
    echo "$key = $value" >>"$CONFIG_FILE"
  done

  [[ "${has_free_keys}" == "true" ]] && echo >>"$CONFIG_FILE"

  for key in $(config_keys); do
    [[ $key == *.* ]] || continue
    value="${config[$key]}"
    IFS="." read -r section_name key_name <<<"$key"

    if [[ "$current_section" != "$section_name" ]]; then
      [[ $current_section ]] && echo >>"$CONFIG_FILE"
      echo "[$section_name]" >>"$CONFIG_FILE"
      current_section="$section_name"
    fi
    
    echo "$key_name = $value" >>"$CONFIG_FILE"
  done
}

Sample usage code

#!/usr/bin/env bash
source config.sh

config_set center Mr. Jones
config_set login.user Operations
config_set login.email [email protected]
config_show

echo "$(config_get login.user)"

config_del login.user
config_show

echo "$(config_get login.user guest)"

[DannyBen]

Are you going to use the ini library directly? or the config?

[Zelaf]

config mostly. But I did install both in-case I need the other library in the future. I am a bit curious about how I could implement a practical example for arrays in my script but I'm unsure if there's a need for it now that I actually understand what they are lol. It was load once but since my project is run with configs and do one thing there's not too much that it does in one go except for generating files.

[DannyBen]

Well - the config library needs the ini library - one uses the other now, as can be seen in this installation manifest:

https://github.com/DannyBen/bashly/blob/f8995c6c0fa4b41cac8b428ed70ebf01fcc6a6d2/lib/bashly/libraries/libraries.yml#L22-L28

[Zelaf]

I see well, good thing I installed both then! Will this be added to bashly add anytime soon?

[DannyBen]

Will this be added to bashly add anytime soon?

Sure. I usually release on the weekend, but if it is important to you - I can release today/tomorrow.

[DannyBen]

Ok - I think this is it: #409

When you add the config library, it will also add the INI library.

I think this is a good conclusion.

1. The config library is the library that most people who need INI file access should use.
2. If they want to use the array itself for whatever reason, they can just use the low level INI library instead.

Makes sense?

[Zelaf]

I've implemented it to my current script and it seems to be working wonders! Thanks a bunch! ❤️

[DannyBen]

Thanks for your involvement. I will release on the weekend.