Issue with sm-ssh-ide in SageMaker Unified Studio

[didhd]

Description

SageMaker Unified Studio is GA release, And I encountered issues with sm-ssh-ide configure --ssh-only command.

1. Permission and Shell Initialization Issues:

sm-ssh-ide: Configuring services...
/usr/local/bin/_activate_current_env.sh: line 8: : command not found
critical libmamba Shell not initialized

The command fails with permission errors when trying to access /home/sagemaker-user:

find: Failed to change directory: /home/sagemaker-user: Permission denied
find: Failed to restore initial working directory: /home/sagemaker-user: Permission denied

2. IAM Role Conflicts:
   Following the IAM setup instructions from IAM_SSM_Setup.md creates problematic role configurations. The provided scripts cause role assumption failures and permission conflicts. This makes it impossible to properly assume roles in the Studio environment.

Environment

• SageMaker Unified Studio (GA version)
• Default Studio environment
• Following official documentation and IAM setup guide

Expected Behavior

• sm-ssh-ide configure should complete successfully
• IAM roles should work correctly without assumption conflicts

Actual Behavior

• Shell initialization fails
• Permission denied errors for /home/sagemaker-user
• IAM role assumption fails after applying the recommended IAM setup

Could you please provide guidance on:

1. Correct permission setup for Unified Studio
2. Updated IAM configuration that doesn't break role assumption
3. Proper shell initialization process for the new Studio environment

[ivan-khvostishkov]

Hi, @didhd , I didn't test it yet in Amazon SageMaker Unified Studio. It should be working in Amazon SageMaker Studio though. Can you try that? I will make my tests to see if there's any major difference.

Also, on IAM setup, at which step you see failures and conflicts? Could you please explain which roles do you use and how did you create them?

[didhd]

Hi @ivan-khvostishkov, I've confirmed it works well in Studio. For now, using Studio instead of Unified Studio seems to be the solution. And regarding the IAM Role, it seems the actually required permissions and trust entity are a bit different.

[moresim]

Hi @ivan-khvostishkov and @didhd, i'm facing the same problem in the now called Amazon SageMaker AI.

In particular, i'm getting errors with the init of the shell

sm-ssh-ide: Configuring services...
/usr/local/bin/_activate_current_env.sh: line 8: : command not found
critical libmamba Shell not initialized

and also to access the sagemaker-user folder

find: Failed to change directory: /home/sagemaker-user: Permission denied
find: Failed to restore initial working directory: /home/sagemaker-user: Permission denied

If I ignore those errors, while trying to connect to the tunneled JupyterLab in my local computer I can't access files because I don't have permissions.

Any idea how to fix it?
Thanks in advance

[jwoschitz]

Hi @ivan-khvostishkov, I can confirm that sm-ssh-ide configure --ssh-only fails in SageMaker AI Studio with SageMaker Distribution 3.1.0 (latest release as time of writing).

Steps to reproduce (in a JL terminal):

1. pip install sagemaker-ssh-helper
2. sm-ssh-ide configure --ssh-only

Fails with:

find: Failed to change directory: /home/sagemaker-user: Permission denied
find: Failed to change directory: /home/sagemaker-user: Permission denied
find: Failed to restore initial working directory: /home/sagemaker-user: Permission denied

The root cause seem to stem from the following code sm-setup-ssh#L62:

  if _is_ssh_ide_inside_studio; then
    find /etc/rsyslog.conf /etc/rsyslog.d/ -type f \
      -exec sed -i -e 's~/var/log/~/var/log/apps/app_container.log__~' {} \;
  else
    [...]

find also tries to traverse the current working dir, which is by default /home/sagemaker-user. sm-setup-ssh is executed by sm-ssh-ide configure implicitly with sudo, but the root user does not have permissions to access /home/sagemaker-user in SMD.

$ ls -ld /home/sagemaker-user/
drwx------ 8 sagemaker-user nogroup 156 Jun 18 11:47 /home/sagemaker-user/

A simple workaround seems to change the working dir in the sm-setup-ssh script before the syslog config patching is performed.

Should I open a separate issue for this (as this is unrelated to Unified Studio)?