[Bug] "Skipping" behavior in offline mode

[JoeDat]

This is a difficult issue to describe.

Occasionally I'll experience what I can only describe as "skipping" behavior in some tracks. I believe the issue only occurs with FLACs, although I don't have many non-FLAC files to experiment with. When viewing the app during the behavior, the player appears to quickly pause and resume the track. The repeated behavior typically happens in short bursts, and can sometimes only last a few seconds. However, if it happens longer, the app seems to give up on the track, starts playing the next track, where sometimes that track plays fine, or exhibits the same behavior. In some extreme cases, it'll do this pause/resume/start next track thing for 2 or 3 songs in quick succession.

I tend to use the app while connected to a BT device such as my Sony WH-1000XM4 headphones, or a JBL PartyBox 110. I don't believe I've heard this issue when not connect via BT, but I admittedly don't play music without an external speaker often.

I can supply debug logs for the last occurrence, however I can't seem to figure out how to get the logs. I can see the log list, but touching a log file does nothing.

[icefields]

hello,
I know exactly what are you talking about.
First, please ensure you’re using the latest version of the app (1.00-65), because I'm continuously working on optimizing this issue.

It sounds like you’re experiencing network connection issues, which become more noticeable with FLAC files (which are very large usually). The app is set to automatically skip to the next song after encountering three connection errors, which is likely why you're seeing these skips.

To improve your experience, you could try downloading the music you want to play. Alternatively, I can adjust the app to increase the number of retry attempts for connection errors.

I would also suggest to keep your ampache backend updated, and maybe try to restart your server to clear cache and memory.

[JoeDat]

Hey @icefields , I'm actually experiencing this in offline mode. I've been testing this morning on PA2 1.00-65. My Ampache backend is on the latest stable (6.6.0). Mobile is a Pixel 7 Pro, in case there's a local resource question. Should be plenty of power.

[icefields]

ok that is a different bug then. The best way for me to check on this would be if you gave me a test account for your server (if you're comfortable with this) so I can check the logs from my LogCat with the app in debug mode

[JoeDat]

Hmm, certainly don't mean any disrespect, but not sure I'm comfortable at this point. I'll think about it.

Trying to understand how the server is at play if the app is in offline mode. The skips occur even when I turn off mobile data and wifi. Bluetooth is at play, so not testing in airplane mode.

I'm gonna run an extended test while hardwired. My gut is telling me it's a bluetooth thing, although I have zero proof of that lol. I'm not having similar issues in other apps. Coming from many years of using DSub and have been going back to it to compare and not experiencing anything similar, again while offline.

[icefields]

No worries, I understand completely.
I don't think it's the same issue I mentioned earlier; it just sounds very similar. The server doesn't play any role in this, especially in offline mode, where there's no connection for playing songs.

I know where to look (and put a debugger breakpoint to analyze the issue) for player errors, so if you change your mind and decide to create test account for me, I'll give you my email, or you can dm me in person on telegram (links on the readme and on power.ampache.dev).

Let me know how the situation evolves if you do more testing.
Thanks again for reporting the bug.

[CrazyStevenz]

Hi @icefields, I have the same issue but for me the app crashed while glitching and trying to skip the offline song that was playing, so I was able to generate a crash.txt file. Not sure how useful it is, but I could forward it to your email (since I expect it might contain some sensitive data from my device).

This issue happens with a bunch of songs, no matter if they're flac or not.

If you're comfortable with a discord call, I could also supply some credentials for you to give it a try.

[icefields]

hello @CrazyStevenz , thanks for reaching out and for providing crash info. The crash report doesn't contain personal info, but it does contain some device info.
You can send it to
powerampache.ducking336 AT silomails DOT com
I'm not on Discord, I used to have an account but I haven't used it in a very long time. If you want to get into a call or DM me, you can use Telegram.
This is PowerAmpache2 official group chat: t.me/powerampache2 , I'm the admin. My personal account cannot be reached directly, but if you drop a message in the chat saying you contacted me on Github about issue-148 I will initiate a private conversation with you.
Timing is good, I'm working on the app today and tomorrow.

[CrazyStevenz]

@JoeDat I had the same issue with my wireless earbuds, but I tested with Bluetooth turned off and the crash still occurs, so not the culprit.

My current way of crashing the app is the following (performed on an offline songs library that included both flac and mp3 songs, with and without cover images):

1. Open the app
2. On my phone, after crashing, the app doesn't connect to the server again, so I need to enable offline mode here via the toggle in the navbar
3. Go to offline songs (every song you play from now will be from here)
4. Press the tune spinner at the top right (which by my understanding shuffles all offline songs and starts playback)
5. Minimize the player and play a different song (songs with a cover image seem to cause the crash sooner)
6. Turn off wifi
7. Exit offline mode (this only needs to be done once, if you enabled offline mode in the beginning)
8. Play other songs with images until you see the "Cannot connect to your server"
9. Play one more song and leave it playing
10. Maximize and minimize the player
11. Enable wifi
12. Play different songs until the "Cannot connect to your server" notice disappears
13. Play one more song and leave it playing
14. Maximize and minimize the player
15. Repeat steps 6-14 for ~3 minutes, playback will start stuttering until eventually the app OOMs and freezes

[JoeDat]

Thanks @CrazyStevenz, I've never encountered any freezes or full on crashes. Just the stutter before it skips to the next track in the playlist. 99% of my catalog contains songs with cover art, so I'm rarely playing anything that isn't. I haven't posted recently because the problem has been elusive for me (of course lol) . But, I'll definitely give the steps above a repro attempt and report back.

[JoeDat]

I got a little lost in your steps a bit, but there seems to be something there regarding air-gapping the device, restoring connectivity, etc. I notice this more when I'm wandering around my property and likely floating in and out of wifi availability (where the phone is constantly trying to find a connection). Manually skipping songs from the notification bar, there's considerable flicker and occasionally corresponding stutter at the very start of the track (something I never see while online). This is very similar behavior I'll notice when it happens in the middle of the track, and the flicker and stutter is seen in both the in app player and notification bar. I've gotten accustomed to running in offline mode when I know I'll be away because I'm coming from DSub and there wasn't another way to see your offline tracks. PA2 grants that capability without explicitly being offline, so I'll do this network connectivity test while PA2 is online to see if I can repro.

I haven't yet been able to repro your OMM and eventual freeze or crash. Maybe a difference in phone hardware between us (ie, I have more available resources?). I'm on a pixel 7 pro.

[CrazyStevenz]

I'm on a Pixel 7 128/8 with GrapheneOS. Maximum ram usage from the app was <500MB before it crashed, and I had over 3GB of ram free.

Funny thing is that, without restarting my phone or changing anything, I can no longer reproduce the issue with my own reproduction steps. I wonder if it's related to the server, like when my Nextcloud instance is really slow the requests pile up and it eventually runs out of memory.

With no solid reproduction steps and if Icefields doesn't by now have an idea of what might be causing this, I don't think we're getting a solution.

[icefields]

@JoeDat @CrazyStevenz Thank you for your detailed reports. I started running some tests using your steps while monitoring the logs, unfortunately LogCat is not telling me much. I also monitor the memory for leaks all the time during my pre-release tests, and haven’t observed any zombie threads or processes consuming excessive memory. But I'll keep digging.

It’s possible the bug could be related to the server, the specific phone model, or the API level, but it’s challenging to pinpoint the exact cause without more detailed logs.
I use a Pixel 7 with GrapheneOS as my personal phone, so I should be able to replicate your steps fairly accurately.
Also, I don't exclude a bug with the library I use for networking or the android media library. I make an effort to keep the app updated with the latest bleeding edge versions of all Android SDK libraries to ensure we benefit from the most current features and improvements, but this, like with every bleeding edge software, might also cause unfixed library-bugs to surface. I recommend keeping the app updated regularly.

I’ll continue to investigate and will keep you updated. If you discover any additional information or observations, please share them here.

[CrazyStevenz]

@icefields Only new thing I can tell you is that after playing one song in online mode, I switched to offline mode, and after ~15 minutes of playing music offline, browsing twitter and switching songs from the media widget and the lock screen, the issue happened again. After the app crashed and I opened it again, I left it in offline mode and the bug didn't occur after hours of playback.

Would a version of the app with extra logging, or a way to setup some kind of debugging on it while it's installed on my phone, help? I want to do whatever's possible to help, since this issue is very infuriating when it happens, and it'll probably affect more people as the app's usage grows.

[icefields]

@CrazyStevenz ok, yes we need to solve this, sounds annoying. I will send you a version with extra logging. Thanks for helping.

[JoeDat]

If you want to include me in this debugging test, I'm all for it as well. I've recently been testing in airplane mode, and I can get the issue to happen there. The server backend can't possibly be a factor.

[icefields]

@JoeDat awesome, thanks! I will make a build with extra logging enabled and I will post the link here in this thread (most likey tomorrow, but I will try tonight if I can).
The apk will be built against its specific bugfix GIT branch, so it will be publicly available if anyone wants to help testing (again, link will be posted here between tonight and tomorrow).

[icefields]

@JoeDat @CrazyStevenz find the apk with extra logging here. If you previously downloaded from FDroid or PlayStore, this will not replace your official version, they will live together on your phone:
https://github.com/icefields/Power-Ampache-2/raw/bug/148-oom_crash/app/Github/release/app-Github-release.apk

The git branch where I'm working on this is bug/148-oom_crash
This apk also contains 2 attemps of fixing for the memory bug (I say 'attempt' because I cannot reproduce it on my phone, so I can't verify the fix worked).
One of the attempts of fixing the bug is already present on the official version (66), which should available on FDroid soon. The other one is introduced in this branch.

Thanks again for helping.

[CrazyStevenz]

@icefields I use Obtainium, so I have the latest apk from Github as soon as you release it.

That said, I tested version 66 and the issue keeps happening, so you can cross that fix out.

I'll install the custom version on my work profile, so I can keep my signed installation and test the debug apk in parallel. I'll let you know how it goes.

[icefields]

@CrazyStevenz that apk is signed with the same key as the Github releases (but it won't replace it because they're the same version), I can make you a FDroid apk tomorrow, so you don't have to go through the trouble of switching profiles.

[CrazyStevenz]

You don't need to switch profiles to use work apps on Android, I just open the "work" version of PA2, so don't worry about that. My main worry is that the issue won't happen on the work profile, if the cause is something unrelated to the app. If I don't observe the issue with this setup I'll try replacing the main app and testing it again. If the issue happens, I'll message you logs and the exact timestamp on Telegram.

[CrazyStevenz]

@JoeDat @icefields Since we are now fairly sure it's not the flac files causing this issue, please remove that part from the title, as it creates confusion for users who are having this issue with mp3s, as can be seen in the linked PR above this message.

[looowizz]

Hi, I'm experiencing this issue too and I think it might have something to do with the connection between the app and media notification because, for me at least, the stuttering seems to go away when the notification crashes/disappears for a longish period. When the notification looks like this screenshot (usually when it just returns from the crashed state) i have no stuttering but it comes back when the notification returns to the default look.

[icefields]

@JoeDat thanks for changing the title, it was indeed inaccurate. But I think the issue is reproducible online as well, anyone please confirm before I change the title again.


@looowizz I was actually running some tests and I noticed an odd behaviour with notifications, the problem now is that I'm using system APIs for the notifications, it's not a custom view that I designed. From your image I see that it works when no album cover is loaded, maybe the image I'm loading is too heavy and causes the memory to run out? I can run some extra tests about that and verify.

Anyway, this said, if anyone here wants to test a new attempt of fixing this, I will share the link to the build with the fix here some time during the weekend.

[looowizz]

Oo interesting. I'd be down to test any builds :)

[icefields]

I'm going link a test release in this thread in the next few days.
@looowizz thanks for supporting us with this.
I got some new helpful reports from @CrazyStevenz that I think will help, and I'm going to try something new that I have in mind with notifications.

[icefields]

I’m going to delay the official release by a few days. I usually work on Power Ampache 2 after hours and on weekends, but this past week has been particularly busy, including the weekend.

I have several fixes in commits that I haven't pushed yet, related to the issues you’ve reported, in addition the attempted fixes already in the debug build. I’ve added a timer to reset the player error counter after a minute of no errors, as currently, the song skips after a certain number of errors without a reset. I’m also checking if the service is active before playing a song, since I suspect it might be getting killed due to memory constraints. There are also a few other minor fixes being addressed.

As I previously mentioned, I will share a debug build here as soon as it's ready.

[icefields]

@CrazyStevenz @JoeDat this is the pre-release that fixes some (hopefully all) of the issue pointed out in this thread.
I have to test the app thoroughly before making the actual release, I'm always a bit scared when I touch the media player. Please check that everything is working correctly if you have some time.
I made some changes to the music player service and the notifications service (@looowizz), and all the changes I mentioned in my previous post.

https://github.com/icefields/Power-Ampache-2/raw/version_1.00-67/app/FDroid/release/app-FDroid-release.apk

[looowizz]

Thank you! Will try it out today :)

[looowizz]

Hi! really sorry to say that I'm still personally experiencing the same issue :/

Another bit of information (not sure if it's useful) is that the app continues trying to play music even when it is swiped away and i have to force close it to stop it.

Here is the log of the issue
Or a pastebin version which will expire in 6 days

[CrazyStevenz]

https://github.com/icefields/Power-Ampache-2/raw/version_1.00-67/app/FDroid/release/app-FDroid-release.apk

@icefields With this version the issue happened pretty much immediately. I started with the tune spinner and on the 3rd song, it played the 1st second and then played from the beginning again. Every song after that became worse and worse. Then, for the first time ever, the app completely froze for about 30 seconds. After the system finally prompted me to force stop it, I managed to also extract an Android ANR log.

Logging

In-app debug logs: I checked the timestamp the issue happened and the printed entries are empty (as in, just a timestamp and an empty body). Empty entries are also printed randomly, so not sure if this is related.

Power Ampache 2 System Log

ANR log

[CrazyStevenz]

The steps required to reliably make the glitch happen are now much fewer.

1. Open the app (Online mode, WiFi).
2. Click the tune spinner.
3. Go to settings, enable offline mode.
4. Turn WiFi off.
5. Go to offline songs, scroll a bit, find a song with a cached image.
6. Turn WiFi on (leave offline mode on).
7. Play the song with the cached image.
8. Song glitches or skips.

To me, it seems like a weird interaction between the image cache and Nextcloud.

[icefields]

@CrazyStevenz @looowizz looks like I solved nothing for you with this update.
I'll go ahead anyway with the release since it includes significant performance improvements and better queue handling. I'll continue to address your issue and include it in the next release (while providing test builds here).

To test this more properly, I created a virtual machine with Ubuntu Server and I installed the Nextcloud Snap package (I'm surprised how easiy it was to set up!), I'll do some more tests today. Thanks for the steps to reproduce and for keeping helping debugging the issue.

@looowizz can you provide:

• phone model and make
• android version (and OS type if degoogled)
• can you confirm you're on Nextcloud? (pardon my laziness if you already told me)

Thanks.

[JoeDat]

To confirm, I'm also still experiencing the issue on the latest test build. I'm on Ampache 6.7.0. The steps above might be sufficient to force the issue to happen, but it might be good to consider that it's not the only way. As reported above, I see this in airplane mode, which if I'm not the only one, kinda turns this issue on its head. I get album art while offline, presumably because I have it embedded in all my metatags. And I do see the same "flickering" behavior as well with the art in notifications, which is a precursor to the eventual stutter/skip that happens. But, again, all while completely airgapped.

If I am the only one with this specific behavior, I'll step aside and see how this shakes out. If the issue here is resolved and I'm still experiencing a completely offline version of this problem, I'll submit a new issue report.

EDIT: Seems like I completely missed the point that images are cached and not read from metatags locally.

[icefields]

@JoeDat thanks. I will keep trying, and I hope it will be resolved soon. The fact that I cannot reproduce this on any of my devices and emulators is making the debugging process very slow.

[looowizz]

Hi!

My phone model is the Google Pixel 4a (5G)
I'm on Android 14 on GrapheneOS
My server is a really old 32 bit laptop running Ampache on Yunohost so not personally running Nextcloud :/

[CrazyStevenz]

https://github.com/icefields/Power-Ampache-2/raw/bug/148-oom_crash/app/Github/release/app-Github-release.apk

@icefields When I installed this version of the app in my work profile 2 weeks ago, I never managed to make it crash. It seemed really weird to me that after all the work you did the crash still happens in the latest RC, so I went back and installed the oom version on my main profile and I managed to crash it almost immediately.

The only major difference between the 2 profiles (on the level I control, not sure what happens under the hood by Android) is that the work profile contains the GrapheneOS sandboxed Google Play Services, while the main profile does not.

[icefields]

@CrazyStevenz that is odd, I have absolutely no interaction with google, not even in the playstore version, and I personally don't have play services on any of my devices. I'm not sure what Android is doing here under the hood or how Work Profiles are handled by the system. On my GrapheneOS I have a completely different user account that has microG, so I never used work profiles.
Just on top of my head, check the firewall if you haven't.
Anyway, interesting update

[CrazyStevenz]

Another update after more than 1 month of usage:

I noticed that even when the app is in offline mode, it still makes network requests for music cover art, which drove up my mobile data usage. I decided to revoke the Network permission from the app, to prevent that from happening, and from that point on, the frequency of this issue was reduced by 99%. It still happens occasionally, when the phone is really hot, or ram usage is high, or I'm switching songs too rapidly, but it nearly never glitches or skips songs anymore, and is how I'm using the app day-to-day now.

[icefields]

Another update after more than 1 month of usage:

I noticed that even when the app is in offline mode, it still makes network requests for music cover art, which drove up my mobile data usage. I decided to revoke the Network permission from the app, to prevent that from happening, and from that point on, the frequency of this issue was reduced by 99%. It still happens occasionally, when the phone is really hot, or ram usage is high, or I'm switching songs too rapidly, but it nearly never glitches or skips songs anymore, and is how I'm using the app day-to-day now.

@CrazyStevenz Thanks so much for keeping testing this!
yes that is correct, "Offline mode" blocks all API calls and retrieves data from the internal database instead of querying the server. However, images are not handled by the internal sqlite database, instead, they are managed by a library called Coil, which initializes at app startup and consequently always attempts to fetch images from the network if the image-cache is expired.
It's important to note that offline mode is not intended to serve as a substitute for airplane mode or a dedicated data-saving feature. It's designed for users who may not have access to the internet or those with home servers not exposed to the public internet. You brought up another use-case which I will need to architect a solution for.
Your observation about the potential impact of large image sizes on memory usage is very insightful, I was suspectint something like that was happening but could not confirm. I wonder if that's an issue with the images served by Nextcloud, I will verify.
Thanks

[JoeDat]

I wonder if that's an issue with the images served by Nextcloud, I will verify.

I mentioned similar behavior earlier regarding network traffic while offline and thought it was interesting, specifically surrounding image art retrieval. It's not just a Nextcloud backend issue, as I'm seeing the same behavior with it struggling to pull album art from Ampache.

[icefields]

@JoeDat @CrazyStevenz
I did some work on the music player and buffer settings, also added the option to use a more advanced player (only downside is that it doesn't play local music).
You can try the new version using this link, or wait for it to be available on F-Droid and the Play Store in a few days. Experiment with the buffer settings under "Player Advanced Settings." Additionally, check out the new advanced player, also found in the same menu.

[CrazyStevenz]

Coming back after a week to say that the situation has improved a lot in the last few updates. Together with the Player Advanced Settings in this update (which I have all set to the lowest values possible, since I only listen to downloaded songs), track glitching has become extremely rare. It still happens maybe once an hour, but it only happens once and doesn't skip the song.

A new issue that appeared 2-3 updates ago is that some songs immediately skip when being played. It can happen to any song, although it usually happens to the first song in a listening session. This might be a different issue though, I'll let you be the judge.

[JoeDat]

Agree with CrazyStevenz assessment, feel like things are much better for me as well. I'll report back here if the situation changes. Keep up the good work with this app!