WebJob error: CacheLoader skipping URLs in SiteList

[Vidocq24601]

Hi Patrick,

When reviewing WebJob Run Details for CacheLoader, some URLs (not all) defined in ProjectNamiCacheLoader.SiteList WebApp configuration are being skipped.

Specifically, the errors in log follow this format:

[03/03/2020 02:47:51 > cbba2e: INFO] Visited - https://example.com
[03/03/2020 02:47:51 > cbba2e: INFO] ERROR Parse Links - Invalid URI: The format of the URI could not be determined.   at System.Uri.CreateThis(String uri, Boolean dontEscape, UriKind uriKind)
[03/03/2020 02:47:51 > cbba2e: INFO]    at System.Uri..ctor(String uriString)
[03/03/2020 02:47:51 > cbba2e: INFO]    at ProjectNamiCacheLoader.Module1.VisitLink(String LinkURL, Int32 LinkDepth) in C:\VS2013_Projects\ProjectNamiCacheLoader\ProjectNamiCacheLoader\Module1.vb:line 64

This issue occurs across separate multi-site network installations. Each multi-site's WebApp configuration settings have been checked, double checked, and then checked again for user error. End result remains that some URLs defined in configuration settings are visited as expected, however other hostnames are being skipped. There does not appear to be a discernible delta between sites successfully visited compared to those sites that are skipped/generate the error above.

Appreciate any guidance identifying root cause and thanks as always for all the support

[patrickebates]

I can think of a couple of reasons for this. One involves the WebJob needing to be updated to .NET 4.7.2. It appears that I've done that locally but not committed to GitHub yet. I will build you some updated bits to test in about 12 hours.

Also, I have seen an issue in the past where A tags with unexpected HREF values could cause problems with parsing the links found in a page. For example, links A tags that don't go anywhere with only a # in the HREF.

[patrickebates]

Try this updated webjob. If it doesn't fix the problem, it should at least returned some more detailed error data.

https://projectnami.blob.core.windows.net/cache/ProjectNamiCacheLoaderV1.1.zip

[Vidocq24601]

Hi Patrick,

Thanks as always for quick response. Updated webjob resolves primary issue (occasionally bailing on hostname entirely when reaching an unexpected HREF value). A few sites had placeholder attributes (#), however looking at the now detailed logs, looks like the majority of troubling HREF values were relative URLs. Using updated webjob, target pages eventually were visited from other site links that were absolute URLs.

Appreciate your help and am comfortable closing unless you'd like additional input.

Unrelated Sidebar

Didn't want to create a separate issue since it is not PN specific, and you're likely already aware, however I have been meaning to share this just in case. When synching from External Repository, MSFT recently disabled adding WebJobs via portal. Might be obvious to others, but still...

Navigating to the WebJob blade will throw this error:

WebJob cannot be added from portal if deployment from source control is configured.

Naturally the workaround to add WebJobs to PN App is to temporarily disconnect deployment. WebJobs such as CacheCleanup/CacheLoader are not removed when reconnected, however this may not be apparent to other users.

Will close shortly barring any feedback requests. Thanks so much again for assistance.

[patrickebates]

If you can provide an example of the relative URL which failed, I would appreciate it.

Also curious where it came from. WordPress is normally good about generating absolutes.