Author Topic: LyricsReloaded (Latest)  (Read 62477 times)

hiccup

  • Sr. Member
  • ****
  • Posts: 7884
Is the VPN workaround something you guys can live with?

No, it's not an option for me to turn on my VPN every time I have MusicBee running.
I will often have other applications running that may present problems when using a VPN.

Here is the error log.
First I ran MusicBee as usual, and the lyrics search failed (05:00).
Then I turned on my VPN (at 05:01), restarted MusicBee, and then the search was successful.

Code
05/08/2022 05:00:07 [INFO] mb_LyricsReloaded in version 1.1.15.0 started!
05/08/2022 05:00:07 [DEBUG] Loading config from field provider_google_com
05/08/2022 05:00:07 [INFO] Provider loaded: Google
05/08/2022 05:00:07 [DEBUG] Received a notification of type PluginStartup
05/08/2022 05:00:07 [DEBUG] Received a notification of type MusicBeeStarted
05/08/2022 05:00:07 [DEBUG] gzip compression detected
05/08/2022 05:00:12 [DEBUG] Lyrics request: Blondie - Hanging On the Telephone - Parallel Lines - Google
05/08/2022 05:00:12 [INFO] Google tries to load the lyrics...
05/08/2022 05:00:12 [DEBUG] The constructed URL: https://www.google.com/search?q=hanging-on-the-telephone+blondie+lyrics
05/08/2022 05:00:13 [DEBUG] gzip compression detected
05/08/2022 05:00:13 [WARN] The pattern </div></div></div></div><div class="hwc"><div class="BNeawe tAd8D AP7Wnd"><div><div class="BNeawe tAd8D AP7Wnd">(?<lyrics>.*)Source: didn't match!
05/08/2022 05:00:13 [INFO] No lyrics found.
05/08/2022 05:00:13 [FAIL] no lyrics found from Google
05/08/2022 05:00:18 [INFO] Plugin disabled
05/08/2022 05:01:02 [INFO] mb_LyricsReloaded in version 1.1.15.0 started!
05/08/2022 05:01:02 [DEBUG] Loading config from field provider_google_com
05/08/2022 05:01:02 [INFO] Provider loaded: Google
05/08/2022 05:01:02 [DEBUG] Received a notification of type PluginStartup
05/08/2022 05:01:02 [DEBUG] Received a notification of type MusicBeeStarted
05/08/2022 05:01:02 [DEBUG] gzip compression detected
05/08/2022 05:01:08 [DEBUG] Lyrics request: Blondie - Hanging On the Telephone - Parallel Lines - Google
05/08/2022 05:01:08 [INFO] Google tries to load the lyrics...
05/08/2022 05:01:08 [DEBUG] The constructed URL: https://www.google.com/search?q=hanging-on-the-telephone+blondie+lyrics
05/08/2022 05:01:09 [DEBUG] gzip compression detected
05/08/2022 05:01:10 [SUCCESS] lyrics found from Google!
05/08/2022 05:01:13 [INFO] Plugin disabled

Mayibongwe

  • Sr. Member
  • ****
  • Posts: 1062
  • Heal The World
It's weird I gotta say.
I momentarily changed the region in my chrome settings to Netherlands and the google lyrics webpage was indeed not displaying any lyrics for any of the songs I searched with.
I don't really know why that happens but I guess it would be why the pattern was/is not matching.

But none of that would matter as long as the plugin had built-in providers that had the same content as Google itself.
From what I've noticed, Google mainly receives its lyrics from Musixmatch and Lyrics Find (I'm yet to see any other source besides those two).

So I have, in addition to Google, added a few more built-in providers.
- Genius
- Deezer (also gets its content from Lyrics Find)
- Musixmatch

With those part of the package as well, users from Europe won't be missing out on anything that Google offers.
hiccup, please confirm if those additions are all working without a need for a VPN.
I'm no longer using these websites' search functions; I'm using the google search engine instead (but it shouldn't be a problem unlike with Google itself as a source)
I already spend hours on end on social media. Might as well spare a few of those to a greater purpose here.

hiccup

  • Sr. Member
  • ****
  • Posts: 7884

sveakul

  • Sr. Member
  • ****
  • Posts: 2469
I haven't tried the new combo DLL yet but noticed in the yml providers zip (which removed MusixMatch and Genius) that Lyrics Freak went from a mod date of 7/29 to 8/3--was the search technique modified for that site?

Mayibongwe

  • Sr. Member
  • ****
  • Posts: 1062
  • Heal The World
Yep. Prior to changing the way Lyrics Freak matches its tracks, I couldn't connect a song like "Coming Home, Pt II" by Skylar Grey.
On the website, it is titled "Coming Home, Part 2" instead. So it wasn't matching that.
But I'm now using the google search engine for it and tracks like those are now being matched regardless of the minor differences in the title or artist.

This is all possible because of the new search loader. I can have the plugin run and extract data from two different searches at a time.
I wanna hear from hiccup first before I make the same change for all providers (just in case the google search engine will also fail for these).
I already spend hours on end on social media. Might as well spare a few of those to a greater purpose here.

phred

  • Global Moderator
  • Sr. Member
  • *****
  • Posts: 9356
I'm yet to see any other source besides those two
I have all my prior YMLs in the providers directory. Without doing the math, I'd say 75% of the retrieved lyrics come from Google, 20% come from Genius, and the rest come from A-Z Lyrics, Oldie Lyrics, and Lyrics Freak.
Overall, I'm happy with the way the plugin is working.
Download the latest MusicBee v3.5 or 3.6 patch from here.
Unzip into your MusicBee directory and overwrite existing files.

----------
The FAQ
The Wiki
Posting screenshots is here
Searching the forum with Google is  here

frankz

  • Sr. Member
  • ****
  • Posts: 3876
Finally had the chance to load up the new version and play with it a little. Minor problem, it's pulling some incorrect lyrics.

The requested song is Another Man by Smash Palace.  The lyrics retrieved are the song Smash Palace by Sharon O’Neill.

Code
05/08/2022 05:48:09 [DEBUG] Received a notification of type TagsChanging
05/08/2022 05:48:09 [DEBUG] Lyrics request: Smash Palace - Another Man - Fast, Long, Loud - Musixmatch
05/08/2022 05:48:09 [INFO] Musixmatch tries to load the lyrics...
05/08/2022 05:48:09 [DEBUG] The constructed identity URL: https://www.google.com/search?q=Musixmatch+another-man+smash-palace
05/08/2022 05:48:09 [DEBUG] Lyrics request: Smash Palace - Another Man - Fast, Long, Loud - Genius
05/08/2022 05:48:09 [INFO] Genius tries to load the lyrics...
05/08/2022 05:48:09 [DEBUG] The constructed identity URL: https://www.google.com/search?q=Genius+another-man+smash-palace
05/08/2022 05:48:09 [DEBUG] Lyrics request: Smash Palace - Another Man - Fast, Long, Loud - Google
05/08/2022 05:48:09 [INFO] Google tries to load the lyrics...
05/08/2022 05:48:09 [DEBUG] The constructed URL: https://www.google.com/search?q=another-man+smash-palace+lyrics
05/08/2022 05:48:09 [DEBUG] gzip compression detected
05/08/2022 05:48:09 [DEBUG] gzip compression detected
05/08/2022 05:48:09 [DEBUG] gzip compression detected
05/08/2022 05:48:09 [WARN] The identity pattern (?<identity>https://www.musixmatch.com/lyrics/.*?)& didn't match!
05/08/2022 05:48:09 [INFO] No lyrics found.
05/08/2022 05:48:09 [FAIL] no lyrics found from Musixmatch
05/08/2022 05:48:09 [WARN] The pattern </div></div></div></div><div class="hwc"><div class="BNeawe tAd8D AP7Wnd"><div><div class="BNeawe tAd8D AP7Wnd">(?<lyrics>.*)Source: didn't match!
05/08/2022 05:48:09 [INFO] No lyrics found.
05/08/2022 05:48:09 [FAIL] no lyrics found from Google
05/08/2022 05:48:09 [DEBUG] The constructed lyrics URL: https://genius.com/Sharon-oneill-smash-palace-lyrics
05/08/2022 05:48:10 [DEBUG] gzip compression detected
05/08/2022 05:48:10 [SUCCESS] lyrics found from Genius!
I think free text searching like this might be a problem for songs like this where there are no legitimate lyrics but words in the band or song title match some other context.

Mayibongwe

  • Sr. Member
  • ****
  • Posts: 1062
  • Heal The World
The requested song is Another Man by Smash Palace.  The lyrics retrieved are the song Smash Palace by Sharon O’Neill.
I think free text searching like this might be a problem for songs like this where there are no legitimate lyrics but words in the band or song title match some other context.
I just tried to look for any website that might have the lyrics for that song and I honestly couldn't find any.
It'd be alarming if the providers returned incorrect lyrics while there was a correct version on their site.

For now, I'm thinking this is a rare occasion that doesn't yet hold weight against the benefits that this kind of free text searching provides.
But I do acknowledge its existence as the exception that it is.

I guess the question is, what would people prefer:
Should the providers return a no match when a song isn't available, rather than return incorrect lyrics?
If the former, then that means songs which had the potential of being matched even when the sites had differing titles/artists, would no longer get matched.
I already spend hours on end on social media. Might as well spare a few of those to a greater purpose here.

hiccup

  • Sr. Member
  • ****
  • Posts: 7884
So I have, in addition to Google, added a few more built-in providers.
- Genius
- Deezer (also gets its content from Lyrics Find)
- Musixmatch
I've started some extensive testing, and things are looking VERY promising.
But since I am making it some project, using a lot of songs (and different languages) for testing, and I would like to come back with useful and (provider-) specific results, it's probably going to be by the end of the day or perhaps tomorrow before I will get back with my report.

Mayibongwe

  • Sr. Member
  • ****
  • Posts: 1062
  • Heal The World
When I began looking at this, I wondered why nobody had ever tried using it as a direct source before.
But I soon realized there was a plugin limitation to it at the time. The plugin was using Firefox to return the websites' html dumps.
It worked fine for every other site, but with google webpages, it was returning some html filtered out content that didn't contain the full lyrics.
I want to make a correction to that statement. I just realized there was in fact no plugin limitation at all.
Firefox being a default user agent at the time wasn't gonna get in the way of Chrome being specified as a user agent for the google.yml
All that I (or anyone at the time) needed to do was add this to the yml :

headers:
    User-Agent: 'Chrome'
I already spend hours on end on social media. Might as well spare a few of those to a greater purpose here.

hiccup

  • Sr. Member
  • ****
  • Posts: 7884
An in-between observation;

I noticed some lyrics missing when using the build-in MusixMatch engine, that I believed I was able to retrieve before.
So I copied my stored yml of MusixMatch to the providers folder, activated it, and deactivated the build-in one.
And indeed I immediately got more results.

Some examples of songs the build-in MM didn't find, and the yml MM does:

Ivan Lins - Novo tempo
Margriet Eshuijs Band - Black Pearl
Chico Buarque - Fortaleza
Simone - Geraldinos e Arquibaldos
Ane Brun - Du gråter så store tåra
Nana Caymmi - Doce Presença
Walter Becker - Door Number Two

the yml:
Code
name: Musixmatch (new)

variables:
    artist:
        type: artist
        filters:
        - strip_diacritics
        - lowercase
        - [regex, "'", ""]
        - [regex, "/", " "]
        - [regex, '\s&(?=\s)', " "]
        - [regex, '(?<=\W|\s)+(feat.+|ft[\W\s]+|(f\.\s)).+', ""]
        - [regex, '[^\sa-z0-9]\s*', ""]
        - [strip_nonascii, -]
    title:
        type: title
        filters:
        - strip_diacritics
        - lowercase
        - [regex, " '|' |/", " "]
        - [regex, "'", " "]
        - [regex, '\.+|,+|/+|(\W+(?=$))|(^\W+)', ""]
        - [regex, '\s&(?=\s)', " and"]
        - [strip_nonascii, -]

config:
    url: "http://www.musixmatch.com/lyrics/{artist}/{title}"
    pattern: ['<p class="mxm-lyrics__content.*?">(?<lyrics>.*?)<div [^>]*"lyrics-report".*?>', s]

post-filters:
- [regex, "<script.*?</script>", "", s]
- strip_html
- utf8_encode
- entity_decode
- clean_spaces

Any idea?

frankz

  • Sr. Member
  • ****
  • Posts: 3876
I just tried to look for any website that might have the lyrics for that song and I honestly couldn't find any.
That's the point, isn't it? If there are no lyrics for the song, the search should fail rather than return incorrect lyrics.  People (me included) set MB to auto-save lyrics when retrieved.  There's no benefit to anyone in retrieving incorrect lyrics.  Will every song by this band called "Smash Palace" return these same incorrect lyrics because a totally unrelated artist recorded a song by that name?
Should the providers return a no match when a song isn't available, rather than return incorrect lyrics?
If the former, then that means songs which had the potential of being matched even when the sites had differing titles/artists, would no longer get matched.
I'm not sure why I would want lyrics from a song to match online lyrics from a song with a different title by a different artist.

I agree that there are massive benefits to the free text Google searching and you are brilliant for being able to implement it.  But for sites where we know exactly how their URLs are constructed, it makes no sense to pull lyrics for songs where the URL is telling us it's the wrong artist all together.  Maybe some sort of check back to make sure the band name is at least similar?

There was never a widespread problem with finding the right page for a song on Genius or MusixMatch - that worked pretty reliably.  The problem was that they kept changing the way the lyrics showed up on the page so you couldn't get them into MB.  This solves the problem of sometimes finding no lyrics by creating a problem of sometimes presenting the wrong lyrics.

Just my $0.02 and presented as a fan of what you've done generally.
Last Edit: August 06, 2022, 03:00:08 PM by frankz

phred

  • Global Moderator
  • Sr. Member
  • *****
  • Posts: 9356
So I copied my stored yml of MusixMatch to the providers folder, activated it, and deactivated the build-in one.
I haven't updated to the new .dll yet, but I did use some of the previous YML files that had been working with frankz's plulgin and with recent releases of this updates/new/current .dll.

Comparing your (hiccup's) YML for Musixmatch with mine I see one line that I have and you don't. And I'm not sure if it's necessary or not. I have been getting acceptable results with MM with the YML I'm using.

The line I have and you don't is
Code
- [regex, '<div class="inline_video_ad_container_container">', "\n", s]
which appears as the second line under post-filters.

Not sure if this will help in your extensive weekend test session, but thought I should mention it.
Download the latest MusicBee v3.5 or 3.6 patch from here.
Unzip into your MusicBee directory and overwrite existing files.

----------
The FAQ
The Wiki
Posting screenshots is here
Searching the forum with Google is  here

Mayibongwe

  • Sr. Member
  • ****
  • Posts: 1062
  • Heal The World
I noticed some lyrics missing when using the build-in MusixMatch engine, that I believed I was able to retrieve before.
My bad. I wasn't accounting for those kind of urls before.
Will fix that for the next update.

Maybe some sort of check back to make sure the band name is at least similar?
There was never a widespread problem with finding the right page for a song on Genius or MusixMatch - that worked pretty reliably.
With Musixmatch and Genius, I can include a check to see if the artist will at least be contained in the url.
With sites like Deezer, that kind of check would not work as the urls are strictly unique IDs (digits throughout).

In a moment, I'll include that check and see if the results are still way off and if they are, then I'll revert back to how these providers functioned at first.
I already spend hours on end on social media. Might as well spare a few of those to a greater purpose here.

Mayibongwe

  • Sr. Member
  • ****
  • Posts: 1062
  • Heal The World
Sadly, the check I thought I'd be able to add turned out to be a lot harder than it sounded.
So I've reverted Genius, Musixmatch and Lyrics Freak back to their original states - redownload the plugin.

To clarify what that means:
- Those three providers are no longer using the google search engine.
- So a user will now get a "no lyrics found" when the song strictly isn't available.
- But that unfortunately, also means song titles with bracketed contents in them (MusicBee strips those off before they get to the plugin)
  and ones with slightly different titles/artists than those available on the sites, would no longer get matched as well.
- So you'll need your songs tagged exactly as they appear on those websites if you want the plugin to make a match on them.
I already spend hours on end on social media. Might as well spare a few of those to a greater purpose here.