qutebrowser and icecat stuck in infinite browser checks (cloudflare) on applicable sites

  • Open
  • quality assurance status badge
Details
2 participants
  • bdju
  • Maxim Cournoyer
Owner
unassigned
Submitted by
bdju
Severity
normal
B
B
bdju wrote on 24 Nov 2022 17:11
(address . bug-guix@gnu.org)
COKNOMQ0RVGD.2O10X1PL1AGHA@masaki
I have hit this issue before, but not in a while. I don't know for sure
when the issue came back as it only matters for the initial login or
visit of the offending sites, so if you manage to get past you'll be
good until your cookie expires or gets deleted somehow. I got logged out
of many sites in qutebrowser recently and was made aware of the issue
being back. I also tested in icecat where some sites I still had a
cookie, but I managed to find one with a browser check and it indeed
never completes (can leave it open hours or even more than a day).

Some sites to test with include:
https://www.livechart.me/(must click login button, main page works)
https://www.gitlab.com/(again must click log in to start the check)

Since this is happening in both icecat and qutebrowser with different
user agents and everything, I suspect a guix-related issue. I found that
at least one other person on IRC was also experiencing the infinite
browser checks. I use a few sites daily that are now unusable on my Guix
System machine due to these browser checks, so a fix would be very much
appreciated if anyone could figure this out.

guix version:
guix (GNU Guix) eb5e650e09dd096c066278918456f3e989f7b9d9
running Guix System as my distro
Issue first noticed (again) under a week ago, but could be older.
In the past I fixed it by spoofing a firefox user agent the same way in
both browsers, but it seemed like it was not working to do that anymore.
Currently I have both browsers set to default UAs again, also not
working.
There is also a relevant closed issue on qutebrowser's github:
M
M
Maxim Cournoyer wrote on 15 Dec 2022 05:32
(name . bdju)(address . bdju@tilde.team)(address . 59546-done@debbugs.gnu.org)
87h6xxcmmz.fsf@gmail.com
Hello,

"bdju" <bdju@tilde.team> writes:

Toggle quote (20 lines)
> I have hit this issue before, but not in a while. I don't know for sure
> when the issue came back as it only matters for the initial login or
> visit of the offending sites, so if you manage to get past you'll be
> good until your cookie expires or gets deleted somehow. I got logged out
> of many sites in qutebrowser recently and was made aware of the issue
> being back. I also tested in icecat where some sites I still had a
> cookie, but I managed to find one with a browser check and it indeed
> never completes (can leave it open hours or even more than a day).

> Some sites to test with include:
> https://www.livechart.me/ (must click login button, main page works)
> https://www.gitlab.com/ (again must click log in to start the check)
>
> Since this is happening in both icecat and qutebrowser with different
> user agents and everything, I suspect a guix-related issue. I found that
> at least one other person on IRC was also experiencing the infinite
> browser checks. I use a few sites daily that are now unusable on my Guix
> System machine due to these browser checks, so a fix would be very much
> appreciated if anyone could figure this out.

I've had that too with Gitlab when using Icecat. Sadly, it has nothing
to do with Guix but with how Cloudfare and the website identifies
browsers.

For example, I had found out that by using a Windows Firefox 83 user
agent, I was able to login into Gitlab (using this plugin:
to Gitlab, and they could apparently fixed it on their side (not yet
deployed) [0]


I think other sites or CloudFare must be similarly faulty, or require
fingerprinting which is guarded against out-of-the-box in IceCat.

Closing, as I doubt Guix has something to do with it. If you find
something to the contrary, let us know!

--
Thanks,
Maxim
Closed
B
B
bdju wrote on 15 Dec 2022 13:39
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)(address . 59546-done@debbugs.gnu.org)
CP2ECEKHJ266.2YZBFAJ7Z2ZP9@masaki
On Wed Dec 14, 2022 at 10:32 PM CST, Maxim Cournoyer wrote:
Toggle quote (46 lines)
> Hello,
>
> "bdju" <bdju@tilde.team> writes:
>
> > I have hit this issue before, but not in a while. I don't know for sure
> > when the issue came back as it only matters for the initial login or
> > visit of the offending sites, so if you manage to get past you'll be
> > good until your cookie expires or gets deleted somehow. I got logged out
> > of many sites in qutebrowser recently and was made aware of the issue
> > being back. I also tested in icecat where some sites I still had a
> > cookie, but I managed to find one with a browser check and it indeed
> > never completes (can leave it open hours or even more than a day).
>
> > Some sites to test with include:
> > https://www.livechart.me/ (must click login button, main page works)
> > https://www.gitlab.com/ (again must click log in to start the check)
> >
> > Since this is happening in both icecat and qutebrowser with different
> > user agents and everything, I suspect a guix-related issue. I found that
> > at least one other person on IRC was also experiencing the infinite
> > browser checks. I use a few sites daily that are now unusable on my Guix
> > System machine due to these browser checks, so a fix would be very much
> > appreciated if anyone could figure this out.
>
> I've had that too with Gitlab when using Icecat. Sadly, it has nothing
> to do with Guix but with how Cloudfare and the website identifies
> browsers.
>
> For example, I had found out that by using a Windows Firefox 83 user
> agent, I was able to login into Gitlab (using this plugin:
> https://gitlab.com/ntninja/user-agent-switcher). I reported the issue
> to Gitlab, and they could apparently fixed it on their side (not yet
> deployed) [0]
>
> [0] https://gitlab.com/gitlab-org/gitlab/-/issues/345328
>
> I think other sites or CloudFare must be similarly faulty, or require
> fingerprinting which is guarded against out-of-the-box in IceCat.
>
> Closing, as I doubt Guix has something to do with it. If you find
> something to the contrary, let us know!
>
> --
> Thanks,
> Maxim

I too have fixed it in the past by switching my user agent. I opened and
closed a similar bug some months or years back. That solution stopped
working. I have consulted with folks in the qutebrowser IRC about this
issue several times and it is not affecting everyone there, so it
definitely seems guix-related to me. Something about our packages must
make the browser(s) look odd to these infernal browser checks. I know
that the issue is cloudflare essentially bullying those of us with more
niche setups, but cloudflare has sadly infected most of the Web, so we
have to play their game. I fear the number of qutebrowser users on guix
is in the single digits and so often I'm running into odd problems with
no solution, that again rarely seems to affect people on the other
distros when I go to ask for help. I think our Qt version is almost
always old which probably doesn't help.
I doubt I have the proper evidence to convince you this is guix's fault,
but just know that if you close this issue my problem may never be
solved (although it may never get solved either way honestly).
Closed
M
M
Maxim Cournoyer wrote on 15 Dec 2022 14:26
(name . bdju)(address . bdju@tilde.team)(address . 59546@debbugs.gnu.org)
87a63odci7.fsf@gmail.com
Hi,

"bdju" <bdju@tilde.team> writes:

[...]

Toggle quote (25 lines)
>> I've had that too with Gitlab when using Icecat. Sadly, it has nothing
>> to do with Guix but with how Cloudfare and the website identifies
>> browsers.
>>
>> For example, I had found out that by using a Windows Firefox 83 user
>> agent, I was able to login into Gitlab (using this plugin:
>> https://gitlab.com/ntninja/user-agent-switcher). I reported the issue
>> to Gitlab, and they could apparently fixed it on their side (not yet
>> deployed) [0]
>>
>> [0] https://gitlab.com/gitlab-org/gitlab/-/issues/345328
>>
>> I think other sites or CloudFare must be similarly faulty, or require
>> fingerprinting which is guarded against out-of-the-box in IceCat.
>>
>> Closing, as I doubt Guix has something to do with it. If you find
>> something to the contrary, let us know!

> I too have fixed it in the past by switching my user agent. I opened and
> closed a similar bug some months or years back. That solution stopped
> working. I have consulted with folks in the qutebrowser IRC about this
> issue several times and it is not affecting everyone there, so it
> definitely seems guix-related to me. Something about our packages must
> make the browser(s) look odd to these infernal browser checks.

That's a good lead; could you please test qutebrowser in Guix vs
qutebrowser on another distribution yourself and confirm this hypothesis
(that it works elsewhere?), and post your finings here? If you can do
that and post your finding, I we can reopen the ticket, as we'll have
something actionable to look at.

--
Thanks,
Maxim
B
B
bdju wrote on 15 Dec 2022 19:04
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)(address . 59546@debbugs.gnu.org)
CP2L93WABQAU.20C8GMOFY5Y0H@masaki
On Thu Dec 15, 2022 at 7:26 AM CST, Maxim Cournoyer wrote:
Toggle quote (7 lines)
> That's a good lead; could you please test qutebrowser in Guix vs
> qutebrowser on another distribution yourself and confirm this hypothesis
> (that it works elsewhere?), and post your finings here? If you can do
> that and post your finding, I we can reopen the ticket, as we'll have
> something actionable to look at.
>

I'm able to pass the browser check in a matter of seconds and log in to
GitLab from my PineBook Pro running postmarketOS (Alpine-based) using
qutebrowser.

Here is the :version output from the working qutebrowser:
------------------------------------------------------------------------
qutebrowser v2.5.2
Git commit:
Backend: QtWebEngine 5.15.3, based on Chromium 87.0.4280.144
Qt: 5.15.6

CPython: 3.11.1
PyQt: 5.15.7

sip: 6.6.2
colorama: 0.4.6
jinja2: 3.1.2
pygments: 2.13.0
yaml: 6.0
adblock: 0.6.0
PyQt5.QtWebEngineWidgets: yes
PyQt5.QtWebEngine: 5.15.6
PyQt5.QtWebKitWidgets: no
pdf.js: no
sqlite: 3.40.0
QtNetwork SSL: OpenSSL 3.0.7 1 Nov 2022

Style: QFusionStyle
Platform plugin: wayland
OpenGL: OpenGL ES
Platform: Linux-5.18.0-aarch64-with, 64bit
Linux distribution: postmarketOS edge (alpine)
Frozen: False
Imported from /usr/lib/python3.11/site-packages/qutebrowser
Using Python from /usr/bin/python3
Qt library executable path: /usr/lib/qt5/libexec, data path: /usr/share/qt5

Paths:
cache: /home/bdju/.cache/qutebrowser
config: /home/bdju/.config/qutebrowser
data: /home/bdju/.local/share/qutebrowser
runtime: /run/user/10001/qutebrowser
system data: /usr/share/qutebrowser

Autoconfig loaded: no
Config.py: /home/bdju/.config/qutebrowser/config.py has been loaded
Uptime: 0:02:55
------------------------------------------------------------------------

guix version info for comparison:
------------------------------------------------------------------------
qutebrowser v2.5.2
Git commit:
Backend: QtWebEngine 5.15.5, based on Chromium 87.0.4280.144
Qt: 5.15.5

CPython: 3.9.9
PyQt: 5.15.5

sip: 6.6.1
colorama: 0.4.4
jinja2: 3.1.1
pygments: 2.12.0
yaml: 6.0
adblock: no
PyQt5.QtWebEngineWidgets: yes
PyQt5.QtWebEngine: 5.15.5
PyQt5.QtWebKitWidgets: no
pdf.js: no
sqlite: 3.36.0
QtNetwork SSL: OpenSSL 1.1.1s 1 Nov 2022

Style: QFusionStyle
Platform plugin: xcb
OpenGL: Intel Open Source Technology Center, 3.0 Mesa 21.3.8
Platform: Linux-6.0.8-gnu-x86_64-with-glibc2.33, 64bit
Linux distribution: Guix System (unknown)
Frozen: False
Imported from
/gnu/store/675pkhpvbvi0yai1bggkkaj3h1xy2xrb-qutebrowser-2.5.2/lib/python3.9/site-packages/qutebrowser
Using Python from
/gnu/store/avmnzy8djp42r5926cwznz6ls9gablf8-python-wrapper-3.9.9/bin/python
Qt library executable path:
/gnu/store/mgkd6lgihvqv9n1rlsqfv2rv0zq9whnh-qtbase-5.15.5/lib/qt5/libexec,
data path:
/gnu/store/mgkd6lgihvqv9n1rlsqfv2rv0zq9whnh-qtbase-5.15.5/share/qt5
OS Version:

--- /etc/os-release ---
NAME="Guix System"
ID=guix
PRETTY_NAME="Guix System"
LOGO=guix-icon

Paths:
cache: /home/brad/.cache/qutebrowser
config: /home/brad/.config/qutebrowser
data: /home/brad/.local/share/qutebrowser
runtime: /run/user/1000/qutebrowser

Autoconfig loaded: no
Config.py: /home/brad/.config/qutebrowser/config.py has been loaded
Uptime: 1 day, 3:11:30
------------------------------------------------------------------------
M
M
Maxim Cournoyer wrote on 16 Dec 2022 14:31
(name . bdju)(address . bdju@tilde.team)
87bko3bhkx.fsf@gmail.com
reopen 59546
quit

Hello,

"bdju" <bdju@tilde.team> writes:

Toggle quote (114 lines)
> On Thu Dec 15, 2022 at 7:26 AM CST, Maxim Cournoyer wrote:
>> That's a good lead; could you please test qutebrowser in Guix vs
>> qutebrowser on another distribution yourself and confirm this hypothesis
>> (that it works elsewhere?), and post your finings here? If you can do
>> that and post your finding, I we can reopen the ticket, as we'll have
>> something actionable to look at.
>>
>
> I'm able to pass the browser check in a matter of seconds and log in to
> GitLab from my PineBook Pro running postmarketOS (Alpine-based) using
> qutebrowser.
>
> Here is the :version output from the working qutebrowser:
> ------------------------------------------------------------------------
> qutebrowser v2.5.2
> Git commit:
> Backend: QtWebEngine 5.15.3, based on Chromium 87.0.4280.144
> Qt: 5.15.6
>
> CPython: 3.11.1
> PyQt: 5.15.7
>
> sip: 6.6.2
> colorama: 0.4.6
> jinja2: 3.1.2
> pygments: 2.13.0
> yaml: 6.0
> adblock: 0.6.0
> PyQt5.QtWebEngineWidgets: yes
> PyQt5.QtWebEngine: 5.15.6
> PyQt5.QtWebKitWidgets: no
> pdf.js: no
> sqlite: 3.40.0
> QtNetwork SSL: OpenSSL 3.0.7 1 Nov 2022
>
> Style: QFusionStyle
> Platform plugin: wayland
> OpenGL: OpenGL ES
> Platform: Linux-5.18.0-aarch64-with, 64bit
> Linux distribution: postmarketOS edge (alpine)
> Frozen: False
> Imported from /usr/lib/python3.11/site-packages/qutebrowser
> Using Python from /usr/bin/python3
> Qt library executable path: /usr/lib/qt5/libexec, data path: /usr/share/qt5
>
> Paths:
> cache: /home/bdju/.cache/qutebrowser
> config: /home/bdju/.config/qutebrowser
> data: /home/bdju/.local/share/qutebrowser
> runtime: /run/user/10001/qutebrowser
> system data: /usr/share/qutebrowser
>
> Autoconfig loaded: no
> Config.py: /home/bdju/.config/qutebrowser/config.py has been loaded
> Uptime: 0:02:55
> ------------------------------------------------------------------------
>
> guix version info for comparison:
> ------------------------------------------------------------------------
> qutebrowser v2.5.2
> Git commit:
> Backend: QtWebEngine 5.15.5, based on Chromium 87.0.4280.144
> Qt: 5.15.5
>
> CPython: 3.9.9
> PyQt: 5.15.5
>
> sip: 6.6.1
> colorama: 0.4.4
> jinja2: 3.1.1
> pygments: 2.12.0
> yaml: 6.0
> adblock: no
> PyQt5.QtWebEngineWidgets: yes
> PyQt5.QtWebEngine: 5.15.5
> PyQt5.QtWebKitWidgets: no
> pdf.js: no
> sqlite: 3.36.0
> QtNetwork SSL: OpenSSL 1.1.1s 1 Nov 2022
>
> Style: QFusionStyle
> Platform plugin: xcb
> OpenGL: Intel Open Source Technology Center, 3.0 Mesa 21.3.8
> Platform: Linux-6.0.8-gnu-x86_64-with-glibc2.33, 64bit
> Linux distribution: Guix System (unknown)
> Frozen: False
> Imported from
> /gnu/store/675pkhpvbvi0yai1bggkkaj3h1xy2xrb-qutebrowser-2.5.2/lib/python3.9/site-packages/qutebrowser
> Using Python from
> /gnu/store/avmnzy8djp42r5926cwznz6ls9gablf8-python-wrapper-3.9.9/bin/python
> Qt library executable path:
> /gnu/store/mgkd6lgihvqv9n1rlsqfv2rv0zq9whnh-qtbase-5.15.5/lib/qt5/libexec,
> data path:
> /gnu/store/mgkd6lgihvqv9n1rlsqfv2rv0zq9whnh-qtbase-5.15.5/share/qt5
> OS Version:
>
> --- /etc/os-release ---
> NAME="Guix System"
> ID=guix
> PRETTY_NAME="Guix System"
> LOGO=guix-icon
> DOCUMENTATION_URL="https://guix.gnu.org/en/manual"
>
> Paths:
> cache: /home/brad/.cache/qutebrowser
> config: /home/brad/.config/qutebrowser
> data: /home/brad/.local/share/qutebrowser
> runtime: /run/user/1000/qutebrowser
>
> Autoconfig loaded: no
> Config.py: /home/brad/.config/qutebrowser/config.py has been loaded
> Uptime: 1 day, 3:11:30
> ------------------------------------------------------------------------

Thanks for sharing. It looks like we have something to investigate.
It's perhaps related to the way we build qtwebengine; do these sites
work in ungoogled-chromium for you?

Re-opening!

--
Thanks,
Maxim
B
B
bdju wrote on 16 Dec 2022 15:56
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
CP3BVWRQA1HB.1QVYWNEFR5DJ2@masaki
On Fri Dec 16, 2022 at 7:31 AM CST, Maxim Cournoyer wrote:
Toggle quote (4 lines)
> Thanks for sharing. It looks like we have something to investigate.
> It's perhaps related to the way we build qtwebengine; do these sites
> work in ungoogled-chromium for you?

Just tested in ungoogled-chromium, and yes it seems both gitlab and
livechart worked fine, passing the brower checks quickly on my Guix
System machine. Very interesting.
I tried copying the user agent from ungoogled-chromium (similar but
slightly different from the qutebrowser default) but it didn't seem to
make a difference.
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 59546@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 59546
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch