Its possible I have pushed a change that fixed this.
Davzz: forum
Davzz [323969] Level 17
- Forum posts (Torn's count)
- 427 observed
- Archived posts
- 64
- Threads started (archived)
- 2
Boards
From the 64 newest archived posts.
- Bugs & Issues
- Announcements
- General Discussion
Threads started
Most-liked posts
Likes as archived at fetch time, adjusted for how long each post had been up. Scored posts only; the full score pass runs after the import.
Hello all,
I apologize for the delay posting here.
We are aware of this problem. Most likely this is due to one of many emergency changes at the weekend to keep the site online. Now that crisis is over, and the site is available, I can assure you that we are working on this. It is not immediately clear to us why this is failing but we are working to figure it out.
An update will follow over the next few days. My apologize for the inconvenience.
Update: I believe this issue may be fixed. Please confirm.
Hello all,
Its pretty clear there is no single cause of this from the HAR files I have, but we have made some further changes that we think reduce the number of connections each browser makes (by fixing/improving cache headers for files that dont change). We also turned off HTTP/3 for TORN. The header change took some time to fully apply, but should be applied everywhere now - I am curious how folks find things from about now-ish.
One thing that we have noticed is that some (but certainly not all) of this is caused by many tabs running. Each tab has 2 connections to the main TORN domain, and browsers have limits on the number of connections they make. If you are seeing this, please check that you do not have lots of tabs open and try to reproduce the problem in a different browser if possible.
Clearly, for those users who do have a ISP that has a rubbish connection to cloudflare we are not going to be able to totally fix this, but hopefully we can help make it better! We thank you all for your patience.
-Alex
Its pretty clear there is no single cause of this from the HAR files I have, but we have made some further changes that we think reduce the number of connections each browser makes (by fixing/improving cache headers for files that dont change). We also turned off HTTP/3 for TORN. The header change took some time to fully apply, but should be applied everywhere now - I am curious how folks find things from about now-ish.
One thing that we have noticed is that some (but certainly not all) of this is caused by many tabs running. Each tab has 2 connections to the main TORN domain, and browsers have limits on the number of connections they make. If you are seeing this, please check that you do not have lots of tabs open and try to reproduce the problem in a different browser if possible.
Clearly, for those users who do have a ISP that has a rubbish connection to cloudflare we are not going to be able to totally fix this, but hopefully we can help make it better! We thank you all for your patience.
-Alex
Hi All,
We are not able to give exact details about the systems we have in place (that would give the attackers a bit of an advantage!) but we have attempted to figure out these limits to absolutely minimize the impact they have. Ignoring fairly brief periods where we get a flood of obviously malicious traffic, we are blocking a very small amount of traffic.
I made some further changes to the system today to try to reduce the number of false positives. If you continue to have problems, please post your source IP (https://www.whatismyip.com/) and the TC time that you were blocked, along with the URL you were blocked and a description of what you are running (e.g. if you have plugins or citywatch or scripts or API use on your machine).
We will continue to do our best to tweak this to minimize the inconvenience. We apologize on behalf of the folks attacking us that we even have to do this; unfortunately the alternative to automated blocking as we are doing now is not ideal.
-Davz
Apologies for the inconvenience. We had some trouble with a subset of the responses - which I believe is now resolved. Please let us know if you see it again.
Recent archived posts
Little_Naz: we are very confident this is not DDOS blocking. Most of the HAR files we have show requests in a 'blocked' state which is the local browser blocking and we have logs on the cloudflare side that clearly show that once the requests folk are complaining about make it there they are very fast.
Its possible that we have changed something that is tripping the browser limits more than in the past - that bit isnt really super obvious to me. Its also possible its something out of our control (e.g. torntools type extensions are used by many users and count towards the browser limits).
If it makes you feel better, here is a bug report with Google engineers complaining about this limit in Chrome: https://issues.chromium.org/issues/40180988. Chrome have made their limit a bit larger since then, but with many tabs (each doing multiple websocket connections and many HTTP requests) its not hard to imagine that we are hitting into this.
Its possible that we have changed something that is tripping the browser limits more than in the past - that bit isnt really super obvious to me. Its also possible its something out of our control (e.g. torntools type extensions are used by many users and count towards the browser limits).
If it makes you feel better, here is a bug report with Google engineers complaining about this limit in Chrome: https://issues.chromium.org/issues/40180988. Chrome have made their limit a bit larger since then, but with many tabs (each doing multiple websocket connections and many HTTP requests) its not hard to imagine that we are hitting into this.
Hello all,
Its pretty clear there is no single cause of this from the HAR files I have, but we have made some further changes that we think reduce the number of connections each browser makes (by fixing/improving cache headers for files that dont change). We also turned off HTTP/3 for TORN. The header change took some time to fully apply, but should be applied everywhere now - I am curious how folks find things from about now-ish.
One thing that we have noticed is that some (but certainly not all) of this is caused by many tabs running. Each tab has 2 connections to the main TORN domain, and browsers have limits on the number of connections they make. If you are seeing this, please check that you do not have lots of tabs open and try to reproduce the problem in a different browser if possible.
Clearly, for those users who do have a ISP that has a rubbish connection to cloudflare we are not going to be able to totally fix this, but hopefully we can help make it better! We thank you all for your patience.
-Alex
Its pretty clear there is no single cause of this from the HAR files I have, but we have made some further changes that we think reduce the number of connections each browser makes (by fixing/improving cache headers for files that dont change). We also turned off HTTP/3 for TORN. The header change took some time to fully apply, but should be applied everywhere now - I am curious how folks find things from about now-ish.
One thing that we have noticed is that some (but certainly not all) of this is caused by many tabs running. Each tab has 2 connections to the main TORN domain, and browsers have limits on the number of connections they make. If you are seeing this, please check that you do not have lots of tabs open and try to reproduce the problem in a different browser if possible.
Clearly, for those users who do have a ISP that has a rubbish connection to cloudflare we are not going to be able to totally fix this, but hopefully we can help make it better! We thank you all for your patience.
-Alex
Hi All,
We have looked at a fair few HAR files from this. We have significantly increased the amount of logging we have in our control from requests before they hit our infrastructure, which helps dig into this stuff. So far, the HAR files i've reviewed all show connection problems between a user and Cloudflare's closest datacenter. Cloudflare is one of, if not the, best connected CDN providers we could have chosen but from time to time folks may have problems between their ISP and cloudflare that are solved by VPNs (which hit another cloudflare POP). Additionally, we are using Cloudflare's 'managed challenge' feature to protect TORN against DDOS attacks that are a sad reality for us - these will sometimes create problems though we have a bit more ability to make changes here if we have clear evidence of specific problems.
If somebody can reproduce this please can you email a sanitised HAR file to alex@torn.com? Please also run the https://speed.cloudflare.com/ at the same time and export the results as a PDF to send. Finally, please try to see if running the free cloudflare VPN (https://one.one.one.one/) on your device makes the problem better or worse.
Thanks!
-Alex
We have looked at a fair few HAR files from this. We have significantly increased the amount of logging we have in our control from requests before they hit our infrastructure, which helps dig into this stuff. So far, the HAR files i've reviewed all show connection problems between a user and Cloudflare's closest datacenter. Cloudflare is one of, if not the, best connected CDN providers we could have chosen but from time to time folks may have problems between their ISP and cloudflare that are solved by VPNs (which hit another cloudflare POP). Additionally, we are using Cloudflare's 'managed challenge' feature to protect TORN against DDOS attacks that are a sad reality for us - these will sometimes create problems though we have a bit more ability to make changes here if we have clear evidence of specific problems.
If somebody can reproduce this please can you email a sanitised HAR file to alex@torn.com? Please also run the https://speed.cloudflare.com/ at the same time and export the results as a PDF to send. Finally, please try to see if running the free cloudflare VPN (https://one.one.one.one/) on your device makes the problem better or worse.
Thanks!
-Alex
Its possible I have pushed a change that fixed this.
^^ Thanks, i've got a handful of users - no need for more HAR files from new users (i'm emailing some of those users directly to ask for more tho!)
If someone can reproduce this are they able to capture a full HAR file with the request and response please?
https://developers.cloudflare.com/support/troubleshooting/general-troubleshooting/gathering-information-for-troubleshooting-sites/
You can email this to alex@torn.com. Please sanitize it to remove your session IDs (see the blog above)
https://developers.cloudflare.com/support/troubleshooting/general-troubleshooting/gathering-information-for-troubleshooting-sites/
You can email this to alex@torn.com. Please sanitize it to remove your session IDs (see the blog above)
There are no currently known issues, and i've gone for a manual look for errors like timeouts and do not see any increase. Can somebody still seeing issues give specific examples, e.g. if it 'hangs to the point of needing F5' what does that mean? If you leave it what error do you get?
Can you link to a screenshot and reproducer for this problem pls?
To try to line this up with the logs on our side, can somebody who can reproduce this run developer tools in Chrome, and send a screenshot / export a HAR file for a failing request? Paste the screenshot here (do not paste the HAR file publicly, but i'll email you to get that).
I've reviewed this, but I fear this is not something we control - all signs here seem to be that your ISP is doing something unhelpful here.
My best suggestion is to try to use a VPN app and see if the problem goes away; my guess is that it will.
My best suggestion is to try to use a VPN app and see if the problem goes away; my guess is that it will.
BTW please run "ping www.torn.com" in a terminal at the same time as capturing any pcap, and send screenshots of that, screenshot of the chrome dev tools, a HAR file from dev tools for the failing request (see https://support.zendesk.com/hc/en-us/articles/204410413-Generating-a-HAR-file-for-troubleshooting) and the pcap from wireshark.
The SPDY errors that many users received were confirmed to be a bug with our DDOS mitigation vendor, which occurred during attacks that should have been mitigated. It only affected modern browsers using HTTP/2. The bug was fixed some weeks ago. The problem was obvious because it happened to very many users.
The error that Stewie is reporting is a new error, and not one we are receiving many reports about - and its most likely a issue making a TCP connection to cloudflare. This cant be a torn problem, and is very unlikely to be a cloudflare problem. If Stewie can reproduce this, what we need is a wireshark tcpdump on the client at the time that the error was seen. Please re-open this thread if you have that and we can look at it - the most likely cause for this really is the internet connection that Stewie has so testing from a phone without wifi at the same time may help narrow that down.
All,
Over the last few months, we have been plagued by denial of service attacks that have taken us out from between a few minutes to a few hours. This is a frustrating new problem for us, and it has taken us too long to respond to this properly. We all share the pain of all players, and apologize that we have not been able to effectively mitigate these attacks as well as we would like.
It is not good practice to share the glory details of what we are doing or have done, but its fair to say that the attacks we have seen are enormous, and sufficient to overwhelm the previous DDOS mitigation systems that we had in place. We are fairly confident that changes made earlier this week will significantly reduce the prevalence of this problem. This has required us to work closely with several of our vendors, including our outer layer of DDOS shield (CloudFlare).
Please continue to report any connectivity issues to torn going forward. We will do our best to keep our uptime as high as possible!
The Torn Staff
This is not going to be a Torn side issue; if there is a problem here it will be some sort of transparent proxy on the user side. The torn SSL config has not changed for several years, and is behind CloudFlare. Cloudflare do not have problems with their SSL serving (or half the internet would see this).
I'm marking this resolved - we made some changes ~10 days ago that we believe made the false positive rate acceptable. If you are still having this issue, please search for existing threads and add your details to it. Thanks!
I'm marking this resolved - we made some changes ~10 days ago that we believe made the false positive rate acceptable. If you are still having this issue, please search for existing threads and add your details to it. Thanks!
I'm marking this resolved - we made some changes ~10 days ago that we believe made the false positive rate acceptable. If you are still having this issue, please search for existing threads and add your details to it. Thanks!
I'm marking this resolved - we made some changes ~10 days ago that we believe made the false positive rate acceptable. If you are still having this issue, please search for existing threads and add your details to it. Thanks!
I'm marking this resolved - we made some changes ~10 days ago that we believe made the false positive rate acceptable. If you are still having this issue, please search for existing threads and add your details to it. Thanks!
I'm marking this resolved - we made some changes ~10 days ago that we believe made the false positive rate acceptable. If you are still having this issue, please search for existing threads and add your details to it. Thanks!