Last update: 2007-11-10 (marcelk) Problem solved. Problem description =================== ADSL 12M/1M from Hinet performs poorly for international connections. This problem is isolated to Hinet. 1. HTTP connections unreliable Test: http://marcelk.bitpit.net/speedtest/ download testdata.dat (55MB) This test usually fails to download to 100%. Usually, there is no error message indicating any failure, the download is simply incomplete. Sometimes there is an error message "Connection reset by peer". It sometimes works at very slow speed (100kB/s) during the time that HiNet proxy is disabled, for example for a couple of hours on 2007-10-27. Status: 2007-11-10 (problem solved) 2. Internet radio (music streaming) doesn't work continously Test: http://mp3.streampower.be/stubru-high.mp3 (10 kB/sec) This radio stream works only for a few minutes, and then Hinet can't keep up. The only solution is to switch to poor quality radio. Status: 2007-10-27 (old problem not related to HTTP; performance is OK now) 3. Video streaming from youtube doesn't work most of the time Test: http://www.youtube.com/watch?v=AvYEg0adRis Often just doesn't load at all. Many reload requests needed. Even then, speed is low. Status: 2007-11-10 (problem solved) Test result overview ==================== Place Computer Connection Testsite Result ----- -------- ---------- -------- ------ Hinet,Taipei Apple 802.11 bitpit.net FAILED Hinet,Taipei Apple TCP bitpit.net FAILED Hinet,Taipei Apple PPPoE bitpit.net FAILED Hinet,Taipei WinXP 802.11 bitpit.net FAILED Hotel,Japan Apple 802.11 bitpit.net OK Hotel,Hsinchu Apple TCP bitpit.net OK Hotel,Hsinchu Apple 802.11 bitpit.net OK Friend Linux 802.11 bitpit.net OK Office,NL WinXP TCP bitpit.net OK Observation summary =================== - Hinet.net ADSL service drops HTTP connections randomly - Hinet serves old version of webpages - Problem is isolated to Hinet Internet connections only - Hinet seems to use a transparant web proxy: connections come in through 211.75.91.20, but my actual IP address is 217.21.240.155 . Likely, Hinet's web proxy is causing the problem. - This proxy reveals itself as "CacheFlow" (Blue Coat?) - The proxy is not for all traffic: traffic to gewis.nl came directly from my client, even before contacting Hinet helpdesk. Log book ======== -------- Date: 2007-10-27 13:00 Dialed 0800-080412 - Selected English service Service representative who answers the phone doesn't speak English... Got forwarded to next operator. Explained problem. Confirmed that computer is directly connected to the ADSL modem - switch off modem - operator resets adsl line - switch on modem - test: still fails downloading (at 19%) Gave my cell phone number. Forwarded to next service engineer Explained problem (again). - switch off modem - engineer resets adsl line - swtich on modem - test: still fails downloading (at 28%) Problem will be escalated to engineering center. Gave by cell phone number (again). Hinet promised to report back 2007-10-27 16:00h -------- -------- Date: 2007-10-27 14:40 Got call from phone company (on house phone, not cell phone). They heard there was a problem with the phone (which is not the case). They didn't know anything about the problem description. They didn't let me explain the problem. They will reset ADSL and call back. -------- -------- Date: 2007-10-27 15:30 Got call from phone company (on house phone, not cell phone). They transfer the problem to technical center on Monday. Tuesday I will be called for appointment. -------- -------- Date: 2007-10-27 16:00 Many hits from hinet network on the speedtest, (Mozilla/3.01) Suddenly, HTTP connections come directly from my own IP address again. Looks like Hinet has disabled their web proxy. Testing: Tranfer: OK (!) Speed: slow (100kB/s) -------- -------- Date: 2007-10-27 19:00 1. Confirmed that gewis.nl was never served through the proxy. 125-225-121-89.dynamic.hinet.net - - [27/Oct/2007:09:00:03 +0200] 125-225-121-89.dynamic.hinet.net - - [27/Oct/2007:11:18:58 +0200] 125-225-121-89.dynamic.hinet.net - - [27/Oct/2007:11:19:24 +0200] 2. HiNet enabled the proxy again(???). The problems returned... Download test initially is ok (100% of file loaded & very fast >300kB/s), BUT when I modify the file on the server, the old file is still being served instead of the new file. This is not a problem when accessing the server from another network --> Hinet problem The Etag has changed, but the CacheFlow Proxy doesn't honor it. (It doesn't even perform a HEAD request to the web site). From hinet connection, stale cache info is served: telnet marcelk.bitpit.net 80 Trying 217.21.240.155... Connected to marcelk.bitpit.net. Escape character is '^]'. HEAD /speedtest/testdata.dat HTTP/1.1 Host: marcelk.bitpit.net HTTP/1.0 200 OK Connection: close Date: Sat, 27 Oct 2007 07:46:40 GMT Server: Apache Last-Modified: Fri, 05 Apr 2002 23:15:03 GMT <--- *WRONG* ETag: "1cbec825-36f5829-e362abc0" Accept-Ranges: bytes Content-Length: 57628713 <--- *WRONG* SHOULD BE 100,000,000 !! Content-Type: chemical/x-mopac-input X-Pad: avoid browser bug <--- Where does this come from? Connection closed by foreign host. From outside hinet: telnet marcelk.bitpit.net 80 Trying 217.21.240.155... Connected to marcelk.bitpit.net. Escape character is '^]'. HEAD /speedtest/testdata.dat HTTP/1.1 Host: marcelk.bitpit.net HTTP/1.1 200 OK Date: Sat, 27 Oct 2007 11:36:53 GMT Server: Apache Last-Modified: Sat, 27 Oct 2007 11:07:29 GMT ETag: "1cbec0a4-5f5e100-791a1e40" Accept-Ranges: bytes Content-Length: 100000000 Content-Type: chemical/x-mopac-input Connection closed by foreign host. When did the problem return? 125.225.121.89 - - [27/Oct/2007:11:41:30 +0200] --> OK 211.75.91.20 - - [27/Oct/2007:12:52:51 +0200] --> WRONG (That is between 17:41 and 18:52 Taiwan time). -------- -------- Date: 2007-10-27 21:15 Something changed again: HTTP connections go direct (cache is bypassed again). Measurement results: Download test 1: Transfer: OK (95M) Speed: slow (85kB/s) Download test 2: (file replaced by different version) Transfer: OK (55M) Speed: slow (88kB/s) This explains why gewis.nl always appear slower than bitpit.net: "bitpit.net got cached, but gewis.nl was always direct (but why?)" This also reveals Hinet's actual Internet speed for 12M/1M ADSL subscribers: *** No more than 100kB/s *** -------- -------- Date: 2007-10-28 9:25 Old problem is back Transfer (55MB): retried several times FAILS at 20% FAILS at 5% FAILS at 6% FAILS at 64% Speed: ~195kB/s Hinet cache behaviour has changed since yesterday: HEAD & GET requests are passed to bitpit.net from 211.75.81.20. But not always. Before, nothing appeared in the logging. But file download is still incomplete. -------- -------- Date: 2007-10-29 12:30 Hinet has disabled the proxy again.. Download test: Transfer: OK (55MB) Speed: slow (70 kB/s) These timestamps correspond to between 10:45 and 11:55 Taiwan time. 211.75.91.20 - - [28/Oct/2007:03:45:36 +0100] <--- WRONG 125.225.121.89 - - [28/Oct/2007:04:55:47 +0100] <--- OK -------- -------- Date: 2007-10-30 9:30 Hinet engineer called. I was at the office, not at home.. Got a confusing story about making some free account changes on the hinet.net website. I explained the problem are caused by Hinet's web caches. I explained that I'm not at home. Engineer explained that the proposed change is supposed to improve international internet connections. Made agreement that hinet would call me this evening at 20:00 to explain what to do. -------- -------- Date: 2007-10-30 18:30 Hinet engineer called again. I was still at work. Engineer asked to call the service center at 0800-0080412 when I'm at home. He will call me back tomorrow to check if it solved my problems. Friendly service so far. -------- -------- Date: 2007-10-30 20:00 I made some http queries. The http connections come in from several transparant proxies now: 211.75.91.25 211.75.91.23 211.75.91.18 My IP now is 125.225.99.51 . Download test: Transfer: FAILED at 16% (55MB) Speed: slow (around 60kB/s) This time there is an error message: ction failed on "POSIX error: Connection reset by peer" Server got a request from 211.75.91.20 -------- -------- Date: 2007-10-30 20:15 Calling support center. Selected '3' for English service. Operator who answers doesn't speak English. Got passed on to an English speaking engineer. So hinet doesn't use the phone menu. A new user account was created with the same password. 72198376@ip.hinet.net (was: 72198376@hinet.net) Connected computer directly to ADSL modem and rebooted the modem. It received a new IP adress 'from a different group' 220.134.88.181 Ended phone call and started testing. Downloading short file: http connection goes through proxy: 211.75.91.24. This file comes in ok. Download test large file: Transfer: FAILED (36%) Speed: slow (~60kB/s) Same error message: Connection reset by peer. Retry: Transfer: FAILED (8%) Connection reset by peer. Connection came from: 211.75.91.24 Retry: Transfer: FAILED (10%) Connection reset by peer. Download from gewis.nl worked (not served by this proxy) Changed account in router also. Will send hinet this log book tomorrow for further investigation. -------- -------- Date: 2007-11-02 23:30 Hinet engineer still hasn't called as promised. I sent a e-mail in reply to one of Hinet's emails. +---------------------------------------------------- To: info@ms1.hinet.net Date: 2 November, 2007 23:38:57 GMT+08:00 Dear hinet customer support, I have applied these changes you adviced me to do. However, it didn't solve my download problems. The problem is caused by HiNet's transparent web proxies operating for TCP port 80 trafic that show two severe problems: 1. Hinet web proxies serve stale information from cache without performing a HEAD check to the actual requested server 2. Hinet web proxies cannot serve large documents from cache without breaking connections For techinical details, please check http://marcelk.bitpit.net/speedtest/hinet-problems.txt Please contact me to explain who hinet will solve this problem with my ADSL service. My cell phone number is 09 7211 8808 Kindest regards, +---------------------------------------------------- Waiting for answer. -------- -------- Date: 2007-10-03 13:40 Dialed 0800-080412 Operator asked my phone number and told Hinet will call back Got called from 04-2210 7020 by hinet engineer. Explained problem again and told about the e-mail. Engineer accessed website. Got some hits from 203.69.169.99 Will call back later Called back. Problem will be transfered. Will be called Monday. -------- -------- Date: 2007-10-03 15:20 Hinet service engineer called (on cell phone). They wanted to visit home an check the last mile. I asked if they had read the e-mail with the problem description. The answer is 'no'. I explained the problem is not in the 'last mile', it is in the international connections. I asked Hinet to read the e-mail first. Hinet engineer asked me to send the e-mail directly to . -------- -------- Date: 2007-10-05 19:00 Received no phone call today. Dialed 0800-080412 to get the status. Supplied cell phone number. I will be called back tomorrow. -------- -------- Date: 2007-10-06 9:40 Hinet engineer called for follow-up. Proposal is to do a line-check on Saturday. I asked if they had read the problem description. Answer: no I asked hinet to read it. That was agreed. Later Hinet called and cancelled the line check, because it was clear that is not where the problem is. (progress!). Problem would be transfered to technical center again. Since I would be working during day time, they would call at home between 8PM and 11PM. Around 21:30 I got a call (from Taichung number). However I was still in office for work due to unexpected emergency. I asked if it is Ok to call back at 23:00, so I could get home for the tests. Engineer said that would be no problem at all. I brought the Windows laptop from office. When home I checked the problems still there: Conclusion: download still fails (both for Apple and WindowsXP) Unfortunately, I received no more call. It must have been busy at the technical center. No problem, can continue later. I called the service center number in Taichung. Service engineer asked to make 'tracert' logging and send to tsscs@ms1.hinet.net H:\>tracert marcelk.bitpit.net Tracing route to marcelk.bitpit.net [217.21.240.155] over a maximum of 30 hops: 1 * * * Request timed out. 2 32 ms 31 ms 31 ms 125-225-96-254.dynamic.hinet.net [125.225.96.254] 3 30 ms 31 ms 31 ms tp-sp2-c12r1.router.hinet.net [168.95.160.50] 4 31 ms 31 ms 32 ms tp-crs11.router.hinet.net [220.128.1.74] 5 32 ms 32 ms 31 ms tp-s2-c12r31b.router.hinet.net [220.128.1.190] 6 496 ms 501 ms 508 ms pa-c12r12.USA-PAIX.router.hinet.net [211.72.108.165] 7 162 ms 162 ms 177 ms POS5-0.IG3.PAO1.ALTER.NET [63.65.130.29] 8 162 ms 161 ms 161 ms 0.so-3-0-0.XL1.PAO1.ALTER.NET [152.63.49.254] 9 163 ms 162 ms 162 ms 0.so-5-2-0.XT1.SCL2.ALTER.NET [152.63.54.134] 10 162 ms 162 ms * 0.so-6-0-0.BR1.SCL2.ALTER.NET [152.63.57.49] 11 177 ms 177 ms 183 ms so-4-0-0.mpr2.sjc7.us.above.net [64.125.13.61] 12 177 ms 176 ms 177 ms so-0-0-0.mpr1.sjc7.us.above.net [64.125.30.169] 13 177 ms 178 ms 178 ms so-1-0-0.mpr2.sjc2.us.above.net [64.125.30.178] 14 * 235 ms * so-1-0-0.mpr1.lga5.us.above.net [64.125.26.142] 15 329 ms 329 ms 329 ms so-1-0-0.mpr1.ams1.nl.above.net [64.125.27.186] 16 330 ms 330 ms 330 ms 62.93.194.107 17 328 ms 325 ms 323 ms son-er1-dc1.signet.nl [217.21.255.75] 18 393 ms 400 ms 400 ms marcelk.bitpit.net [217.21.240.155] Trace complete. H:\> -------- -------- Date: 2007-11-08 Got a call from HiNet engineer that they had changed something with an international line. He requested me to test if it solved the problem. I was at work, so I said I would do it in the evening. 20:30 Back at home I tested the downloading again. Performance was very bad: download was usually dropped at the first MB (<2%). I tested several times, same poor result. -------- Date: 2007-11-09 I called Hinet and explained about the problem still being unsolved. The operator said that she checked the line but it is ok. I asked to be transferred to the engineer who had been handling my isse. The operator told that she would inform him and I would be called back later. Later I was called back. I explained the download still failed after yesterday's action. The engineer revealed something interesting: -- They had tested the download from their own ADSL, and it FAILED. Good! Hinet confirms they can reproduce the problem! That is just one step away from isolating and solving it. One would think so, but not at the HiNet engineering department: HiNet got ugly instead: -- HiNet claims that it is caused by the site (marcelk.bitpit.net) I explained no other ISP suffers from this problem, and they can now see for themselves it is failing. -- "We have checked the proxy's and they are OK" I told that I would be glad to that invite HiNet tester to come to my house and uphold that statement while I'm performing the HTTP queries manually, show him the RFC, and show the tcpdump logging from outside Hinet. The engineer did not respond to this offer. -- Then it was "nobody else is complaining", only a few subscribers. First of all, many sites have poor download performance with HiNet ADSL. This is just a test site to tackle the problem. If more subscribers complain about this proxy behaviour, then that indicates maybe something indeed is wrong. But more importantly, I like it to be solved for me. Well, I don't care what everybody else puts up with. The engineer told he would contact the other support center again and call me back. Somewhat later I got a call back: Hinet offered to disable the proxy for my new IP address, because it is static anyway. The proxy would still be used for dynamic IP range. I agreed with the offer and told I would test it in the evening or on Friday. -------- -------- 2007-11-09 HiNet has indeed disabled the proxy. However, the direct download speed has also dropped to almost zero. Before I would het ~100kB/s when there was no proxy involved (see the test of 2007-10-27 21:15). But now it is down to 20kB/s. I tried my upload speed to marcelk.bitpit.net: 150kB/s! Lets do some cross-testing to see what is going on: We have several sites with testdata.dat (55MB): 217.21.240.147 http://solar.bitpit.net/~marcelk/testdata.dat 217.21.240.150 http://www.sebster.com/testdata.dat 217.21.240.155 http://marcelk.bitpit.net/speedtest/testdata.dat 131.155.71.116 http://gewis.nl/~nargileh/speedtest/testdata.dat We can download from two different HiNet IP numbers (static and dynamic) and from one independent ADSL site: unox 220.134.88.181 We can download using two different protocols: HTTP and SSL(scp) Download client Testdata server Method kB/s --------------- --------------- ------ ----- 220.134.88.181 217.21.240.150 http 81 220.134.88.181 217.21.240.155 http 26 220.134.88.181 217.21.240.150 http 76 220.134.88.181 217.21.240.155 http 26 194.109.162.75 217.21.240.155 http 1160 194.109.162.75 217.21.240.150 http 502 194.109.162.75 217.21.240.150 http 495 194.109.162.75 217.21.240.155 http 1100 194.109.162.75 217.21.240.150 http 508 194.109.162.75 217.21.240.155 http 1150 194.109.162.75 131.155.71.116 http 1090 194.109.162.75 131.155.71.116 http 1140 194.109.162.75 217.21.240.150 http 508 194.109.162.75 217.21.240.155 http 1140 Independent download comparison (different ISP): 194.109.162.75 217.21.240.147 http 1170 194.109.162.75 217.21.240.155 http 1120 194.109.162.75 217.21.240.150 http 498 194.109.162.75 217.21.240.147 http 1130 194.109.162.75 217.21.240.155 http 1170 220.134.88.181 217.21.240.147 http 27 220.134.88.181 217.21.240.150 http 86 wait until 2AM 220.134.88.181 217.21.240.155 http 39 ------ ------ ------ ------- -------- -------- 2007-10-09 8:00 AM 220.134.88.181 217.21.240.155 http 94 220.134.88.181 217.21.240.155 http 77 -------- -------- 2007-11-10 18:00 Sent a confirmation to HiNet support -------- To: tccsc@ms1.hinet.net, cskao422@cht.com.tw, info@ms1.hinet.net Dear HiNet supporters, I like to confirm that the download problems with my HiNet ADSL account are indeed fully solved since Thursday after the following two changes: 1. swithing from dynamic to the static IP adresss 2. HiNet disabled the transparant HTTP proxies for HTTP traffic from this address. Also the downloads from youtube.com and other sites are reliable now. Thanks for the support, -------- --------