8:40 a.m.
On Sat, 29 Nov 2014 23:57:22 +0000, Commit Mailer wibbled on for an age:
correctly calculate writeout bandwidth and properly impose limits
Is this really right?
(22.696681) content/llcache.c llcache_persist 2414: Wrote 884 bytes in 225ms bw:3928
http://aminet.net/pics/at.gif
(22.696759) content/llcache.c llcache_persist 2420: Overran timeslot
(22.696828) content/llcache.c llcache_persist 2426: Cannot write minimum bandwidth
(22.697699) amiga/misc.c ami_misc_req 51: Disc cache write bandwidth is too slow to be
useful, disabling cache
(22.497406) content/llcache.c llcache_persist 2414: Wrote 1480 bytes in 114ms bw:12982
http://news.bbcimg.co.uk/view/1_4_38/cream/hi/news/img/services.gif
(22.497485) content/llcache.c llcache_persist 2420: Overran timeslot
(22.497549) content/llcache.c llcache_persist 2426: Cannot write minimum bandwidth
(22.497751) amiga/misc.c ami_misc_req 51: Disc cache write bandwidth is too slow to be
useful, disabling cache
(16.366240) content/llcache.c llcache_persist 2414: Wrote 3931 bytes in 263ms bw:14946
http://i2.cdnds.net/13/47/hearst.png
(16.366319) content/llcache.c llcache_persist 2420: Overran timeslot
(16.366384) content/llcache.c llcache_persist 2426: Cannot write minimum bandwidth
(16.366585) amiga/misc.c ami_misc_req 51: Disc cache write bandwidth is too slow to be
useful, disabling cache
Even if I spawn it off to another process (so store returns
immediately) I still get the not enough bandwidth message.
Chris
10:01 a.m.
New subject: netsurf: branch master updated. release/3.2-500-gd08acbc
On Sun, Nov 30, 2014 at 01:40:47PM +0000, Chris Young wrote:
On Sat, 29 Nov 2014 23:57:22 +0000, Commit Mailer wibbled on for an
age:
> correctly calculate writeout bandwidth and properly impose limits
Is this really right?
(22.696681) content/llcache.c llcache_persist 2414: Wrote 884 bytes in 225ms bw:3928
http://aminet.net/pics/at.gif
(22.696759) content/llcache.c llcache_persist 2420: Overran timeslot
(22.696828) content/llcache.c llcache_persist 2426: Cannot write minimum bandwidth
(22.697699) amiga/misc.c ami_misc_req 51: Disc cache write bandwidth is too slow to be
useful, disabling cache
well under 4000 bytes a second is so slow that it is not useful. The
disc caching really needs to be several multiples of the network
conenction speed to be useful. Although the minimum bandwidth setting
is a passe din parameter and you can chnage the default value in
desktop/netsurf.c to experiment.
(22.497406) content/llcache.c llcache_persist 2414: Wrote 1480 bytes in 114ms bw:12982
http://news.bbcimg.co.uk/view/1_4_38/cream/hi/news/img/services.gif
(22.497485) content/llcache.c llcache_persist 2420: Overran timeslot
(22.497549) content/llcache.c llcache_persist 2426: Cannot write minimum bandwidth
(22.497751) amiga/misc.c ami_misc_req 51: Disc cache write bandwidth is too slow to be
useful, disabling cache
(16.366240) content/llcache.c llcache_persist 2414: Wrote 3931 bytes in 263ms bw:14946
http://i2.cdnds.net/13/47/hearst.png
(16.366319) content/llcache.c llcache_persist 2420: Overran timeslot
(16.366384) content/llcache.c llcache_persist 2426: Cannot write minimum bandwidth
(16.366585) amiga/misc.c ami_misc_req 51: Disc cache write bandwidth is too slow to be
useful, disabling cache
Even if I spawn it off to another process (so store returns
immediately) I still get the not enough bandwidth message.
Is it possible that your implementation of a milisecond monotonic
counter in libnsutils is problematic? If it is returning microseconds
instead of miliseconds that would cause this erroneus behaviour.
Chris
--
Regards Vincent
http://www.kyllikki.org/
11:09 a.m.
New subject: netsurf: branch master updated. release/3.2-500-gd08acbc
On Sun, 30 Nov 2014 15:01:54 +0000, Vincent Sanders wrote:
> (22.696681) content/llcache.c llcache_persist 2414: Wrote 884
bytes in 225ms bw:3928 http://aminet.net/pics/at.gif
> (22.696759) content/llcache.c llcache_persist 2420: Overran timeslot
> (22.696828) content/llcache.c llcache_persist 2426: Cannot write minimum bandwidth
> (22.697699) amiga/misc.c ami_misc_req 51: Disc cache write bandwidth is too slow to
be useful, disabling cache
well under 4000 bytes a second is so slow that it is not useful.
Agreed...
The
disc caching really needs to be several multiples of the network
conenction speed to be useful. Although the minimum bandwidth setting
is a passe din parameter and you can chnage the default value in
desktop/netsurf.c to experiment.
I'll have a look at that.
Is it possible that your implementation of a milisecond monotonic
counter in libnsutils is problematic? If it is returning microseconds
instead of miliseconds that would cause this erroneus behaviour.
I thought it might be wrong, so I rewrote it using a different timer.
I get very similar results to before.
When I return immediately, I'm still getting things like:
Wrote 94314 bytes in 1305ms
even though above it says:
Wrote 91944 bytes in 1ms
So there's something weird with the timer, or something slow is
happening between the timing interval.
I've just had a thought that some other task might be nicking the CPU
whilst the timing is happening - We're measuring how long it takes,
not how much CPU time it takes. Perhaps the monotonic timer should be
measuring CPU time for the task it's running under?
Chris
1:18 p.m.
New subject: netsurf: branch master updated. release/3.2-500-gd08acbc
On Sun, Nov 30, 2014 at 04:09:51PM +0000, Chris Young wrote:
On Sun, 30 Nov 2014 15:01:54 +0000, Vincent Sanders wrote:
> Is it possible that your implementation of a milisecond monotonic
> counter in libnsutils is problematic? If it is returning microseconds
> instead of miliseconds that would cause this erroneus behaviour.
I thought it might be wrong, so I rewrote it using a different timer.
I get very similar results to before.
You've verified that the timer returns sane answers, presumably? What
does this print out?:
#include <inttypes.h>
#include <stdio.h>
#include <time.h>
#include <unistd.h>
#include <nsutils/time.h>
int main(void)
{
int i;
uint64_t now_mt;
struct timeval tv;
for (i = 5; i > 0; --i) {
nsu_getmonotonic_ms(&now_mt);
gettimeofday(&tv, NULL);
printf("%ld.%ld : %lu\n", tv.tv_sec, tv.tv_usec, now_mt);
usleep(1000);
}
return 0;
}
When I return immediately, I'm still getting things like:
Wrote 94314 bytes in 1305ms
even though above it says:
Wrote 91944 bytes in 1ms
That variance is insanely large. I presume I/O performance is consistent
on Amiga OS?
I've just had a thought that some other task might be nicking the
CPU
whilst the timing is happening - We're measuring how long it takes,
not how much CPU time it takes. Perhaps the monotonic timer should be
measuring CPU time for the task it's running under?
I don't think so, no. If the system scheduler is preventing tasks
running for prolonged periods, then performance is going to suck,
regardless. Thus, disabling the cache under these conditions isn't the
world's worst idea.
J.
2:19 p.m.
New subject: netsurf: branch master updated. release/3.2-500-gd08acbc
On Sun, 30 Nov 2014 18:18:31 +0000, John-Mark Bell wrote:
> I thought it might be wrong, so I rewrote it using a different
timer.
> I get very similar results to before.
You've verified that the timer returns sane answers, presumably? What
does this print out?:
1417374580.535953 : 1304710
1417374580.537786 : 1304712
1417374580.539558 : 1304714
1417374580.541320 : 1304716
1417374580.543073 : 1304717
(after I changed the printf to %llu for now_mt)
> When I return immediately, I'm still getting things like:
> Wrote 94314 bytes in 1305ms
> even though above it says:
> Wrote 91944 bytes in 1ms
That variance is insanely large. I presume I/O performance is consistent
on Amiga OS?
Nothing to do with I/O, that's not writing anything to disk - just
spawning a process and returning, so it's basically doing nothing.
> I've just had a thought that some other task might be
nicking the CPU
> whilst the timing is happening - We're measuring how long it takes,
> not how much CPU time it takes. Perhaps the monotonic timer should be
> measuring CPU time for the task it's running under?
I don't think so, no. If the system scheduler is preventing tasks
running for prolonged periods, then performance is going to suck,
regardless. Thus, disabling the cache under these conditions isn't the
world's worst idea.
Yes, but the performance of the whole browser will suck so disabling
the cache is the wrong solution.
In this case the OS has seemingly decided that some other task is more
important whilst NetSurf is timing something critical, and
re-allocated the CPU for rather more milliseconds than the bandwidth
checker would like.
Using clock() seems to be a better solution and rather more portable.
Chris
2:52 p.m.
New subject: netsurf: branch master updated. release/3.2-500-gd08acbc
On Sun, Nov 30, 2014 at 07:19:55PM +0000, Chris Young wrote:
On Sun, 30 Nov 2014 18:18:31 +0000, John-Mark Bell wrote:
> > I thought it might be wrong, so I rewrote it using a different timer.
> > I get very similar results to before.
>
> You've verified that the timer returns sane answers, presumably? What
> does this print out?:
1417374580.535953 : 1304710
1417374580.537786 : 1304712
1417374580.539558 : 1304714
1417374580.541320 : 1304716
1417374580.543073 : 1304717
(after I changed the printf to %llu for now_mt)
OK, so modulo usleep(1000) sleeping for ~2ms, rather than 1, that looks
about right.
> > When I return immediately, I'm still getting things
like:
> > Wrote 94314 bytes in 1305ms
> > even though above it says:
> > Wrote 91944 bytes in 1ms
>
> That variance is insanely large. I presume I/O performance is consistent
> on Amiga OS?
Nothing to do with I/O, that's not writing anything to disk - just
spawning a process and returning, so it's basically doing nothing.
I must be missing something fundamental here, then. What is spawning a
process? (I'd also suggest that it's not doing nothing if, in some
circumstances, it's taking >1000 times longer to complete than in
others)
> I don't think so, no. If the system scheduler is preventing
tasks
> running for prolonged periods, then performance is going to suck,
> regardless. Thus, disabling the cache under these conditions isn't the
> world's worst idea.
Yes, but the performance of the whole browser will suck so disabling
the cache is the wrong solution.
Why? We care about throughput, which needs to be measured against some
real clock; if we don't take scheduler context switches into account,
then that will skew the results such that, as far as the bandwidth
calculator is concerned, everything is fine, whereas in reality, it's
not.
In this case the OS has seemingly decided that some other task is
more
important whilst NetSurf is timing something critical, and
re-allocated the CPU for rather more milliseconds than the bandwidth
checker would like.
We're talking about a difference of over 1000ms here, though. That
doesn't sound like normal behaviour for any scheduling algorithm.
Using clock() seems to be a better solution and rather more portable.
That then means we're not measuring the right thing, though.
J.
7:08 p.m.
New subject: netsurf: branch master updated. release/3.2-500-gd08acbc
On Sun, 30 Nov 2014 19:52:03 +0000, John-Mark Bell wrote:
> > That variance is insanely large. I presume I/O performance
is consistent
> > on Amiga OS?
>
> Nothing to do with I/O, that's not writing anything to disk - just
> spawning a process and returning, so it's basically doing nothing.
I must be missing something fundamental here, then. What is spawning a
process?
The store operation I've set to offload to another process. For the
purposes of testing this that function should then return almost
immediately. I get the "too slow" error whether I do that or not.
> > I don't think so, no. If the system scheduler is
preventing tasks
> > running for prolonged periods, then performance is going to suck,
> > regardless. Thus, disabling the cache under these conditions isn't the
> > world's worst idea.
>
> Yes, but the performance of the whole browser will suck so disabling
> the cache is the wrong solution.
Why? We care about throughput, which needs to be measured against some
real clock; if we don't take scheduler context switches into account,
then that will skew the results such that, as far as the bandwidth
calculator is concerned, everything is fine, whereas in reality, it's
not.
My point was that if this is affecting disk writing, it is also
affecting network access. If NetSurf for some reason is getting
randomly suspended during the caching code for an extended period, it
will also be getting randomly suspended during network access for an
extended period. Therefore the net gain for using the disk cache is
the same, even if it is sometimes taking a longer time than deemed
normal.
The disk cache throughput should be measured against network
throughput, not against some arbitrary ideal. I'd argue that it
shouldn't be disabling itself based on one bad result either, which is
what is happening here.
> In this case the OS has seemingly decided that some other task
is more
> important whilst NetSurf is timing something critical, and
> re-allocated the CPU for rather more milliseconds than the bandwidth
> checker would like.
We're talking about a difference of over 1000ms here, though. That
doesn't sound like normal behaviour for any scheduling algorithm.
People do tell me Exec's task scheduler sucks, so it's not
inconceivable that it has handed the CPU to something (or many things)
more important for a second at the exact point we're trying to
measure. There are lots of processes that run at a higher priority
than NetSurf, although mostly they don't hog the CPU.
Chris
10:45 a.m.
New subject: netsurf: branch master updated. release/3.2-500-gd08acbc
<snip other discussion which i have not addressed>
The disk cache throughput should be measured against network
throughput, not against some arbitrary ideal. I'd argue that it
shouldn't be disabling itself based on one bad result either, which is
what is happening here.
Eventually I will add the network fetch bandwidth measurement, however
as has been discovered it was hard enough to manage this in the
backing store code so I have not done this yet.
However I have changed the disabling test to be scheduled 100 time
quanta in the future when a single write goes slow and then check
against the overall write bandwidth so the low bandwidth writes must
occur over a substantial time period before we disable the cache now
instead of just a single event.
Hope that addresses the "lumpy" nature of writeouts causing spurious
backing store disables.
--
Regards Vincent
http://www.kyllikki.org/
2856
days inactive
2860
days old
netsurf-dev@netsurf-browser.org
7 comments
3 participants
participants (3)
-
Chris Young -
John-Mark Bell -
Vincent Sanders