Login
Username:

Password:

Remember me



Lost Password?

Register now!
Sections
Who's Online
55 user(s) are online (46 user(s) are browsing Forums)

Members: 0
Guests: 55

more...
Support us!
Recent OS4 Files
OS4Depot.net





ftell() speed : newlib & clib2
Home away from home
Joined:
2007/9/11 11:31
From Russia
Posts: 5526
Found lately , that when i use newlib's ftell() it is slow when used offten on files more than few MB.

Real world difference is that for example doing the same (open a 20mb .b3d file and do a lot of ftell() on it) take on my x5000 (with ssd/sata/etc) for about 40 seconds, while on icore5 it take for about 2-3 seconds.

I didn't write any test case at moment which can meassuer speed of ftell() , but what i do is made such a simple test case:

#include<stdio.h> 
  
int main() 

    
/* Opening file in read mode */
    
FILE *fp fopen("test.txt","r"); 
  
    
/* Reading first string */
    
char string[20];    
    
fscanf(fp,"%s",string); 
  
    
/* Printing position of file pointer */
    
printf("%ld"ftell(fp)); 
    return 
0
}


Now, create simple "test.txt" with something like "aa" inside, so test-case will not crash/fail.

And then, i build version for newlib, and for clib2, and take a snoopy log of it. And result are:

newlib:

00027 ftell_newlib    o.k. = Open("test.txt",OLD) = [0x1984643C] [22uS]
00028 ftell_newlib    o.k. = ExamineObject(0x6316E848 [test.txt]) [16uS]
00029 ftell_newlib    FAIL IsInteractive("test.txt")
00030 ftell_newlib    o.k. = IsFileSystem("") [4uS]
00031 ftell_newlib    o.k. = ExamineObject(0x6316E7A8 [test.txt]) [12uS]
00032 ftell_newlib    o.k. = GetFilePosition(0x1984643C [test.txt]) = [3uS]
00033 ftell_newlib    o.k. = GetFilePosition(0x1984643C [test.txt]) = [2uS]
00034 ftell_newlib    o.k. = ChangeFilePosition(0x1984643C [test.txt],2,OFFSET_BEGINNING) [3uS]
00035 ftell_newlib    o.k. = GetFilePosition(0x1984643C [test.txt]) = [1uS]
00036 ftell_newlib    FAIL ExamineObject(0x6316E538 [<null>]) [4617uS]
00037 ftell_newlib    :        SetCurrentDir("<untracked>")
00038 ftell_newlib    DIR  ParentOfFH(0x1984643C"test.txt" [5uS]
00039 ftell_newlib    o.k. = ExamineObject(0x6316EC78 [test.txt]) [12uS]
00040 ftell_newlib    : <----- RunCommand(0x18E89915 "ftell_newlib",,"",1) = [0x00000000] [21304uS]



clib2:

00027 ftell_clib2     o.k. = Open("test.txt",OLD) = [0x19ECFD14] [18uS]
00028 ftell_clib2     FAIL IsInteractive("test.txt")
00029 ftell_clib2     o.k. = IsFileSystem("") [3uS]
00030 ftell_clib2     DIR  ParentOfFH(0x19ECFD14"test.txt" [5uS]
00031 ftell_clib2     o.k. = ExamineFH("test.txt") [459uS]
00032 ftell_clib2     : <----- RunCommand(0x18E89915 "ftell_clib2",,"",1) = [0x00000000] [4949uS]


As can be seen, in newlib version there is 4 native call happens for that simple test case:

GetFileposition()
GetFileposition()
ChangeFileposition()
GetFileposition()


While for clib2, there is no such calls.

In both cases that simple test case works of course, and show size of text file in console , so ftell() definately works in clib2 too, just , it seems to use lot less functions ?

Now, when i use newlib's ftell in actuall app , i can see is log a gazilion of those "4 calls" repeated all the time, which, probably reassons for them being slow in result.

Is anyone can think about how to fix it in newlib ? Maybe some buffers should be disabled, or something..

Or maybe anyone can think of some simple test case in which we can measuer ftell() time for both clib2 and newlib ?

_________________
Join us to improve dopus5!
zerohero's mirror of os4/os3 crosscompiler suites
   Report Go to top

Re: ftell() speed : newlib & clib2
Home away from home
Joined:
2007/9/11 11:31
From Russia
Posts: 5526
As another test, in the code of game where all those ftell() used, i changed it so file firstly readed to the memory fully, and only then used by fteels and stuff, like:

#ifndef MEMORY
        
B3DFile f;
#else
       
long sz f->getSize();
       
c8buff = new c8[sz];
       
f->read(buffsz);
       
B3DFile = new io::CMemoryReadFile(buffszf->getFileName(), true);
       
f->seek(0);
#endif


And speed now changed RADICALLY. I mean, its just a few seconds instead of a minute.

That mean, that ftell() are VERY SLOW. I mean, very. Do not know through, if it only newlib's one, or its filesystem's issue.. That can be checked only by some test-case which can meassure speed of ftell().

Maybe some buffering should be disabled , which is enalbed by default and which make it be that slow ?

_________________
Join us to improve dopus5!
zerohero's mirror of os4/os3 crosscompiler suites
   Report Go to top

Re: ftell() speed : newlib & clib2
Just can't stay away
Joined:
2009/5/1 17:57
From Czech Republic
Posts: 1058
@kas1e

Quote:
That mean, that ftell() are VERY SLOW. I mean, very. Do not know through, if it only newlib's one, or its filesystem's issue.

You should definitely tell the DOS guys, i.e. Colin Wenzel and Tony Wyatt.

_________________
Smoke me a kipper, I'll be back for breakfast!

AmigaOne X5000 @ 2GHz / 4GB RAM / Radeon RX 560 / ESI Juli@ / AmigaOS 4.1 Final Edition
SAM440ep-flex @ 667MHz / 1GB RAM / Radeon 9250 / AmigaOS 4.1 Final Edition
   Report Go to top

Re: ftell() speed : newlib & clib2
Home away from home
Joined:
2007/9/11 11:31
From Russia
Posts: 5526
@Trixie
If only i have some simple test case showing the problem :) And after thinking a bit more, its probably can be all filemanagment operations guilty : fseek(), fread() and ftell().

_________________
Join us to improve dopus5!
zerohero's mirror of os4/os3 crosscompiler suites
   Report Go to top

Re: ftell() speed : newlib & clib2
Quite a regular
Joined:
2013/10/17 15:21
From Hungary
Posts: 600
What happens if you do a rewind(fp) (or fseek(fp, 0, SEEK_SET) if that's not available) before calling ftell? I know this defeats the purpose, but I wonder how many DOS calls you get at the beginning of the file.

_________________
I see the jimmies have been rustled.
   Report Go to top

Re: ftell() speed : newlib & clib2
Just can't stay away
Joined:
2006/11/30 11:30
From Finland
Posts: 1689
@kas1e

The first two GetFilePosition() and the ChangeFilePosition() calls are from newlib's ftell() implementation calling fflush() at the beginning, which flushes any buffered writes and ensures that the file position is the same as the position in the internal read/write buffer (in your code this leads to the ChangeFilePosition() call to move one byte backwards because one extra byte had been read into the read buffer).

Clib2 also caches the current file position which means that the GetFilePosition() calls can be avoided, however it also means that any code that may directly or indirectly change the file position needs to keep this cached value up to date and valid or it can lead to buggy behaviour.

FWIW I'm currently looking into updating newlib to cache the file position in a similar way, which as I said above would at least make the GetFilePosition() calls unnecessary.

Another difference between clib2 and newlib is that newlib uses the newer DOS API calls with large file support, which means that it will be faster with new style vector port file systems such as NGFS and ram-handler but slower with legacy packet based file systems such as FFS2 and to some extent SFS which use packet packet based I/O and in FFS2 case probably does not even support the 64-bit packets leading to the first packet failing with ERROR_ACTION_NOT_KNOWN and DOS having to use a legacy packet as fallback.

   Report Go to top

Re: ftell() speed : newlib & clib2
Home away from home
Joined:
2007/9/11 11:31
From Russia
Posts: 5526
@salas00
Quote:

FWIW I'm currently looking into updating newlib to cache the file position in a similar way, which as I said above would at least make the calls unnecessary.


What make me most curious, is HOW slow is it , i mean, few seconds when its in the memory, and about minute (!) when its as usuall from a file for 20mb sized file

But if get rid of GetFilePosition(), then it will mean things will be x3 times faster (through, still not few seconds). If , of course, take in account that issue with slow speed is amount of calls, and not something in the code of calls.

I can't say at moment how fast clib2 version in compare with newlib, because didn't wrote any "real meassure" test case, so i only can compare snoopy output.

From another side, i see that each calls take a very little microseconds, which can't be issue with speed. Maybe issue is amount of calls in general, not their speed when they executes. I.e. time which DOS spend between 2 calls.

Quote:

Another difference between clib2 and newlib is that newlib uses the newer DOS API calls with large file support, which means that it will be faster with new style vector port file systems such as NGFS and ram-handler but slower with legacy packet based file systems such as FFS2 and to some extent SFS which use packet packet based I/O and in FFS2 case probably does not even support the 64-bit packets leading to the first packet failing with ERROR_ACTION_NOT_KNOWN and DOS having to use a legacy packet as fallback.


Yeah, while it sounds good, it seems that at moment something wrong : i do all test on NGFS , and that is what show very-slow results.

For sake of tests, i create an SFS2 volume, transfer all the game there, and run from it -> the same slow behaviour. So i assume it can be not issue with filesystems but instead with newlib realisation of those ftell/fseek/fread/etc.

_________________
Join us to improve dopus5!
zerohero's mirror of os4/os3 crosscompiler suites
   Report Go to top

Re: ftell() speed : newlib & clib2
Home away from home
Joined:
2007/9/11 11:31
From Russia
Posts: 5526
@Salas00
Tested today's newlib: sadly no changes. The same slow, and i can see that when all this reading happens, hdd-led blinks, so it doing something, just slow by some reassons.

But i do check "ftell_newlib" test case, to see, what native calls is used now, and yeah no more of those calls, its now like this:

00027 ftell_newlib    o.k. = Open("test.txt",OLD) = [0x18DAFC06] [23uS]
00028 ftell_newlib    o.k. = ExamineObject(0x633C2848 [test.txt]) [1288uS]
00029 ftell_newlib    FAIL IsInteractive("test.txt")
00030 ftell_newlib    o.k. = IsFileSystem("") [4uS]
00031 ftell_newlib    o.k. = ExamineObject(0x633C27A8 [test.txt]) [1395uS]
00032 ftell_newlib    o.k. = ChangeFilePosition(0x18DAFC06 [test.txt],2,OFFSET_BEGINNING) [5uS]
00033 ftell_newlib    FAIL ExamineObject(0x633C2538 [<null>]) [17812uS]
00034 ftell_newlib    :        SetCurrentDir("<untracked>")
00035 ftell_newlib    DIR  ParentOfFH(0x18DAFC06"test.txt" [4uS]
00036 ftell_newlib    o.k. = ExamineObject(0x633C2C78 [test.txt]) [12uS]
00037 ftell_newlib    : <----- RunCommand(0x18FF7571 "ftell_newlib",,"",1) = [0x00000000] [41085uS]


But still things the same slow, and hdd always blinks , like it doing something with file, just all the time and doing so slow :(

When i run the same game from SFS2 partition, then holy-omiga, its VEEEERY SLOW. It take not just 100 seconds as with NGFS, but i wait 15 minutes when run from SFS2 and it even didn't finished up ! I just reboot as tired to wait. And while when i run it from NGFS, hdd led "blinks" , then when i run it from SFS2, hdd lef always red, like something very heavy happens.

Definately something wrong there, and i not sure if it newlib fault at all. Maybe it can be something about those new filesystem things maybe (i mean vector/dos-packet). Maybe DOS itself involved..

_________________
Join us to improve dopus5!
zerohero's mirror of os4/os3 crosscompiler suites
   Report Go to top

Re: ftell() speed : newlib & clib2
Just can't stay away
Joined:
2006/11/30 11:30
From Finland
Posts: 1689
@kas1e

Well the fflush() call does make using ftell() in any kind of file i/o loop a bad idea, as it throws away any buffered reads (so they have to be read again unnecessarily) and if you are writing it is also bad. A better idea right now would be to keep track of the current file position yourself by how many bytes you've read/written.

I checked newlib 3.1.0 sources for ftell()/ftello() and they don't use fflush() any more so I will see if maybe the changes can be back-ported easily into our newlib.

Of course updating newlib.library to newlib 3.1.0 would be even better but that will probably take a lot of work.

   Report Go to top

Re: ftell() speed : newlib & clib2
Home away from home
Joined:
2007/9/11 11:31
From Russia
Posts: 5526
@salas00
Do you think it is newlib's issue ? I mean, not DOS, not filsystem one , but fflush() ?

Strange why on NGFS it take 100 seconds, and on SFS2 the same i even didn't waiting finishing (but i wait 10-15 minutes).

ps. in my case i only use reading for sure.

_________________
Join us to improve dopus5!
zerohero's mirror of os4/os3 crosscompiler suites
   Report Go to top

Re: ftell() speed : newlib & clib2
Just can't stay away
Joined:
2006/11/30 11:30
From Finland
Posts: 1689
@kas1e

In your Snoopy log the ChangeFilePosition() call is only taking 3 microseconds (that's 0.000003 seconds) so I don't think NGFS is the culprit.

Using SFS\2 on my Sam460 it's taking about 18 microseconds and with ram-handler 7 microseconds.

   Report Go to top

Re: ftell() speed : newlib & clib2
Home away from home
Joined:
2007/9/11 11:31
From Russia
Posts: 5526
@Salas00
Tested new newlib : yeah ! backporting of new fteel/etc from newlib 3.1.0 did the trick, now all fast !

now with usage of original code it take 19 seconds to load
with usage of change-to-memory-buffer it take 17 seconds to load
from sfs2 the same 19 seconds.

In other words you did it, thanks a bunch !

Through for release i of course still will use change-to-memory-buffer way, because when newlib will hit public no one know , but its good that you deal with anyway. I assume that will help a lot for other unix ports which rely on ftell() when readin/writing data.


Edited by kas1e on 2019/9/30 6:46:54
_________________
Join us to improve dopus5!
zerohero's mirror of os4/os3 crosscompiler suites
   Report Go to top





[Advanced Search]


Powered by XOOPS 2.0 © 2001-2016 The XOOPS Project