Login
Username:

Password:

Remember me



Lost Password?

Register now!
Sections
Who's Online
71 user(s) are online (62 user(s) are browsing Forums)

Members: 0
Guests: 71

more...
Support us!
Recent OS4 Files
OS4Depot.net





Is newlib's realloc() have issues ? & optimisation discussion
Home away from home
Joined:
2007/9/11 11:31
From Russia
Posts: 5200
There is one function from original code which works fine on linux/win32, which aim to get the ogg file and decode it to the memory:


void OpenALSound::CreateSampleFromOGG(const IDataStreamStreambool Looping) {
  
int Length Stream.Size();
  
printf ("DEBUG AUDIO: Length = %dn"Length);
  
  
auto  pBuffer = new byte[Length];
  
OggVorbis_File vf;
  
ov_callbacks ov_call = {nullptrnullptrnullptrnullptr};
  
Stream.Read(LengthpBuffer);
  
  
ov_open_callbacks(nullptr, &vf, (const char *)pBufferLengthov_call);
  
int bitstream 0;
  
vorbis_info *info ov_info(&vf, -1);
  
long size 0;
  
ALenum format = (info->channels == 1) ? AL_FORMAT_MONO16 AL_FORMAT_STEREO16;
  
ALuint freq info->rate;
  
printf("DEBUG AUDIO: info->channels = %dn"info->channels);
  
printf("DEBUG AUDIO: info->rate = %dn"info->rate);
  
  
long rc 0;
  
long allocated 4*1024*16;
  
ALubyte *retval static_cast<ALubyte*>(malloc(allocated));
  
char *buff static_cast<char*>(malloc(allocated));

#ifdef USE_TREMOR
  
while ( (rc ov_read(&vfbuffallocated, &bitstream)) != 
#else
  #ifdef __amigaos4__
  
while ( (rc ov_read(&vfbuffallocated121, &bitstream)) != 
  
#else    
  
while ( (rc ov_read(&vfbuffallocated021, &bitstream)) != 
  
#endif
#endif
  
{
    
printf("AUDIO DEBUG : rc = %dn",rc);
    
printf("AUDIO DEBUG : size = %dn",rc);
    if (
rc <= 0) {
      break;
    }
    
size += rc;

    
ALubyte *tmp static_cast<ALubyte*>(realloc(retvalsize)); 
    if (
retval == nullptr)
      {
        
free(retval);
        
retval nullptr;
        break;
      }
    
retval tmp;
    
memmove(retval+size-rcbuffrc);
  }
  
alGenBuffers(1, &buffer);
  
alBufferData(bufferformatretvalsizefreq);
  
free(retval);
  
free(buff);
  
ov_clear(&vf);
}


When i compiled it for aos4 via gcc 8.2.0, with usage latest beta newlib 53.50 and with -O3 optimisation (don't tested on puclib ones, but probabaly that not related), i have got some weird issues, named that buffer epanding (reallocation) is never ends. It keep bring me "2048" for rc and size in a loop, till not eat all my memory (whole 2gb) and then just crashes.

Once i change "long allocated = 4*1024*16;" on "long allocated = 1024*1024*16;", so to avoid many reallocs by enlarge the buffer , it start to works ! It just eat as expected about 30-40mb of memory for my decompressed ogg file, and continue futher.

Question is : WTF? Is there something amigaos4 specific should be done about ? Is static_cast may cause issues there ?

Any help and ideas apprecated. Some shorted test-cases will be interesting to check as well, just dunno where to look at.

Thanks !


Edited by kas1e on 2019/1/16 15:01:41
_________________
Join us to improve dopus5!
zerohero's mirror of os4/os3 crosscompiler suites
   Report Go to top

Re: Is newlib's realloc() have issues ?
Just can't stay away
Joined:
2009/5/1 17:57
From Czech Republic
Posts: 1027
@kas1e

There was a thread on the Hyperion forum about Newlib's realloc() having some problem but I don't remember the details. I vaguely recollect Steven Solie mentioned that the bug occurs rarely and that fixing it is not high priority.

_________________
Smoke me a kipper, I'll be back for breakfast!

AmigaOne X5000 @ 2GHz / 4GB RAM / Radeon RX 560 / ESI Juli@ / AmigaOS 4.1 Final Edition
SAM440ep-flex @ 667MHz / 1GB RAM / Radeon 9250 / AmigaOS 4.1 Final Edition
   Report Go to top

Re: Is newlib's realloc() have issues ?
Home away from home
Joined:
2007/9/11 11:31
From Russia
Posts: 5200
@Trixie
Oh, i found it !

http://forum.hyperion-entertainment.c ... f=14&t=1591&hilit=realloc

Damn !

I even see you asking there what was in the 4458 , but there is indeed nothing important, only asking for necessary stuff to be implemented in kernel, before realloc() thing can be fixed.

Through i can see in that thread Salas00 bring some workarounds for it, so maybe i better reuse it instead and be done with it..

Through if it can be replaced like this, why not replace it like this in newlib itself ? Probabaly there will be other issues arise then of course..


Edited by kas1e on 2019/1/11 11:20:19
_________________
Join us to improve dopus5!
zerohero's mirror of os4/os3 crosscompiler suites
   Report Go to top

Re: Is newlib's realloc() have issues ?
Home away from home
Joined:
2007/9/11 11:31
From Russia
Posts: 5200
@Trixie
Well i was over optimistic, it seems that i can't simple made replacement for realloc() in the big projects , which indirectly or directly call malloc() (as newlib is shared library, and it all can cause crashes, as need to replace all other memory functions too).

In turn it mean that it too much hassle and better or get rid of realloc() at all in that code, or, keep 1024x1024x16, which mean it eat up for realloc() by itself about 100mb, which then returned right after reallocs loop done.

I even trying 2048x1024x16, and then it eat up just 30-40mb only.

In other words : don't use newlib's realloc() ever if possible ! It works, and memory after realloc() loop ends are freed, but in the process it take A LOT of memory, and can easyly fillup whole memory if you trying to realloc looots of small chunks.

_________________
Join us to improve dopus5!
zerohero's mirror of os4/os3 crosscompiler suites
   Report Go to top

Re: Is newlib's realloc() have issues ?
Home away from home
Joined:
2006/11/26 21:45
From a story that hasn't been written yet
Posts: 3356
@kas1e

Is this realloc() bug looked after/has an entry in the database or is this some kind of "will never get fixed, just don't use realloc() at all" situation?

I guess this may be the reason why a certain engine fails/crashes/freezes in ScummVM as the engines behaviour on how to function completely covers your description...

_________________
If slaughterhouses had glass walls, everyone would be a vegetarian. ~ Sir Paul McCartney
-
Did everything just taste purple for a second? ~ Philip J. Fry
-
Ain't got no cash, ain't got no style, ladies vomit when I smile. ~ Dr. John Zoidberg
   Report Go to top

Re: Is newlib's realloc() have issues ?
Home away from home
Joined:
2007/9/11 11:31
From Russia
Posts: 5200
@Raziel
As far as i understand, newlib's realloc() works, it just for its own needs it take much memory and in worse situations when there a looot of small chunks, it can lead to filling up of whole memory and a freeze as result.

You may try to run some memory monitor (i use zzd10h's dockie for RAM), and run scummwm over it. If before you crash you will see that all your memory is filled up - then it is. But if not, then it something else

_________________
Join us to improve dopus5!
zerohero's mirror of os4/os3 crosscompiler suites
   Report Go to top

Re: Is newlib's realloc() have issues ?
Just popping in
Joined:
2007/7/30 21:55
From Sweden
Posts: 186
I overrode malloc, free, and realloc in JamVM. Can’t remember the details, but I think I ended up using the memory pools (even though the code had Agee other methods in ifdef’s).

Hmm... looking at the code I seem to have changed the functions to call my memory functions, thought I’d made some defines for it.

Anyway, below is the code. Look for mymalloc, myfree, and myrealloc:
src/os/amiga/memory.c

And here is where I had to change it. JamVM already had its own memory allocation functions, so it didn’t call them directly:
src/alloc.c

_________________
Maintainer and developer for Jamiga2 - Java for Amiga
   Report Go to top

Re: Is newlib's realloc() have issues ?
Home away from home
Joined:
2007/5/19 13:23
From England
Posts: 3551
@kas1e
Why are you using -O3, seems like asking for trouble? I'd try using just -Os (or even no -O at all). And then add better optimisation at the end, after everything is working.

_________________
Author of the PortablE programming language.
I love using Amiga OS4.1
It is pitch black. You are likely to be eaten by a grue...
   Report Go to top

Re: Is newlib's realloc() have issues ?
Home away from home
Joined:
2007/9/11 11:31
From Russia
Posts: 5200
@ChrisH
With -O3 optimisation i tested already many big projects which prove that at least on recent gcc (8.2.0) all fine. Maybe there can be problems in some situations, but for my own needs i find none.

In end of all that realloc() is wellknown bug/feature not related to optimisation, which not fixed for long.

@Salass00
Do you think it possible to deal with realloc in newlib, so it will not eat lots of memory when realloc() do reallocation of many small chunks ? Or its better to completely avoid it ?

_________________
Join us to improve dopus5!
zerohero's mirror of os4/os3 crosscompiler suites
   Report Go to top

Re: Is newlib's realloc() have issues ?
Home away from home
Joined:
2006/12/4 23:15
Posts: 2080
@kas1e
Quote:

@ChrisH
With -O3 optimisation i tested already many big projects which prove that at least on recent gcc (8.2.0) all fine. Maybe there can be problems in some situations, but for my own needs i find none.


I think you are right here, using optimisation enables a quite afew imporant warnings that are not other detected, and the only time it causes problems is when trying to trace Grim Reapers, then you can disabled it for that case and rebuild.

Quote:

@Salass00
Do you think it possible to deal with realloc in newlib, so it will not eat lots of memory when realloc() do reallocation of many small chunks ? Or its better to completely avoid it ?


I would avoid it or use clib2 as advised else where .

_________________
Blender For OS4.x : Blues : Walker Broad
   Report Go to top

Re: Is newlib's realloc() have issues ?
Just popping in
Joined:
2015/6/11 8:51
From Cologne
Posts: 242
@ChrisH
Quote:
Why are you using -O3, seems like asking for trouble? I'd try using just -Os (or even no -O at all). And then add better optimisation at the end, after everything is working.

I second kas1e and broadblues. And I for myself do more or less the opposite of your approach:
I always use -O3 (or even more) optimizations and only lower or disable them if I run into trouble and need to debug in depth.
The reason is that very high compiler optimization modes usually don't cause any problems at all (at least if you're using more or less recent gcc) - if your code is truely clean and doesn't contain subtle bugs, which often won't show up when optimizations are disabled.
Really, in 95% of the cases where -O3 seemed to be the cause for problems it turned out that in reality it was a side-effect of a bug made by myself. So from my experience -O3 is a nice tool to uncover bugs

_________________
[Facebook] [YouTube Channel]
   Report Go to top

Re: Is newlib's realloc() have issues ? & optimisation discussion
Home away from home
Joined:
2007/9/11 11:31
From Russia
Posts: 5200
And right, there is some prove with which i meet today about way of doing optimisation (when no optimisation just hide real issues):

Porting one game, and it seems to works. Added -O3, and it start crashes on some `strcpy(s,&s[strlen(s)-4]);` string. While with no optimisation it works for sure (optimisation with -O3 probabaly not only replace it with memmove, but doing something else ?).

Then, i create simple test case which show the problem (test case with parts of code from the game):

#include <math.h>
#include <stdio.h>
#include <stdlib.h>
#include <ctype.h>
#include <dirent.h>
#include <string.h>

DIR d;
struct dirent dp;
char musicfile[500][255];


int strsound char s1[] )
{
    
int i 0;
    
char s[10];
    if(
strlen(s1) > 4) {
        
strcpy(s,&s[strlen(s)-4]);
        while (
s[i]) {
        
s[i] = toupper(s[i]);
        ++
i;
    }
        if(
strcmp(s,".MP3") || strcmp(s,".OGG")) {
        return(
1);
        }
    }
    return(
0);
}



int main()
{
        
int i;
        
        if((
opendir("music"))) {
            
0;
            while ((
dp readdir(d))!= NULL && <= 500) {
                if(
strsound(dp->d_name)) {
                    
strcpy(musicfile[i++],dp->d_name);
                    
fprintf(stderr,"DEBUG: musicfile = %sn"musicfile[i-1]);
                }
            }
        }
        return 
0;
    
}


To run it you need directory "music" and put there some .mp3 files (or just text files with .mp3 or .ogg names).

Then, if you compile it without optimisation, it didn't crash. Will works fine (visually). If you compile it with -O3, it will crashes. But not only that , -Wall will say you when optimisation enabled:

Quote:

In function ‘int strsound(char*)’,
inlined from ‘int strsound(char*)’:
test.c:18:9: warning: ‘char* strcpy(char*, const char*)’ accessing 1 byte at offsets 0 and [0, 5] may overlap 1 byte at offset 0 [-Wrestrict]
strcpy(s,&s[strlen(s)-4]);
~~~~~~^~~~~~~~~~~~~~~~~~~


So, anyone can see what the problem is with that code and why it fail with -O3 ? Replacing strcpy(s,&s[strlen(s)-4]); with just memmove(s, s+strlen(s)-4, 5); make it works even with -O3.

Anyone see error in original code ?

_________________
Join us to improve dopus5!
zerohero's mirror of os4/os3 crosscompiler suites
   Report Go to top

Re: Is newlib's realloc() have issues ? & optimisation discussion
Just popping in
Joined:
2015/6/11 8:51
From Cologne
Posts: 242
@kas1e
Just don't do it, it's illegal, results are undefined / depend on implementation if you use overlapping src / dst for strcpy, simple as that. Your favorite strcpy-man-page will tell you.
That is, if the code would be correct at all, actually it's completely bogus:

strcpy(s,&s[strlen(s)-4]);

At that point s is completely undefined, so the overlapping issue is only the top of the iceberg.
What the author meant is most likely

strcpy(s,&s1[strlen(s1)-4]);

Why your snippet produces anything working at all with -O0 is beyond me, to be honest, must be some weird funny stack-coincidence.

But well, yet another reason why to embrace -O3

_________________
[Facebook] [YouTube Channel]
   Report Go to top

Re: Is newlib's realloc() have issues ? & optimisation discussion
Home away from home
Joined:
2007/9/11 11:31
From Russia
Posts: 5200
@Daniel
Yeah, what mean that usage of -O3 is very good idea all the time.

I also found another bug because of -O3 in that trashy memory game, when no optimisation enabled, app can be opened and in window mode, and in fullscreen. When -O3 enabled, then again something trashes, and only fullscreen works, and mouse cursor distored. So it for sure 100% bugs, which caused not by -O3, but because of -O3 they didn't left unnoticed.

ps. and check the mail for some good news :)

_________________
Join us to improve dopus5!
zerohero's mirror of os4/os3 crosscompiler suites
   Report Go to top

Re: Is newlib's realloc() have issues ? & optimisation discussion
Just popping in
Joined:
2015/6/11 8:51
From Cologne
Posts: 242
@kas1e
Quote:
ps. and check the mail for some good news :)

Holy cow! Yeah, that day work was really worth it

_________________
[Facebook] [YouTube Channel]
   Report Go to top





[Advanced Search]


Powered by XOOPS 2.0 © 2001-2016 The XOOPS Project