Login
Username:

Password:

Remember me



Lost Password?

Register now!

Sections

Who's Online
100 user(s) are online (45 user(s) are browsing Forums)

Members: 0
Guests: 100

more...

Headlines

 
  Register To Post  

Is newlib's realloc() have issues ? & optimisation discussion
Home away from home
Home away from home


See User information
There is one function from original code which works fine on linux/win32, which aim to get the ogg file and decode it to the memory:


void OpenALSound::CreateSampleFromOGG(const IDataStreamStreambool Looping) {
  
int Length Stream.Size();
  
printf ("DEBUG AUDIO: Length = %d\n"Length);
  
  
auto  pBuffer = new byte[Length];
  
OggVorbis_File vf;
  
ov_callbacks ov_call = {nullptrnullptrnullptrnullptr};
  
Stream.Read(LengthpBuffer);
  
  
ov_open_callbacks(nullptr, &vf, (const char *)pBufferLengthov_call);
  
int bitstream 0;
  
vorbis_info *info ov_info(&vf, -1);
  
long size 0;
  
ALenum format = (info->channels == 1) ? AL_FORMAT_MONO16 AL_FORMAT_STEREO16;
  
ALuint freq info->rate;
  
printf("DEBUG AUDIO: info->channels = %d\n"info->channels);
  
printf("DEBUG AUDIO: info->rate = %d\n"info->rate);
  
  
long rc 0;
  
long allocated 4*1024*16;
  
ALubyte *retval static_cast<ALubyte*>(malloc(allocated));
  
char *buff static_cast<char*>(malloc(allocated));

#ifdef USE_TREMOR
  
while ( (rc ov_read(&vfbuffallocated, &bitstream)) != 
#else
  #ifdef __amigaos4__
  
while ( (rc ov_read(&vfbuffallocated121, &bitstream)) != 
  
#else    
  
while ( (rc ov_read(&vfbuffallocated021, &bitstream)) != 
  
#endif
#endif
  
{
    
printf("AUDIO DEBUG : rc = %d\n",rc);
    
printf("AUDIO DEBUG : size = %d\n",rc);
    if (
rc <= 0) {
      break;
    }
    
size += rc;

    
ALubyte *tmp static_cast<ALubyte*>(realloc(retvalsize)); 
    if (
retval == nullptr)
      {
        
free(retval);
        
retval nullptr;
        break;
      }
    
retval tmp;
    
memmove(retval+size-rcbuffrc);
  }
  
alGenBuffers(1, &buffer);
  
alBufferData(bufferformatretvalsizefreq);
  
free(retval);
  
free(buff);
  
ov_clear(&vf);
}


When i compiled it for aos4 via gcc 8.2.0, with usage latest beta newlib 53.50 and with -O3 optimisation (don't tested on puclib ones, but probabaly that not related), i have got some weird issues, named that buffer epanding (reallocation) is never ends. It keep bring me "2048" for rc and size in a loop, till not eat all my memory (whole 2gb) and then just crashes.

Once i change "long allocated = 4*1024*16;" on "long allocated = 1024*1024*16;", so to avoid many reallocs by enlarge the buffer , it start to works ! It just eat as expected about 30-40mb of memory for my decompressed ogg file, and continue futher.

Question is : WTF? Is there something amigaos4 specific should be done about ? Is static_cast may cause issues there ?

Any help and ideas apprecated. Some shorted test-cases will be interesting to check as well, just dunno where to look at.

Thanks !


Edited by kas1e on 2019/1/16 15:01:41
Join us to improve dopus5!
AmigaOS4 on youtube
Go to top
Re: Is newlib's realloc() have issues ?
Just can't stay away
Just can't stay away


See User information
@kas1e

There was a thread on the Hyperion forum about Newlib's realloc() having some problem but I don't remember the details. I vaguely recollect Steven Solie mentioned that the bug occurs rarely and that fixing it is not high priority.

The Rear Window blog

AmigaOne X5000 @ 2GHz / 4GB RAM / Radeon RX 560 / ESI Juli@ / AmigaOS 4.1 Final Edition
SAM440ep-flex @ 667MHz / 1GB RAM / Radeon 9250 / AmigaOS 4.1 Final Edition
Go to top
Re: Is newlib's realloc() have issues ?
Home away from home
Home away from home


See User information
@Trixie
Oh, i found it !

http://forum.hyperion-entertainment.c ... f=14&t=1591&hilit=realloc

Damn !

I even see you asking there what was in the 4458 , but there is indeed nothing important, only asking for necessary stuff to be implemented in kernel, before realloc() thing can be fixed.

Through i can see in that thread Salas00 bring some workarounds for it, so maybe i better reuse it instead and be done with it..

Through if it can be replaced like this, why not replace it like this in newlib itself ? Probabaly there will be other issues arise then of course..


Edited by kas1e on 2019/1/11 11:20:19
Join us to improve dopus5!
AmigaOS4 on youtube
Go to top
Re: Is newlib's realloc() have issues ?
Home away from home
Home away from home


See User information
@Trixie
Well i was over optimistic, it seems that i can't simple made replacement for realloc() in the big projects , which indirectly or directly call malloc() (as newlib is shared library, and it all can cause crashes, as need to replace all other memory functions too).

In turn it mean that it too much hassle and better or get rid of realloc() at all in that code, or, keep 1024x1024x16, which mean it eat up for realloc() by itself about 100mb, which then returned right after reallocs loop done.

I even trying 2048x1024x16, and then it eat up just 30-40mb only.

In other words : don't use newlib's realloc() ever if possible ! It works, and memory after realloc() loop ends are freed, but in the process it take A LOT of memory, and can easyly fillup whole memory if you trying to realloc looots of small chunks.

Join us to improve dopus5!
AmigaOS4 on youtube
Go to top
Re: Is newlib's realloc() have issues ?
Home away from home
Home away from home


See User information
@kas1e

Is this realloc() bug looked after/has an entry in the database or is this some kind of "will never get fixed, just don't use realloc() at all" situation?

I guess this may be the reason why a certain engine fails/crashes/freezes in ScummVM as the engines behaviour on how to function completely covers your description...

People are dying.
Entire ecosystems are collapsing.
We are in the beginning of a mass extinction.
And all you can talk about is money and fairytales of eternal economic growth.
How dare you!
– Greta Thunberg
Go to top
Re: Is newlib's realloc() have issues ?
Home away from home
Home away from home


See User information
@Raziel
As far as i understand, newlib's realloc() works, it just for its own needs it take much memory and in worse situations when there a looot of small chunks, it can lead to filling up of whole memory and a freeze as result.

You may try to run some memory monitor (i use zzd10h's dockie for RAM), and run scummwm over it. If before you crash you will see that all your memory is filled up - then it is. But if not, then it something else

Join us to improve dopus5!
AmigaOS4 on youtube
Go to top
Re: Is newlib's realloc() have issues ?
Just popping in
Just popping in


See User information
I overrode malloc, free, and realloc in JamVM. Can’t remember the details, but I think I ended up using the memory pools (even though the code had Agee other methods in ifdef’s).

Hmm... looking at the code I seem to have changed the functions to call my memory functions, thought I’d made some defines for it.

Anyway, below is the code. Look for mymalloc, myfree, and myrealloc:
src/os/amiga/memory.c

And here is where I had to change it. JamVM already had its own memory allocation functions, so it didn’t call them directly:
src/alloc.c

Maintainer and developer for Jamiga2 - Java for Amiga
Go to top
Re: Is newlib's realloc() have issues ?
Home away from home
Home away from home


See User information
@kas1e
Why are you using -O3, seems like asking for trouble? I'd try using just -Os (or even no -O at all). And then add better optimisation at the end, after everything is working.

Author of the PortablE programming language.
Go to top
Re: Is newlib's realloc() have issues ?
Home away from home
Home away from home


See User information
@ChrisH
With -O3 optimisation i tested already many big projects which prove that at least on recent gcc (8.2.0) all fine. Maybe there can be problems in some situations, but for my own needs i find none.

In end of all that realloc() is wellknown bug/feature not related to optimisation, which not fixed for long.

@Salass00
Do you think it possible to deal with realloc in newlib, so it will not eat lots of memory when realloc() do reallocation of many small chunks ? Or its better to completely avoid it ?

Join us to improve dopus5!
AmigaOS4 on youtube
Go to top
Re: Is newlib's realloc() have issues ?
Home away from home
Home away from home


See User information
@kas1e
Quote:

@ChrisH
With -O3 optimisation i tested already many big projects which prove that at least on recent gcc (8.2.0) all fine. Maybe there can be problems in some situations, but for my own needs i find none.


I think you are right here, using optimisation enables a quite afew imporant warnings that are not other detected, and the only time it causes problems is when trying to trace Grim Reapers, then you can disabled it for that case and rebuild.

Quote:

@Salass00
Do you think it possible to deal with realloc in newlib, so it will not eat lots of memory when realloc() do reallocation of many small chunks ? Or its better to completely avoid it ?


I would avoid it or use clib2 as advised else where .

Go to top
Re: Is newlib's realloc() have issues ?
Not too shy to talk
Not too shy to talk


See User information
@ChrisH
Quote:
Why are you using -O3, seems like asking for trouble? I'd try using just -Os (or even no -O at all). And then add better optimisation at the end, after everything is working.

I second kas1e and broadblues. And I for myself do more or less the opposite of your approach:
I always use -O3 (or even more) optimizations and only lower or disable them if I run into trouble and need to debug in depth.
The reason is that very high compiler optimization modes usually don't cause any problems at all (at least if you're using more or less recent gcc) - if your code is truely clean and doesn't contain subtle bugs, which often won't show up when optimizations are disabled.
Really, in 95% of the cases where -O3 seemed to be the cause for problems it turned out that in reality it was a side-effect of a bug made by myself. So from my experience -O3 is a nice tool to uncover bugs

Go to top
Re: Is newlib's realloc() have issues ? & optimisation discussion
Home away from home
Home away from home


See User information
And right, there is some prove with which i meet today about way of doing optimisation (when no optimisation just hide real issues):

Porting one game, and it seems to works. Added -O3, and it start crashes on some `strcpy(s,&s[strlen(s)-4]);` string. While with no optimisation it works for sure (optimisation with -O3 probabaly not only replace it with memmove, but doing something else ?).

Then, i create simple test case which show the problem (test case with parts of code from the game):

#include <math.h>
#include <stdio.h>
#include <stdlib.h>
#include <ctype.h>
#include <dirent.h>
#include <string.h>

DIR d;
struct dirent dp;
char musicfile[500][255];


int strsound char s1[] )
{
    
int i 0;
    
char s[10];
    if(
strlen(s1) > 4) {
        
strcpy(s,&s[strlen(s)-4]);
        while (
s[i]) {
        
s[i] = toupper(s[i]);
        ++
i;
    }
        if(
strcmp(s,".MP3") || strcmp(s,".OGG")) {
        return(
1);
        }
    }
    return(
0);
}



int main()
{
        
int i;
        
        if((
opendir("music"))) {
            
0;
            while ((
dp readdir(d))!= NULL && <= 500) {
                if(
strsound(dp->d_name)) {
                    
strcpy(musicfile[i++],dp->d_name);
                    
fprintf(stderr,"DEBUG: musicfile = %s\n"musicfile[i-1]);
                }
            }
        }
        return 
0;
    
}


To run it you need directory "music" and put there some .mp3 files (or just text files with .mp3 or .ogg names).

Then, if you compile it without optimisation, it didn't crash. Will works fine (visually). If you compile it with -O3, it will crashes. But not only that , -Wall will say you when optimisation enabled:

Quote:

In function ‘int strsound(char*)’,
inlined from ‘int strsound(char*)’:
test.c:18:9: warning: ‘char* strcpy(char*, const char*)’ accessing 1 byte at offsets 0 and [0, 5] may overlap 1 byte at offset 0 [-Wrestrict]
strcpy(s,&s[strlen(s)-4]);
~~~~~~^~~~~~~~~~~~~~~~~~~


So, anyone can see what the problem is with that code and why it fail with -O3 ? Replacing strcpy(s,&s[strlen(s)-4]); with just memmove(s, s+strlen(s)-4, 5); make it works even with -O3.

Anyone see error in original code ?

Join us to improve dopus5!
AmigaOS4 on youtube
Go to top
Re: Is newlib's realloc() have issues ? & optimisation discussion
Not too shy to talk
Not too shy to talk


See User information
@kas1e
Just don't do it, it's illegal, results are undefined / depend on implementation if you use overlapping src / dst for strcpy, simple as that. Your favorite strcpy-man-page will tell you.
That is, if the code would be correct at all, actually it's completely bogus:

strcpy(s,&s[strlen(s)-4]);

At that point s is completely undefined, so the overlapping issue is only the top of the iceberg.
What the author meant is most likely

strcpy(s,&s1[strlen(s1)-4]);

Why your snippet produces anything working at all with -O0 is beyond me, to be honest, must be some weird funny stack-coincidence.

But well, yet another reason why to embrace -O3

Go to top
Re: Is newlib's realloc() have issues ? & optimisation discussion
Home away from home
Home away from home


See User information
@Daniel
Yeah, what mean that usage of -O3 is very good idea all the time.

I also found another bug because of -O3 in that trashy memory game, when no optimisation enabled, app can be opened and in window mode, and in fullscreen. When -O3 enabled, then again something trashes, and only fullscreen works, and mouse cursor distored. So it for sure 100% bugs, which caused not by -O3, but because of -O3 they didn't left unnoticed.

ps. and check the mail for some good news :)

Join us to improve dopus5!
AmigaOS4 on youtube
Go to top
Re: Is newlib's realloc() have issues ? & optimisation discussion
Not too shy to talk
Not too shy to talk


See User information
@kas1e
Quote:
ps. and check the mail for some good news :)

Holy cow! Yeah, that day work was really worth it

Go to top

  Register To Post

 




Currently Active Users Viewing This Thread: 1 ( 0 members and 1 Anonymous Users )




Powered by XOOPS 2.0 © 2001-2023 The XOOPS Project