Login
Username:

Password:

Remember me



Lost Password?

Register now!

Sections

Who's Online
111 user(s) are online (66 user(s) are browsing Forums)

Members: 0
Guests: 111

more...

Headlines

 
  Register To Post  

AmigaOS 4 port of zstd V1.4.4
Just can't stay away
Just can't stay away


See User information
From www.zstd.net:
Quote:

Zstandard is a real-time compression algorithm, providing high compression ratios. It offers a very wide range of compression / speed trade-off, while being backed by a very fast decoder (see benchmarks below). It also offers a special mode for small data, called dictionary compression, and can create dictionaries from any sample set. Zstandard library is provided as open source software using a BSD license.


So far I've compiled the libzstd static library and the zstd command-line program for AmigaOS 4.

If anyone wants to help with testing, the current zstd binary can be downloaded from:

https://www.dropbox.com/s/gxpjdsp97bvs ... td-1.4.4-20200307.7z?dl=0

There is no large file support as yet, as I haven't made the necessary changes the file I/O code.

My intention is to turn the static library into an AmigaOS shared library, as I have done with for example zlib, libzip, libpng and pixman.

Go to top
Re: AmigaOS 4 port of zstd V1.4.4
Just can't stay away
Just can't stay away


See User information
@salass00

Hi, when runnign witour args from commandline, it shows an assignwedfe requester for
"stdin*\:" and "stdout*\:" .

"zstd --help" shows usage command/options without problem.

And as it have somekind of benchmark options:

####zstd -b
3#Synthetic 50% : 10000000 -> 3231410 (3.095), 1.78 MB/s , 49.5 MB/s
#


Edited by jabirulo on 2020/3/7 10:43:28
Go to top
Re: AmigaOS 4 port of zstd V1.4.4
Just can't stay away
Just can't stay away


See User information
@jabirulo

Quote:

Hi, when runnign witour args from commandline, it shows an assignwedfe requester for
"stdin*\:" and "stdout*\:" .


That's because of how dumb and lazy the code is for handling stdin input and stdout output. AFAICT the code looks for files called "/*stdin*" and "/*stdout*" on Linux/Windows/MacOS as well.

Just specify explicit input and output files and it should work correctly.

To get a list of the options available use "zstd -h".

To compress the file "testfile" to "testfile.zstd" you use:
zstd testfile -o testfile.zstd

To decompress it back you use:
zstd -d testfile.zstd -o testfile

I tested with compressing and decompressing a ~665MB AVI file on my Sam460 and the MD5 sums of the original and the decompressed file did match.

Go to top
Re: AmigaOS 4 port of zstd V1.4.4
Just can't stay away
Just can't stay away


See User information
@salass00
tested with a plain textfile and it goes close to 7za compressor, very nice new (de)compressor!!!

#7za a test HD6750_slowVGAInit.txt

7-Zip (a) 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=C,Utf16=off,HugeFiles=off,32 bits,1 CPU BE)

Scanning the drive:
1 file, 133052 bytes (130 KiB)

Creating archive: test.7z

Items to compress: 1


Files read from disk: 1
Archive size: 15246 bytes (15 KiB)
Everything is Ok


#zstd HD6750_slowVGAInit.txt -o test.zsdf
HD6750_slowVGAInit.txt : 13.34% (133052 => 17753 bytes, test.zsdf)
#


EDIT1: another test, this time with an ISO image:
#zstd KDH8:Sam460InstallCD-53.58.iso -o test.zstd
KDH8:Sam460InstallCD-53.58.iso : 57.02% (564692992 => 321987237 bytes, test.zstd)
#set
Process 6
RC 0
Result2 0
_LastCommand zstd
_LastCommandArgs KDH8:Sam460InstallCD-53.58.iso -o test.zstd
_RunTime 229.610135
..
#l
LICENSE 1530 ----rwed 04-Nov-19 17:54:32
test.zstd 321987237 ----rw-d 23-Nov-14 22:07:54
zstd 758064 ----rwed Hoy 07:38:20
#

Go to top
Re: AmigaOS 4 port of zstd V1.4.4
Home away from home
Home away from home


See User information
@salass00

Nice! I've considered using zstandard to compress the firmware files that get embedded into the graphics drivers (the total size of which keeps growing). How easy was it to get the library to build?

Hans

http://hdrlab.org.nz/ - Amiga OS 4 projects, programming articles and more.
https://keasigmadelta.com/ - more of my work
Go to top
Re: AmigaOS 4 port of zstd V1.4.4
Just can't stay away
Just can't stay away


See User information
@Hans

Quote:

How easy was it to get the library to build?


Quite easy. The library source code is very portable so I didn't have to make any changes there.

I did write my own makefile because the included one was a bit messy and I like to have more control over what is done, but this didn't take too much time either.

The main issue I encountered was that my initial test with zstd CLI program generated a file after decompression that was different from the original one (much of the data had been replaced with 0xFEFECAFE).

My initial thought was that zstd doesn't correctly clear memory before using it, but after more testing I found the problem was with zstd's sparse mode and a bug in newlib.library that causes files to be extended with uninitialized data rather than zero bytes. I've fixed this by disabling the sparse mode if newlib.library < V53.59 is used.

Go to top
Re: AmigaOS 4 port of zstd V1.4.4
Home away from home
Home away from home


See User information
@salass00

Neat.

Did that newlib bug already made it into the bug tracker?

People are dying.
Entire ecosystems are collapsing.
We are in the beginning of a mass extinction.
And all you can talk about is money and fairytales of eternal economic growth.
How dare you!
– Greta Thunberg
Go to top
Re: AmigaOS 4 port of zstd V1.4.4
Just can't stay away
Just can't stay away


See User information
@Raziel

It is already fixed in V53.59. It was such an obvious programming error that I'm surprised I hadn't noticed it while going over the code before.

Go to top
Re: AmigaOS 4 port of zstd V1.4.4
Just can't stay away
Just can't stay away


See User information
The zstd.library now compiles but is as yet untested.

I was going to use the zstd CLI program for the first test but it uses some experimental functions which are only available with static linking so it's not suitable.

As libzstd uses a few global variables for debug levels and such I'm compiling it using the small data model and creating a separate copy of the data segment for each library opener. If this approach works well enough here, I might try to use it for other shared library adaptations in the future (if used in AmiSSL it would get rid of the baserel dependency which is one of the reasons it can't be compiled using a gcc newer than 4.0.4 right now).

Also while compiling the library using small data model I noticed that the .sdata section was much bigger than the eight bytes that I expected it would be from libzstd. This, it turned out, was because gcc was putting constant global data in the .sdata section instead of .rodata as usual. I've fixed this by adding the -mno-readonly-in-sdata option.

Go to top

  Register To Post

 




Currently Active Users Viewing This Thread: 1 ( 0 members and 1 Anonymous Users )




Powered by XOOPS 2.0 © 2001-2023 The XOOPS Project