seek on big files

Sleeper · Post by **Sleeper** » Sun Dec 24, 2006 8:33 pm

Sorry for maybe stupid question..
Why do i get nil when use "seek" builtin function on big binary files? (up to 6 GB)

> (set 'f (open "storage.tmp" "r"))
3
> (seek f)
0
> (seek f -1)
nil
> (seek f 5893610690)
1598643394

I'm sure that position 5893610690 exists in that file.
Sometimes i get negative values in return...
NewLISP manual says:

Code: Select all

    (open "newlisp_manual.html" "read")
    (seek file -1)  ; seek to EOF
    → 593816     

    (set 'fle (open "large-file" "read") 
    (seek file 30000000000)  → 30000000000

newLISP supports file position numbers up to 9,223,372,036,854,775,807.

Lutz · Post by **Lutz** » Sun Dec 24, 2006 9:50 pm

I just retested on 5 and 6GB files on the Mac PPC with OS X and an X86 machine with FreeBSD, and things are fine on those platforms and using newLISP v.9.0.11.

I cannot test on Windows before tomorrow or the day after. But you can test yourself running:

c:\sleeper\> newlisp qa-lfs

you can find the test script qa-lfs in the source distribution at:
newlisp-x.x.x/qa-lfs

The script generates a 5GB file and then seeks to different positions verifying the contents. After the script completes you can do:

Code: Select all

> (open "largefile" "r") 
3
> (seek 3 -1)
5000000000

you can change the script so it goes to 6GB or more. With 5GB you are already in 64-Bit territory. The script works in 1K blocks and needs several minutes to complete. WinXP was last tested with newLISP 9.0 and was fine back then on a Celeron running XP home edition.

Lutz

Sleeper · Post by **Sleeper** » Sun Dec 24, 2006 10:37 pm

I get the same errors with newlisp 9.0.9 and 9.0.11 on WinXP:

Code: Select all

...
2140000 record -> Ok 02140000
2150000 record -> Error reading 02140001
...
4290000 record -> Error reading 02140215
4300000 record -> Error reading ########
...
4990000 record -> Error reading ########

Lutz · Post by **Lutz** » Sun Dec 24, 2006 10:46 pm

It seems to flip over to failure at the 32nd bit after 0x7fffffff, as if LFS is not working on WinXP, I am pretty sure I tested this, but will try again as soon as I get my hands on a Windows machine.

Lutz

Lutz · Post by **Lutz** » Sun Dec 24, 2006 10:49 pm

Perhaps your disks are formatted for FAT32 which would not support large files? Your disk has to be formatted with NTFS.

Lutz

Sleeper · Post by **Sleeper** » Sun Dec 24, 2006 11:05 pm

perhaps something is wrong with my system..
newlisp 9.0 - same results.
i have Intel Core 2 Duo and NTFS-formatted disk

Lutz · Post by **Lutz** » Mon Dec 25, 2006 10:13 pm

The large file support on Win32 never worked, sorry for the false information earlier, but is fixed now in 9.0.12 and posted in the development directory.

Lutz

ps: still I think you need a NTFS formatted disk drive and it will not work on FAT32

Lutz · Post by **Lutz** » Mon Dec 25, 2006 10:34 pm

Note that the md5-checksums.txt file in the Win32 installer is messed up but will be fine in the next version.

Lutz

HPW · Post by **HPW** » Mon Dec 25, 2006 10:41 pm

Maybe the file should be named like the version for example:

md5-checksums_9_0_12.txt

Lutz · Post by **Lutz** » Mon Dec 25, 2006 10:52 pm

I reposted the installer and source package, the checksums are fine now.

There will also be a checksum for the entire installer. That ensures, that when installing from the installer the checksums inside are authentic.

Once the files are expanded on the machine, security is in the hands of the machine owner, as even the md5-checksums.txt file could be replaced/renamed together with false .exe and .dll files.

Lutz

Sleeper · Post by **Sleeper** » Tue Dec 26, 2006 11:00 am

Thanx Lutz!
Now all works fine.