RetroBSD

2.11BSD operating system for microcontrollers
It is currently Sun Sep 15, 2019 8:30 pm

All times are UTC




Post new topic Reply to topic  [ 22 posts ]  Go to page 1, 2  Next
Author Message
 Post subject: ls -l total
PostPosted: Wed Jan 13, 2016 8:32 pm 
Contributor
User avatar

Joined: Thu Nov 08, 2012 7:04 am
Posts: 2401
Location: Rapa Nui
Is the total value ok here?
Code:
# ls -l
total 24
-rw-r--r--  1 root  wheel  3007 Dec 30 04:50 rc4.c
-rw-r--r--  1 root  wheel  8334 Jan 11 13:41 rc4.s
-rw-r--r--  1 root  wheel  1039 Jan 11 13:58 t9.c
-rw-r--r--  1 root  wheel  1134 Jan 11 14:47 t9.s
# dd bs=1k count=1000 if=/dev/zero of=AAA
1000+0 records in
1000+0 records out
1024000 bytes transferred in 1 secs (1024000 bytes/sec)
# ls -l
total 1028
-rw-r--r--  1 root  wheel  1024000 Jan 11 14:49 AAA
-rw-r--r--  1 root  wheel     3007 Dec 30 04:50 rc4.c
-rw-r--r--  1 root  wheel     8334 Jan 11 13:41 rc4.s
-rw-r--r--  1 root  wheel     1039 Jan 11 13:58 t9.c
-rw-r--r--  1 root  wheel     1134 Jan 11 14:47 t9.s
#

_________________
Pukao Hats Cleaning Services Ltd.


Top
 Profile  
 
 Post subject: Re: ls -l total
PostPosted: Wed Jan 13, 2016 9:15 pm 
Committer
User avatar

Joined: Thu Oct 11, 2012 8:45 am
Posts: 1801
Location: Room 217, Floor 8, Arm 8, Wheel S7, Mars Base Alpha 3
Why do you suppose it might not be?

From the FreeBSD ls manual:
Code:
     The listing of a directory's contents is preceded by a labeled total num-
     ber of blocks used in the file system by the files which are listed as
     the directory's contents (which may or may not include . and .. and other
     files which start with a dot, depending on other options).

_________________
Why not visit my shop? http://majenko.co.uk/catalog
Universal IDE: http://uecide.org
"I was trying to find out if it was possible to only eat one Jaffa Cake. I had to abandon the experiment because I ran out of Jaffa Cakes".


Top
 Profile  
 
 Post subject: Re: ls -l total
PostPosted: Wed Jan 13, 2016 9:36 pm 
Contributor
User avatar

Joined: Thu Nov 08, 2012 7:04 am
Posts: 2401
Location: Rapa Nui
:?

_________________
Pukao Hats Cleaning Services Ltd.


Top
 Profile  
 
 Post subject: Re: ls -l total
PostPosted: Wed Jan 13, 2016 10:08 pm 
Committer
User avatar

Joined: Thu Oct 11, 2012 8:45 am
Posts: 1801
Location: Room 217, Floor 8, Arm 8, Wheel S7, Mars Base Alpha 3
1024000 / 1024 = 1000
3007 / 1024 = 2.9 (3)
8334 / 1024 = 8.1 (9)
1039 / 1024 = 1.01 (2)
1134 / 1024 = 1.1 (2)

1000 + 3 + 9 + 2 + 2 = 1016

1028 - 1016 = 12 blocks unaccounted for which are probably the inode blocks for the files.

Use "ls -ls" and see if the total blocks equals the sum of all the blocks. On FreeBSD it gives me:

Code:
total 1042
1024 -rw-r--r--  1 matt  matt  1024000 Jan 13 22:04 AAA
   4 -rw-r--r--  1 matt  matt     3007 Jan 13 22:04 rc4.c
  10 -rw-r--r--  1 matt  matt     8334 Jan 13 22:04 rc4.s
   2 -rw-r--r--  1 matt  matt     1039 Jan 13 22:05 t9.c
   2 -rw-r--r--  1 matt  matt     1134 Jan 13 22:05 t9.s

1024 + 4 + 10 + 2 + 2 = 1042.

_________________
Why not visit my shop? http://majenko.co.uk/catalog
Universal IDE: http://uecide.org
"I was trying to find out if it was possible to only eat one Jaffa Cake. I had to abandon the experiment because I ran out of Jaffa Cakes".


Top
 Profile  
 
 Post subject: Re: ls -l total
PostPosted: Thu Jan 14, 2016 5:09 am 
Committer
User avatar

Joined: Wed Oct 10, 2012 11:01 pm
Posts: 1081
Location: Sunnyvale, CA
In LiteBSD, the block size is 4k (not 1k as in RetroBSD).
So the total block count is:
1024000 / 4096 = 250
3007 / 4096 = 0.7 (1)
8334 / 4096 = 2.03 (3)
1039 / 4096 = 0.25 (1)
1134 / 4096 = 0.28 (1)
Plus one indirect block for AAA file (1)

250 + 1 + 3 + 1 + 1 + 1 = 257

"ls -ls" shows the total and per file block counts in kbytes, not blocks.
So we have 257*4 = 1028.
Code:
# ls -ls
total 1028
1004 -rw-r--r--  1 root  wheel  1024000 Jan  3 14:25 AAA
   4 -rw-r--r--  1 root  wheel     3007 Jan  3 14:26 rc4.c
  12 -rw-r--r--  1 root  wheel     8334 Jan  3 14:26 rc4.s
   4 -rw-r--r--  1 root  wheel     1039 Jan  3 14:26 t9.c
   4 -rw-r--r--  1 root  wheel     1134 Jan  3 14:27 t9.s


Top
 Profile  
 
 Post subject: Re: ls -l total
PostPosted: Thu Jan 14, 2016 7:25 am 
Contributor
User avatar

Joined: Thu Nov 08, 2012 7:04 am
Posts: 2401
Location: Rapa Nui
:shock:
Thanks!

_________________
Pukao Hats Cleaning Services Ltd.


Top
 Profile  
 
 Post subject: Re: ls -l total
PostPosted: Thu Jan 14, 2016 10:21 am 
Committer
User avatar

Joined: Thu Oct 11, 2012 8:45 am
Posts: 1801
Location: Room 217, Floor 8, Arm 8, Wheel S7, Mars Base Alpha 3
Aha, 4k is the magic number ;) So that is where the extra few blocks went to...

Sent from my SM-T555 using Tapatalk

_________________
Why not visit my shop? http://majenko.co.uk/catalog
Universal IDE: http://uecide.org
"I was trying to find out if it was possible to only eat one Jaffa Cake. I had to abandon the experiment because I ran out of Jaffa Cakes".


Top
 Profile  
 
 Post subject: Re: ls -l total
PostPosted: Thu Jan 14, 2016 9:40 pm 
Contributor

Joined: Mon Nov 12, 2012 1:34 pm
Posts: 1092
Hi All,

Not really related to the topic perhaps, but I have always wondered why more than one file can't be in one block? Many files are under 2K in size, so they 'could' share a block? Especially since most of the files stored will never be changed for the life of the OS.

And the studies I have seen show LOTS of wasted space in filesystems?

Not saying it makes sense, but I have always wondered.

Lots of fun :).

Wiz


Top
 Profile  
 
 Post subject: Re: ls -l total
PostPosted: Thu Jan 14, 2016 10:49 pm 
Committer
User avatar

Joined: Thu Oct 11, 2012 8:45 am
Posts: 1801
Location: Room 217, Floor 8, Arm 8, Wheel S7, Mars Base Alpha 3
And how would you manage that?

In the index for a file you have a number. That number says "This file starts at block X". How can you have two files in the index that both say "This file starts at block X" and have them be different files?

_________________
Why not visit my shop? http://majenko.co.uk/catalog
Universal IDE: http://uecide.org
"I was trying to find out if it was possible to only eat one Jaffa Cake. I had to abandon the experiment because I ran out of Jaffa Cakes".


Top
 Profile  
 
 Post subject: Re: ls -l total
PostPosted: Fri Jan 15, 2016 4:18 am 
Contributor

Joined: Mon Apr 29, 2013 1:56 am
Posts: 196
wiz wrote:
Not really related to the topic perhaps, but I have always wondered why more than one file can't be in one block? Many files are under 2K in size, so they 'could' share a block? Especially since most of the files stored will never be changed for the life of the OS.


Growing files, deleting files, recovering files, searching for a suitable hole and defragmenting blocks will all become more complex. On top of that, depending on the underlying media, you may increase the chances of losing data. E.g. something terrible happens to the block when you're accessing a file in it and voila another file in this block is damaged.


Top
 Profile  
 
 Post subject: Re: ls -l total
PostPosted: Fri Jan 15, 2016 5:28 am 
Committer
User avatar

Joined: Wed Oct 10, 2012 11:01 pm
Posts: 1081
Location: Sunnyvale, CA
Actually, the LiteBSD file system (UFS) does support sharing of a block between multiple files - so called fragments. Here is a quote from article "A Fast File System for UNIX", written by authors of 4.2BSD Marshall McKusick, William Joy, Samuel Leffler, and Robert Fabry:
Quote:
To be able to use large blocks without undue waste, small files must be stored in a more efficient way. The new file system accomplishes this goal by allowing the division of a single file system block into one or more fragments. The file system fragment size is specified at the time that the file system is created; each file system block can be broken optionally into 2, 4, or 8 fragments, each of which is addressable. The lower bound on the size of these fragments is constrained by the disk sector size, typically 512 bytes.

When the root filesystem is created by ufstool, this feature is disabled for simplicity and cpu efficiency. Disk space is not much critical these days.


Top
 Profile  
 
 Post subject: Re: ls -l total
PostPosted: Fri Jan 15, 2016 8:16 am 
Contributor
User avatar

Joined: Thu Nov 08, 2012 7:04 am
Posts: 2401
Location: Rapa Nui
What is the max size of a partition in LiteBSD then?

_________________
Pukao Hats Cleaning Services Ltd.


Top
 Profile  
 
 Post subject: Re: ls -l total
PostPosted: Fri Jan 15, 2016 8:56 am 
Committer
User avatar

Joined: Wed Oct 10, 2012 11:01 pm
Posts: 1081
Location: Sunnyvale, CA
Pito wrote:
What is the max size of a partition in LiteBSD then?
Frankly, I don't know.
I'm afraid it's only 4Gbytes, as file offsets are limited by 32 bits.

Update: This is wrong: in LiteBSD the file offset, or the type off_t, is defined as 64-bit value. Sorry for confusion.


Top
 Profile  
 
 Post subject: Re: ls -l total
PostPosted: Fri Jan 15, 2016 4:54 pm 
Contributor
User avatar

Joined: Thu Nov 08, 2012 7:04 am
Posts: 2401
Location: Rapa Nui
Hmm, I want to run win10 in vmware under LiteBSD, so not sure 4gb would be enough :)

_________________
Pukao Hats Cleaning Services Ltd.


Top
 Profile  
 
 Post subject: Re: ls -l total
PostPosted: Fri Jan 15, 2016 4:58 pm 
Committer
User avatar

Joined: Thu Oct 11, 2012 8:45 am
Posts: 1801
Location: Room 217, Floor 8, Arm 8, Wheel S7, Mars Base Alpha 3
Pito wrote:
Hmm, I want to run win10 in vmware under LiteBSD, so not sure 4gb would be enough :)

You want a brain transplant :P

Even Wiz makes more sense than you right now... ;)

_________________
Why not visit my shop? http://majenko.co.uk/catalog
Universal IDE: http://uecide.org
"I was trying to find out if it was possible to only eat one Jaffa Cake. I had to abandon the experiment because I ran out of Jaffa Cakes".


Top
 Profile  
 
 Post subject: Re: ls -l total
PostPosted: Fri Jan 15, 2016 6:45 pm 
Contributor

Joined: Mon Nov 12, 2012 1:34 pm
Posts: 1092
Hi Serge and all,

I am wondering how big the virtual user and kernel space is? I assume there is some sort of MMU or LiteBSD imposed limit?

Since we can easily get 64GB flash memories, I suppose we could think about how we might easily use that space? I would seem that this should not be too difficult at least for some sort of direct access scheme? i.e.- Making the unallocated space into one huge block addressable memory with some simple user interface [a syscall or two] would seem to do the trick for many applications.

Wiz


Top
 Profile  
 
 Post subject: Re: ls -l total
PostPosted: Fri Jan 15, 2016 8:19 pm 
Committer
User avatar

Joined: Wed Oct 10, 2012 11:01 pm
Posts: 1081
Location: Sunnyvale, CA
On 32-bit processors, the virtual address has a 32-bit width - as one might easily guess. :)
So we can directly address up to 4Gbytes.

It's not directly related to the max filesystem size though. Sorry, I've caused some confusion declaring the 4Gb limit caused by 32-bit file offsets - that's not true. FS size can be limited by one of three factors:
1. File offset, or off_t type, as defined by the kernel API. In LiteBSD/mips, it's a 64-bit value, so we can handle up to 2^64 byte media. It's a huge amount, essentially unlimited.
2. 32-bit block pointers, used in UFS1 filesystem, allow up to 2^32 blocks per FS. With 4kbyte blocks, we have a 2^44 byte limit, or 16Tbytes.
3. PC-compatible partition table uses 32-bit values for partition size and offset, measured in 512-byte sectors. The max size of partition is 2^41 bytes, or 2Tbytes.

Hence we have a practical limit of 2Tbytes per FS on SD card. I believe, it should be enough for tiny embedded systems like LiteBSD.


Top
 Profile  
 
 Post subject: Re: ls -l total
PostPosted: Sat Jan 16, 2016 12:12 am 
Contributor

Joined: Mon Nov 12, 2012 1:34 pm
Posts: 1092
Hi Serge,

Thanks.

This should make huge programs easily possible.

I have one data file that is now approaching 2GB [several years of logged soil conductivity data]. But it doesn't need to be all in memory at once. But I guess it could be.

I doubt that a single PIC32 can do much with that data set?

So maybe its just another thing to ponder :).

Lots of fun :)

Wiz


Top
 Profile  
 
 Post subject: Re: ls -l total
PostPosted: Fri Jan 22, 2016 6:29 am 
Committer
User avatar

Joined: Wed Oct 10, 2012 11:01 pm
Posts: 1081
Location: Sunnyvale, CA
Hi Wiz,

In case the data need not be present in memory all at once, LiteBSD allows to have file sized up to 16Tbytes.

If you need to map all the data directly in the address space of the program, you can have up to 2Gbytes (as user space is limited to 0...0x7fffffff range).

But note, that the resident set size of the user program is limited by the amount of available physical memory. Typically it's about 200kbytes on PIC32MZ under LiteBSD. Exceeding this size will slow down the program significantly, sometimes up to three orders of magnitude. Paging is a powerful mechanism, still it cannot help you to draw a camel through the eye of a needle. :)

--Serge


Top
 Profile  
 
 Post subject: Re: ls -l total
PostPosted: Fri Jan 22, 2016 2:07 pm 
Contributor

Joined: Mon Nov 12, 2012 1:34 pm
Posts: 1092
Hi Serge,

Points well taken.

So I guess one thing to consider for things like text editors. The commonly used core code should all be in one RAM loadable block.

The commands not used very often can be in 'high RAM'. Or maybe this doesn't really matter since paging is fine enough grained to make this a non-issue?

Can we have more than one program sharing the available physical RAM without paging if they are small enough?

I assume we can now easily edit very large files.

Nice :)

Lots of fun :). Lots of ponder :).

Wiz


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 22 posts ]  Go to page 1, 2  Next

All times are UTC


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
cron




Powered by phpBB® Forum Software © phpBB Group

BSD Daemon used with permission