RetroBSD

2.11BSD operating system for microcontrollers
It is currently Thu Dec 13, 2018 6:36 am

All times are UTC




Post new topic Reply to topic  [ 12 posts ] 
Author Message
PostPosted: Tue May 12, 2015 7:34 pm 

Joined: Tue May 12, 2015 7:20 pm
Posts: 5
Hi,
a few weeks ago I was looking at smaller C as I am currently hacking my own back end on to a C compiler. By a happy coincidence, this has led me to RetroBSD, which I'm very interested in (it's an amazing project, I'm somewhat astounded :D).

I've had a look at the available boards and the fubarino looks awesome. I'd also (of course) like a RAM disk, and what would really be nice is a piggybacked one that just drops on top/underneath the fubarino (with the same footprint). Did anything ever come of Pito's "Furamdisk"?

I have some other general questions about RetroBSD that I was hoping might get easy answers:

1. How much RAM is left free once the system has booted (for applications)?
2. What is the maximum size of applications (i.e. can they be larger than the free RAM size, and spill over into swap)?
3. How much like a recent BSD/Unix is RetroBSD? Can I expect stuff that just uses libc and system calls to basically work? Or have system calls changed a lot since then?
4. With swap, shouldn't TCP/IP be possible?
5. What about those ethernet/wifi modules that have built in TCP/IP stacks?

Thanks a lot, looking forward to having a play with one of these! :D


Top
 Profile  
 
PostPosted: Tue May 12, 2015 11:50 pm 
Committer
User avatar

Joined: Thu Oct 11, 2012 8:45 am
Posts: 1801
Location: Room 217, Floor 8, Arm 8, Wheel S7, Mars Base Alpha 3
Quote:
1. How much RAM is left free once the system has booted (for applications)?

IIRC 96KB is available as "user" space, into which has to fit the currently running program and all its data.
Quote:
2. What is the maximum size of applications (i.e. can they be larger than the free RAM size, and spill over into swap)?

Including all its data, 96KB.
Quote:
3. How much like a recent BSD/Unix is RetroBSD? Can I expect stuff that just uses libc and system calls to basically work? Or have system calls changed a lot since then?

Quite different. The basis is the same, but the way it works as regards memory management is quite different.
Quote:
4. With swap, shouldn't TCP/IP be possible?

No. Because of the lack of an MMU there is no virtual memory. On RetroBSD you have one process loaded into memory at a time, and it only has the user memory to play with. When the swapper switches tasks it removes the running process from memory and stores it in swap, loading a new task out of swap. That is the sole use of swap on RetroBSD - to store the tasks that aren't the current running one.
Quote:
5. What about those ethernet/wifi modules that have built in TCP/IP stacks?

Absolutely, yes, and we have a user-land library for the Wiznet W5100 and a few small applications that link against it.

_________________
Why not visit my shop? http://majenko.co.uk/catalog
Universal IDE: http://uecide.org
"I was trying to find out if it was possible to only eat one Jaffa Cake. I had to abandon the experiment because I ran out of Jaffa Cakes".


Top
 Profile  
 
PostPosted: Wed May 13, 2015 6:23 am 

Joined: Tue May 12, 2015 7:20 pm
Posts: 5
Thanks for your fast response!

majenko wrote:
Quote:
1. How much RAM is left free once the system has booted (for applications)?

IIRC 96KB is available as "user" space, into which has to fit the currently running program and all its data.
Quote:
2. What is the maximum size of applications (i.e. can they be larger than the free RAM size, and spill over into swap)?

Including all its data, 96KB.


This makes sense. Thinking about it now I don't know how I thought the PIC was going to address memory in a RAM disk connected via parallel IO. So do programs need to use hard coded addresses (actually in their source)?

majenko wrote:
Quote:
3. How much like a recent BSD/Unix is RetroBSD? Can I expect stuff that just uses libc and system calls to basically work? Or have system calls changed a lot since then?

Quite different. The basis is the same, but the way it works as regards memory management is quite different.


I guess you don't have malloc and friends (or perhaps a very simple implementation of them)? Presumably if you need more RAM you can use a large RAM disk to access a file in chunks? I suppose you could implement some sort of library to do "bank switching" from a RAM disk. Perhaps if you have pointers larger than your address space (not sure if this is true) you could even use the top few spare bits to indicate which bank to use? It would be interesting if you could modify the build system (cc, ld) to support this without using a library.

majenko wrote:
Quote:
4. With swap, shouldn't TCP/IP be possible?

No. Because of the lack of an MMU there is no virtual memory. On RetroBSD you have one process loaded into memory at a time, and it only has the user memory to play with. When the swapper switches tasks it removes the running process from memory and stores it in swap, loading a new task out of swap. That is the sole use of swap on RetroBSD - to store the tasks that aren't the current running one.


Yeah, I see this follows from the other answers now.

majenko wrote:
Quote:
5. What about those ethernet/wifi modules that have built in TCP/IP stacks?

Absolutely, yes, and we have a user-land library for the Wiznet W5100 and a few small applications that link against it.


This is cool! The wiznet W5100 is quite large, I wonder if the newer modules that do this would be easy to add (also, ones with wifi).


Top
 Profile  
 
PostPosted: Wed May 13, 2015 7:17 am 
Contributor

Joined: Mon Apr 29, 2013 1:56 am
Posts: 196
munchausen wrote:
a few weeks ago I was looking at smaller C as I am currently hacking my own back end on to a C compiler.


I'm curious, what architecture (CPU) is that for? And what compiler you're modifying/extending?


Top
 Profile  
 
PostPosted: Wed May 13, 2015 7:27 am 
Committer
User avatar

Joined: Wed Oct 10, 2012 11:01 pm
Posts: 1079
Location: Sunnyvale, CA
munchausen wrote:
So do programs need to use hard coded addresses (actually in their source)?
Yes, as we don't have MMU, the user programs are linked to static address range 7f008000-7f01ffff.

munchausen wrote:
I guess you don't have malloc and friends (or perhaps a very simple implementation of them)?
We have malloc() and the rest of libc, inherited from 2.11bsd - much like as it worked on pdp11 in early 80s.

munchausen wrote:
Presumably if you need more RAM you can use a large RAM disk to access a file in chunks? I suppose you could implement some sort of library to do "bank switching" from a RAM disk. Perhaps if you have pointers larger than your address space (not sure if this is true) you could even use the top few spare bits to indicate which bank to use? It would be interesting if you could modify the build system (cc, ld) to support this without using a library.
This sounds much like overlay approach. It's possible at user space level, and we already have a library of this kind - see libicache: https://github.com/RetroBSD/retrobsd/tree/master/src/libicache

munchausen wrote:
majenko wrote:
munchausen wrote:
4. With swap, shouldn't TCP/IP be possible?
No.
Yeah, I see this follows from the other answers now.
For networked applications, we have another project named LiteBSD, based on 4.4bsd. It requires pic32mz though.

munchausen wrote:
This is cool! The wiznet W5100 is quite large, I wonder if the newer modules that do this would be easy to add (also, ones with wifi).
Other network modules with embedded TCP/IP should be easy to add. In case of SPI or UART interface it's all at user space, nothing special is required from the kermel.


Top
 Profile  
 
PostPosted: Wed May 13, 2015 8:37 am 

Joined: Tue May 12, 2015 7:20 pm
Posts: 5
Thanks again for the replies. I'm going to check out libicache. Could you point me at some of the example programs that use it?

alexfru wrote:
munchausen wrote:
a few weeks ago I was looking at smaller C as I am currently hacking my own back end on to a C compiler.


I'm curious, what architecture (CPU) is that for? And what compiler you're modifying/extending?


I've created my own architecture called "minx", standing for min x86. I've developed a type-directed decompiler and am currently going about proving its correctness... the focus is actually on recovering high level types (structs, arrays) from binary code, the decompiler sort of falls out as a side effect. However, because I don't want to get concerned with control flow recovery, argument recovery, and other "side issues" I'm using an abstraction of x86 that has all these problems solved already (and has a reduced instruction set): it has no stack, an infinite number of registers, and function definitions where arguments are explicit about which registers they use for returning values and reading parameters. In order to actually do real tests on the decompiler I need to be able to build real C programs, so that's why I'm writing a backend for a completely fictional machine!

I wanted to use smaller C, as I was basically looking for the easiest compiler to modify that actually really works (there are a lot of half finished C compilers out there), and the size of smaller C appealed to me as indicating that it might be easy to modify. While I'm impressed with smaller C, I confess that I found the code really hard to follow, and after looking at it for a while I still didn't have a good handle on how it was working. I think this may be for efficiency reasons as much as anything though (like I noticed you don't really use structs, which sort of flattens everything but is hard to follow). I've gone for ucc instead (https://github.com/esrever10/ucc and http://ucc.sourceforge.net/) which is designed for teaching and readability, and has a very clear layout and easy to read code. I have wondered how ucc might fare on retrobsd compared to smaller C, as it has a tiny footprint (not sure if its small enough though) but does optimise, which I don't think smaller C does really?


Top
 Profile  
 
PostPosted: Wed May 13, 2015 9:21 am 
Committer
User avatar

Joined: Thu Oct 11, 2012 8:45 am
Posts: 1801
Location: Room 217, Floor 8, Arm 8, Wheel S7, Mars Base Alpha 3
munchausen wrote:
Thanks for your fast response!
majenko wrote:
Quote:
3. How much like a recent BSD/Unix is RetroBSD? Can I expect stuff that just uses libc and system calls to basically work? Or have system calls changed a lot since then?

Quite different. The basis is the same, but the way it works as regards memory management is quite different.


I guess you don't have malloc and friends (or perhaps a very simple implementation of them)? Presumably if you need more RAM you can use a large RAM disk to access a file in chunks? I suppose you could implement some sort of library to do "bank switching" from a RAM disk. Perhaps if you have pointers larger than your address space (not sure if this is true) you could even use the top few spare bits to indicate which bank to use? It would be interesting if you could modify the build system (cc, ld) to support this without using a library.

Yes of course we have malloc(), that is about dynamic memory, not virtual memory.

I think the function family you are thinking of is mmap() and friends, and no, we don't have those.

_________________
Why not visit my shop? http://majenko.co.uk/catalog
Universal IDE: http://uecide.org
"I was trying to find out if it was possible to only eat one Jaffa Cake. I had to abandon the experiment because I ran out of Jaffa Cakes".


Top
 Profile  
 
PostPosted: Wed May 13, 2015 11:26 am 

Joined: Tue May 12, 2015 7:20 pm
Posts: 5
majenko wrote:
Yes of course we have malloc(), that is about dynamic memory, not virtual memory.

I think the function family you are thinking of is mmap() and friends, and no, we don't have those.


I guess it's simpler without paging or memory protection or other vm stuff as you can basically just give back the next large enough free address (i.e. doesn't hit the stack or other user data and doesn't have the kernel in it) and make a note of the size.


Top
 Profile  
 
PostPosted: Wed May 13, 2015 12:49 pm 
Contributor
User avatar

Joined: Thu Nov 08, 2012 7:04 am
Posts: 2401
Location: Rapa Nui
Quote:
Did anything ever come of Pito's "Furamdisk"?

There were several attempts to create a ramdisk, for swap and as a disk partition as well.
You may search for really many topics in this regard at this forum.
There was a small batch of ramdisks produced (ie w/ 4MB sram, 8MB psram, 16MB sdram, 2MB mram), I do prefer the psram cpld based 8MB one, as it requires only 11 signals..
If you want to see the ramdisk in real app:
viewtopic.php?f=3&t=3222&start=40#p41094
viewtopic.php?f=3&t=20597
Arduino uno/due - more details on implementation at http://forum.arduino.cc/index.php?topic=220918.0

_________________
Pukao Hats Cleaning Services Ltd.


Top
 Profile  
 
PostPosted: Wed May 13, 2015 4:33 pm 

Joined: Tue May 12, 2015 7:20 pm
Posts: 5
Pito wrote:
Quote:
Did anything ever come of Pito's "Furamdisk"?

There were several attempts to create a ramdisk, for swap and as a disk partition as well.
You may search for really many topics in this regard at this forum.
There was a small batch of ramdisks produced (ie w/ 4MB sram, 8MB psram, 16MB sdram, 2MB mram), I do prefer the psram cpld based 8MB one, as it requires only 11 signals..
If you want to see the ramdisk in real app:
viewtopic.php?f=3&t=3222&start=40#p41094
viewtopic.php?f=3&t=20597
Arduino uno/due - more details on implementation at http://forum.arduino.cc/index.php?topic=220918.0


Hi Pito, thanks for the reply. I was wondering specifically about the "furamdisk" shown here: viewtopic.php?f=3&t=4442#p4442

This one is really nice because it has the same board size as the fubarino, so I could have a little retrobsd USB stick with a ramdisk (with a little adapter like this http://www.ebay.co.uk/itm/301190480904). I guess you didn't make any though? Is your CPLD firmware freely available? Then I could make one myself :)


Top
 Profile  
 
PostPosted: Wed May 13, 2015 8:29 pm 
Contributor
User avatar

Joined: Thu Nov 08, 2012 7:04 am
Posts: 2401
Location: Rapa Nui
Nope, nope.

_________________
Pukao Hats Cleaning Services Ltd.


Top
 Profile  
 
PostPosted: Thu May 14, 2015 9:46 am 
Contributor

Joined: Mon Apr 29, 2013 1:56 am
Posts: 196
munchausen wrote:
Thanks again for the replies. I'm going to check out libicache. Could you point me at some of the example programs that use it?


The latest code is at https://github.com/alexfru/icacheMips. Serge included the code in RetroBSD and I have since fixed a bug (in SLTIU) but the fix only exists on my github. I think nobody's using icache. As for documentation, there's an old thread on it.

munchausen wrote:
I've created my own architecture called "minx", standing for min x86. I've developed a type-directed decompiler and am currently going about proving its correctness... the focus is actually on recovering high level types (structs, arrays) from binary code, the decompiler sort of falls out as a side effect. However, because I don't want to get concerned with control flow recovery, argument recovery, and other "side issues" I'm using an abstraction of x86 that has all these problems solved already (and has a reduced instruction set): it has no stack, an infinite number of registers, and function definitions where arguments are explicit about which registers they use for returning values and reading parameters. In order to actually do real tests on the decompiler I need to be able to build real C programs, so that's why I'm writing a backend for a completely fictional machine!


Got it.

munchausen wrote:
I wanted to use smaller C, as I was basically looking for the easiest compiler to modify that actually really works (there are a lot of half finished C compilers out there), and the size of smaller C appealed to me as indicating that it might be easy to modify. While I'm impressed with smaller C, I confess that I found the code really hard to follow, and after looking at it for a while I still didn't have a good handle on how it was working. I think this may be for efficiency reasons as much as anything though (like I noticed you don't really use structs, which sort of flattens everything but is hard to follow). I've gone for ucc instead (https://github.com/esrever10/ucc and http://ucc.sourceforge.net/) which is designed for teaching and readability, and has a very clear layout and easy to read code. I have wondered how ucc might fare on retrobsd compared to smaller C, as it has a tiny footprint (not sure if its small enough though) but does optimise, which I don't think smaller C does really?


The compiler became self-compilable before the introduction of structures. And to this day the core does not make use of struct, union, typdef or enum internally, although it supports them in input code. It was a goal to self-compile early. Which is one of the reasons why there aren't any external lexers and parsers used (it wouldn't have been able to compile lex, yacc and such early on). So, it's an historical thing.

But at the same time it let me use memory quite efficiently as all the various declarations are kept inside of an array, quite tightly and without the heap metadata and alignment/padding overhead, not to mention the overhead of the memory/heap manager itself.

I have very briefly looked at something like that ucc, but I'm not sure if it's the same one. Nonetheless, I doubt you can fit it or pretty much any other reasonably complete C compiler into where Smaller C fits.

Today Smaller C occupies something like 92xxx bytes of RAM on RetroBSD. The rest 5xxx bytes are used for the stack, the arguments and, probably, the environment variables.

In those 92xxx bytes there is no:
    * AST/declaration logger
    * preprocessor (an external program is used)
    * long long
    * floating point
    * bit fields
    * other minor/rare language features
    * structure passing/returning by value (TBD for MIPS, done for x86 only)
    * malloc(), free() and such (fixed-size static arrays and stack is all there is)
    * full printf() (a simplified one is used)
    * full fopen(), fgetc(), fputc(), fclose() and such (simplified ones are thin wrappers around system calls)
    * extra warnings
    * optimization logic/passes

If you add all the language/compiler features and the complete libc and include optimization, the compiler simply won't fit. If you add only some of those, you'll have to heavily rework the compiler code in order to reduce memory footprint. You'll have to use files as storage instead of RAM and say bye-bye to your beloved pointers and array indices and replace them with fseek(). :) Not to mention longer compile times.

Smaller C's source code isn't quite nice for several reasons:
    * it is my first compiler and in September 2012 it was a simple expression calculator (exprval() is the heart of that calculator), out of which it grew and many things were learned in the process (comments containing the word WRONG in caps should give you an idea)
    * I probably wanted it to self-compile a bit too early and had to deal with less-expressive arrays, single file compilation, etc
    * I typically added stuff as opposed to rewriting large parts (I'd be more willing to refactor stuff if I had good automated tests, I'm considering that, but so far getting by/away without them)
    * I had too much free time on my hands when writing GenExpr1() in cgx86.c and used it improperly, which is why the code is a mess that even I don't dare to touch these days (I'm hoping to rewrite it one day, though)
    * There are already over 160 functions in smlrc.c + x86.c and some globals, all of which fill up SyntaxStack[][] as declarations at compile time and adding more requires extending the array, thus making the compiler's data section larger and I'm trying to avoid it

In contrast, if you look at neatcc, you may find its source code nice, but it's bug-ridden. To begin with, error checking is virtually non-existent and the compiler may crash or do a garbage transformâ„¢ (garbage in, garbage out) without you having any clue.

Anyway, if you're still open to giving Smaller C another try, I could provide some help (btw, why didn't you contact me earlier?). If you're interested in making a code generator (backend) quickly, there's GenExpr0() in cgx86.c. It's a straightforward implementation that receives an expression in the Reverse Polish Notation (RPN) in the stack[][] array and generates 1:1 code to evaluate this RPN at runtime using the CPU stack, (E)AX (stack top, effectively) and a few temporary/hardwired registers (e.g. need BX to access memory in 16-bit code, need CL for shift counts, need DX for division, etc). That's less than 500 lines of code and the functions called from GenExpr0() are small and trivial. I keep this code around for reference. Occasionally it gets stale, but it isn't hard to update/fix it (which is what I just did) and the RPN representation is rather stable and hasn't changed for probably a year now.

All other CG/backend implementations do the same thing, but they implement various shortcuts, thus reducing the number of unnecessary instructions to a more reasonable count, and take care of some calling convention details (e.g. MIPS passes first 4 machine words in 4 regs and always allocates stack space for 4 words even if no arguments are passed (the latter is to make printf() happy), while TR3200 preserves all used regs).

IOW, you can throw in a small and dirty code generator for another 32-bit CPU quickly. It will be grossly inefficient, but it should work nonetheless so long as the RPN transformed into equivalent instructions correctly. And then you can improve it. You can traverse the RPN in stack[][] either linearly or recursively or both ways, depending on what you want to do. You can even turn stack[][] into a tree of linked structures if structures appeal better to you and correspond better to pictures in textbooks on compilers. :)


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 12 posts ] 

All times are UTC


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
cron




Powered by phpBB® Forum Software © phpBB Group

BSD Daemon used with permission