corelang/README.md

The Core Language

A compiled language that assumes C as base reality but it also has lots of ideas taken from Jai, Go like type system, order indepent declarations using Ion algorithm. Syntax currently is whitespace significant. Made to practice language development. It has lot's of ideas from modern programming languages that you would not find in any compiler book. It supports modules combined with ordered independent declarations and lazy typechecking. Also runtime reflection, slices and other standard features you would find in C.

The language is currently very debuggable. It can produce readable C code with line directives. This allows you to debug the programs with Visual Studio with full source mapping, exactly like you would debug C programs.

Using Windows API example

• More examples can be found in /examples and /modules, especially checkout:
  • examples/raymarcher.core
  • examples/drawing_to_screen_using_windows_api.core

#import "KERNEL32.core"
#import "Base.core"

PAGE_SIZE :: 4096
Memory :: struct
  commit : SizeU
  reserve: SizeU
  data   : *U8

Reserve :: (size: SizeU): Memory
  // C like compound expressions with named arguments
  result := Memory{reserve=AlignUp(size, PAGE_SIZE)}

  // Named arguments to function calls
  result.data = VirtualAlloc(
    flProtect        = PAGE_READWRITE,
    dwSize           = result.reserve,
    flAllocationType = MEM_RESERVE,
    lpAddress        = 0)->*U8
  return result

Commit :: (m: *Memory, size: SizeU): Bool
  commit_size := AlignUp(size, PAGE_SIZE)
  total_commit := m.commit + commit_size
  clamped_commit := ClampTopSizeU(total_commit, m.reserve)
  adjusted_commit := clamped_commit - m.commit
  if adjusted_commit != 0
    result := VirtualAlloc(
      lpAddress        = (m.data + m.commit)->*void,
      dwSize           = adjusted_commit,
      flAllocationType = MEM_COMMIT,
      flProtect        = PAGE_READWRITE,
    )
    Assert(result != 0)
    m.commit += adjusted_commit
    return true
  return false

Release :: (m: *Memory)
  result := VirtualFree(m.data->*void, 0, MEM_RELEASE)
  if result != 0
    m.data    = 0
    m.commit  = 0
    m.reserve = 0

Simple math library example (operator overloading!)

Vec3   :: struct                   ;; x: F32; y: F32; z: F32
Length :: (a: Vec3): F32           ;; return sqrtf(a.x*a.x + a.y*a.y + a.z*a.z)
Negate :: (a: Vec3): Vec3          ;; return {-a.x, -a.y, -a.z}
Dot    :: (a: Vec3, b: Vec3): F32  ;; return a.x*b.x + a.y*b.y + a.z*b.z
"*"    :: (a: Vec3, b: Vec3): Vec3 ;; return {a.x*b.x, a.y*b.y, a.z*b.z}
"*"    :: (a: Vec3, b: F32) : Vec3 ;; return {a.x*b, a.y*b, a.z*b}
"+"    :: (a: Vec3, b: Vec3): Vec3 ;; return {a.x+b.x, a.y+b.y, a.z+b.z}
"/"    :: (a: Vec3, b: Vec3): Vec3 ;; return {a.x/b.x, a.y/b.y, a.z/b.z}
"/"    :: (a: Vec3, b: F32) : Vec3 ;; return {a.x/b, a.y/b, a.z/b}
"-"    :: (a: Vec3, b: Vec3): Vec3 ;; return {a.x-b.x, a.y-b.y, a.z-b.z}

Unicode conversion example

Utf8ToUtf32 :: (c: *U8, max_advance: S64): U32, S64
  out_str: U32
  advance: S64
  if (c[0] & 0b10000000) == 0
    if max_advance >= 1
      c0 := c[0]->U32
      out_str = c0
      advance = 1

  elif (c[0] & 0b11100000) == 0b11000000
    if (c[1] & 0b11000000) == 0b10000000 // Continuation byte required
      if max_advance >= 2
        c0 := c[0]->U32; c1 := c[1]->U32
        out_str = (c0 & 0b00011111) << 6 | (c1 & 0b00111111)
        advance = 2

  elif (c[0] & 0b11110000) == 0b11100000
    if (c[1] & 0b11000000) == 0b10000000 && (c[2] & 0b11000000) == 0b10000000 // Two continuation bytes required
      if max_advance >= 3
        c0 := c[0]->U32; c1 := c[1]->U32; c2 := c[2]->U32
        out_str = (c0 & 0b00001111) << 12 | (c1 & 0b00111111) << 6 | (c2 & 0b00111111)
        advance = 3

  elif (c[0] & 0b11111000) == 0b11110000
    if (c[1] & 0b11000000) == 0b10000000 && (c[2] & 0b11000000) == 0b10000000 && (c[3] & 0b11000000) == 0b10000000 // Three continuation bytes required
      if max_advance >= 4
        c0 := c[0]->U32; c1 := c[1]->U32; c2 := c[2]->U32; c3 := c[3]->U32
        out_str = (c0 & 0b00001111) << 18 | (c1 & 0b00111111) << 12 | (c2 & 0b00111111) << 6 | (c3 & 0b00111111)
        advance = 4

  return out_str, advance

What's missing ?

• Constant evaluation and constant folding - Folding and precomputing every expression that can be calculated at compile time. Which allows to check if a given constant expression is actually something that can be computed at compile time

  • Constant expressions with concrete types without casting

• Untyped types - Context dependent type assignment of constant expressions, this is a feature I really loved in Go, it makes constants work very well with a very strict type system and it makes errors like overflow of constants in C due to bad size specifier impossible

  • Infinite precision integers in constants
  • Resolution of all untyped types in the typechecking stage
  • Special case of booleans and their type propagation

• Module system

  • Lazy evaluation of modules (unused code is not compiled or typechecked)
  • Import module, load project file distinction

• Order independent declarations - The ordering of functions in code files or modules does not matter, compiler figures all that stuff out for you. "main" can be wherever you want it to be and all functions should be available without problems

• Synchronize generated C code with original source using line directives so that debuggers work

• Expressions

  • Compounds with named fields and numbered fields
  • Functions calls with named arguments
  • All the standard binary, unary expressions
  • Pointer arithmetic and pointer as array
  • Dot access expression needs a redesign because it doesn't handle expressions after the dot it requires names instead
  • Casting might need a redesign not sure

• Runtime reflection

  • Dumping info
  • Is the design of this correct? That's a lot of data.

• Builtin data structures

  • Arrays
  • Slices
  • Dynamic arrays
  • Hash tables
  • Tuples
    • Beginnings
    • Multiple return values from a function
    • But do we actually want this?
    • Some kind of tuple expressions
    • Using tuples as single values without unpacking

• Generics / Parametric polymorphism

• Operator overloading

  • Binary operators
  • Unary operators
  • Bulletproof
  • Assignment expressions?

• Platforms

  • Windows
  • Unix based

• Language constructs

  • Standard constructs like if, for loop etc.
  • Jai like using statement
  • Defer statement
  • Unions (or something like unions)
  • Unnamed blocks

Building

1. Install Visual Studio and Clang
2. Run build.bat

Resources

Stuff that helped me a lot programming the compiler. Hopefully they also will be helpful to you!

• https://bitwise.handmade.network/ - series by Per Vognsen where he actually creates a C like language, very helpful, very hands on!
• https://hero.handmade.network/episode/code/day206/ - this episode of handmade hero started me on the entire compiler journey a long, long time ago.
• https://www.youtube.com/watch?v=TH9VCN6UkyQ&list=PLmV5I2fxaiCKfxMBrNsU1kgKJXD3PkyxO - I have rewatched this playlist many this, searching for keywords and ideas. Jonathan Blow's compiler was a big inspiration of mine when learning programming languages.
• A Retargetable C Compiler: Design and Implementation by Christopher W. Fraser and David R. Hanson - sometimes looked at this as a reference to figure stuff out. Very helpful resource on compiler construction, the way it's written reads more like documentation but you use what you have.
• https://github.com/JoshuaManton/sif - looked at this as a reference from time to time, author seems like a Jonathan Blow fan so it was a good resource informed by similar resources as I used.
• https://github.com/c3lang/c3c - I sometimes looked at C3 compiler as a reference, the author also let me use his big int library he wrote sometime in the past! Thanks a lot!
• https://go.dev/blog/constants - article on golang type system, untyped types, constants that kind of stuff.