neugram / ng

scripting language integrated with Go

Neugram

Neugram is a scripting language and shell.

It is an early-stage experimental hobby project. You can install from HEAD with:

go get -u neugram.io/ng

The language uses Go's syntax for expressions and follows its type system closely. The intention is lightweight interaction with Go packages.

There is a bit of a shell user guide.

For more information, see the neugram website.

ng's Issues

ng: add implementation of type-switch

expression switch has been mostly implemented.
now that type-assert has been also implemented, we should also implement type-switches as described here:
https://golang.org/ref/spec#Switch_statements

ng/typecheck: make sure fallthrough isn't allowed in type-switch

fallthrough in type-switch isn't allowed as per the Go specs.

ng: add flag to start in $$-expression mode

ng: no detection of duplicate declarations

running the following:

ng> c := float64(1)
ng> c := 10
ng> c
int(10)

I would have expected (following Go expectations) that the second line had bombed, telling me c had been already declared.

ng/eval: panic calling closure wrapping an interface{} over an int

running the following script:

func f() (int, int) {
	return 42, 66
}

func g(i, j int) interface{} {
	return interface{}(i+j)
}

x, y := f()
o := g(x, y) // <<< BOOM
print(o)

panicks like so:

$> ng ./test.ng
(func() (int, int))(0x68d360)
(func(int, int) interface {})(0x68d360)
ng eval panic: reflect: function created by MakeFunc using closure returned wrong type: have int for interface {}
goroutine 1 [running]:
runtime/debug.Stack(0x72, 0x0, 0x0)
	/usr/lib/go/src/runtime/debug/stack.go:24 +0xa7
runtime/debug.PrintStack()
	/usr/lib/go/src/runtime/debug/stack.go:16 +0x22
neugram.io/ng/eval.(*Program).Eval.func1(0xc42015e120, 0xc420187860, 0xc420187848)
	/home/binet/work/gonum/src/neugram.io/ng/eval/eval.go:436 +0x204
panic(0x9ac960, 0xc4200fca80)
	/usr/lib/go/src/runtime/panic.go:491 +0x283
reflect.callReflect(0xc420100940, 0xc420185f88)
	/usr/lib/go/src/reflect/value.go:529 +0x7cb
reflect.makeFuncStub(0x2a, 0x42, 0x0, 0x0, 0xc420186280, 0x682285, 0xc4200f5fb0, 0xc420100940, 0xc420100ae0, 0x1000000020, ...)
	/usr/lib/go/src/reflect/asm_amd64.s:17 +0x33
reflect.Value.call(0xc42013c600, 0xc420100940, 0x13, 0xa3b53a, 0x4, 0xc420110270, 0x2, 0x2, 0xc42013c600, 0xc42013c601, ...)
	/usr/lib/go/src/reflect/value.go:434 +0x905
reflect.Value.Call(0xc42013c600, 0xc420100940, 0x13, 0xc420110270, 0x2, 0x2, 0x2, 0x2, 0x2)
	/usr/lib/go/src/reflect/value.go:302 +0xa4
neugram.io/ng/eval.(*Program).evalExpr(0xc42015e120, 0xea60a0, 0xc420110210, 0x0, 0x0, 0x0)
	/home/binet/work/gonum/src/neugram.io/ng/eval/eval.go:1337 +0x1eb8
neugram.io/ng/eval.(*Program).evalStmt(0xc42015e120, 0xea65e0, 0xc4200f8e40, 0x0, 0x0, 0x0)
	/home/binet/work/gonum/src/neugram.io/ng/eval/eval.go:483 +0x128e
neugram.io/ng/eval.(*Program).Eval(0xc42015e120, 0xea65e0, 0xc4200f8e40, 0xc42014c0c0, 0x0, 0x0, 0x0, 0xea5960, 0xc4200fca90)
	/home/binet/work/gonum/src/neugram.io/ng/eval/eval.go:458 +0x366
main.handleResult(0x1, 0xc4200fca60, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/home/binet/work/gonum/src/neugram.io/ng/ng.go:383 +0x7ff
main.runFile(0xc4200fe068, 0x9, 0xc4200fe068, 0x0)
	/home/binet/work/gonum/src/neugram.io/ng/ng.go:276 +0x2a3
main.main()
	/home/binet/work/gonum/src/neugram.io/ng/ng.go:120 +0x256
ng: ng eval panic: reflect: function created by MakeFunc using closure returned wrong type: have int for interface {}
interface {}

Terminal is not properly reset when killing ng (with Ctrl-C) on MacOS

At least on MacOS Sierra, with go1.9.2, Ctrl-C when "ng" is "stuck" will kill the program but not reset the terminer (liner.State.Close() not called ?)

Error elision

Neugram is intended as a scripting language. This proposal adds the equivalent of a default set -e mode to the language.

Unhandled non-nil errors turn into panics. Starting with

func f() error
func g() (int, error)

then

// nil error is ignored, non-nil error panics
f()
x := g()

// returned error is always ignored
_ = f()
x, _ := g()

// returned error value is left to the programmer
err := f()
x, err := g()

Libraries should still handle errors just as in Go.

TODO

• Throw, not panic. Make it impossible to recover, so it doesn't get used in libraries.

ng: implement `export` in shell mode

The env map is available in the global scope in program mode, but it's a pain to work with from the shell and generally undiscoverable. Everyone (including me) expects export, so let's do that.

print($$ls$$) hangs

$ cat foo.ng
#!/usr/bin/env ng

print($$ls$$)

Run on my Google workstation (Ubuntu Linux).

ng/eval: creating plugins may fail for system-wide Go installs

on my ArchLinux box running Go-1.9.2 installed system-wide with the package manager, I get:

[22:57 binet@zoidberg ng]$ go test -v ./...
ok  	neugram.io/ng	1.868s
Returning Eval: []reflect.Value{reflect.Value{typ:(*reflect.rtype)(0x901b20), ptr:(unsafe.Pointer)(0xc420018698), flag:0x82}}
Returning Eval: []reflect.Value{reflect.Value{typ:(*reflect.rtype)(0x901b20), ptr:(unsafe.Pointer)(0xc4200187c0), flag:0x82}}
Returning Eval: []reflect.Value{reflect.Value{typ:(*reflect.rtype)(0x901b20), ptr:(unsafe.Pointer)(0xc420018908), flag:0x82}}
Returning Eval: []reflect.Value{reflect.Value{typ:(*reflect.rtype)(0x901be0), ptr:(unsafe.Pointer)(0xc420018ad0), flag:0x86}}
Returning Eval: []reflect.Value{reflect.Value{typ:(*reflect.rtype)(0x901b20), ptr:(unsafe.Pointer)(0xc420018ce8), flag:0x82}}
Returning Eval: []reflect.Value{reflect.Value{typ:(*reflect.rtype)(0x901b20), ptr:(unsafe.Pointer)(0xc420018ed8), flag:0x82}}
--- FAIL: TestPrograms (4.75s)
	eval_test.go:209: testdata/array1.ng:neugram panic: plugin: building wrapper failed for Go package "md5": exit status 1
		go install runtime/internal/sys: mkdir /usr/lib/go/pkg/linux_amd64_dynlink: permission denied
	eval_test.go:209: testdata/http1.ng:neugram panic: plugin: building wrapper failed for Go package "http": exit status 1
		go install runtime/internal/sys: mkdir /usr/lib/go/pkg/linux_amd64_dynlink: permission denied
	eval_test.go:209: testdata/import4.ng:neugram panic: plugin: building wrapper failed for Go package "ioutil": exit status 1
		go install runtime/internal/sys: mkdir /usr/lib/go/pkg/linux_amd64_dynlink: permission denied
	eval_test.go:209: testdata/import5.ng:neugram panic: plugin: building wrapper failed for Go package "md5": exit status 1
		go install runtime/internal/sys: mkdir /usr/lib/go/pkg/linux_amd64_dynlink: permission denied
	eval_test.go:209: testdata/import6.ng:neugram panic: plugin: building wrapper failed for Go package "frame": exit status 1
		go install runtime/internal/sys: mkdir /usr/lib/go/pkg/linux_amd64_dynlink: permission denied
FAIL
FAIL	neugram.io/ng/eval	4.761s

we should probably devise a way to get around this limitation.

support stdin

$ echo '$$ id $$' | ng
ng: terminal mode: inappropriate ioctl for device

ng: import-alias does not update scope with correct prefix

ng> import foo "fmt"
ng> foo.Printf("ff")
ng: typecheck: undeclared identifier: foo

ng: missing complex128 type

trying to import "gonum.org/v1/gonum/mat", I get:

ng> import "gonum.org/v1/gonum/mat"
typecheck: unknown go type: complex128
typecheck: unknown go type: complex128
typecheck: unknown go type: complex128
typecheck: unknown go type: complex128
typecheck: unknown go type: complex128
typecheck: unknown go type: complex128
typecheck: unknown go type: complex128
ng: neugram panic: plugin: wrapper gen failed for Go package "mat": genwrap: bad generated source: 

[...]

Operator overloading

Operator overloading is viewed quite reasonably with skepticism.
There are two concrete motivations for it in neugram.

The first is being able to build a user interface to a tool like TensorFlow in neugram that is as compelling (or more) as the standard python user interface. This involves performing a series of arithmetic operations on objects that build up a model of a program to execute.

The second is support for matrices. Not only a single, fixed in-memory model for matrcies, but large sparse matrices, or matrices that exist in GPU memory and whose operations need to be batched and communicated to a different program.

Both of these problems require arithmetic and indexing operations backed by arbitrary computation. Operator overloading can do this.

Proposal

Any type can have specifically-named methods the type checker will notice and use to implement operations on those types. Critically, named interfaces can also have these operator methods.

// Matrix is an n-dimensional float64 matrix.
interface Matrix {
	Dim() []int
	OpIndex(dims ...int) float64         // m[3,4]     => m.OpIndex(3,4)
	OpIndexSet(val float64, dims ...int) // m[3,4] = 0 => m.OpIndexSet(0, 3, 4)
	OpMul(v Matrix) Matrix               // m*n        => m.OpMul(n)
	OpAdd(v Matrix) Matrix               // m+n        => m.OpAdd(n)
	OpSub(v Matrix) Matrix               // m-n        => m.OpSub(n)
}

TODO

• Start method names with non-letter ⨂ instead of Op, to avoid conflicts?
• Allow a concrete type to have both interface ops and concrete ops?
  For example OpMul(x *Mat2) *Mat2 and OpMul(x Matrix) Matrix?
• Reference num proposal, for how to get the matrix type generic enough.

(The first comment of this issue is kept up-to-date with the current proposal.
When commenting on it, quote any relevant sections and respond to the quote.)

ng: support comma-ok idiom for chan-receive

right now, there is no support for the comma-ok idiom for chan-receive:

$> ng
ng> ch := make(chan int)
ng> v, ok := <-ch
ng: typecheck: arity mismatch, left 2 != right 1

ng: provide a Jupyter kernel

integrating ng with Jupyter would be great.

we'd need to implement a Jupyter kernel for ng to be able to do that.

consider using mvdan/sh as a shell parser

Neugram has its own shell syntax parser. It's complicated, and not complete.

There is a nice shell implementation in https://github.com/mvdan/sh (by @mvdan) we could use. It would be great to reduce the lines of code in Neugram.

Potential issues:

• We aren't interested in control flow syntax. No big deal, we can still report errors well when we encounter those AST nodes.
• Passing control from one tokenizer to another and back could lead to some tricky glue code.
• Neugram shell blocks end on $$, a valid shell word. This may need a hack in the shell parser.
• A lot of the complexity of shell is the evaluator, which we won't get to use.

Worth investigating at some point.

ng: fatal error: concurrent map read and map write

running the eval tests, I got (a couple of times):

fatal error: concurrent map read and map write

goroutine 186 [running]:
runtime.throw(0x991dc9, 0x21)
	/usr/lib/go/src/runtime/panic.go:605 +0x95 fp=0xc420cfdb20 sp=0xc420cfdb00 pc=0x5b4615
runtime.mapaccess1(0x923200, 0xc42028d6b0, 0xc420cfe368, 0x904460)
	/usr/lib/go/src/runtime/hashmap.go:355 +0x238 fp=0xc420cfdb68 sp=0xc420cfdb20 pc=0x591068
neugram.io/ng/eval.(*Program).evalExpr(0xc4202f01b0, 0xdb9c00, 0xc4201be960, 0x0, 0x0, 0x0)
	/home/binet/work/gonum/src/neugram.io/ng/eval/eval.go:1135 +0x2bd9 fp=0xc420cfe428 sp=0xc420cfdb68 pc=0x8b1da9
neugram.io/ng/eval.(*Program).evalStmt(0xc4202f01b0, 0xdba0c0, 0xc420498100, 0x0, 0x0, 0x0)
	/home/binet/work/gonum/src/neugram.io/ng/eval/eval.go:474 +0x1322 fp=0xc420cfeee0 sp=0xc420cfe428 pc=0x8a88c2
neugram.io/ng/eval.(*Program).evalStmt(0xc4202f01b0, 0xdba140, 0xc420333420, 0x0, 0x0, 0x0)
	/home/binet/work/gonum/src/neugram.io/ng/eval/eval.go:538 +0x8fe fp=0xc420cff998 sp=0xc420cfeee0 pc=0x8a7e9e
neugram.io/ng/eval.(*Program).evalFuncLiteral.func1(0xc4202841e0, 0x1, 0x1, 0x0, 0x0, 0x0)
	/home/binet/work/gonum/src/neugram.io/ng/eval/eval.go:1452 +0x473 fp=0xc420cffa68 sp=0xc420cff998 pc=0x8c5cb3
reflect.callReflect(0xc420284140, 0xc420cffc28)
	/usr/lib/go/src/reflect/value.go:514 +0x38b fp=0xc420cffc08 sp=0xc420cffa68 pc=0x65445b
reflect.makeFuncStub(0x3, 0x904460, 0xc4202a2040, 0xc4202a2018, 0xc420cfff20, 0x6535d5, 0xc4205a23c0, 0xc420284140, 0xc4202a2040, 0x800000008, ...)
	/usr/lib/go/src/reflect/asm_amd64.s:17 +0x33 fp=0xc420cffc28 sp=0xc420cffc08 pc=0x65dea3
runtime.call32(0xc4205a23c0, 0xc420284140, 0xc4202a2040, 0x800000008)
	/usr/lib/go/src/runtime/asm_amd64.s:509 +0x3b fp=0xc420cffc58 sp=0xc420cffc28 pc=0x5e3c7b
reflect.Value.call(0x903ba0, 0xc420284140, 0x13, 0x98568d, 0x4, 0xc420284180, 0x1, 0x1, 0x0, 0x0, ...)
	/usr/lib/go/src/reflect/value.go:434 +0x905 fp=0xc420cfff30 sp=0xc420cffc58 pc=0x6535d5
reflect.Value.Call(0x903ba0, 0xc420284140, 0x13, 0xc420284180, 0x1, 0x1, 0x8f4180, 0xc4203ed960, 0x642c33)
	/usr/lib/go/src/reflect/value.go:302 +0xa4 fp=0xc420cfff98 sp=0xc420cfff30 pc=0x652bb4
runtime.goexit()
	/usr/lib/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc420cfffa0 sp=0xc420cfff98 pc=0x5e64e1
created by neugram.io/ng/eval.(*Program).evalStmt
	/home/binet/work/gonum/src/neugram.io/ng/eval/eval.go:597 +0x4066

this actually reminded of a question I had: what is the concurrency model of ng?
shouldn't we have some t Thread or g Goroutine argument to the p *Program methods ?

provide a ngfmt command like gofmt

we should probably have the equivalent of gofmt for ng scripts. (ngfmt ?)

Table type

This issue covers the proposed design and tradeoffs in the neugram table type.

A data table, or frame, is an two-dimensional data set arranged in columns.
It is intended to be used like an R data.frame, Matlab table, or an SQL table.
An ideal table has a small number of columns and any number of rows.

Columns can be named. The intention is a typical program understands the set of columns statically, but this is not required (and so not baked into the static type system).

Table is an abstract type. Any concrete type satisfying an equivalent method set can be used as a table.

The table does not have to be in memory, or on the local machine. Operations can return errors. Neugram will ship with a table implementation built on SQL. In general, tables can be understood as language-specific syntax for working with SQL tables. Nowhere near all SQL features are supported and there are some important semantic differences that make only a subset of SQL tables suitable treating as a neugram table, specifically, rows are ordered in a neugram table, and neugram has no notion of primary keys.

Type and slice syntax

The syntax for a table type is [|]T, where T is the type of the table cells.
A table that can hold any type is [|]interface{}

For slicing, a table uses the same comma-separated syntax described in the Go multi-dimensional slice proposal: https://github.com/golang/proposal/blob/master/design/6282-table-data.md

Methods

Every table over data of type T must implement:

interface {
	Cols() []string          // reports number of columns and their names
	Len() (int, error)       // reports the number of rows
	Get(x, y int) (T, error) // gets the value of cell
}

A table can optionally implement more methods:

interface { Set(x, y int, value T) error }
interface { Slice(xlow, xhigh, ylow, yhigh int) [|]T }
interface { SliceCol(name ...string) [|]T }
interface { CopyFrom(src [|]T) (int, error) }
interface { CopyTo(dst [|]T) (int, error) }

Builtin make memory tables

The builtin function make can be used to create in-memory data tables.

x := make([|]T, 9, "Grape", "Vintage")
// x is a table with two columns named "Grape" and "Vintage", and 9 rows.

Composite literals

presidents := [|]interface{}{
	{|"ID", "Name", "Term1", "Term2"|},
	{1, "George Washington", 1789, 1792},
	{2, "John Adams", 1797, 0},
	{3, "Thomas Jefferson", 1800, 1804},
	{4, "James Madison", 1808, 1812},
}

TODO

• append: Don't support it? We do want an equivalent to SQL insert.
• Extend slicing/indexing syntax to allow using column names.
• [|]interface{} is extremely common but clunky. ([|]val, [|]any?)
• potential slicing variants:

presidents["Name"] == presidents[1] == [|]interface{}{
	{|"Name"|},
	{"George Washington"},
	{"John Adams"},
	{"Thomas Jefferson"},
	{"James Madison"},
}

x = [|]num{
	{|"Col0", "Col1", "Col2"|},
	{0.0, 0.1, 0.2},
	{1.0, 1.1, 1.2},
	{2.0, 2.1  2.2},
}

x[1] == x["Col1"] == [|]num{
	{|"Col1"|},
	{0.1},
	{1.1},
	{2.1},
}

x[,2] == [|]num{
	{|"Col0", "Col1", "Col2"|},
	{2.0, 2.1  2.2},
}

x[0|2] == x["Col0"|"Col2"] == [|]num{
	{|"Col0", "Col2"|},
	{0.0, 0.2},
	{1.0, 1.2},
	{2.0, 2.2},
}

x[0:1] == x[0|1] == x["Col0"|"Col1"]

x[1,0:1] == [|]num{
	{|"Col1"|},
	{0.1},
	{1.1},
}

(The first comment of this issue is kept up-to-date with the current proposal.
When commenting on it, quote any relevant sections and respond to the quote.)

ng: functions returning the empty interface{} do not behave

ng> func g() interface{} { x := 42; return x }
(func() interface {})(0x650d10)
ng> g()
ng eval panic: reflect: function created by MakeFunc using closure returned wrong type: have int for interface {}

ng/parser: remove equalType

ng/parser has an internal function equalType that does almost the same thing than ng/tipe.Equal.

consolidate the two and remove ng/parser.equalType.

plugin.Open: plugin was built with a different version of package runtime

$ go version
go version go1.9 linux/amd64
$ go get -u neugram.io/ng
$ go install net/http
$ ng
ng> import "net/http"
ng: neugram panic: plugin: failed to open Go package "http": plugin.Open: plugin was built with a different version of package runtime

Not really sure what to do from here. What can I do to troubleshoot this error?

ng: panic: SetBool using unaddressable value

ng> f := func() bool { return false }
ng> t := func() bool { return true }
ng> t() && !f()
ng eval panic: reflect: reflect.Value.SetBool using unaddressable value

ng: support compiling scripts to binaries

With the Go generating backend from issue #5, it would be relatively easy to support compiling ng->go->binary.

Windows Support

I'm going to start this as a placeholder. In my brief searching, I don't see a suitable package to replace pkg/term for Windows terminal support.

See pkg/term/#8

go get -u neugram.io/ng
# neugram.io/ng/eval/shell
C:\go\src\neugram.io\ng\eval\shell\job.go:48:10: undefined: syscall.Termios

eval: convert tuples into varags

print($$ ls $$) should print both results from the shell command

/usr/bin/ng conflicts with another "ng"

There is another /usr/bin/ng ( https://packages.debian.org/en/sid/ng-common ).

Although the other "ng" is no longer under active development, any chance to reconsider the executable name?

ng: suggest ctrl+d when user types `exit`

Is this intended to be a replacement for bash, etc? Reading this makes me think so:

If you are familiar with another Unix shell, such as the Bourne shell, then you already know a lot about the Neugram shell.

However it doesn't have any common shell commands like: ls, clear or even exit. Are those going to be eventually included?

Compilation on BSDs

s/+build darwin/& freebsd/

Probably the same fix will work on other BSDs.

A generic number type: num

Whether or not neugram adopts a full parametric type system, they come with a readability cost. Intermediate generalized types are given names like T, or U. A programmer reading code has to track possible meanings of these meaningless names, which is unfortunate.

This proposal is about introducing a single implicit type parameter: num. The type can only hold the value of one of the builtin machine words.

Context implicitly defines num. A function or type written over num is parameterized, and can be used for any value of num.

Functions and types can be parameterized over a single type parameter. The type parameter can only resolve to numeric scalar types and must have the name num. Its appearence anywhere in the type declaration means the function is generic:

min := func(x, y num) num {
	if x < y {
		return x
	}
	return y
}
print(min(float64(4.5), float64(4.1))) // prints 4.1
print(min(integer(4), integer(3)))     // prints 3
print(min(integer(4), float64(3.2)))   // compile error

When declaring a variable using type inference from a constant,
the variable adopts the type of num.

add2 := func(x num) num {
	two := 2
	return x + two
}
add2(float32(4.5)) // two was a float32
add2(int64(4))     // two was an int64

In a scope where num has not been defined (including the top-level),
num takes on the default parameterization of float64.

> x := num(4.3) // x is a float64

(The first comment of this issue is kept up-to-date with the current proposal.
When commenting on it, quote any relevant sections and respond to the quote.)

Undefined math.Erfcinv upon 'go get'

Running a fresh go get at 2017-10-30, 10AM GMT0

$ go get -v -u neugram.io/ng

Fetching https://neugram.io/ng?go-get=1
Parsing meta tags from https://neugram.io/ng?go-get=1 (status code 200)
get "neugram.io/ng": found meta tag get.metaImport{Prefix:"neugram.io/ng", VCS:"git", RepoRoot:"https://github.com/neugram/ng"} at https://neugram.io/ng?go-get=1
neugram.io/ng (download)
Fetching https://neugram.io/ng/vendor/github.com/peterh/liner?go-get=1
Parsing meta tags from https://neugram.io/ng/vendor/github.com/peterh/liner?go-get=1 (status code 404)
Fetching https://neugram.io/ng/eval?go-get=1
Parsing meta tags from https://neugram.io/ng/eval?go-get=1 (status code 404)
Fetching https://neugram.io/ng/eval/environ?go-get=1
Parsing meta tags from https://neugram.io/ng/eval/environ?go-get=1 (status code 404)
Fetching https://neugram.io/ng/eval/gowrap?go-get=1
Parsing meta tags from https://neugram.io/ng/eval/gowrap?go-get=1 (status code 404)
Fetching https://neugram.io/ng/eval/gowrap/genwrap?go-get=1
Parsing meta tags from https://neugram.io/ng/eval/gowrap/genwrap?go-get=1 (status code 404)
Fetching https://neugram.io/ng/eval/gowrap/wrapbuiltin?go-get=1
Parsing meta tags from https://neugram.io/ng/eval/gowrap/wrapbuiltin?go-get=1 (status code 404)
Fetching https://neugram.io/ng/vendor/mat?go-get=1
Parsing meta tags from https://neugram.io/ng/vendor/mat?go-get=1 (status code 404)
Fetching https://neugram.io/ng/eval/shell?go-get=1
Parsing meta tags from https://neugram.io/ng/eval/shell?go-get=1 (status code 404)
Fetching https://neugram.io/ng/expr?go-get=1
Parsing meta tags from https://neugram.io/ng/expr?go-get=1 (status code 404)
Fetching https://neugram.io/ng/tipe?go-get=1
Parsing meta tags from https://neugram.io/ng/tipe?go-get=1 (status code 404)
Fetching https://neugram.io/ng/token?go-get=1
Parsing meta tags from https://neugram.io/ng/token?go-get=1 (status code 404)
Fetching https://neugram.io/ng/format?go-get=1
Parsing meta tags from https://neugram.io/ng/format?go-get=1 (status code 404)
Fetching https://neugram.io/ng/stmt?go-get=1
Parsing meta tags from https://neugram.io/ng/stmt?go-get=1 (status code 404)
Fetching https://neugram.io/ng/parser?go-get=1
Parsing meta tags from https://neugram.io/ng/parser?go-get=1 (status code 404)
Fetching https://neugram.io/ng/typecheck?go-get=1
Parsing meta tags from https://neugram.io/ng/typecheck?go-get=1 (status code 404)
neugram.io/ng/eval/gowrap/wrapbuiltin
# neugram.io/ng/eval/gowrap/wrapbuiltin
../../../neugram.io/ng/eval/gowrap/wrapbuiltin/wrap_math.go:33:38: undefined: math.Erfcinv
../../../neugram.io/ng/eval/gowrap/wrapbuiltin/wrap_math.go:34:38: undefined: math.Erfinv
../../../neugram.io/ng/eval/gowrap/wrapbuiltin/wrap_math.go:90:38: undefined: math.Round

go get does not work

$ go get -u neugram.io/ng
package neugram.io/ng: unrecognized import path "neugram.io/ng" (parse https://neugram.io/ng?go-get=1: no go-import meta tags ())

Can't import anything

Hey, I am running osx 10.12.6 (16G29). I get the following errors when I try to import sth.

ng> import "net/http"
typecheck: unknown go type: unsafe.Pointer
typecheck: unknown go type: unsafe.Pointer
typecheck: unknown go type: unsafe.Pointer
ng: neugram panic: plugin: building wrapper failed for Go package "http": exit status 1
-buildmode=plugin not supported on darwin/amd64

ng> import "log"
ng: neugram panic: plugin: building wrapper failed for Go package "log": exit status 1
-buildmode=plugin not supported on darwin/amd64

But, yeah fucking awesome project! Can't wait to get it running :)

ng: importing packages with a dot "." fails

hi,

trying importing a package with a dot "." in the import string fails:

ng> import "github.com/astrogo/fitsio"
typecheck: unknown go type: unsafe.Pointer
typecheck: unknown go type: unsafe.Pointer
typecheck: unknown go type: complex128
typecheck: unknown go type: complex128
typecheck: unknown go type: unsafe.Pointer
typecheck: unknown go type: complex128
typecheck: unknown go type: unsafe.Pointer
typecheck: unknown go type: unsafe.Pointer
typecheck: unknown go type: unsafe.Pointer
typecheck: unknown go type: unsafe.Pointer
typecheck: unknown go type: unsafe.Pointer
ng: neugram panic: plugin: wrapper gen failed for Go package "fitsio": genwrap: bad generated source: 16:8: expected 'STRING', found '.' (and 10 more errors)
  0: 
  1: // Generated file, do not edit.
  2: 
  3: package main
  4: 
  5: import (
  6: 	"reflect"
  7: 
  8: 	"neugram.io/ng/eval/gowrap"
  9: 
 10: 
 11: 	io "io"
 12: 
 13: 	reflect "reflect"
 14: 
 15: 	github.com_astrogo_fitsio "github.com/astrogo/fitsio"
 16: 
 17: 	image "image"
 18: 
 19: 	image_color "image/color"
 20: 
 21: )
 22: 
[...]

I guess we could do the same than in the old C backend of the gc toolchain and replace . with the UTF dot ? (or just replace it with _)

Proposal: introduce separate syntax for error elision

I've been working on a few of the outstanding issues with error elision, but as I started to think about it I realized it quickly gets very hairy.

I define error elision as: "given an expression that yields a tuple ([T1, T2, ...,], error), we support using this in places that expect just (T1, T2, ...). In that case, the runtime dynamically checks if the actual expression returned a non-nil error and panics in that case.

However, by this definition, one could argue these two cases should be supported:

func concat(a, b string) string { return a + b }
concat($$ echo foo $$, $$ echo bar $$)

func closeAll(files ...*os.File) { /* ... */ }
closeAll(os.Open(...), os.Open(...))

Since the error elision definition above supports turning (T, error) into just T, the function calls should be able to be used in single-value contexts. However, this quickly leads to a typechecking nightmare. While it's hard to give an example likely to show up in real code, how would we typecheck a call like this:

f1 := os.Open(...)
f2 := os.Open(...)
func foo(a, b, c error) { }
foo(f1.Close(), f2.Close())

There are two valid interpretations to which Close gets its error elided.

It would be nice to first clarify what error elision semantics would be desired. Am I giving them too much flexibility, and they should just be used in much simpler situations? If the definition to error elision I gave is desirable, I think the best approach to that would be to introduce syntax to let the programmer define the intent.

I think it would end up simplifying the code base, too, given that a fair amount of complexity is needed to determine whether or not error elision is needed or even desired. By asking the programmer to make their intent clear, this difficulty largely goes away.

I don't have a particular syntax suggestion, but I will give a few examples just to illustrate what I mean. I suggest we define the syntax $$ foo $$ to mean shell execution without error elision, and introduce $! foo !$ to mean with error elision. And then something similar for function calls. Alternatively, introduce $$ foo $$! and foo()! or $$ foo $$? and foo()? which are more parallel in syntax.

ng: add select implementation

there is no select support.

• introduce select keyword
• introduce select statement
• introduce select expression

ng: importing a package with a "-" in import path fails

ng> import "go-hep.org/x/hep/rootio"
[...]
ng: neugram panic: plugin: wrapper gen failed for Go package "rootio": genwrap: bad generated source: 20:9: expected 'STRING', found '-' (and 10 more errors)
  0: 
  1: // Generated file, do not edit.
  2: 
  3: package main
  4: 
  5: import (
  6: 	"reflect"
  7: 
  8: 	"neugram.io/ng/eval/gowrap"
  9: 
 10: 
 11: 	wrap_io "io"
 12: 
 13: 	wrap_reflect "reflect"
 14: 
 15: 	wrap_sync "sync"
 16: 
 17: 	wrap_time "time"
 18: 
 19: 	wrap_go-hep_org_x_hep_rootio "go-hep.org/x/hep/rootio"
 20: 
 21: 	wrap_bytes "bytes"
 22: 
 23: )
[...]

ng: support comma-ok idiom for maps

right now, there is no support for the comma-ok idiom for maps:

$> ng
ng> m := make(map[string]float64)
ng> v, ok := m["foo"]
ng: typecheck: arity mismatch, left 2 != right 1

PS1 support

Warning when installing Neugram

macOS 10.13.1 High Sierra
Golang 1.9.2

Hello, I've just tried to install using go get -u neugram.io/ng and I receive the following message:

 neugram.io/ng
ld: warning: PIE disabled. Absolute addressing (perhaps -mdynamic-no-pic) not allowed in code signed PIE, but used in type..eqfunc.[106]string from /var/folders/1d/1rqcnlq51zd4j_vqhp1kv3540000gp/T/go-link-645849390/go.o. To fix this warning, don't compile with -mdynamic-no-pic or link with -Wl,-no_pie

Should I be alarmed? Should I carry out any action to fix this? Thanks!

ng: printing math/big.Float values panics

the following panics:

ng> import "math/big"
ng> f := big.NewFloat(42)
ng> f.String()
"42"
ng> f
panic: reflect.Value.Interface: cannot return value obtained from unexported field or method

goroutine 1 [running]:
reflect.valueInterface(0x8dfac0, 0xc4203c0f60, 0x1aa, 0x8dfa01, 0x8ac4f2, 0x6)
	/usr/lib/go/src/reflect/value.go:942 +0x1dd
reflect.Value.Interface(0x8dfac0, 0xc4203c0f60, 0x1aa, 0xd8a5c0, 0x8dfac0)
	/usr/lib/go/src/reflect/value.go:931 +0x44
neugram.io/ng/format.(*debugPrinter).printv(0xc4203a5668, 0x8dfac0, 0xc4203c0f60, 0x1aa)
	/home/binet/work/gonum/src/neugram.io/ng/format/debug.go:188 +0x64d
neugram.io/ng/format.(*debugPrinter).printv(0xc4203a5668, 0x9369e0, 0xc4203c0f60, 0x199)
	/home/binet/work/gonum/src/neugram.io/ng/format/debug.go:177 +0xa71
neugram.io/ng/format.(*debugPrinter).printv(0xc4203a5668, 0x955400, 0xc4203c0f60, 0x16)
	/home/binet/work/gonum/src/neugram.io/ng/format/debug.go:120 +0x149b
neugram.io/ng/format.WriteDebug(0xc4201c2000, 0x955400, 0xc4203c0f60)
	/home/binet/work/gonum/src/neugram.io/ng/format/debug.go:258 +0x12d
neugram.io/ng/format.Debug(0x955400, 0xc4203c0f60, 0x196, 0x955400)
	/home/binet/work/gonum/src/neugram.io/ng/format/debug.go:263 +0x58
main.handleResult(0x1, 0xc4203506c0, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/home/binet/work/gonum/src/neugram.io/ng/ng.go:366 +0x133
main.loop()
	/home/binet/work/gonum/src/neugram.io/ng/ng.go:331 +0x602
main.main()
	/home/binet/work/gonum/src/neugram.io/ng/ng.go:119 +0x35c

should we add a mode to format.Debug that inspects into the given empty interface and use fmt.Stringer if the value implements it ?

Go generating backend

Now that plugin.Open works for importing Go packages, we can implement importing Neugram packages with a Go generating backend.

That is, when we see import 'mypkg.ng', create a Go package from mypkg.ng, build it, and then use plugin.Open.

This works around the fact that Go's reflect package cannot yet create methods, so we get good methodik support in imported packages.

ng: missing "_" blank identifier

hi,

not sure ng wants to support this but, here it goes:

ng> import "fmt"
ng> fmt.Println("boo")
boo
int(4)
ng> _ = fmt.Println("boo")
ng: typecheck: undeclared identifier: _

ng> _,_ = fmt.Println("boo")
ng: typecheck: undeclared identifier: _

ng>

ng: devise a way to query documentation from the REPL

it would be nice if we could, without leaving the REPL, query the documentation related to a given identifier:

• documentation of a package
• documentation of a type, var, const, func, method, ...

python does it with the builtin doc() function.
jupyter does it with a magic hook (appending ? to an identifier)
julia does it via a dedicated mode (enabled by typing ? at the prompt)

we could also do it by positionning the cursor on top of the identifier and hitting some special key (Shift+Tab, Ctrl+Space, or something)...

shell mode: implement clear

Looks like we have to look in the terminfo database on clear for the right sequence to send.

Deadlock when the scanner encounters an unknown rune

Unlucky me, this was one of the first things I tried (getting help in the REPL using \?):

• Build and exec ng
• type \ and hit enter
• the REPL hangs

I tried to investigate the issue a bit, and I can reproduce it easily in a go test: in parser/parser_test.go:572, simply adding this line:

var stmtTests = []stmtTest{
	{`\`, &stmt.For{Body: &stmt.Block{}}},

and running TestParseStmt results in the following:

fatal error: all goroutines are asleep - deadlock!
Parsing stmt "\\"
Scanner.Next unknown r=92 ("\\") s.off=2

goroutine 1 [chan receive]:
testing.(*T).Run(0xc4200bc0f0, 0x117a368, 0xd, 0x1182888, 0x1069401)
	/usr/local/Cellar/go/HEAD-c4b65fa/libexec/src/testing/testing.go:813 +0x2c6
testing.runTests.func1(0xc4200bc000)
	/usr/local/Cellar/go/HEAD-c4b65fa/libexec/src/testing/testing.go:1036 +0x64
testing.tRunner(0xc4200bc000, 0xc420057df8)
	/usr/local/Cellar/go/HEAD-c4b65fa/libexec/src/testing/testing.go:769 +0xc0
testing.runTests(0xc42000a820, 0x1247b80, 0x3, 0x3, 0x100efd9)
	/usr/local/Cellar/go/HEAD-c4b65fa/libexec/src/testing/testing.go:1034 +0x296
testing.(*M).Run(0xc4200b0000, 0x0)
	/usr/local/Cellar/go/HEAD-c4b65fa/libexec/src/testing/testing.go:954 +0x15c
main.main()
	_testmain.go:48 +0x151

goroutine 5 [chan receive]:
neugram.io/ng/parser.(*Parser).ParseLine(...)
	/Users/jawher/lib/go/src/neugram.io/ng/parser/parser.go:43
neugram.io/ng/parser.ParseStmt(0xc42001c270, 0x1, 0x8, 0x0, 0x0, 0x0, 0x0)
	/Users/jawher/lib/go/src/neugram.io/ng/parser/parser.go:124 +0x111
neugram.io/ng/parser_test.TestParseStmt(0xc4200bc0f0)
	/Users/jawher/lib/go/src/neugram.io/ng/parser/parser_test.go:782 +0x138
testing.tRunner(0xc4200bc0f0, 0x1182888)
	/usr/local/Cellar/go/HEAD-c4b65fa/libexec/src/testing/testing.go:769 +0xc0
created by testing.(*T).Run
	/usr/local/Cellar/go/HEAD-c4b65fa/libexec/src/testing/testing.go:812 +0x2a5

The parser gets stuck in:

func (p *Parser) ParseLine(line []byte) Result {
	p.s.addSrc <- append(line, '\n') // TODO: skip the append?
	<-p.s.needSrc // PARSER STUCK IN HERE
	r := p.res
	p.res = Result{State: r.State}
	return r
}

I'm not yet familiar with the code, so I wasn't able to file a fix (debugging and fixing channel related deadlocks can be fun 😆).

ng: type assertion not implemented

ng> import "io"
ng> import "bytes"
ng> func f(r io.Reader) bool {
..>  _, ok := r.(*bytes.Buffer)
neugram: parser: panic: TODO parse type assertion (off 114)

ng> import "io"
ng> import "bytes"
ng> func f(r io.Reader) *bytes.Buffer { return r.(*bytes.Buffer) }
neugram: parser: panic: TODO parse type assertion (off 73)

ci: add codecov to monitor code coverage

ng: implement export

refs #62