There are several ways to bring complex arithmetic into Lua. lcomplex allows you to write code like this:
> require 'complex'
> z = complex.new(3,4)
> = z*z
-7+24i
> = z + 10
13+4i
> = z:real()
3
> = z:imag()
4
> = z:conj()
3-4i
> = z:abs()
5
> = z^(1/3) -- the cube root of z
1.6289371459222+0.52017450230455i
> i = complex.I
> = 10+4*i
10+4i
Another alternative, if complex arithmetic has to be as fast as possible and you don't mind working with a patched Lua, is the LNUM patch.
lbc provides arbitrary precision arithmetic based on the BC library:
> require 'bc'
> bc.digits()
> bc.digits(20)
> PI = bc.number "3.14159265358979323846"
> = PI
3.14159265358979323846
> = 2*PI
6.28318530717958647692
> = bc.sqrt(PI)
1.77245385090551602729
Matrix operations are provided by LuaMatrix
NumLua is a binding mostly based on the Netlib libraries.
GSL Shell is an interactive Lua binding to the GNU Scientific Library (GSL). It comes in two flavours; as a custom build of Lua using the LNUM patch for complex numbers and with a few enhancements, and as a extension library that can be called from vanilla Lua. It uses the powerful Anti-Grain Geometry (AGG) library to produce good looking graphs.