Haskwhal

Reading about hlint

Wed, 16 Sep 2009 04:55:39 GMT

Via Fatvat's take on HLint I learned about HLint and Neil Mitchell's post on the tool.

I haven't spent much time with it, but it seems like a good coding mentor for the intermediate Haskell programmer. As a noob, some of the feedback I'm afraid to accept, such as the changes that push it towards point free form, whereas I'm still clinging to my explicit parameters.

Haskell has many cool development tools, the more I read about the ecosystem.

hello world in CGI FastCGI and Happstack FastCGI

Fri, 04 Sep 2009 01:09:23 GMT

Just as a sanity check for Help! Can you run Happstack as a CGI? I made sure your Haskell CGI and FastCGI setup is working.

CGI

-- ghc --make -o CgiPlay.cgi CgiPlay.hs 
import Network.CGI

cgiMain :: CGI CGIResult
cgiMain = output "Hello World!"

main :: IO ()
main = runCGI (handleErrors cgiMain)

FastCGI

-- ghc -package fastcgi -threaded --make -o FCgiPlay.fcgi FCgiPlay.hs 
import Control.Concurrent
import System.Posix.Process (getProcessID)

import Network.FastCGI

test :: CGI CGIResult
test = do setHeader "Content-type" "text/plain"
          pid <- liftIO getProcessID
          threadId <- liftIO myThreadId
          let tid = concat $ drop 1 $ words $ show threadId
          output $ unlines [ "Process ID: " ++ show pid,
                             "Thread ID:  " ++ tid]

main = runFastCGIConcurrent' forkIO 10 test

Happstack on FastCGI

import Happstack.Server.FastCGI
import Happstack.Server.SimpleHTTP (askRq, getData, RqData, ToMessage)
import Happstack.Server (look, fromData, FromData, simpleHTTP, Conf(..), toResponse, ServerPartT)

simpleCGI :: (ToMessage a) => ServerPartT IO a -> IO ()
simpleCGI = runFastCGIConcurrent 10 . serverPartToCGI
 
main = simpleCGI handleRequest

handleRequest = return $ toResponse "Howdy From Happstack on FastCGI"

Help! Can you run Happstack as a CGI?

Wed, 02 Sep 2009 14:55:07 GMT

Does anyone already have code for running a Happstack app as a CGi instead of a HTTP Server?

Searching for a CGI adapter or backend to Happstack, I didn't find anything. The documentation mentions you can do it... but it isn't clear if it works out of the box.

I was able to find the happstack-fastcgi package. For fun, I tried converting my app to fast-cgi, following the blog post announcing the project. No dice.
I changed

main = handleSqlError $ trace "Starting up, try port 8080" (simpleHTTP (Conf 8080 Nothing) $ handleRequest)

into

main = simpleCGI handleRequest

simpleCGI :: (ToMessage a) => ServerPartT IO a -> IO ()
simpleCGI = runFastCGIConcurrent 10 . serverPartToCGI

I compiled with

ghc --make -package cgi -package xhtml -o main.fcgi Main.hs


when I run it via apache I get
[Tue Sep 01 23:11:12 2009] [error] [client 172.16.46.1] FastCGI: incomplete headers (0 bytes) received from server "/home/some/web/main.fcgi"


I'd really like to find out how to run my app under plain old CGI, so that I can host it at NearlyFreeSpeech. They don't support FastCGI, nor long running processes like the Server mode.

Segmentation Fault and small files for errors

Tue, 01 Sep 2009 15:59:02 GMT

I don't know if I should be proud of myself or not... but my toy application has Seg Faulted ghc :/

I found to suspicious things

Calling commit on a connection that had already been disconnected

Trying read an int out of the textual representation of a
```
Data.ByteString.Lazy.Char8
```
and insert it into the database

I'm not quite sure, but fixing those two things have fixed the issue.

Another thing I'm noticing is to keep files extremely small, because while compile errors are verbose, runtime error reporting in ghci is very weak. Seg Fault gave me nothing... The error
Prelude.read: no parse
Gives me just a filename and no line number.

I'm excited I've reached a milestone in my toy application... I can

Send out HTML

Include data from the Database in the HTML

Process input from a form

Save the data to the db

Do a redirect

It's taken quite a few hours in 30 minute chunks to get this far. It's a good problem for exercising basic Haskell knowledge.

Being explicit helps compilation

Sat, 29 Aug 2009 20:43:49 GMT

It seems like I should be exact as possible with type signatures as I'm learning. For example I was using:
doWeight :: (ServerMonad m) => String -> m Response
which is all fine and good, until I change the String into a Maybe Stats. Then I ran into some compilation errors and I figured it was with my Maybe plumbing or maybe the IO of the database connection. My compilation error was

WeightController.hs:36:0:
    Ambiguous constraint `ServerMonad m'
        At least one of the forall'd type variables mentioned by the constraint
        must be reachable from the type after the '=>'
    In the type signature for `doWeight':
      doWeight :: (ServerMonad m) => ServerPartT IO Response
Failed, modules loaded: Config, StatsDal, Stats, FiveBeeX, WeightView.

It seems like the biggest clue, which I often miss, is to pay attention to every part of the compilation error message.... what line and character? Ambiguous constraint... and then the hint of the actual type signature ServerPartT IO Response.

After fumbeling around, I remembred by looking at another one of my function type signatures that m Response was indeed ServerPartT IO Response, so without changing the implementation, but just the type signature to
doWeight :: Maybe Stats -> ServerPartT IO Response
everything compiles and works. I'm just so used to the issue being the implementation, maybe always being explicit with the Type signature will reduce these compilation errors and red herrings.

liftIO and a Formal Haskell Learning Curve

Thu, 27 Aug 2009 15:35:29 GMT

I’ve come to know Control.Monad.Trans and liftIO for the first time.

I was trying to glue together Database.HDBC results and a Happstack.Server Response. My method’s which wouldn’t compile looked like

currentWeight :: ServerPartT IO Response
currentWeight = let conn = connectMySQL defaultMySQLConnectInfo { mysqlHost = host, mysqlDatabase = database, mysqlUser = username, mysqlPassword = password, mysqlUnixSocket = unixSocket}
                -- this works...
                --in return $ toResponse "f"
                -- but not this..
                in (currentStat conn) >>= showStats
                where
                  showStats :: Stats -> ServerPartT IO Response
                  showStats stats = return $ toResponse $ show stats

The function currentStat retrieves a row from the database and creates a Stats piece of data.

I got some help on #haskell from ksf and Twey. I now understand that I needed to “lift” my expressions of type IO into the ServerPartT monad. liftIO was the recommended function.

It makes sense now, but I didn’t know you could “stack contexts” like this.

There was also some discussion of fmap, <$>, etc... but it was all a bit much for me at this point. There are so many advanced features to the language, I think it’s tough to give newbies advice. It’s seems easy to bake in lots of advanced features instead of the most basic, brain dead example, so that they will see the core of the answer and can then later “rework” it as they learn Applicative, combinators, partially applied functions, etc.

One role that RWH plays, but that I think could be improved upon, is to have a Formal Haskell Learning Curve. It would be a series of “levels” in understanding the Haskell language. At beginner level I you study the syntax for declaring functions, data types, how to run your code, and part of the Prelude, recursion, etc. Level II you learn the IO monad, etc at higher levels it begins to branch. At level 4 you understand stacking monads, and a bunch of other stuff I don’t know yet :) Then when people are helping you they can try to stay on your level in the explanation.

We do this intuitively already when we teach someone a topic. With a formal learning curve, we can say... don’t tackle HDBC until your a level 3 and don’t mess with Happstack until your a level 4, etc.

They also pointed me to the Typeclassopedia, which I am reading now. This is good supplementary material, but covers way to many “levels” and steepens the learning curve.

W00t! I’ve got data being displayed in my browser via MySQL...

currentWeight :: ServerPartT IO Response
currentWeight = connectDb >>= (\conn ->
                    liftIO (currentStat conn) >>= showStats)
                where
                  connectDb = liftIO $ connectMySQL defaultMySQLConnectInfo { mysqlHost = host, mysqlDatabase = database, mysqlUser = username, mysqlPassword = password, mysqlUnixSocket = unixSocket}
                  showStats :: Stats -> ServerPartT IO Response
                  showStats stats = return $ toResponse $ show stats

The key here was lifting the connection and lifting my results from the database each time, before using the >>= function.

Where can I put my debug statements?

Mon, 24 Aug 2009 16:21:33 GMT

I'm not much on debuggers and rely on the REPL or println statements to learn what a piece of code is doing (or not doing). A frustration I've been having when learning Haskell, is not knowing where I could use Debug.Trace's trace function in my code. The module Debug.Trace provides two methods.

putTraceMsg :: String -> IO ()
trace :: String -> a -> a

I chose to go with putTaceMsg... because what the heck is that a -> a all about? Okay, so I plopped it into my code and voila, it compiles and I get to see my output. A day later I added it to a different chunk of code and it won't compile.

I was frustrated and confused. Then I noticed that it worked in an IO() context, but not in a pure function... Hmm.

I got a clue from reading totherme's post Development with Types and it's comments. I re-read chapters 4 and 7 of RWH focusing on Types, I see that I've been having trouble internalizing "everything is an expression. I need to unlearn the imperative statement oriented languages.

I think I'm starting to see what the basis is for where I can use trace. I've got to focus on the "shape" of the main function or within my pure functions. Like lego bricks, the input Type and output Type of each component must match. The overall type must be satisfied by the expression. An expression can be broken up by if, let, where, case, and function calls. The various branches, must all have the same type. Doh, but realizing this difference from weakly typed, statement oriented languages really takes some time.

Okay, so now I understand why trace has the signature String -> a -> a. This allows you to "hide" your tracing by not changing the Type of your expression. You are effectively wrapping a sub-expression in a trace.

Example: To inject a bit of tracing, just wrap an expression in trace. Before and after, it retains the same shape:

  some f

becomes

  
  trace "hello world" (some f)

Related learning, while trying to figure this out: I've been avoiding declaring the Type signature of functions, until I had them "figured out". It sounds like I need to go the other way around, so that I'll be sure that the pile of function calls maintains the correct shape.

Haskwhal <- Eval Scheme, Apply Learnings

Fri, 21 Aug 2009 23:27:39 GMT

Introducing Haskwhal, Missteps of a Narwhal calf...

Hello again... I originally created this journal to track my progress through The Structure and Interpretation of Computer Programs. Until this post, it was named "Eval Scheme, Apply Learnings".

I'm moving on to study Haskell from the excellent book Real World Haskell. A new name is in order... so it's Haskwhal. To tidy up, I've re-tagged all the previous entries with Eval Scheme, Apply Learnings. As a Lifelong Learner I try to learn a new programming language every 18 months, to learn new concepts and warp my mind a bit more.

Going forward I'm looking forward to doing a better job of using the "Friends" features of livejournal to find other newbie Haskellers, asking questions here in the journal, document my failures and successes, and generally learn in the open.


case journalEntry of
  Just learned -> apply $ whatWas $ learned 
  Nothing -> getAClue $ readComments

Finished SICP last April

Sat, 07 Jun 2008 17:53:36 GMT

Shame on me... I have been super busy with my new gig. So I finished my independent study on SICP and got my grade back... A boo ya. I turned in a ton of work and it has been a very enlightening and worth while exercise. I really enjoyed working through SICP and STPL. I have not been doing any Lisp stuff since late-April. I will be focusing much more on Lisp in C's clothing aka JavaScript. I probably won't be updating this livejournal anymore ( and I wasn't very good at posting frequently during this study anywhoos ). Follow me at ozten.com

Gnuplot in Action

Fri, 11 Apr 2008 03:47:32 GMT

My friend and colleague Philipp K. Janert is writing a book on gnuplot. I am a technical reviewer and have thoroughly enjoyed the experience so far.

The title is Gnuplot in Action: Understanding Data with Graphs. It really coming along swimmingly! gnuplot is a really interesting graphing application. Much like ruby's irb, it gives you an interactive REPL ( read evaluate print loop ) where you can explore a dataset visually. There isn't a contemporary GUI, so figuring out how to setup and use the program has been a chore in the past, just working from man pages.

This book revels the cultural aspects of how to really use the program, or incorporate it into systems by setting up the environment and writing little reusable commands. Before I had seen gnuplot, I thought you would have to fire up a spreadsheet program or Mathematica to do graphs. In reality, you can be trying to troubleshoot an issue by analyzing the frequency of errors in a server log, and then pipe the results into a file and display it with Gnuplot to spot trends and see relationships.

Another nice aspect of the book is the background information on logarithms and logarithmic plots, a review of color spaces, and some details around how to expose quantitative information in your graphs. It also touches on how to integrate your graphs into LaTeX.

I was surprised at how many backends gnuplot comes with. I installed it on Mac OS X Tiger and was able to generate SVG, png, and Cocoa output. I had recently added gnuplot to my toolbag, so the chance to review this book couldn't have come at a better time.

Check the sample chapter Essential gnuplot.

SICP 5.5 thoughts proper tail recursion

Sat, 15 Mar 2008 18:22:27 GMT

It's interesting how proper tail recursion is such a subtle and deep feature that is easy to get wrong in a language. Many languages don't support it, such as Java, and so you don't miss it if you don't have it.

Recursion can make solutions for some problems much easier to think about and to implement. With iterative recursion, there is no additional memory overhead since the stack doesn't grow. But without proper tail recursion this problem solving technique is severely crippled.

It makes me wonder about other subtle and deep potential features of programming languages are out there lurking.

5.3 update and Leopard quick view for DrScheme

Wed, 12 Mar 2008 05:12:41 GMT

SICP 5.1 - 5.3 have been really fun and really insane. Writing assembly code is really addictive, but at the same time like serving prison time. You really appreciate how closely the C programming language really is just an improved assembly. Having to think in such linear chunks, manage registers, and the state of the stack is t-e-d-i-o-u-s.

It is pretty mind-blowing to keep going to deeper levels of computation and keep re-implementing the meta-circular evaluator. It's also nice to see how to two types of garbage collection work at a high level. There is a pun buried in the idea of using two arrays for implementing car and cdr memory. I laughed out loud in a "I think I am losing it" kind of way as I worked through 5.4.

On a practical note, I am now using Mac OS X Leopard... and want scheme code to show up as text in the "Quick View" such as TimeMachine and Finder CoverFlow.

I cracked open /Applications/PLT Scheme v372/DrScheme.app/Contents/Info.plist and added the following:
After


<string>372</string>

Add

	
	<key>UTExportedTypeDeclarations</key>
	<array>
		<dict>
			<key>UTTypeConformsTo</key>
			<array>
				<string>public.text</string>
				<string>public.plain-text</string>
			</array>
			<key>UTTypeDescription</key>
			<string>Scheme text file</string>
			<key>UTTypeIdentifier</key>
			<string>org.plt-scheme</string>
			<key>UTTypeTagSpecification</key>
			<dict>
				<key>com.apple.ostype</key>
				<string>TEXT</string>
				<key>public.filename-extension</key>
				<array>
					<string>scm</string>
					<string>ss</string>
				</array>
			</dict>
		</dict>
	</array>

which should be before


</dict>
</plist>

Save file and touch the DrScheme.app or move it so Tiger picks up the changes.

Random SICP updates

Thu, 06 Mar 2008 04:38:41 GMT

Mac OS X Leopard breaks SICP.pdf. So I just upgraded to 10.5 and now I can't use Preview's search to find specific snippets of code in the SICP book. This is a bummer, as I typically jump around the pdf cross referencing different parts of the book.

I download Adobe's Reader 8 and it does find things like (assign new-cdrs correctly, so oddly enough I will be using Adobe Reader often for the first time in about X years.

Chapter 5 is really great at putting together the lower levels of computation and how the Lisp rubber hits the road. 5.3's description really drove home how garbage collection works in general, and how clever the idea of Lisp pairs are. Having two arrays one for car and one for cdr is very smart. The pun of pairs as values in the car or cdr made me laugh out loud on the bus.

BTW a great update of the Steps Toward The Reinvention of Programming is worth reading.

Streams documentation

Mon, 18 Feb 2008 07:58:31 GMT

I am most of the way through the exercises in Chapter 4, but am totally stalling on finishing. For some yak shaving goodness, I thought I would document the Query system and all the layers on top of Scheme which are need to implement this.

PLT Streams code is originally by Berthold K.P. Horn for 2007 class

I am starting from the bottom up... Without further ado

Doh, LJ doesn't allow iframe... so click to open in a new window SICP Streams Documentation.

Hopefully once I have it all documented in a Visual Language, then I can polish of the last questions quicker.

Package systems a special case of first class functions

Thu, 27 Dec 2007 05:07:57 GMT

SICP Lecture 7B makes a great point during the second Q&A session about solving modularity and abstraction problems by reorganizing a solution around PGEN which parametrizes a general algorithm for a specific input.

Early in the book, the similarity of class based object systems and package management systems with Scheme style first class functions kept peaking my interest. Sussman points out that first-class functions can solve modularity issues in the general case with a very simple minimal mechanism. Without first class functions, one can solve specific known problems without special language features such as packages, modules, etc.

One case the immediately comes to mind is the Java language feature of anonymous inner classes for getting around certain awkward pieces of code ( usually introduced by static typing requirements ).

Finished 3.4 and playing with backtracking

Tue, 16 Oct 2007 04:50:50 GMT

So I finished Section 3.4 on concurrency. Sorry no posts for 2.3 - 3.4. Study group has been going well and I have been playing with Scheme instead of writing summaries here... Doh!

An example:
Implementing a backtracking stategy in Scheme:

(define matrix
  #{
    #{40  53 216}
    #{252 52 85}
    #{150 36 70}
    })
(define (matrix-width)
  (vector-length (vector-ref matrix 0)))
(define (matrix-height)
  (vector-length matrix))
(define (energy@ matrix x y)
  ;(display (format "energy@ ~a ~a~n" x y))
  (vector-ref (vector-ref matrix y) x))
(define (energy@-test)
  (values
   (energy@ matrix 0 0)   ;=> 40
   (energy@ matrix 2 2))) ;=> 70

;problem - find lowest energy path from top row "0"
; to bottom row "2". A path can only go 1 space away
; so 0,1 (value 53 above) can go to
;1,0 (252)
;1,1 (52)
;1,2 (85)
; since they are the left, down, and right spaces from 0,1
;
; The lowest energy path is found by adding cells in path
; Example perhaps
; x:0,y:1    x:1,y:2    x:2,y:1  could have the lowest sum
; when we add up the values 53 + 85 + 36

(define (brute-find-lowest-no-cache)
  (define (column-iter n)
    (define (try-path x y solution)
      (if (or (or (>= y (matrix-height))
                  (< x 0))
              (>= x (matrix-width)))
          solution
          (let* ((try-left (> x 0))
                 (left-sum (if try-left 
                               (+ (try-path (- x 1) (+ 1 y) (energy@ matrix x y)) solution)
                               30000))
                 (try-down #t) ; see assertion below
                 (down-sum (if try-down 
                               (+ (try-path  x (+ 1 y) (energy@ matrix x y)) solution)
                               (error "Should never happen, can always go down without hitting an edge")))
                 (try-right (< x (- (matrix-width) 1)))
                 (right-sum (if try-right 
                                (+ (try-path (+ 1 x) (+ 1 y) (energy@ matrix x y)) solution)
                                30000)))
            (cond ((and try-left (and (< left-sum down-sum) (< left-sum right-sum)))
                   left-sum)
                  ((and try-right (and (< right-sum down-sum) (< right-sum left-sum)))
                   right-sum)
                  (else
                   down-sum)))))
    ;(display (format "Doing ~a~n" n))
    (if (>= n (matrix-width))
        0
        (begin
          ;(display (format "x=~a~n" n))
          (display (format "col ~a lowest=~a~n" n (try-path n 0 0)))
          (column-iter (+ 1 n)))))
  (column-iter 0))
;(brute-find-lowest-no-cache)

(define (get-cache-key x y)
  ;todo try sha-digest on this output, profile...
  (format "~a:~a" x y))
(define *cache* #f)
(define (brute-find-lowest)
  (let ((cache (make-hash-table 'equal)))
    (define (column-iter n)
      (define (try-path x y solution)
        (let* ((cache-key (get-cache-key x y))
               (cached-answer (hash-table-get cache cache-key #f)))
          (if cached-answer
              (begin
                ;(display (format "Using cached answer for cache-key=~a cached-answer=~a~n" cache-key cached-answer))
                (+ cached-answer solution))
              (if (or (or (>= y (matrix-height))
                          (< x 0))
                      (>= x (matrix-width)))
                  solution                  
                  (let* ((try-left (> x 0))
                         (left-sum (if try-left
                                       (let* ((new-solution                                                (try-path (- x 1) (+ 1 y) (energy@ matrix x y))) 
                                              
                                              (new-cache-key (get-cache-key x y)))
                                         ;(display (format "at ~a put ~a~n" new-cache-key new-solution))
                                         ;(hash-table-put! cache new-cache-key new-solution)
                                          new-solution )
                                       30000))
                         (try-down #t) ; see assertion below
                         (down-sum (if try-down 
                                       (let* ((new-solution
                                               (try-path  x (+ 1 y) (energy@ matrix x y)) 
                                               )
                                              (new-cache-key (get-cache-key x y)))
                                         ;(display (format "at ~a put ~a~n" new-cache-key new-solution))
                                         ;(hash-table-put! cache new-cache-key new-solution)
                                         new-solution
                                         )
                                       (error "Should never happen, can always go down without hitting an edge")))
                         (try-right (< x (- (matrix-width) 1)))
                         (right-sum (if try-right                                      
                                        (let* ((new-solution
                                                (try-path (+ x 1) (+ 1 y) (energy@ matrix x y)) 
                                                )
                                               
                                               (new-cache-key (get-cache-key x y)))
                                          ;(display (format "at ~a put ~a~n" new-cache-key new-solution))
                                          
                                          new-solution )
                                        30000)))
                    
                    (cond ((and try-left (and (< left-sum down-sum) (< left-sum right-sum)))
                           (hash-table-put! cache (get-cache-key x y) left-sum)
                           (+ left-sum solution))
                          ((and try-right (and (< right-sum down-sum) (< right-sum left-sum)))
                           (hash-table-put! cache (get-cache-key x y) right-sum)
                           (+ right-sum solution))
                          (else
                           (hash-table-put! cache (get-cache-key x y) down-sum)
                           (+ down-sum solution))))))))
      ;(display (format "Doing ~a~n" n))
      (if (>= n (matrix-width))
          0
          (begin
            ;(display (format "x=~a~n" n))
            ;(display (format "col ~a lowest=~a~n" n (try-path n 0 0)))
            (column-iter (+ 1 n)))))
    (column-iter 0)
    (set! *cache* cache)
    ))

(define lowest-seen 10000)
(define (backtracking-find-lowest)
  (define (column-iter n)
    (define (try-path x y solution-so-far)
      (if (or (or (or (>= y (matrix-height))
                      (< x 0))
                  (>= x (matrix-width)))
              (>= solution-so-far lowest-seen))
          solution-so-far
          
          (let* ((try-left (> x 0))
                 (left-sum (if try-left 
                               (+ (try-path (- x 1) (+ 1 y) (energy@ matrix x y)) solution-so-far)
                               30000))
                 (try-down #t) ; see assertion below
                 (down-sum (if try-down 
                               (+ (try-path  x (+ 1 y) (energy@ matrix x y)) solution-so-far)
                               (error "Should never happen, can always go down without hitting an edge")))
                 (try-right (< x (- (matrix-width) 1)))
                 (right-sum (if try-right 
                                (+ (try-path (+ 1 x) (+ 1 y) (energy@ matrix x y)) solution-so-far)
                                30000)))
            (cond ((and try-left (and (< left-sum down-sum) (< left-sum right-sum)))
                   left-sum)
                  ((and try-right (and (< right-sum down-sum) (< right-sum left-sum)))
                   right-sum)
                  (else
                   down-sum)))))
    ;(display (format "Doing ~a~n" n))
    (if (>= n (matrix-width))
        0
        (let ((solution (try-path n 0 0)))
          ;(display (format "x=~a~n" n))
          (if (< solution lowest-seen)
              (begin
                ;(display "New lowest ever")(newline)
                (set! lowest-seen solution)))
          (display (format "col ~a lowest=~a~n" n solution))
          (column-iter (+ 1 n)))))
  (column-iter 0))
;(backtracking-find-lowest)

So I coded up a brute force solution, a backtracking solution, and then backtracking with caching. In Land of the Lost terminology that is memoization which simply means caching results. Then I ran

(time-apply brute-find-lowest '())

backtracking.scm has the gory details including a real jpeg converted into an "energy map" for testing with a larger data set.

Behind on blog updates

Wed, 10 Oct 2007 00:05:23 GMT

So I am still on pace with Reading SICP and doing exercises, but haven't really had time to update this blog with ruminations.

I find the text a much deeper wading into Lisp and computational processes, which is what I needed after reading the Lisp Tutorial and the Little Schemer books.

The study group has been going well with pretty consistent attendance. I hope my taking a break for one month to go to South Africa doesn't disturb the flow too much.

Unit Testing Scheme code

Tue, 18 Sep 2007 03:49:03 GMT

Today I played around with three testing frameworks for PLT DrScheme.

SRFI-78 "check.ss", SRFI-64 "testing.ss", and lastly SchemeUnit "test.ss".

I started with SRFI-78 since it was a more recent standard. It has a very minimal API.
(require (lib "check.ss" "srfi" "78"))
(check-set-mode! 'report-failed ) ;report-failed)
(load "image-filters.scm")
(check (get-nth-filter 0) => identity-filter)
(check (get-nth-filter 1) => sharpen-filter)
(check (get-nth-filter 1) => identity-filter)
(check-report)

(get-nth-filter 1)
=> ((0 -2 0 -2 11 -2 0 -2 0) 3 0)
; *** failed ***
; expected result: ((1 1 1 1 1 1 1 1 1) 9 0)

; *** checks *** : 2 correct, 1 failed. First failed example:

(get-nth-filter 1)
=> ((0 -2 0 -2 11 -2 0 -2 0) 3 0)
; *** failed ***
; expected result: ((1 1 1 1 1 1 1 1 1) 9 0)

Loading the file causes the tests to be run. I didn't like the output, but found "check-set-model" would keep it to only reporting failed tests. This is pretty good, but very minimal.

As I read more, I realized that this library wasn't the only SRFI testing standard, so I also loaded up 64 and gave it a shot.

(require (lib "testing.ss" "srfi" "64"))
(load "image-filters.scm")
(test-begin "Image Filters")
(test-eqv identity-filter (get-nth-filter 0))
(test-eqv sharpen-filter (get-nth-filter 1))
(test-eqv identity-filter (get-nth-filter 1))
(test-error "Bad input" (get-nth-filter 100))
(test-end)

Again, loading this fail causes the test runner to run with the results:
%%%% Starting test Image Filters (Writing full log to "Image Filters.log")
:14: FAIL
Nth filter : 100
# of expected passes 3
# of unexpected failures 1

Both of these libraries dis a xUnit style library SchemeUnit, so I had to give this one a go also.

(require (planet "test.ss" ("schematics" "schemeunit.plt" 2)))
(load "image-filters.scm")

(define file-tests
(test-suite
"Test image-filters.scm"
(test-equal? "First element is zero indexed" (get-nth-filter 0) identity-filter)
(test-equal? "Normal case" (get-nth-filter 1) sharpen-filter)
(test-equal? "Pretend error" (get-nth-filter 1) identity-filter )))

Which comes with two test runners, a text and gui. Let's run them both...

Text Test Runner

Test image-filters.scm > Pretend error
Pretend error has a FAILURE
name: check-equal?
location: #>struct:object:...pper stepper-tool.ss:618:8=""<:31:3
actual: ((0 -2 0 -2 11 -2 0 -2 0) 3 0)
expected: ((1 1 1 1 1 1 1 1 1) 9 0)
2 success(es) 1 failure(s) 0 error(s) 3 test(s) run
1

and Gui Test Runner

I think SRFI-64 and SchemeUnit are both usable. I am more familiar with xUnit APIs so I am going to start coding against SchemeUnit for now.

Sections 2.1 and 2.2

Thu, 13 Sep 2007 05:11:33 GMT

SICP Section 2.1
The text introduces the topic of Data Abstraction. Coming into this discussion, I see this as a core feature of Object Oriented Programming and an element for enabling programming in the large.

It is also another tool for adding abstraction to a system. Given selectors, constructors, and invariant rules or contracts, one can happily go about programming by “wishful thinking” caring only about this interface and not the actual data representation on the backend. This is important as one either builds layers of abstraction, or attempts to collaborate on a system with other programmers.

The text runs though silly implementations of basic “primitives” of Lisp, to show that as long as selectors and constructors use the same silly implementation everything will work, albeit slowly.

This has massive benefits for prototyping and agile system development. Once can sketch-out a component of the system, but not commit to it’s details or spend time on making it efficient, until it is something that is worth investing in. Also this supports David Parnas style information hiding and module de-composition. If one structures the abstract data types so that they use only the constructors and selectors of other abstract data types, then one can rework the implementation without recoding these parts.

Section 2.2
The text continues to blur the lines between data and code, when introducing the concept of hierarchical data and the closure property. This term “closure property” used in the text been superseded by a new concept of scoping rules and environment. I would call this feature composition or regularity. A cons is a pair of boxes that point to something. A cons pair may in-fact point to two other cons pairs. Thus the definition achieves “closure” and powerful abilities emerge out from this.

I think about GUI programming where every widget is composable and recomposable. Example you can place a frame inside of a panel inside of a tabbed widget and so on. The ability to nest them brings power and the regularity brings simplicity.

We can model sequences or hierarchical structures with the same simple glue.

The section on “A Picture Language” is a very cool example. In this picture language the only primatives are painters. A painter given a rectangle, paints itself into that rectangle. The means of combination are functions, created with a rectable, that compose other painters. We are given two painters “Rogers” and an Escheresque “Wave” drawing where the line segments wrap around the image, for maximum trippiness factor.

Very quickly the text begins creating composition combinators that do nothing more than compose other painters, but which create complexity from simple parts very quickly. Thus the “beside” combinator breaks the field in half and populates each half recursively with it’s two arguments.

These painters are sort of data, in that they represent a photo, but at the same time they are a procedure that given a rectangle, render onto the screen.

Again, similar to writing of Parnas, the text stress the importance of stratified design, such that each layer in a system builds on the abstraction below it only. This allows one to make a more robust system and to focus on less details at any one time.

SICP Section 1.3

Thu, 13 Sep 2007 04:37:46 GMT

High order procedures are a very powerful mechanism for designing procedures. In languages that support it, one can pass procedures into procedures as arguments, or similarly have a procedure which returns a procedure as it’s result.

One first blush this sound like a lot of complicated gobbledygook, but being able to pass in procedures allows one to factor code to capture common patterns and name it as a new abstraction, via a procedure, which can be written, tested, and documented once. An example from the Ruby world would be File#open method given a block. This procedure accepts a block executes the block in the context of an open file, and lastly closes the file. The user doesn’t need to remember to do with, if they pass in their “read file” procedure which takes one argument “file-handle” then they don’t have to duplicate standard file code.

Being able to return procedures from procedures, I image, would allow one to parameterize the new procedure. One of the annoying things about Java’s object constructors is that they can not create and return a subclass. So one ends up adding static “creation” methods, so that you can parameters the object you give out. High order procedures don’t have this limitation and you can dynamically build a function based on the inputs the the current procedure.

In a way this is run time code generation. I think this makes it harder to reason about the code, and it use probably really shines when it clarifies an idiom and captures it, but it could be needless complexity or obfuscation when used without purpose.

This “first class” notion of functions, where you can name them by variables, pass as arguments to procedures, returned as results from procedures, and be included as elements of data structures, increases the faculty for building abstractions.

counting car

Sun, 09 Sep 2007 18:04:19 GMT

How many occurrences of Car in PLT Scheme is a great blog post. I have an idea for learning new PL languages and their idioms. Get a sample of "good" projects, and then do static code analysis. This would allow you to study the documentation of syntax in the order it is used the most, and discover those idioms faster.

I have never implemented a system to try this out. Languages where there is a large body of quality projects under open source licensing would be a requirements.

SICP Exercises 1.1 - 1.7

Sun, 09 Sep 2007 04:56:46 GMT

I have been slow in posting these, but here is exercises 1.1 - 1.7

SICP 1.2 recursion fun

Tue, 04 Sep 2007 04:30:52 GMT

Reading section 1.2 and watching Lecture 1-b. Up until now, I had the false understanding that any procedure which calls itself was considered recursive in terms of both written procedure and runtime process. Aside, that doesn't mean that I shied away from using recursion, and I did have an intuitive feel that some forms of recursion performed better than others.

Why structure a procedure using recursion? Often code written using recursion is easier to understand, for certain classes of problems, like process tree based structures ( XML Nodes ). The book also discusses an algorithm for making change from different coins, where the recursive algorithm is easier to understand.

I had previously missed the distinction between how a procedure is specified in source code, and how the processes runs and evolves through time. SICP refines the definition of recursion specifically for processes into iterative processes and recursive processes. Iterative, doesn't mean that it can't be written using recursion. An operation may call itself, but it does so in constant space in a straight forward manner, but like a for loop changing the state of loop variables.

So how do we frame procedures, so that we get the runtime process we want?

The metaphor of "Bureaucracy" in the video drove the lesson home of explicit state, versus hiding state away in the machinery of the computation. In the bureaucracy lite version, you ask someone to solve a problem. They ask someone else to solve a similar problem and to give the results back to you directly. They have processed one step and are now out of the picture, have passed on the final destination and updated state of the problem. This is an interative process that chips away at the answer in constant space. The procedure may call back into itself, but it is still an iterative processes.

The heavy bureaucracy has you ask someone to solve a problem. They process part of the problem and delegate another part to someone else. When that person is finished they answer back to their manager, instead of passing the finished work to you. They then finish processing the partially solved problem and give you the answer, perhaps claiming credit for all of their child processes work. This hidden state is all of the operations delegating to further operations and awaiting a response to post process and hand back up the chain of command.

I find the domain of SICP ( mathematics ) very challenging, so it takes me a long time to solve the math focused exercises, but I can get the gist of what the problem is trying to get me to think about, and thus make some guesses at the directions for massaging the equations from recursive to iterative and back again.

Thoughts on SICP 1.1

Thu, 30 Aug 2007 17:45:24 GMT

Section 1.1 of SICP starts of with some simple, straight forward basics of programming languages, but at the same time provided some deep food for thought.

Judging for this limited exposure, Scheme has a very minimal syntax and is incredibly regular. We learn how to use expressions and names. Names are similar to variable names from a language like Java, but are more general in that they also serve the purpose of naming data or functions, by placing them into the current environment.

Next we see that we can combine expressions and that a grouping of operator followed by operands is called a combination. Again very simple and regular. The postfix notation doesn't bother me since I have spent some time playing with Common Lisp for doing generative graphics.

The section on substitution models is really fascinating and I am having a hard time keeping the two notions straight. The two notions being applicative order and normal order. I have created a mnemonic "nORmal order is ||". This means most substitution follows the textual pattern, except some things are conditionally or lazily computed, just like || in Java. If we say

if( foo() || bar() ){
  baz()
}

in Java, it is poosible that bar() may never be computed.

I had rarely considered this formally as a first class notion. I have created a template language in the past and when implementing looping and conditional logic I had to implement a normal order substitution model, but just kind of stumbled through it, without the formal perspective.

Scheme seems to run in applicative order normally, except for when you declare that you want to step out into normal order.

The chapter moves on to look at Newton's Method for approximating square roots. These mathematical passages are challenging to me, and I better buck up if I am going to finish this book. Perhaps I will learn a fair amount of math along the way. It would be great to rewrite the domain aspect of this text using the domain of web programming instead of math :) Of course you would have to dive into input output very early, which isn't desirable from a pedagogical point of view.

Lastly this chapter defines several functions in a nested fashion within an overall function. This left me in awe for a few minutes. Wowsers. This is so simple, regular, and powerful! This immediately reminded me of Java classes and how they must be implemented under the covers. Except that Java has many more restrictions and semantics on top of the class keyword. Here we are left with the raw power and goodness of nested function definition. Parameters to the top-most function are visible within the scope of the nested function. The nested function isn't defined in the global environment, and thus doesn't pollute it's namespace. The is something really interesting going on here, which I think will become clearer as I gain more experience with these constructs, but all ready it sets my mind alight.

The ability to name functions and nest functions allows us to build powerful black box abstractions. No notion of interface or documentation has been introduced yet, but the idea is that you can tell someone if you calll operator X with arguments y and z then you can expect foo as a result.

This allows construction of programs in a layered fashion, whereby each piece follows contracts between interfaces and the overall complexity of the system is broken down into manageable chunks that can be

tested
thought about
reused

Exercises 1.5 and 1.6 were great. Working them out on paper, as well as exploring some possibilities from Scheme REPL really helps to understand the substitution model.

R6RS ratified

Wed, 29 Aug 2007 22:00:03 GMT

A controversial R6RS has been ratified. I like how open the voting is, so I can read what concerns were voiced. 65.7% voted in favor of the new standard. Yikes, that is a narrow margin. According to the document 60% were required to pass.

Of course as a complete newbie to Scheme, I have no opinion. I would tend to agree with the gist of the ideas that R4RS serves as a better research tool, and that R6RS would make a better production language. It is disconcerting that many of the Nahs stated that the process ignored many of the SRFIs.