Biosimilarity: Channel-based communication, monadically

The last time we checked in we looked at messaging from a monadic perspective. This time we want to look at a broader class of communication from the point of view of the monadic design pattern. At the heart of this idea is recognizing that what is currently the rage of key-value databases is actually a situation that has a wide range of potential interpretations all captured by a single parametrically polymorphic pattern. Consumers consume data from keyed locations in the store while producers deposit data to keyed locations in the store. The basic control abstraction that supports this situation is the delimited continuation. If a consumer is asking for data from keyed locations where no data exists the store grabs the consumer's thread's continuation up to the point of the request and stores it at the keyed the location. When a producer provides data to some or all of these locations the store first checks to see if there are any continuations waiting on these keys and -- if there are -- makes the data available to them first. This essentially resumes the consumer's thread's flow of control at the point in it's computation where it requested the data from the keyed location.

This basic flow of control abstraction is subject to variation in numerous ways including

varying the means of matching keyed location request to keyed location submission;
varying whether the request consumes the data, removing it from the location or simply passes a copy to a requester
varying whether a stored continuation is removed from the location when data is supplied or simply passed the data

The last variation covers a simple pub/sub mechanism. When continuations are persistent (i.e. not removed on matching data production), this is a simple subscription mechanism. When data is persisted this is storage. When it is not, this is message-passing. When key-matching has a certain level of sophistication this arrangement supports basic broadcast capability. If you look at the code sample and the traces in the screenshots you will see that we've used regular expressions as a basic matching. In the specialK repo we use unification.

All of these variations can be captured in a single configurable mechanism -- the configuration of which can be treated as policy governing communication between producers and consumers. To my mind this is really the pattern underlying the web. The web, in microcosm, is a parametrically polymorphic key-value db. Let's look at some code, however, before getting too philosophical.

The core of the implementation is in the code below. The full implementation is in the sansBNFC branch of the specialK repo.

package com.biosimilarity.lift.model.store

import com.biosimilarity.lift.lib._

import scala.concurrent.{Channel => Chan, _}
import scala.concurrent.cpsops._

import scala.util.continuations._
import scala.collection.MapProxy
import scala.collection.mutable.Map
import scala.collection.mutable.HashMap

trait MonadicTupleSpace[Place,Pattern,Resource]
//       extends MapLike[Place,Resource, This]
extends MonadicGenerators with FJTaskRunners
{
self : WireTap with Journalist =>

type RK = Option[Resource] => Unit @suspendable
//type CK = Option[Resource] => Unit @suspendable
type Substitution <: Function1[Resource,Option[Resource]]

case class IdentitySubstitution( )
extends Function1[Resource,Option[Resource]] {
override def apply( rsrc : Resource ) = Some( rsrc )
}

case class PlaceInstance(
place : Place,
stuff : Either[Resource,List[RK]],
subst : Substitution
)

def theMeetingPlace : Map[Place,Resource]
def theChannels : Map[Place,Resource]
def theWaiters : Map[Place,List[RK]]
def theSubscriptions : Map[Place,List[RK]]

def fits( ptn : Pattern, place : Place ) : Boolean
def fitsK( ptn : Pattern, place : Place ) : Option[Substitution]
def representative( ptn : Pattern ) : Place

//def self = theMeetingPlace

override def itergen[T]( coll : Iterable[T] ) =
Generator {
gk : ( T => Unit @suspendable ) =>
val collItr = coll.iterator

while( collItr.hasNext ) {
gk( collItr.next )
}
}

def locations(
map : Either[Map[Place,Resource],Map[Place,List[RK]]],
ptn : Pattern
) : List[PlaceInstance] = {
def lox[Trgt,ITrgt](
m : Map[Place,Trgt],
inj : Trgt => ITrgt
) : List[(Place,ITrgt,Substitution)]
= {
( ( Nil : List[(Place,ITrgt,Substitution)] ) /: m )(
{
( acc, kv ) => {
val ( k, v ) = kv
fitsK( ptn, k ) match {
case Some( s ) => acc ++ List[(Place,ITrgt,Substitution)]( ( k, inj( v ), s ) )
case None => acc
}
}
}
)
}
val triples =
map match {
case Left( m ) => {
lox[Resource,Either[Resource,List[RK]]](
m, ( r ) => Left[Resource,List[RK]]( r )
)
}
case Right( m ) => {
lox[List[RK],Either[Resource,List[RK]]](
m, ( r ) => Right[Resource,List[RK]]( r )
)
}
}
triples.map(
( t ) => {
val ( p, e, s ) = t
PlaceInstance( p, e, s )
}
)
}

// val reportage = report( Luddite() ) _
def mget(
channels : Map[Place,Resource],
registered : Map[Place,List[RK]],
consume : Boolean
)( ptn : Pattern )
: Generator[Option[Resource],Unit,Unit] =
Generator {
rk : ( Option[Resource] => Unit @suspendable ) =>
shift {
outerk : ( Unit => Unit ) =>
reset {
val map = Left[Map[Place,Resource],Map[Place,List[RK]]]( channels )
val meets = locations( map, ptn )

if ( meets.isEmpty )  {
val place = representative( ptn )
tweet( "did not find a resource, storing a continuation: " + rk )
registered( place ) =
registered.get( place ).getOrElse( Nil ) ++ List( rk )
rk( None )
}
else {
for(
placeNRrscNSubst <- itergen[PlaceInstance](
meets
)
) {
val PlaceInstance( place, Left( rsrc ), s ) = placeNRrscNSubst

tweet( "found a resource: " + rsrc )
if ( consume ) {
channels -= place
}
rk( s( rsrc ) )

//shift { k : ( Unit => Unit ) => k() }
}
}
tweet( "get returning" )
outerk()
}
}
}

def get( ptn : Pattern ) =
mget( theMeetingPlace, theWaiters, true )( ptn )
def fetch( ptn : Pattern ) =
mget( theMeetingPlace, theWaiters, false )( ptn )
def subscribe( ptn : Pattern ) =
mget( theChannels, theSubscriptions, true )( ptn )

def putPlaces(
channels : Map[Place,Resource],
registered : Map[Place,List[RK]],
ptn : Pattern,
rsrc : Resource
)
: Generator[PlaceInstance,Unit,Unit] = {
Generator {
k : ( PlaceInstance => Unit @suspendable ) =>
// Are there outstanding waiters at this pattern?
val map = Right[Map[Place,Resource],Map[Place,List[RK]]]( registered )
val waitlist = locations( map, ptn )

waitlist match {
// Yes!
case waiter :: waiters => {
tweet( "found waiters waiting for a value at " + ptn )
val itr = waitlist.toList.iterator
while( itr.hasNext ) {
k( itr.next )
}
}
// No...
case Nil => {
// Store the rsrc at a representative of the ptn
tweet( "no waiters waiting for a value at " + ptn )
channels( representative( ptn ) ) = rsrc
}
}
}
}

def mput(
channels : Map[Place,Resource],
registered : Map[Place,List[RK]],
consume : Boolean
)( ptn : Pattern, rsrc : Resource ) : Unit @suspendable = {
for( placeNRKsNSubst <- putPlaces( channels, registered, ptn, rsrc ) ) {
val PlaceInstance( wtr, Right( rks ), s ) = placeNRKsNSubst
tweet( "waiters waiting for a value at " + wtr + " : " + rks )
rks match {
case rk :: rrks => {
if ( consume ) {
for( sk <- rks ) {
spawn {
sk( s( rsrc ) )
}
}
}
else {
registered( wtr ) = rrks
rk( s( rsrc ) )
}
}
case Nil => {
channels( wtr ) = rsrc
}
}
}

}

def put( ptn : Pattern, rsrc : Resource ) =
mput( theMeetingPlace, theWaiters, false )( ptn, rsrc )
def publish( ptn : Pattern, rsrc : Resource ) =
mput( theChannels, theSubscriptions, true )( ptn, rsrc )

}

import java.util.regex.{Pattern => RegexPtn, Matcher => RegexMatcher}

object MonadicTSpace
extends MonadicTupleSpace[String,String,String]
with WireTap
with Journalist
with ConfiggyReporting
with ConfiggyJournal
{

override type Substitution = IdentitySubstitution

override val theMeetingPlace = new HashMap[String,String]()
override val theChannels = new HashMap[String,String]()
override val theWaiters = new HashMap[String,List[RK]]()
override val theSubscriptions = new HashMap[String,List[RK]]()

override def tap [A] ( fact : A ) : Unit = {
reportage( fact )
}

def representative( ptn : String ) : String = {
ptn
}

def fits( ptn : String, place : String ) : Boolean = {
RegexPtn.matches( ptn, place ) || RegexPtn.matches( place, ptn )
}

def fitsK(
ptn : String,
place : String
) : Option[Substitution] = {
if ( fits( ptn, place ) ) {
Some( IdentitySubstitution() )
}
else {
None
}
}

}

3 comments:

Thomas Lockney said...: Yikes, something seems to be seriously messed up with the code formatting -- all the indentation is collapsed. Any chance of that getting fixed?; February 16, 2011 at 1:13 PM
Chris League said...: Here's a link to the well-formatted version of this code on github.; February 21, 2011 at 3:53 PM
leithaus said...: Ah, cheers Chris! In my browser the formatting looks ok. i'll have to look into this, more.; February 26, 2011 at 12:00 AM

Biosimilarity

Monday, February 14, 2011

Channel-based communication, monadically

3 comments:

Hot links

About Me

hoogle

My Blog List

Blog Archive

Biosimilarity

Monday, February 14, 2011

Channel-based communication, monadically

3 comments:

Hot links

About Me

Subscribe To

hoogle

My Blog List

Blog Archive