Brian's MultiValue Blog

mvStudio First View

2011-12-19T05:24:00.000-08:00

Well it's been a while since the last blog, so what' been happening?

The big news is that I've been hard at work on my new development toolkit, codenamed 'mvStudio' though that may well change between now and its general release next year. This is already being used in anger by a couple of my clients and the feedback has been overwhelmingly positive so far.

mvStudio is the natural inheritor to uvCase, a development environment I wrote years ago whilst working for ADG (that dates it). uvCase was a Windows based 4GL for UniVerse that was probably ahead of its time, and some of the better features have found their way into mvStudio.

Architecturally, mvStudio consists of a runtime client written using .NET WinForms connecting to a database server over UniObjects. So why not web based? Well the issue there is that mvStudio is specifically designed to work with legacy systems, and has lots of tricks up its sleeve for code reuse and for making the most out of the sort of - shall we say - less than optimal file design that somtimes accompanies mature applications that have undergone years of organic growth.

Anyway, the issues with putting web based front ends onto those is that they use a different model to legacy applications, and can cause some fun and games when you want to mix and match them with existing locking, security, workfile creation and user management. Plus you end up having to buy connection pools to stay legal.

That is not to say you can't add web front ends to legacy systems - and for that you can always use my mvScript tools! But these are often for new features, adding new eCommerce capabilities for instance.

mvStudio is designed to handle the nuts and bolts that make up the whole of the application, and so coexistence with existing code and operations is very high on the priority list.

Talking the MultiValue Language

Another high priority is familiarity. mvStudio uses a small number of highly flexible tools that roughly translate to the things well known to MultiValue developers: screens, menus, listings, command and the like.

The number of such tool types has been deliberately kept to a minimum to make for a shallow learning curve - but each of these is very flexible indeed. A screen can be a dialog, can be linked to a real or virtual data source, can integrate with subroutines for its data. So can a listing - which can also be used for export and reporting. A command can render text or HTML complete with two-way javascript integration so even a simple command can be used to present complex information and to use standard javascript libraries like Google Visualisation, as in the screen shot above.

Back on the subject of familiarity, linking all of these togther is the scripting language, which is the real heart of mvStudio.

Rather than requiring developers to learn a new and alien language, the mvStudio language is modelled on an event-oriented, tool-oriented version of UniVerse Basic.

So you have all the familiar structures and functions: nice loops (with an MV style understanding of true and false, not the Windows one!); familiar conditionals and case structures; dynamic array operators; database I/O statements (open, read, write, delete, readu, release etc); calls to subroutines; execute and capture; functions like DCount() - all the regular paraphernalia.

To that familiar base has been added the GUI support - Windows-specific functions such as message boxes, property and method acccess on tools and objects, runtime overrides of the ways that the tools operate.

The snippet below gives an example of how this works - first a search dialog runs, returning a set of user-entered details as a dynamic array, then this is used to construct a new TCL SELECT statement to override the regular Command property for a listing:

This is just a very small taster on mvStudio and I'll be blogging more about mvStudio over the coming weeks.

Happy Holidays.

Fun with NS-BASIC

2011-08-07T04:07:00.000-07:00

Over the past few weeks I've been boring my associates with my troubles regarding app development with Titanium. Whilst there is no denying that Titanium is a very powerful and capable platform for creating professional apps, and whilst there is no denying that the native apps it produces are themselves extremely powerful and capable, there is also no denying that using it for Android is just .. frustrating.

So a breath of fresh air has been playing with NS-BASIC. Unlike Titanium this is a webkit based tool, so it lacks some of the high end features that can be built into a Titanium app. But, that said, it makes use of the PhoneGap APIs to extend it and developers can use the excellent new PhoneGap packaging service, currently in beta (and therefore currently free) to build native style packages for Android, iPhone and Blackberry. It also comes with a simple IDE (one that offers a form painter, unlike Titanium) and a reasonable handbook. There is no free version, but at $99 it's a snap and a lot cheaper than a Titanium subscription. And there are other savings to be made.. because it's webkit based, now that PhoneGap is avialable as an online service you don't need to buy a Mac to develop cross platform apps - something that you need to do with Titanium.

Other nice things are the fact that you do not need to mess around with the ugly Android emulators - you can simply run and debug your apps using a local copy of Chrome, which is very much quicker and easier, and developing is responsive. It feels a bit rough and ready in places - there is clearly a long way to go - but it makes the whole job of app development feel more appealing.

In fact, the only thing wrong with NS-BASIC is the name. On the face of it - yes, you are programming in a style of Basic, and one that is not too far removed from the old Visual Basic, which makes it look friendly and approachable. That is then converted in JavaScript behind the scenes, which is where the really useful part of NS-BASIC comes in - you can, if you wish, totally bypass this Basic generation and code your app directly in JavaScript. Given the extensive JavaScript libraries that are out there this opens up a world of possibilities. In fact the places where the language struggles is in the differences between JavaScript and Basic, and so coding in JavaScript makes more sense if you are used to the foibles of both.

And you still have access to the SQLite databases for local data storage, and to calling web services for access to live data. So I should be able to fetch data from an mvScript page from NS-BASIC as I can from Titanium. I'll let you know how I get on.

mvScript, Ajax and JSON

2011-07-20T02:10:00.000-07:00

I've been asked to write a website for a new client. Nothing unusual in that, except that I have not yet met the client and the requirements, as they stand, are - well - thin, to say the least. I have a bunch of data in a spreadsheet with instructions that this needs to be served up to registered users. There will be a search box and a heirarchical navigation of some kind.

The solution will need to be available on all standard desktops and mobile platforms, and the data will change regularly.

For the mobile platform, I'm leaning towards a Titanium app. just because. I've spent quite some time recently working with Titanium and learning its foibles, to the extent that I'm writing a new book on Android Development with Titanium - not, if anyone from Appcelerator is watching (unlikely) - to diver away from their training, but because just getting up and running in the first instance is so very frustrating. A good product let down with pitiful documentation and very poor hand holding .. where have we heard that one before?

For the desktops, a web site seems the obvious choice, which means using one of my favourite technologies - the DoJo toolkit. This now, in fact, comes with mobile controls that emulate the look and feel of a real mobile application, but on the web - and unlike other webkit-based stacks it really does look and feel like a real app. It's also easier to prototype the look and feel using this than Titanium.

Which is where mvScript comes in.

The final website will be delivered using mainstream technologies - in this case it will probably be using SQL Server and WCF services to deliver data in JSON format. Which is a good combination but painful to prototype - there's too much configuration involved and too much general messing about. Once the data requirements have been hammered out and the functionality solidified I'm sure it will form a good basis. But we're not there yet.

I've written before about the flexibility of the UniVerse platform, and how it lends itself to rapid development especially during the prototype cycle. I can see a role for it even in dedicated Oracle and SQL Server shops just as a prototyping tool - the fact that it is typeless, handles complex data structures and has completely dynamic metadata makes changing the data structures a breeze. So in no time at all, and assisted with a copy of mvStudio, I had the database designed and a set of Windows forms to maintain it.

But what of the front end?

I've possibly made a mistake in marketing mvScript as a simple solution. It is - you can create dynamic web pages in just a couple of minutes, using regular DHTML and it's UniVerse-like scripting language. It can call regular UniVerse subroutines for data, can read and write files, execute commands, and has all the dynamic array handling and logic constructs of a standard UniVerse application. Except it delivers these up as web pages.

But what I've failed to communicate, is that mvScript is not limited to DHTML. You can create anything in mvScript that can be served up through a web server - and that applies especially to web services and to JSON.

Building JSON output from UniVerse code is childs play - just create some template lines and call CHANGE() to substitute real data. That's the great beauty of JSON - it really lends itself to the multivalue model in terms of its data format, and it is incredibly simple.

       SUBROUTINE adManfacturers(InData, OutData, ErrText)
$OPTIONS PICK
       ErrText = ""
       Results = ''
       Open 'MANFACTURERS' To MANUFACTURERS Else
          ErrText = 'Cannot open MANUFACTURERS'
          RETURN
       End
       Template = \{ identifier: 'name', label: 'fullname', items: [ {DATA} ]}\
       ItemTemplate=\{ name: '{ID}', type:'manufacturer', fullname:'{FULLNAME}', image:'{IMAGE}' }\
       GoSub DoSelect
       RETURN
*---------------------------------------------------------------------------
* DoSelect
*---------------------------------------------------------------------------
DoSelect:
       ExLine = 'SSELECT MANUFACTURERS'
       Perform ExLine
       ReadList List Else List = ''
       Dc = Dcount(List,@FM)
       For P = 1 To Dc
          GoSub Add
       Next
       Convert @FM To "," In Results
       OutData = Change(Template,'{DATA}',Results)
       Return
Add:
       Id = List

       Read ManufRec From MANUFACTURERS, Id Then
          Line = ItemTemplate
          Line = Change(Line,'{ID}',Id)
          Line = Change(Line,'{FULLNAME}', ManufRec<1>)
          Line = Change(Line,'{IMAGE}', ManufRec<2>)
          Results<-1> = Line
       End
       Return

And creating mvScript pages to source data in JSON format is equally simple. How simple?

That simple!

DoJo, and many other similar projects such as the popular jQuery, make handling JSON and the whole process of AJAX simple. In the case of DoJo it's just a call to a URL and instructing the library in the format of the expected output. All DoJo visual components are data aware and can be sourced from JSON formatted data, so building a tree view, for example, is accomplished by calling the mvScript page and populating a data source:

function getManufacturers(){
dojo.xhrPost(
   {url : "http://localhost/ad/adTreeJSON.wsp",
   handleAs: "json",
     load: function(data, args){
           mStore = new dojo.data.ItemFileWriteStore({data : data});
            var mGrid = dijit.byId("grdManufacturers");
           mGrid.setStore(mStore);
          },
      error: function(error, args){
           alert('Error reported is ' + error);
         }
    }
    );
}

Result - a nicely populated tree view, list or table.

If you want to see mvScript and DoJo in action together, take a look at the Better and Better site at the U2UG. And while you're there, feel free to post suggestions.

The U2UG board have been spending the last meetings discussing the future of the U2 products with Rocket and suggestions for getting new developers involved. But that's food for another post.

Another Bad Delphi Day

2011-07-04T04:31:00.000-07:00

Some days I just hate Delphi. Other days I love it - the speed, the power. That can even get me past the ugliness of the Pascal syntax, the horrible unit structure, the disappointing upgrades and just why the h*ll has Borland/CodeGear/Embarcadero never added split screen editing when the whole structure of the interface/implementation sections just cries out for it.

Today was a bad day.

mvPDF runs as a service application. This presents its own problems - it’s virtually impossible to debug, for a start. So I have three versions: the regular version (service), the lite version (service) and a test desktop version where I can see what is going on.

Today, I’ve been testing out some changes to the printing functionality. mvPDF, as the name suggests, was designed to produce documents, and it does this very well. But you can also use mvPDF to generate prints directly through Windows, supporting all the features - the basic API, spooler capture, merge forms, reports, markup, PCL parser etc.

The initial problem is that Delphi can only render one real print job at a time. Its printing interface is nice to use, but very limited in terms of device control. And it uses a globally defined Printer object that causes all kinds of mayhem when you have two threads trying to print to different printers at the same time. Which is of course, not good for a reporting service.

So mvPDF uses a virtual printer class to generate the print jobs, then has a separate spooler thread that sends the jobs to the designated Windows printer. By this point, all the work of building the job has been done and so each print takes a few milliseconds to spool. It’s also safer - if a printer backs up too far (e.g. if your printer runs out of paper) you can safely queue a few hundred documents but after a time the Windows spooler just stops accepting prints.

All soak tested fine with the test (desktop) version of mvPDF, so on to testing the service. And hit another Delphi problem.

When you try to print from a service application on Windows 7, you get the nice error:

There is no default printer currently selected.

Now, there is a reason why there is no default printer - whilst users should always have a default printer, a Windows 7 service runs as Local System, which won’t. It is also completely irrelevant, since the code sets the destination printer explicitly. But Delphi still complains, and so the only solution is to have Delphi set a default printer on the way in.

Except you can’t. Because to do so you need to get the device details. And to get the device details you need to access a Delphi TPrinter. And as soon as you do that it crashes with the same no default printer error. Because whenever you set the printer index, whatever you set it to, it checks the current state of the printer and that gets the current printer index and that crashes with this error. Nice one.

There are get-arounds, of course. You can run mvPDF as anything other than Local System - which is fine - or I can use the Windows API to get the device information, and bypass Delphi entirely. Or fix Delphi’s printer unit so it doesn’t have this stupid check. One of which I’ll have to do. But it’s another couple of hours and another job to do.

Connecting to UniVerse over SSL

2011-03-19T04:43:00.000-07:00

As I wrote in my last post, I have been writing a custom secure terminal emulator for connecting to a client system running UniVerse. This entails using an encrypted telnet connection over SSL, a feature that is available through the UniVerse telnet service on Windows or the uvtelnetd daemon on UNIX.

Whether you roll your own emulator – and my reasons for my doing so are specific to this client – or use a standard emulator that supports the encrypted communication such as WinTegrate, actually setting up the server end of the connection is relatively straight forward. If you don't want to have to learn about certificates and encryption here is a step-by-step quick guide to get you up and running.

Still Running TELNET?

First off, why should you be using SSL? Simply, if you are accessing your system over a regular TELNET connection, whether for administration tasks or to access text based legacy applications - it is open to the world – or at least to anyone on your network – to listen in. Network sniffers are easy to find (man tcpdump) and genuinely useful for developers engaged in supporting network protocols, and so listening in on a communication between a client and server is simple enough.

TELNET was clearly designed in the days before people were concerned about network security and did not have auditors breathing down their necks asking why plain text passwords were being sent over the wire. Anything you type over a regular TELNET connection is plainly visible – even your credentials. And that should be a cause for concern.

Using Telnet over SSL

The preferred mechanism for talking to UniVerse from a terminal style connection is to run TELNET over SSL. This uses the standard TELNET protocol and all the negotiations involved, so retaining compatibility with existing software, but the data passed between client and server is encrypted. Moreover, you can add endpoint verification at each end of the connection to ensure that both client and server are who they say they are.

Enabling Telnet over SSL

There are a number of steps that need to be followed before you can connect to UniVerse using TELNET over SSL. These can be performed using the administration client or, if that is not available or you just don’t like it (understandably), UniVerse Basic has a set of tools for working with certificates and for accessing security contexts and so I have scragged together a simple Basic program that can do this for you: download it from here.

This is provided as a self-instsalling program so to use this:

Create an installation file as a type 19 directory, preferably called inst.bp.
Copy the downloaded installation file into that directory, compile and run.
It will create two files, one named blutil.bp and the other blstd.bp.
The program is named BUILD_SSL and is written to the blutil.bp file.

e.g. on sensible PICK flavour:

>CREATE.FILE inst.bp 1,1 1,1,19

(copy the setup_ssl.uvexe using Windows explorer, ftp etc. into the inst.bp directory.

>BASIC inst.bp setup_ssl.uvexe

>RUN inst.bp setup_ssl.uvexe

>SETUP_SSL

The SSL layer is based on certificates used to authenticate the server and to provide the encryption keys: it can also make use of client side certificates to verify the identities of the clients. So the first steps involve creating the necessary certificates and then attaching these to the UniVerse telnet service.

The steps to do this are:

Create a certificate request.
Create or assign a certificate.
Create a security context
Attach the security context to the service.
Enable the service

The SETUP_SSL program will walk you through the first four of these stages.

Step 1. Creating a Certificate Request

Certificates may be generated in-house or, for additional security, by a trusted third party such as VeriSign depending on the level of trust that is required. If your only requirement is to encrypt the communication so it cannot easily be snooped, an in-house self-signed certificate may well be all you need, and can be used in the first instance for testing anyway.

Regardless of where the certificate originates, its life begins with a certificate request. This encodes details about the organization and the preferred certificate format. You can use a regular tool to create the request, but this – and the other certificate functions – is now available as a set of UniVerse functions anyway so it is just as easy to use those.

I’m going to assume that you will be using the SETUP_SSL program, but the prompts generally follow the same form as the UniVerse admin client.

The certificate request is encoded into a file, conventionally given a csr suffix. There are no standard naming conventions here so to keep everything simple I suggest using the name of the server for the root of the file name, and the following suffixes:

csr	certificaterequest
cer	certificate
pvt	private key
pub	public key

You also have some choices to make regarding the format of the certificate, although in some cases this will be dependent on your certificate authority or other provider.

The screen above (you can click to show full size) shows a typical certificate request, one that you can use as a template for your certificates. Note the Common Name in this case refers to the name of the server, since that is what will be identified using this certificate, and not the name of the company. The organization unit and email are both optional.

Once accepted this will create the three files – the request, private and public key files – in the nominated directory, here c:\ssl. Obviously if you are running on UNIX you will need to substitute equivalent UNIX paths.

Step 2 Creating the Certificate

Once you have defined your request, you can either go ahead and create the certificate yourself if that is sufficient, or send it to a third party trusted authority to create the certificate on your behalf – for a fee. If you can generate the certificate yourself, this is simply a matter of determining the length of time for which it will be valid and supplying the request and private key files you created in the first step:

Step 3 Creating the Security Context

So far so good – you have a certificate that can be used to identify you and the keys required to unlock it. But the certificate alone does not contain the necessary information to dictate how it will be used in the context of securing the SSL connection. That is the job of the Security Context Record, or SCR.

The security contexts are held in a UniVerse account and I suggest holding these in either a dedicated account or in the uv account: it does not make any difference as far as the ability to access accounts is concerned, it just has to live somewhere that UniVerse can find it. Frankly I don’t know why Rocket designed it this way and did not just make it part of the uv account like the key store.

Each SCR is held as a separate record in the &SECUCTX& file, and so needs to have a unique record key. The SCR binds together the certificate you created earlier, the private key required to access it and the authority that it will confer on a UniVerse service. Needless to say, the &SECUCTX& file is itself encrypted.

There are three levels at which you can choose to apply the authentication.

No Client Authentication. In this mode, the the server will send its certificate to the client to begin the encryption process, but it does not request a return certificate from that client. This may well still be sufficient for your needs – if your only requirement is to encrypt the data in transit, this will meet that requirement.

Client Authentication in GENEROUS mode (see Auth Type in the figure above). The client will send its certificate to the server, which will check the validity of the certificate but does not verify the identity of the client itself. rocket suggests that this should only be used for testing and development.

Client Authentication in STRICT mode. Here the client will send its certificate and the server must verify the identify of the client against the list of trusted names along with other details in the certificate. You will need to create a certificate for each client that will identify the client to the server using the Common Name entry: a list of these ‘trusted peers’ is maintained as part of the security context. This will of course involve a major administrative effort both in setting this up and in keeping the record up to date as new clients are added, especially where your user community runs to hundreds if not thousands of users.

Step 4. Assigning the Security Context to a Service

Once the security context has been created, the last piece for the UniVerse side is to attach that to one or more of the UniVerse services. For telnet over SSL this will be the uvtelnetd service, and the same context can also be applied to help secure other connections, such as UniObjects.

The attachment is made through a file named .unisecurity. This is a text file that is created in the unishared directory – until you run this it is not created by default so there is no point looking for it.

The .unisecurity file binds together the name of the service with the account holding the SCR and the name of the SCR record itself. It also contains the password to the SCR in plain text which is not the best idea but that is how it is.

Step 5. Starting on Windows

On Windows, that is all you need to do: your TELNET access to UniVerse is provided through the UniVerse telnet service whether you are running over SSL or not, so the service is good to go. You may need to unblock port 992 on your firewall. Find an emulator that supports Telnet over SSL and connect on port 922: notice the difference in the logon banner.

Remember that limited period evaluation copies of WinTegrate are available for download from the Rocket website if you wish to test out the SSL connection and the other WinTegrate features.

Step 6. Enabling the uvtelnetd service on UNIX

On UNIX you have a little more pain, as you will need to specifically enable the uvtelnetd service. There are two approaches: for those flavours of UNIX that use inetd or xinetd and for Solaris that doesn’t.

The first step in both cases is to define the uvtelnetd service in the /etc/services file. You will need to add or modify the entry for the telnets service on port 992 so that it is connected to the UniVerse telnet daemon and not to any pre-existing telnet service. The /etc/services file is relatively similar across all UNIX flavours and you may find a line that reads:

telnets 992/tcp # telnet protocol over TLS/SSL

This should be modified, or a new line added, to specify the path to the uvtelnetd binary and any startup flags: for your first test you may wish to enable detailed logging through the –d3 flag:

telnets 992/tcp no-wait/uv../bin/uvtelnetd –d3

On Linux and most UNIX, including older versions of Solaris, the service should be configured through the xinted or inetd configuration files respectively.

On most Linux, the xinetd.conf file is populated using individual entries in the xinetd.d directory, so you will need to create a new entry for telnets similar to the one below:


# This is the configuration for the tcp telnets service.

service telnets
{
        disable = no
        socket_type     = stream
        wait            = no
        user            = root
        server          = /usr/uv/bin/uvtelnetd
        server_args     = -d3
        log_on_failure  += USERID
}

On Solaris 10 the inetd configuration has been replaced with the service control management. This uses an XML based definition of each service (a “manifest”) that is imported into a services repository using the service configuration commands. The intent is to provide a single model for controlling all of the different types of service available.

The best starting point is therefore to take the existing telnet service definition as a template and modify it to build the telnets definition. You should find this in the /var/svc/manifest/network directory. Copy this to create the telnets.xml file and modify it as below:

service_bundle type='manifest' name='SUNWtnetr:telnets'> 

<service 
        name='network/telnets' 
        type='service' 
        version='1'> 

        <create_default_instance enabled='false' /> 

        <restarter> 
                <service_fmri value='svc:/network/inetd:default' /> 
        </restarter> 

        <exec_method 
                type='method' 
                name='inetd_start' 
                exec='/opt/universe/bin/uvtelnetd -d3' 
                timeout_seconds='0'> 
                <method_context> 
                lt;method_credential user='root' group='root' /> 
                </method_context> 
        </exec_method> 

        <exec_method 
                type='method' 
                name='inetd_disable' 
                exec=':kill' 
                timeout_seconds='0'> 
        </exec_method> 

        <property_group name='inetd' type='framework'> 
                <stability value='Evolving' /> 
                <propval name='name' type='astring' value='telnets' /> 
                <propval name='endpoint_type' type='astring' value='stream' /> 
                <propval name='proto' type='astring' value='tcp6' /> 
                <propval name='wait' type='boolean' value='false' /> 
                <propval name='isrpc' type='boolean' value='false' /> 
        </property_group> 

        <stability value='Unstable' /> 

        <template> 
                <common_name> 
                        <loctext xml:lang='C'> Telnets server </loctext> 
                </common_name> 
                <description> 
                        <loctext xml:lang='C'> 
                          UniVerse Telnet over SSL. 
                        </loctext> 
                </description> 
                <documentation>                        
                </documentation> 
        </template> 
</service> 

</service_bundle>

To install this as a service you will need to call upon the service controller /usr/sbin/svccfg to validate and import the manifest into the services repository. Note that in common with most UNIX commands, this performs its operations silently unless an error is encountered.

# /usr/sbin/svccfg 
svc:> validate telnets.xml 
svc:> import telnets.xml

Finally you will need to enable the service before it can be launched:

# /usr/sbin/svcadm enable telnets

You can test this by listing the currently installed services using the svcs command:

# svcs | grep teln 
online         Dec_30   svc:/network/telnet/tcp6:default 
online         13:46:51 svc:/network/telnets:default

You should now be able to connect to the uvtelnetd service by using a terminal emulator that supports telnet over SSL and connecting on port 992. Note that the first time you connect to the service there may be a delay of up to a minute: this Is not the case for subsequent connections.

Catching Up

2011-03-11T01:12:00.000-08:00

Has it really been this long since I posted?

Must do better: but the past few weeks have been insanely busy juggling a lot of small projects (including converting ENTER to handle SIMPLE definitions, and for a real PRIME client - would you believe it?) and one dragging on - adapting mvPDF to directly output PCL for faster printing.

The regular Windows PCL drivers are notoriously slow (just Google for PCL and slow) and bloat-producing, so this has meant changing the report generation in particular to directly generate PCL macros and to use better compression for the content. You did know mvPDF did reporting, and not just raw PDFs, didn't you?

It has been a nightmare. I will never code PCL again.

But amongst these tribulations one particular client project has been especially interesting in its own painful way, and containing a number of features deserving of comment.

SWINE (Secure Windows Emulator)

There seems to come a time in every serious MultiValue developers' life when he or she has to write a terminal emulator: it just goes with the territory. Back in the mists of time one of my colleagues created WinTerm so I had some idea how difficult this seemingly simple technology can be.

The new terminal emulator in question is a highly customized emulator written for a specific client.

Amongst its features are:

Encrypted communication with UniVerse using Telnet over SSL. On Solaris.
Programmable look and feel.
Scripting language to control such aspects as exporting, capture and opening URLs.
Security and other constraints that I'm not going to share.

Aside from that, it's written in C# which gives its own peculiar restrictions, especially when it comes to performance and graphics. And it would be nice if it could automatically correct my typing but that's a bit beyond..

It does give me an opportunity to blog about:

SSL and UniVerse, and how to set this up on UNIX-like systems.
Scripting using the Irony compiler toolkit.
Getting almost sensible performance out of .Net (hint, avoid GDI+).

So watch this space over the coming days.

U2 UniVersity

The U2 UniVersity visits the UK this month, so if you are involved in U2 and you have not signed up - shame on you! This is the main event and I look forward to meeting y'all there.

U2 User Group

After a long wait we are finally ready for the new board elections, and I can step down once again as President. The presidential roles are managed on a three year rolling cycle - the Vice President is elected, becomes President after the first year and Past President the year after. It gives a good blend of continuity and change.

The election is also a good time to announce the launch of one of our pet projects - an incubator for new (and old) U2 developers. This is an open source collection of applications all featuring the same underlying database and giving examples of how to use the U2 technologies around it - similar to the sample applications provided by Microsoft for SQL Server.

I've started the ball rolling with a simple C# and UniObjects.NET implementation, and my colleagues will be adding various other pieces - reporting using DataVu, terminal based versions, and so forth. Over time we want the community to embrace and extend this so it becomes a useful resource for everyone.

Away from the City

2010-12-23T05:04:00.000-08:00

Like most of us in the UK I've been trapped by snow - that unexpected event that occurs with reassuring predictability every couple of years - and the indignities of a 6 1/2 hour train journey home after work with equally predictable cancellations, engineering works and miserable vacuity of those nominally in charge of our third world rail infrastructure.

But then, that's what I get for working in Canary Wharf (London) and living in deepest rural Oxfordshire. I invested this year in building an office in the garden, thanks to Green Retreats, so I can VPN in whilst working away from the house - much to my wife's relief. Perfect timing.

So I took the dog out for a walk to admire the snow and remember just why I live here.

Seasons' Greetings to All.

Brian

UniObjects, VBScript and Windows 7 (64)

2010-12-04T07:44:00.000-08:00

As some of you will know from previous articles, I'm a big fan of VBScript. Yes, I started my grown-up programming life with UNIX and have spent a good number of hours using - and teaching - various forms of shell programming as a result. But VBScript is one of the true hidden gems of Windows, a full and well designed scripting language that sits on every desktop and has just the right mixture of control and invisibility. The VB-like syntax gives you the control when you need it and by offering almost no features of its own, it lets you control your scripting environment, applications, operating system and devices by calling on other, more powerful language elements to do the dirty work by acting as a COM client.

One particularly useful application of this, is that VBScript allows you to create easily-maintained scripts to automate a whole batch of UniVerse (or UniData) operations by leveraging UniObjects. Since UniObjects is a simple class wrapper around an excellent protocol for accessing the U2 databases, it plays very nicely with VBScript as a means of executing commands, calling subroutines and all the other paraphernalia that goes along with automating otherwise tedious and long running tasks.

Like cutting releases. Since almost all my products have some degree of client/server operation this naturally means having to build setups that span client and server code, and so I keep a nice library of cutting scripts, one for each product.

These will typically perform a series of actions such as:

requesting a product version number.
logging in the server.
updating the source code version stamps.
recompiling all source code for all platforms.
building the server side setups.
recompiling all client side code.
updating the version stamp on the setup (normally InstallShield).
building the setup.
copying and renaming the setup files into a shipping directory.
zipping up the setup.
updating various log files.

So being able to use VBScript to automate these actions is very, very important to me.
And suddently not being able to call UniObjects from VBScript - typically at a point of urgent need - was .. well.. a problem.

Unfortunately, my nice new sleek laptop running 64 bit windows 7 wouldn't start UniObjects. No errors or breakages in the script, just a refusal to create the UniObjects instance - or any other third party COM component I wanted to access.

The reason was, of course, obvious.

UniObjects is a 32 bit COM component and I'm now using a 64 bit version of the Windows Scripting Host (WScript or CScript). Thankfully, Microsoft has provided two versions of the scripting host, with a 32 bit version sitting in the %windir%\SysWOW64 directory. This starts up UniObjects nicely and I can get back to running my cutting scripts, backups and all the other automation scripts. So if you are having problems with your COM scripts, that is the place to turn.

But an error message might have been nice.

As would retaining the ScriptPW class that allows you to prompt for a password. Gone in Windows 7.

Incidentally VBScript is also great for automated testing. I've been developing a new version of mvQuery for Sandri Technologies, who earlier this year bought the product outright. This involves replacing all the old VB and C++ code with shiny new .NET code, upgrading the Delphi code to the latest version - which is not backwardly compatible, thanks to its new owners - and redesigning the look and feel to fit a more up to date presentation.

But one feature definitely retained is the COM interface, which may seem anacrhonistic in this age of managed code. But the COM interface was added so we could perform automated regression testing: it has the ability to call into the client and perform a suite of activities from running queries to setting filters, running reports and exports and all the result set operations - and because this is a two way interface controlled through a scripting language, not only can it call these actions but it can spot check the results against what is expected.

One more advantage of VBScript for COM automation. Whenever I create a new COM controller in Delphi, this upsets my antivirus which wipes it immediately before I can get to run it. Which is damn inconvenient. But no such problems with VBScript... so far.

UniVerse Release 11

2010-10-25T13:28:00.000-07:00

Two important new announcements from Rocket this week.

The first is the imminent release of UniVerse 11. This is the first full release to bear the Rocket brand and the first full release of UniVerse in some time, so in many ways this sets the tone for future expectations - and I have to say, this sounds very positive. It is clear that Rocket is listening to its users and that the promised development freedom is beginning to bear fruit.

The U2 team have clearly not been resting on their laurels. There are some welcome new additions and extensions to the products, and the good news is that this takes in both the front end features and also some core functionality that has been missing or simply become outdated.

At the core level, we see changes to the MQ interface to support the current API in place of the deprecated AMI interface. This is a change that won’t affect many sites but for those who rely on message queues it is a welcome relief and sends a strong signal in terms of support. I’ll be writing in Spectrum about this new interface. Added to this are extensions to make the UCI layer fully ODBC 3 compatible – another welcome piece of backfill.

Then there are some pieces that will truly strengthen the product. Replication has had an overhaul – this has often been seen as a less than optimal part of the product: following the changes Rocket can now announce high availability with fast switchover and read-only copies for reporting and data warehousing down to the account level.

Data encryption has been extended with keys and indices now getting the encryption treatment and additional protection for master keys available – sites can now require more than one password to unlock the master key, preventing a single disgruntled employee from wrecking the master.

Sign-on now supports both PAM and LDAP for authentication.

Then we come to the big ticket items. Rocket is continuing the theme of embracing Eclipse for administration and development seen in BDT and WebDE, and now added a new administration interface to the mix (XAdmin) – of which I will post more later.

And we also see the release of EDA. For those not familiar with this latest term, this adds External Data Access to UniVerse, allowing seamless access to external sources such as SQL Server and Oracle under the guise of UniVerse tables. UniVerse is not the first to add this facility – it has been part of jBASE and mvON for many years. Again, this deserves a post in its own right.

As if getting the new Universe release out was not enough, members of the U2 User Group will already have received a newsflash regarding another tool to come out of Rocket in the same timeframe – DataVu. This is the first evidence of the collaboration between the U2 team and the other technologies that sit under the Rocket banner – in the case, an adaptation of CorVu, a BI tool for key indicators. David Jordon has been looking into this – see the post on the U2UG website for more details.

Good news all round, and thumbs up to the Rocket crew.

mvScan going open

2010-07-31T10:53:00.000-07:00

mvScan was originally a tool that I had developed for my use, to document a UniVerse system by iterating through the account and file structures, building impact maps and filling out tables with information culled from the entries found to make it easier for someone to search through their system.

Information such as the relationships between files, programs, subroutines, external functions, SUBR() dictionary calls, translates, includes, common blocks, use of variables, string tables, calling menus and paragraphs. All captured, linked together and available for enquiry or to build documentation, assisted in my case with my autodoc headers and version stamps.

However, and I stress this, it was a tool for my own use.

After various questions in various forums - especially the u2-user list - from people wanting something along these lines to document their systems, I decided to package it up and add it to my product list. But with every system being so different, however adaptable it is (and it is designed to be highly adaptable) I'm always worried whether it has had enough of a real world thrashing for me to feel confident offering it as a packaged product.

So I've decided the best way forward is for me to open it up. That way, people who want to run it on their systems can do so and feed back any updates and changes that result from applying it to their specific structure and code organization.

So watch this space for announcements. If this goes well, there's plenty of other stuff I want to open source.

Message Queues

2010-07-30T05:47:00.000-07:00

Good news for those deploying UniVerse as part of a distributed infrastructure.

Rocket has announced that the upcoming Release 11 of UniVerse - the first and eagerly awaited major release of UniVerse since the move to Rocket - will include an upgrade to the current support for IBM WebSphere message queues (MQ).

UniVerse has long offered support for MQ through a serviceable Basic interface built upon AMI. However, this route for accessing MQ was superceded some years back and is now deprecated: the patches required to use AMI are not available for all operating systems and it is becoming increasingly difficult to track these down. So the investment in providing a link to MQ that supports more current protocols is very welcome, both technically and as a signal of intent.

From what I have been told, the old functions will continue to operate as they do currently so that sites can continue to run their existing MQ routines without the need to do any immediate development. The new functionality will be provided through a new set of MQI functions. This means that sites wanting to make use of the new features will need to do some development work, but it is probably the safer option.

Tell me Less

2010-06-24T07:54:00.000-07:00

Deep joy: I need to evaluate some new software. Like most developers it something I do on a regular basis -a triumph of hope over expectation that inevitably leaves my various PCs cluttered with garbage, and worse.

Talking of which..

Dear Software Provider

I'm about to evaluate your software.

If it is any good, I'll buy it. If not, I don't want to hear from you.

I do not want to fill in five pages of 'Required Fields' telling you about my business, size of business, number of employees, market sector, and especially not my phone number. If I have a problem, I'll call you.

You are not doing me a favor by letting me download this.
I'm giving up my valuable time to see if your miserable product is worth my hard earned cash.

So guess what - I'm not going to waste more of it filling out stupid questionaires. You don't need to know what databases I use, what languages I prefer and what my shoe size is. And that drop-down list of 'relationships' to say whether I'm an end user, VAR or supplier - I'm none of those yet. Just let me see the damn software.

Ask me what you need to know: an email address to send a registration. Nothing more. Because I'm not giving you any other marketing-related crap that your employees can leave on a bus.

And no, I do not want to speak with an advisor. If I need an advisor to test your software, you need to think about redesigning your website. Or your software. Or both.

Got it? No?

Then you're off the list. I'm off to your competitor who will ask me my email and nothing more.

Some of us have better things to do with our time.

UniVerse Free Edition

2010-03-21T03:20:00.000-07:00

Exciting news for UniVerse and UniData developers: we are told that Rocket are preparing now to release free versions of these databases. This will share the same technical limitations as the current personal edition, but significantly will no longer be restricted to non-commercial use and will relax the restrictions on distribution.

This is a huge and very welcome (and very timely) step forward.

There has never been a way for new developers to release a free or very low cost, entry-level product using this technology because of the cost of the database licence. It has been all but impossible to release free demonstration copies of commercial products, because of the licence restrictions that prevented distribution of the personal edition, without signing up as a business partner.

Why does that matter? Because we need new blood in this industry, and new developers are not going to run with a technology that put barriers in the way of getting their products in front of potential clients: not when there are other, free and easily distributable alternatives. New developers don't want to sign up to partnership agreements, or to have to direct clients to external websites to download underlying technology: they want to get their new products in front of clients in the easiest way possible. But simply getting in at the bottom rung - so bread and butter to the rest of the industry - has simply not been a practicable option in this market. Perhaps this is down to the history of a product that began its life as an emulator, running software applications that had already been developed elsewhere. But it has long been a clear barrier to new start-up developers entering the U2 marketplace.

Compare this with the most obvious rival, SQL Server. There the pattern has always been, to release entry-level products using the JET database engine (the one shared with Access) or more recently with SQL Server Express, in the knowledge that over time at least some of those clients will grow and upscale their investment: at which point they have a natural and largely automated upgrade path to the full SQL Server. Unfortunately, the MultiValue world has been largely blind to this model throughout its history.

Which is one of the two reasons why I am so excited about this move.

The other, is that from the very start, one of the key objectives of the U2 User Group has been to disseminate knowledge about the model. But we have been constantly held back by the fact that we could not distribute the software alongside learning materials.

We could not, for example, produce a CD holding a learner pack that had pre-installed the demonstration database and samples, to give away at venues: downloading the database and installing software is fine for existing MultiValue users of various flavours, but we want to get the U2 model in front of people who are not familiar with the concepts and to make it as easy as possible for them to explore and to experience it.

Now Rocket are working on a way to allow us to do exactly that.

So watch this space for progress reports.

BCI.IMPORT posted on website

2010-03-06T03:44:00.001-08:00

In my last blog I wrote about using UniVerse as a prototyping tool for rapidly modelling complexity. Creating a prototype application may require a different approach to regular development, with the onus being on dividing a process into stages to ensure that each one can be individually verified. Therefore when modelling financial calculations in particular, I take the approach of generating each step as a workfile, which is less efficient but allows the data from each stage to be assessed and proven.

To assist me in doing this I have been leveraging a set of dictionary-driven tools, along with the much-neglected REFORMAT (often a better choice than INSERT and UPDATE) to handle each stage.

As promised, the first of these is now available on the website, in the hope that it may prove useful to other developers.

BCI.IMPORT, as the name suggests, is an import routine using BCI, the Basic Calling Interface built into UniVerse and enabling access to remote data sources using ODBC. Whilst ODBC has its limitations, the BCI implementation is sufficiently near to the metal that it can be a very fast choice when directed to natively relational databases such as SQL Server. It requires ODBC drivers, which are an additional expense if running on UNIX platforms.

BCI.IMPORT runs either a SQL statement or stored procedure and assigns the results to one or more UniVerse files deriving the keys and fields to be updated from the dictionary. It supports a range of update operations for multivalued data in particular, and provides simple structural validation. More importantly, it enables you to specify expressions using the incoming data following the UniVerse I-Descriptor format, giving full reign to massage and to enrich the incoming data before assigning it.

BCI.IMPORT is free, and can be downloaded from the Freebies section of the website. As always, comments and suggestions welcome.

Prototyping with UniVerse

2010-02-27T02:12:00.000-08:00

Whenever I write about UniVerse - or MultiValue in general - I always refer to it as a business engine, which encapsulates the difference between this and the mainstream relational database model in practice. I have blogged before about the advantages of keeping the data definitions and business logic in the same space when it comes to speed of development and consistency of access.

All the same, in my day job I'm frequently working with SQL Server (it's practically impossible to make a living as an independent MultiValue consultant in the UK). So it's good when the opportunity arises to use both technologies together and to bring their respective strengths to the fore.

If the relational model is predicated on structure and rigourous design, the MultiValue model is founded on agility and the ease of managing complex data. And so in situations where change is the top of the agenda, and given the facility with which it can read and write SQL Server data, UniVerse can become a powerful tool for solving problems in SQL Server.

Two projects over the last few months have allowed me to do just that.

The first was a data cleansing exercise, pulling data that had orginated in different applications from different vendors and patching up differences and errors. Here the key piece is the granularity offered by the low level UniVerse read and write operations, especially when hunting down data that might be in one of several different places. True, it can be done in SQL Server by using temporary tables and lots of updates through potential joins, but UniVerse does it more neatly and with lower overheads.

The second has been more interesting. I've been asked to develop a new risk calculation engine that will eventually be written in C# for SQL Server. But before getting to that point, the calculations themselves must be modelled; different methods assessed; data checked for consistency; missing data provided; results scrutinized and business processes adapted. All of which argued for prototyping the financial calculations first.

Here, once again, is a job for UniVerse. The fact that UniVerse is metadata driven makes it very adaptable - just what you need in a prototype, especially when that prototype involves modelling different alternate operations to see which one fits the business needs best. You can change a file structure in seconds by adding or amending dictionaries, and calculations can be embedded into virtual field descriptors: more lightweight and powerful than using SQL views or stored procedures. Again, where data is difficult to come by, or needs to be split or merged in more complex ways, it may be synthesized or extracted using UniVerse Basic. So, despite the fact that the final delivery will be pure SQL Server, to model and remodel the prototype has been pure UniVerse.

One thing that has helped, is that over time I have put together a good series of dictionary-driven commands that allow me to script file and dictionary creation, importing from remote data source using BCI, exporting in various formats, pivoting and amending data. These are slower than the equivalent Basic routines, of course, but are more quickly adapted and easier to document as the prototype develops.

Over the coming months I will be adding these to the website, so watch the blog for details and instructions.

New Book Release

2009-12-29T07:47:00.000-08:00

Well it's taken far, far longer than planned (courtesy of having been placed on the back shelf whilst I've been busy wth my day job working on financial trading system) but at last book 3 of my Self Paced Training Series in UniVerse (and UniData) has been sent to the publisher, sent back, re-edited, turned around, sent again and is now finally available for purchase!

The third volume treats the subject of client/server development for UniVerse, focussing on development using the Microsoft .NET platform with UniObjects for native transactional systems an using ODBC and OleDb with ADO.NET for mainstream access.

The sticking point for this, and the reason for the long gestation, has been the IBM ADO.NET provider. This was shaping into a nice new middleware for UniVerse with nascent support for Entity Framework when the sale to Rocket was announced. I had held off writing the final sections until the initial Entity Framework bugs had been ironed out, then waited a bit longer until there was news on what would happen to this provider after the sale, to finally drop the chapter when it became apparent that this would no longer be a core part of the U2 product line.

So now, readers can enjoy the new title:

IBM UniVerse Client Developer - Self Paced Training

- at least, until I get the name changed to Rocket throughout - when the first Rocket version of UniVerse (11) is released.

Silverlight 4 CAS restrictions (loadFromResourcesSwitch)

2009-12-03T04:18:00.000-08:00

A whole morning wasted on this one..

I've been upgrading a bunch of silverlight stuff to version 4, which should be straight forward, except that I had included some third party controls in the projects.

Now whenever I try to build the project, VS2010 came back with the less than helpful error:

The "ValidateXaml" task failed unexpectedly...
System.IO.FileLoadException: Could not load file or An attempt was made to load an assembly from a network location which would have caused the assembly to be sandboxed in previous versions of the .NET Framework. This release of the .NET Framework does not enable CAS policy by default, so this load may be dangerous. If this load is not intended to sandbox the assembly, please enable the loadFromRemoteSources switch. See http://go2.microsoft.com/fwlink/?LinkId=131738 for more information.

Problem? Well apart from something new to have to fix, the files concerned are not being loaded from a remote source: they are all referenced libraries on my local machine, with Copy Local settings.

Which led to three questions:

a) where exactly does this mythical switch go?
b) why does Microsoft think this is a remote file?
c) why is Microsoft once again insistent on breaking everything by treating everyone as idiots?

According to the knowledge base, the way to fix is to add a switch to the config file for the project.

<runtime >
<loadfromresourcesswitch enabled="true" >
</runtime >

Small problem here - Silverlight projects don't have a config file. I added the switch to the config file for the web application hosting it, but to no avail. I still can't build the project in the first place.

The Solution.

After a whole morning of hitting my head against a brick wall, I found the solution. The files are local, but these were custom controls originally downloaded from the vendors website. Which means that Windows had marked them as remote files (adding the 'Mark of the Web' to the file). That mark remains for ever, unless you explicitly Unlock the files using the Properties tab in Windows Explorer.

So the .NET runtime was confused into thinking they were remote files.

So I hope this helps someone else.

Now to fix the other stuff that the upgrade has broken - like all the service references..

More Government Unhelpfulness

2009-10-24T04:08:00.000-07:00

It's that happy time of year when I celebrate the anniversary of setting up my own company and the fact that I have made it through another year. Which means the usual round of fees and insurances (including of course, the mandatory employers liability insurance even though I am a one person company, just in case I sue myself for emotional damages) and the attendant paperwork, shareholders meeting and submitting my Annual Return.

Normally this is a simple procedure, since the only share holders are myself and my wife. However, Companies House have now added a new box - from this month - in which you must 'disclose the particulars of rights attached to any shares'.

Whatever that means.

From the limited blurb I get that this covers such things as voting rights, dividend distribution and the like, all as per the articles of association (you know, that long tedious piece of unpenetrable legalese you were sent when you bought your company from a web shop). Since this is a new provision that affects EVERY limited company, from the largest to one-person companies like mine, you would think that Companies House would offer some guidance on how this is to be completed or what they expect it to contain.

Wrong.

It's just a memo box. No guidance, template answers, standard wording, nothing.
Above all, nothing to suggest the level of detail they are requiring.

Internet search shows everyone else - including accountants - are equally confused.

What level of detail will they accept? Is a simple statement of 'all shares have equal voting rights and dividends are divided on the basis of shareholding' sufficient? Or do you need to reproduce all the relevant text from the articles of association? Which they have anyway, so what exactly is the point?

One small grain of comfort: after the website has predictably timed out whilst you try to find this garbage, after searching once again for those auth codes you filed back after logging in, you can at least resume back at the page where you left off.

And, correct me if I'm wrong, the point of the changes to the companies act was to make life simpler for small companies.

U2 University

2009-10-16T01:14:00.000-07:00

I'm just back from another U2 University event, this time held in the rather surprising location of Liverpool. This has proven to be another well run and highly informative symposium, with a broad range of talks spanning such diverse topics as Data Encryption, Web services, a talk by Susan Joslyn on standards and regulation, and the chance to hobnob with old and new friends. The U2 community is a tight-knit and friendly one, and so the opportunity to get together is always welcome.

The news about the acquisition seems to now have been accepted as a 'good thing'. Panic is decidedly over, and there is a strong sense that everyone wants to make this work.

The second piece of good news is that the product continues to move forward, keeping with - or in many cases well in front of - the technology curve.

In particular this year the spotlight has been on data security. As companies in the US and beyond grapple with PCI compliance, UniVerse and UniData are ready for the challenge with SSL being rolled into the final pieces of the middleware jigsaw (more below) to protect data in transit, and the now mature automated data encryption to protect data at rest.

One side effect of this will be a major change to WebDE (formerly RedBack) to fully meet with the challenges of encryption and NLS. Rather than beef up the legacy RedBack code that makes up the scheduler/responder architecture, IBM/Rocket have taken the sensible decision to move the whole thing over to UO.Net (RedPages) and UOJ (RedBeans).

This will then use connection pooling in place of webshares, which introduces a few potential issues:

the question of how to licence existing customers transferring from multiple web servers, since the connection pooling will be managed in the middle tier and not at the database.

the effect on upgrading method calls where subroutines are cached in memory (UniVerse) without bouncing the web servers themselves.

IBM/Rocket are aware of these issues and are working to address them. The move in architecture should lead to quicker response times, and better focus as IBM/Rocket consolidate all the main architectures (web services, WebDE etc.) into a single middleware family in UO.

Top of the list on cool new technologies must be the introduction of EDA to UniVerse. EDA (Extensible Data Architecture) allows U2 products to directly map data at the file level to external data sources using a driver structure based loosely around the ODBC concepts. Once in place, this becomes entirely transparent to the application. Files are mapped individually, and - a very good piece of thinking here - only specific columns need to be mapped, with the addition fields being stored as a long string or blob in dynamic array format. This reduces the overheads on type mapping and updates.

Clearly IBM/Rocket have been thinking ahead with this one. In addition to moving data out permanently, they have also realized the potential for replicating data though this gateway to external databases for BI and reporting purposes. This uses the existing replication architecture to set up a publisher/subscriber model that
may reside in the same machine, and directing the subscriber to push data through EDA to a remote data source.

The fact that you do not need to individually map every column, also opens up the opportunities to use this as a general gateway - something I have asked for long ago. I can see this opening up services such as continuous backup and more intelligent auditing, since the drivers do not necessarily need to be directing to data sources.

EDA is already present for UniData with an initial driver for DB2: now that the link with IBM has been dissolved, the way is free to extend this more quickly to other databases such as SQL Server and mySQL.

UniVerse will gain the new facilities in version 11, which is being slated for release next year.

All in all, a good event and my thanks to the U2 team for organizing it so well.

Rocket Launch

2009-10-01T01:27:00.000-07:00

Rocket have posted a news flash here about the acquisition. The deal has closed, and the U2 products are now under new ownership.

This promises to be an exciting time for us all. The potential for this platform is huge, especially in nervous financial times, and we all wish the new owners and all of the U2 teams involved the very best.

Expect to see the U2-sphere bursting with excitement as the news can now be posted.

This should be a good transition: Rocket has all the pieces in place to really set this product on the right path and let it flourish. THe people who have managed the products through the IBM years (not forgetting those still with IBM) have done an excellent job despite the constraints, and whilst we don't know what constraints Rocket will put upon them, everything heard so far seems good news.

Let's make sure we all give these guys our full support.

Update on the Update

2009-09-30T09:08:00.000-07:00

I've spoken with IBM UK and there is good news: it appears the rumour of legal action is nothing more than baseless speculation - there is nothing planned outside of the usual TUPE requirements. So that can be well and truly laid to rest.

General Update

2009-09-28T02:58:00.000-07:00

Plenty of opportunities for curmudgeon this week, after staying silent for a while. So a general round up seems in order.

U2 and Rocket

Well the dust is settling after the shock announcement by IBM of the sale of the U2 assets to Rocket Software. The feeling in the community seems to have settled down to a grim resignation offset by a feeling that we should give these guys a chance. At the same time, the sale is still underway and not closed, and for the time being the U2 products are still part of IBM, which limits what any of us can say (IBM lawyers haven't gone on strike as far as we know).

Still, some useful information (or rumour) seems to have trickled out.

Here is what people are saying:

It seems that the push came from within the U2 division.

IBM was not planning on getting rid of U2, but U2 wanted out of IBM.

All the U2 staff have been invited to join Rocket.

The senior U2 management will be moving with the product.

There were nine companies in the frame for purchasing U2.

Rocket was chosen because they offered an opportunity to take the products forward.

Rocket has a 'hands-off' style of management that should let the products breathe.

Most of the U2 people in the US are excited about the opportunities.

So there we have it. I can possibly look forward to editing all of my books, course ware, articles and other materials to remove the IBM name. No doubt screen shots and registry locations will change yet again. And there is still no solid news on what will happen to the ADO.NET provider in all of this.

I will post more when I have spoken with Andy Youniss, the head of Rocket, this week.

FSA Transaction Reporting

On the consulting front I'm embarking on a project to boost the FSA transaction reporting and compliance for a client. Not that their reporting is bad, I hasten to add, but the FSA is getting very tight on this with the announcement of a GBP 2.4 million fine for Barclays for misreporting, and the knowledge that they will be increasing their fines for companies and individuals is getting people worried.

Still, I must say I'm looking forward to this project - it should prove exciting!

All this getting tough is about time, some would say, but typically there is a fly in the ointment. Whilst the FSA puts great responsibility on firms to ensure that their reporting and controls are accurate and robust, the same cannot be said of their own guidelines. The FSA have now released a new version of their transaction reporting handbook, which explicitly excludes any guidance for data formats, and their web site is minimalist in the extreme when it comes to pertinent information. Whilst they will monitor companies to ensure they have sufficient controls in place, they offer no guidelines or advice on what they expect and what they would deem to be adequate.

This recalls the very first project I ever worked on, as a young and green analyst/programmer. I was hired by a company producing software for local government at the time the dreaded and hated Community Charge (aka poll tax) was brought out in a flurry of hurried and badly formed legislation, and I was accordingly set to work on that, taking over the development of the commercial side (NNDR) a year later. With the political to and fro the legislation was subject to constant revision, and whilst the key dates remained in force and enshrined in law, the guidance from the goverment lagged behind - in some cases by months.

The first legally enforced deadline was the registration period, during which local authorities had to ensure that all properties and individuals within their area were registered onto the new systems ready for billing, which AFAICR was in the April.

At that time (we're taking early eighties and PICK operating systems) most local authorities lacked the resources to print out and despatch the registration forms inhouse, and so half inch tapes containing the necessary details needed to be despatched to the printers some time in advance. This was duly undertaken and the registration completed on time.

So what was the problem with that? The guidance from the government for what information was required to complete the registration came out some six weeks later.

LME SMART

After long planning, various LME member firms are going live on SMART as we speak. SMART is the new interface to the LME trading platforms, and a project that has occupied me on and off for some months, which has meant radical changes to the ways in which trades are submitted and followed through their various registration, matching and clearing states.

This has seen a switch from the synchronous LCH protocols (where everyone knew where they were) to an asynchronous FIX based solution providing feedback using execution reports. The change was necessary with the new division of responsibilities between te LME and LCH, and as part of the LME's ongoing work to spearhead new development of their systems.

The advantage of course is that FIX is a known quantity, and the messages can be based on the FIX standards (with the usual 'tuning' to the specific industry that seems always to take place). The result has been the creation a general purpose FIX engine with validation and extraction layers that can be adapted to this, and to future FIX based interfaces.

So far, touch wood, the roll out seems to be going as planned, though the LME have 'temporarily' issued a severe throttling rate for messages to ensure their systems are not overwhelmed during the early phases. Good idea that, introduce a protocol known for its speed and then throttle back. Fortunately, forewarned is forearmed and so the solution we put together has the necessary throttling built in, but it's hardly a glowing endorsement.

IBM Sells U2 (more)

2009-09-16T09:29:00.000-07:00

Tony G has written an excellent blog that distills some of the initial reaction and brims with perceptive comments. Read it here.

Another thought just struck me: maybe they will do away with the 'U2' moniker. That would be such a relief. As long as nobody there thinks 'rocket base' is a good alternative..

IBM Sells U2 Products

2009-09-16T02:26:00.000-07:00

Well the news that greeted me this morning was certainly a surprise. IBM is selling off the U2 brands - UniVerse and UniData - to Rocket Software.

The loss of the IBM branding will a hard blow for the U2 community.

On the other hand, there are silver linings to this cloud. The same people will be involved in driving the product forward, and now unshackled from the restrictions of working through IBM. What restrictions Rocket apply will remain to be seen, but there is positive news in this.

Yes, there are already murmurings, but let's give these guys the benefit of the doubt. They have chosen U2, and so are coming in with some commitment. IBM have done a good job in the work they have done with the product, especially with the XML features. But they could have gone much further, and marketing always seemed compromised by the DB2 banner.

There are practical questions raised, of course. What happens to the ADO.NET support, since that is a generic IBM provider? What happens to training - does this mean that independent trainers like myself can now get a foot back into the market? What about strategic partnerships for products? Rocket already have a portfolio that includes reporting and data migration, amongst others: perhaps they will realize the value of the tools that face end users, or maybe they have their own plans for that space.

It could have been worse. IBM could have sold off the U2 products to one of the major vendors (assuming they had the money), which would have fractured the community. Rocket is, to the best of my very limited knowledge, independent in this regard. They could have sold it to another database vendor, with competing products: once again, through Rocket have storage solutions, the U2 products look like a brand that could carve a space in their portfolio, if the will is there to do so.

I just hope that the move to Rocket opens up as many opportunities as it closes.

Fingers crossed and keep faith.
We've been through this before, and U2 has remained intact.

C# Finger Stress

2009-08-29T02:07:00.000-07:00

I hear that my comments on Silverlight found their way to a TEXMUG meeting. Thanks for following along - I never thought anyone read this stuff! So I'll be posting more on my travails with Silverlight and UniVerse as I go..

But now for more pressing curmugeon:

I spent yesterday debugging some old code.

This was for a system I had written some four years ago for a client: it has been working and live throughout, but in the way that things sometimes happen they had hit an unforeseen circumstance ("events, dear boy") that had caused a hiccup. Nothing major (since I'm blogging live about my software!) but a once-in-four-years annoyance and four years on is probably a good time to review code anyway.

The interesting thing for me was that this was written in VB.Net.

Now I switched to using C# pretty much exclusively for my .Net coding soon after that and today all my .Net work is in C# - and so looking back at some VB.Net code was a pleasure. It is so much easier to read and to debug, especially when coming back to code you haven't seen in a long time.

Intellectually, I know that there is nothing that you can do in C# that you can't do in VB.Net, and that under the hood they are effectively the same. Yes, I have heard the argument trotted out that C# is quicker but that was only the case back in 1.1.

In fact, there is nothing (apart from the XML auto-documentation) that should really cause anyone to prefer C# over VB.Net. Particularly if considering the entire product lifecycle: if everyone who developed were forced to support their own code, and they might think twice about intelligibility!

Sadly, industry doesn't see it that way. Look at the Microsoft marketing, and it is clear that VB.net is positioned as a hobby language and C# as a tool for professionals. Well, they had to appease the C++ community somehow, I guess. C# is the language par-excellence for geekdom, and if you can't churn out some decent lamda expressions to create anonymous types and delegates in C#, you're not even on the first step.

So why did I turn to C#? Because it just makes me seem more professional and in the some of the areas I deal with, more of the examples are in C#. Clearly, VB.Net programmers are not supposed to do complicated stuff. Look at the blog-sphere on things like dynamic type creation if you don't believe me. Chances of finding good examples in VB are down there with articles about MultiValue on CodeProject.

MultiValue databases also use a variant of Basic, and just like VB.net this is a far cry from the old Dartmouth stuff. MultiValue Basic - in all its various forms - is a hugely capable language specialized for business processing. It is not perfect, but it has stood the test of time and offers tremendous advantages in terms of focussed business performance. It is also an elegant language for expressing business rules, and the use of external subroutines and functions as the main building blocks of an application makes it highly flexible and adaptable. It lacks OO features (and who really needs them in server procedures?) but even this is being addressed by OpenQM in their version.

But the same snobbery rears its head when you talk about UniVerse Basic. Oh, it's Basic, it can't be a real language. Urm, you must mean compared to the syntactic richness and highly developed business features available for writing stored procedures in the likes of TSQL? Yeah, really. And if run time typing is a problem, take a look at PHP.

Good MultiValue code - written with an eye to maintainability - is a joy.

[Note the word 'good' in that sentence. There is plenty of unreadable and unmaintainable rubbish in the MV world as well. But there is less excuse.]

Certainly, VB.Net is more legible and syntactically simpler than C#. Is that enough to relegate it to a hobby language?

Well, aside from the debugging yesterday, I've been using the two side by side for some time building the example code for a book. One thing is clear: my poor fingers definitely prefer VB.Net! For productive, professional programming the layout of C# is simply not keyboard-friendly. And given the relative extents to which you need to cast types in C# over VB.Net, it is not necessarily shorter, either:

MyAwkwardTypeName myType = new MyAwkwardTypeName();
MyOtherThing myThing = (MyOtherThing) ((MyContainerClass) someObject).property;

Couple that with the huge role played by property gets and sets (and now I'm using Silverlight, the fact that so much of the UI is based on data binding to properties and decorating the resultant code with endless attributes) and I wonder whether anyone has researched into the effect of different computer languages in creating RSI.

Take a (very) simple construct like the one below:

[DataMember]
public Boolean myProperty{
get{ return _myProperty; }
set{ setMyProperty(value); }
}

Now type that out twenty times (for different properties, duh). Remember the intellisense, so you can make use of the cursor and tab keys as well (or Esc to stuff it out of the way again). Do that often enough and your fingers will start to feel the stress. However good a typist you are - and I'm not, as anyone who has seen live demos of my software will attest.

If there is a league table for RSI, I bet C# is at the front of the list.

But I'll probably keep using it and grinning through the pain. That's the price of geek-dom after all.

Hating Silverlight

2009-08-22T03:21:00.000-07:00

It's official. I hate Silverlight.

Or at least, I hate some of the Microsoft'ed-ness of it. A great idea, excellent technology, and yet at every step you hit your head against a wall. And with Silverlight 3 it's a different set of walls to Silverlight 2 and Silverlight 1..

Why?

Here's a typical piece of Microsoft thinking:

We have decided in our wisdom that all syncronous calls to services are evil. So you can't do them. The only service calls we will support are asynchronous. So there. We don't know about you, we know nothing about your applications, but we know better. So we are not going to give you the choice.

Now don't get me wrong: I understand the arguments for why async should generally be preferred in many situations, and I fully intend to use the async model where appropriate. But I'm intelligent enough (I hope) to know when and where to use these. Like with sockets: you can (still) choose to make a socket call asychrounously or syncronously depending on the context. At least there I have the choice.

But not with Silverlight. No, never, we know best.

Now, I need to do something very simple at the start of one of my applications. I neeed to present a login dialog with a list of connections provided by a service, request the connection name, user name and password, and send these back to the same service for validation.

Here is how it works in a Windows Forms scenario:

The user clicks a Connect menu item.

This fetches a list of connection names from the service using a remoting call.

This then creates the connection dialog, and populates the drop down list.

The connection dialog is displayed modally.

On acceptance, the details are passed to a Connect function on the service via remoting.

All handled in a single function, so easy to follow and to debug.

Now the Silverlight version:

Well, first of all there is no synchronous call to get the connection names. So the user clicks Connect, I fire off a async call to get the connections, and lock the UI in the meantime since they can't legitimately do anything else at this point.

A second event handler is fired when the call returns, so at that point I can get hold of the connection names. Create a child window, populate the drop down list of connections and show. Except that this doesn't support a modal show (just what is the point of a dialog that can't be shown modally?)

So a third event handler to pick up the dialog close. Now I can call the connect function - once again as an async call.

Which means coding a fourth event handler for the connect call response..

And the code becomes more and more fragmented. And that is just for one simple action.

So I'm left with a choice: carry on writing the entire application using the async model with reams of functions cluttering up the place until it all becomes an unmaintainable mess, or write another pointless container for the service calls that will perform these on a non-UI thread, blocking as it goes to simulate syncronous operation by using a ManualResetEvent:

myEvent.Reset();
client.getConnectionsCompleted += (sender, args) => { myEvent.Set(); };
client.getConnectionsAsync(new ReportStudioSL.RSMainService.getConnectionsRequest
(connections, errText));
myEvent.WaitOne();

and so on and so forth for every call that needs to be logically handled syncronously. And there are quite a few of these...

So today, I hate Silverlight. Officially.

Microsoft Patches Linux

2009-07-23T07:31:00.001-07:00

So Microsoft have reportedly released 20,000 lines of code for the Linux kernel, to improve performance when hosted under their Virtualization technology, Hyper-V.

Cynical: possibly. Self-serving: definitely. Useful: quite probably.

No doubt there will be rants from the anti-Microsoft fraternity, but this is open source and anyone can play. The code should be treated entirely on its merits, regardless of who posted it. If it is good, accept it. If not, reject or improve it. That's what it's all about.

I run Linux under Microsoft's Virtual PC, so I'm not complaining.

Painting Pictures Part 2

2009-06-26T09:45:00.000-07:00

Just to follow up from the previous post, I now have the beta version of the new mvScan ready, including the graphics generated from UniVerse. In the event I also decided to put a Windows front end onto the map files, but I expect the generated pages to be the most useful resource - and they don't require a database licence.

So here are a couple of shots of the new charts in action.

First a field listing, breaking the dictionary out into its various associations:

And a logic diagram following from menus through executed programs and called subroutines:

Naturally these are all hyperlinked alongside the text, so developers can drill around their systems.

So I guess I'll be writing a Spectrum article on UniVerse and graphics in the near future.

Painting Pictures

2009-06-21T04:40:00.000-07:00

As part of the grand scheme of releasing new versions of all my products in advance of getting down to completing my new reporting tools, this week I've turned my attention to mvScan. This is a set of analytical tools for scanning a UniVerse system and determining all the impacts: programs accessing files, subroutine calls (including external function calls), routines called from dictionaries, PROCs accessing files, tools called from menus - all the bits and pieces that make up a live UniVerse application.

The analytical part derived from some tools I've been using in a consulting mode, and so when I came to take a new look at them last month, they were capable but not particularly user-friendly. And whilst they generate a goldmine of information, navigating that takes some work. So I needed a new way to present the results.

Part of that job was already done: the tools produce technical documentation in the form of HTML pages - handy since anyone can view them and they can drill up and down through the relationships. But I wanted something more graphical to present these links as a chart or diagram.

UniVerse and Graphics

Now whilst UniVerse has plenty of APIs for working with data and for interacting with remote systems, graphics has never been part of the mix. Normally I would tackle this by adding an external library, something along the lines of mvPDF that runs as a service backed by calls made from UniVerse Basic. Using this technique I could simply harness existing Windows graphics libraries to generate images. But it seemed a little cumbersome for what I was looking for, and in any case there is an additional complication in the fact that I want these to include the same links. Clicking a subroutine in a diagram should take you to the map for that subroutine, and so forth. This would mean not only creating an image, but an image map to overlay, which seemed to be the wrong choice.

So I decided that I needed a way to generate web images directly from UniVerse Basic. That means using a format that can be easily assembled by a language that is predominently string based and lacks the sort of feedback required for image placement.

Which leads to two real possibilities: markup languages like XAML or a dedicated image language like SVG.

Vector Graphics

I had not really appreciated SVG (Scalable Vector Graphics) before this exercise. This is an XML based definition of - well, scalable vector graphics. And it has the great benefit of being extremely simple.

SVG is presented as an XML document. Despite being a W3C recommendation, it is noticeably absent in Internet Explorer, but can be viewed using a free plugin from Adobe. The syntax consists of graphics primitives: lines, rectangles, ellipses, polygons, polylines, paths and text. Graphics elements can be combined into reusable definitions - perfect for a diagram holding a limited pallate of symbols. And the rendering of those elements: fill, stroke, colours etc. all follows regular CSS. Even better, you can include regular HTML style anchor tags to provide hot-spotting.

So I'm now working towards my goal of generating diagrams directly from UniVerse Basic. When the new evaluation version of mvScan is released, you can see how I got on.

Testing Times

2009-06-14T05:01:00.001-07:00

This week has been a flurry of activity as I've taken advantage of an enforced break from contract work to bring a number of my products up to date. Over the next month or so, look out for new versions of mvPDF, mvScript, mvInstaller, mvDeveloper and some new utilities such as Conduit, which is an export pipeline between IBM UniVerse and SQL data sources.

First out of the gate will be a new mvScript, with a completely rewritten connection handler to take advantage for the first time of the IBM Pooled Connection licences and with the first round of support for QM, which I'm getting to like more and more as I work with it.

A large part of my development philosophy has always been to reduce the amount of manual intervention required to get a product through QA and out of the door. For example, when building a product I run cutting scripts that will rebuild all the source on the server, reversion everything, generate the server side setups (using mvInstaller, naturally), then progress to updating any resources in the client, build the client and package it all up in the shipping directory. All of which is automated to the point where it should happen consistently every time.

The same is true wherever possible with regard to testing. I've already written elsewhere about how I try to have all my server subroutines use the same calling convention, making it easy to write test rigs that can stack in arguments and unit test by checking the results. Added to that, products like mvScript have their own test beds: for mvScript this is a set of pages that cover every function and statement in the language and the runtime facilities such as the emulated COMMON blocks.

But with the extension to the connection pool, I now have another layer to test. And with two connection pools, new database support for QM and three connection models (pooled, connect on demand or pooled licencing) that is new raft of things to automate.

Using CallHTTP

Fortunately, once more IBM UniVerse comes to the rescue. UniVerse is stuffed full of different APIs for performing all kinds of useful operations, usually representing an easy-to-use abstraction of some underlying protocols. The socket API is a case in point, providing a single high level model for socket communication that irons out some of the annoying difficulties across different platforms (as anyone who has coded socket routines on UNIX will attest).

One of the APIs that is built upon the socket support is CallHTTP. As the name suggests, this provides a quick and simple route for making HTTP requests: spanning the various HTTP methods (GET, POST, TRACE etc) and allowing form post variables to be sent along with the request. Perfect then for soak testing the connection pool.

So here's how I'm now testing the new mvScript connectivity. I have a file of requests, holding details of pages to fetch (URL, query string, post variables, additional http headers and proxy data - not all of these needed, but I'll allow for them for future operations). Then a file of responses, that will capture the HTTP status, headers and data returned from the calls, along with any errors from the calling routine.

Next, to run this I will need to fire off some phantom processes to ensure that requests can be sent concurrently - there is no point testing connection pooling with sequential requests! I will be slightly limited by the need for each of these to consume an iPhantom licence since the request involves a socket call, but so be it.

Then to the request routine itself, which is laudably simple since I'm not using secure socket connections, which take a bit more work to set up.

The first step is to configure the HTTP environment. This sets the HTTP version (1.0 or 1.1) and details of any proxy servers and/or ports, using the setHTTPDefault function:

Now the call is set up in two stages: first the request is created and any parameters attached, and then the request is submitted.

Creating the request requires the URL and HTTP method:

The hRequest is a handle to the new request, to which you can add new HTTP headers calling the setRequestHeader() function, and form post parameters through the addRequestParameter() function.

Finally the request can be submitted, using the submitRequest() function. This requires a timeout in milliseconds, and returns the HTTP status, HTTP headers and data in three separate variables. You can supply additional post data as an alternate to using the addRequestParameter above:

Simple. So now I'm off to soak test my new connection pool.

mvDeveloper Interface

2009-05-28T04:16:00.000-07:00

Friends of my free Windows editor for MultiValue platforms, mvDeveloper, may be pleased to learn that I've finally got off my backside regarding specifying a new interface API.

mvDeveloper was designed to be as lightweight as possible, allowing me to carry it around on a USB stick and connect it to any passing MultiValue database (generally UniVerse or UniData) as required by my work. Specifically, it was designed such as to require nothing special installed on the server side.

However, this cavalier approach does not sit with those sites who have implemented rigorous (or otherwise) source code control or source management systems. So for the current release I added Handler subroutines, that can be called for read/write and build operations, but really that was something of a stop-gap approach.

So at a timely prompt from Susan Joslyn of SJ+ Associates, the author of the highly esteemed PRC system for code management, I've written a proposal for a generic interface that should hopefully meet the needs of any such systems out there.

If anyone is interested in commenting, please drop me an email at the usual address.

Transformations

2009-05-10T02:33:00.000-07:00

Out here in the Oxfordshire countryside, the last few weeks have been a time of huge changes. Only a month ago the trees and thorn bushes along the river that winds its way around the village, were bare and lifeless: then two weeks in which they were transformed, covered in white blossoms - at first incongruous and then glorious. Then, just as quickly, the blossom was gone and in there place a vista of green: how many different greens there are in the world!

In the Chinese Five Element system, Spring is the time of the Wood Element and signified by the colour green: a time for growth, momentum and transformation.

It has been a time of change here as well. Another financial year survived marking four years working for myself, and the end of a long and enjoyable contract that had seen me back working on commodity trading systems, which is where ADG and my adventures with UniVerse really began. And now bringing with it a chance to spend some time working on my products again (at least until the next job comes along) - and transformation is the theme here once again.

First up for new treatment is mvPDF. I'm launching a new "mvPDF Lite" for these credit-crunch times, offering the most requested features at a bargain price: transforming existing MultiValue reports into PDF documents and Windows printouts, by using spooler capture and a page designer to generate new backgrounds, and now with improvements to the PCL parsing and support for EPSON/Brother/HP compatible barcodes.

More exciting, I'm also adding report transformation into the product, based on software I originally designed for mvQuery. This captures existing reports and strips the report content, allowing this to be reformatted and extended to generate new and more current reports without reprogramming. For most companies, the existing reports represent a substantial investment in time and knowledge, and throwing these out to replace them with new reporting solutions - however glossy - is simply not a viable option. mvPDF and the new mvPDF Lite will allow clients to retain this investment, whilst transforming a vital part of their instrastructure.

The second transformative product that is getting a new round of attention is Conduit: a program I wrote for a customer again to assist with their reporting, that makes it simple to strip data from a UniVerse system and to post it to an external data source: typically SQL Server. Conduit runs as a Windows service, generating the data for export from dictionary driven defintions, and with sufficient control to handle both batch and continuous updates, and the ability to generate and call stored procedures to optimise the posting performance. The transformation in this case is not just between data sources: the Conduit data engine is a powerful beast for generating real and virtual data and handling the necessary normalizng of UniVerse data without the need to use the clumsy UniODBC drivers.

So whilst the contract work dies down I won't be twiddling my thumbs.. time to get down to some serious product development and hit the infinite to-do list while the Spring is here.

Writing in mvScript

2009-04-26T02:52:00.000-07:00

Okay, I'll admit it: mvScript was written for fun. But Tony G has recently hassled me about why I don't market it, which is a nod in its favour, so I guess it's time I told a few more people about it. So here we go.

I wanted an easy way to create dynamic content from UniVerse – mostly web pages, but possibly other content such as XML for AJAX – without having to tangle with the cost and overheads of existing web transports. Also, I wanted something that felt natural to UniVerse (and potentially to other MultiValue) developers. Thus mvScript was born.

mvScript, as the name suggests, is a scripting language. Similar to ASP or PHP, it is a language that can generate dynamic content for web pages. There are two significant differences however: first, mvScript is compiled and interpreted on the database server; and second, mvScript is modelled after UniVerse Basic.

In this post I am going to give a very brief introduction to mvScript: which I believe to the be the simplest way to create websites for MultiValue platforms. You can download an evaluation copy, along with the demonstration database, from http://www.brianleach.co.uk/. You will need to create a virtual directory in Internet Information Server to host your pages. Once installed and configured you should be ready to play.

Creating an mvScript Page

Pages in mvScript are generally created as regular HTML or XHTML. You can use your favourite web design tool (I use Microsoft Expression) or even Notepad. So let's begin with a simple greeting:

The scripts are identified by the tags < % and % >. The code between should seem familiar: these are regular UniVerse Basic statements to show the text and current time in external format. To ensure that this is handled by mvScript, save the page in the virtual directory with a wsp extention and request it in your browser making sure you give the full URL so that it runs through IIS (if you just give the file location, it won't). Each time you refresh the page, the time will change.

So how did this work? mvScript pages are passed through an IIS DLL and compiled on demand on the server into an intermediate code format. This code is interpreted by a run machine on the server and the resulting page returned to the browser. The Crt statement in Basic displays to screen: in mvScript Crt (and “=”) update the dynamic page content.

Performing Data Operations

The advantage becomes clear when you work with UniVerse data. Each page requires a single round trip to the server regardless of how much it needs to interact with the database. mvScript uses familiar constructs to access the data in your system: opening files, reading and writing records, handling dynamic arrays. See how we can show a record from the demonstration database:

Entering Values

The page above is less than useful in the real world: it only displays a fixed id! We really need to enter the id or to select it from a list. Web pages have two routes for supplying data: the querystring or form post variables. The QueryString is the key/value pairings part of a URL that sits after a question mark, as in: http://mysite/mypage.htm?key=value.

mvScript can access the querystring and form post values through a Query() and Form() function respectively. So you can extend the page to fetch a title id by adding: id=Query(“id”), and call the page adding the id to the URL.

Calling Subroutines

mvScript, like all good front-end MultiValue tools, is designed primarily to call Basic subroutines. Any client/server or web based application should be backed by subroutines. These offer the fastest, most direct route to handling data and can be most easily tested. A well designed subroutine library builds into an API that can document the interface to your application: one that can be reused across a variety of interfaces, both current and future, to protect your investment.

Having created a page to view a title record, now we need a way to select candidate records. Because of space restrictions I'm going to build these into a table in Basic:

Now we can host this in an mvScript page. mvScript uses the same Call syntax as UniVerse Basic, so you can hide the complex parts away in your UniVerse Basic code:

So there you have it – the start of a simple application. Not pretty, but mvScript pages are fully formed web pages in their own right, capable of using all the ususal web adornments including style sheets, javascript and AJAX, whilst keeping the important part – the database logic – close to home.

For any seasoned MultiValue developer, it doesn't get much easier than that.

The Return of ODBC

2009-03-29T06:40:00.000-07:00

Interesting news from Redmond this week. It seems that after a decade of dormancy, Microsoft have been looking once again at ODBC. Even redeveloping driver code and extending the specification. Yes, that's right - the cross platform database access protocol that they dumped like a hot potato in favour of OleDB/ADO/ADO.Net and whatever is the current flavour of the month, is now back on the agenda.

Perhaps their data strategists finally just ran out of acronyms and needed to recycle.

Good news? Not necessarily.

The story of ODBC in UniVerse is - well, mixed.

Today, there are two ODBC centred routes in UniVerse: the BCI (Basic Calling Interface) and the UniODBC driver. And they are both very different beasts.

BCI is one of a batch of protocols for making calls out from the server. Remember the key thing about MultiValue databases in general, and IBM UniVerse in particular, is that the business processing is handled within the database: no three tier solutions required. So BCI is in a genuinely useful place. Using BCI, developers can make calls to external ODBC data sources, and perform all the regular ODBC style operations: fetching data, running inserts, creating and hitting stored procedures. And as ODBC goes, it's pretty quick, though for UNIX based UniVerse it requires third party ODBC drivers from companies like Merant and EasySoft.

The UniODBC driver runs in the other direction. That is used for calling into UniVerse from external systems (and into UniData also, using a different model). This is the latest in a long line of ODBC protocols that began with a third party product called HyperStar, which explains the naming behind many of the commands and control files that mostly begin with 'HS'.

Now these products reflect the twin natures of ODBC with UniVerse, and the problems faced when trying to integrate MultiValue data with the outside world.

BCI runs within UniVerse, and so has direct access to UniVerse data and the multivalued data structures. It is driven by code, so developers can use all the facilities of the language to process that data, either for posting or upon retrieval. Any normalizing that needs to take place to accommodate primitive data tables, can be handled through code and tailored exactly to the circumstances. If you're looking to export quantities of data to third party systems, BCI is a very attractive route.

UniODBC on the other hand is driven from the outside, MultiValue-intolerant world. So the database has to present UniVerse data as if it came from a standard relational system. That means it must 'dynamically normalize' the data that is presented: multivalue associations are automatically presented as child tables, to be joined back to the parents as desired through the ODBC SQL. On top of this there is a lot of administration to be undertaken: dictionaries need to be cleaned up, preferably replaced with reporting views of the data; associations fully defined; data types specified.

Now, interestingly the SQL that runs through UniODBC is not the same SQL that runs inside UniVerse. Despite being a MultiValue database, Universe also has first line SQL capabilities built, once again, directly into the run machine. The native SQL is
something I will be touching on later, as it is the most under-utilized feature of UniVerse: largely, I believe, because of the experience UniVerse developers have had with the ODBC SQL. Native UniVerse SQL is multivalued, and handles the native structures with aplomb making it fast. It also understands the dictionary structures, including virtual fields in the same way as the MultiValue enquiry language, RetrieVe. So for many tasks it can represent the best of both worlds.

The UniODBC driver, on the other hand, has to work hard to translate between these two models, which means overheads in terms of operation and complexity for developers. It can also hit the lock table if used poorly.

In recent years, changes to the world of data access have moved away from simple tabular data. In particular, the rise of XML as a transport has been a huge benefit to MultiValue systems, since the XML schema more neatly encapsulates the nested entities present in MultiValue data. The ability to export data without having to
normalize it actually puts MultiValue at an advantage since the native features can come into play.

So whilst some improvements in driver manager performance may be a good thing for many, the re-emergence of ODBC may not necessarily be such a welcome sight.

Fundamentals of MultiValue (6)

2009-03-24T03:49:00.000-07:00

The View from 20,000 feet

In this mini-series I have tried to highlight some of the differences between the MultiValue model as expressed by IBM UniVerse and IBM UniData, and the mainstream relational database model. By the nature of a blog these posts have been high level summaries: as the blog progresses many of these areas will be plumbed in more detail.

Even so, we have covered a lot of ground and so I thought the time had come to further summarize where we have reached. Call this an Executive Summary, if you like.

Put simply, Why should you consider UniVerse for your next project?

MultiValue is Ubiquitous

The first thing most people say when asked about MultiValue is - "never heard of it".
Don't let that put you off: it may not be as pervasive as SQL Server, but you will find MultiValue applications in every industry, every government sector, every size and shape of application.

So why don't you get to hear about it? Two reasons.

First - it is simple, works and is cheap. That's bad news for industry commentators, consultants and anyone who earns a living from technology. Running MultiValue is easy, getting return on investment for running your business on MultiValue is quick, but making money out of MultiValue is hard.

Second - it is classed as an 'embedded database'. That means you don't - and shouldn't - see it. All you see is the application and the database just works.

MultiValue is Stable

The MultiValue data model has been powering mission critical systems for over 40 years. In that time it has been improved and extended but it survives because it got the key things right.

Application Engine

MultiValue servers are application engines and database systems rolled into one.

This makes for a low cost of ownership: no need for three tier solutions, dedicated application servers and complicated rollouts. Inside the MultiValue environment, data and business processing share the same space: making it quicker, cleaner and faster to respond to changes in the way your business operates.

Low, Low Administration

Remember what I just said about an embedded database? These are systems that keep running invisibly, and cannot afford to stall. Did you know that most MultiValue sites have no dedicated DBAs for their MultiValue systems?

No wonder the commentators don't like it.

Real World Data Model

The secret behind the technology: MultiValue data models the real world.

No messy and costly joins, no need to normalize your data, no need to scatter information across hundreds of tables only to put it back together every time you come to use it. Making your systems easy to understand.
MultiValue data storage is flexible, scalable and resource friendly.

Virtual Data

Why store what you can derive? And why force developers - and power users - to write complex enquiries to perform even simple calculations?

Information in the MultiValue world is open through the Dictionary: a reusable store of definitions giving you the information you need, without resorting to costly and time consuming programming. Give your business professionals their own dictionaries and they can build the information they want to see in the way they want to see it.

What's in a Name

The word "MultiValue" was coined to address the fragmentation a generation of vendors had made of marketing the model.

But today, one of the world's most important names stands behind the leading MultiValue platforms.

That name: IBM.

Fundamentals of MultiValue (5)

2009-03-14T04:27:00.000-07:00

In the previous blogs in this mini-series I have outlined the file store, metadata and the enquiry languages that form three of the four pillars of the MultiValue model. In this blog I will look briefly at the remaining pillar: the development language.

The MultiValue model is not just about data. It's all about what you do with it.

Modern databases are generally tied to one task: fetching and storing data. Some of these may use stored procedures to empower this task, but they are not complete application engines. If you want to build a business application, you need to invest in n-tier solutions and application servers to hold your business logic.

That is hard work, and expensive too.

The MultiValue model puts the application at the heart of the database. Away with three tier solutions: at your disposal you have a fully formed, high level business language that can handle all the complexity of real solutions. By keeping the business logic in the same space as the database, you not only reduce your costs and simply the development, you can gain in efficiency and scale.

I'm not going to go through the whole development language here. For that, you can download my free primer here, along with the free Personal Edition of UniVerse from IBM.

But there are some elements of the language that are pure joy, so I'm going to share some of those.

Direct Data Access

Updating the data store is a fast, low level operation. No intervening SQL to worry about: just read a record by specifying the key, change it and write it back. To add a new record, just use a different key. Simple and fast. And let the database take care of the rest.

Locking and Contention

Contention management is the achilles heel of many a relational application. Optimistic locking may be all well in theory, but the practicalities of merging changes (or more often than not, simply rejecting them as the easy option) are subjects for whole volumes in themselves. Yes, tools like Visual Studio may selecting and updating data easy, but even they do not help when it comes to handling contention.

Because the update logic is on the server, MVDBMS developers can use a granual pessimistic locking model that is robust and far simpler. And if you do need optimistic merging, it's a lot easier with multivalued data than across parent/child tables.

Runtime Binding

MVDMBS applications are built up from libraries of external subroutines and functions. These are loaded dynamically on request, so adding to an existing application is childs play. No need to stop and relink the whole app or deploy the whole caboodle to an application server.

Features

Device control, socket interfaces, XML parsing, SOAP requests, SQL gateways, external C calls: you name it, the language can handle it.

The Language

The MVDBMS language is a highly modified and extended version of Basic, that has been specially designed to work with the conventions and data structures that make up the MVDMBS platform. All using a nice high level syntax that is simple to learn and immensely productive.

Why do people use PHP? Because it is simple, it works, and it gets the job done quickly. Exactly the same reaons people use this language.

After all, what is the benefit of complicating things with difficult syntax?
It's the business rules lying behind them that really count.

Molly - definitely a multi-valued dog

2009-02-26T00:51:00.000-08:00

This week our lives have been turned upside down by the arrival of the latest, four footed member of the family: a five year old labrador collie cross from a rescue centre. Size of a big labrador and temperament of a collie, and we're quickly learning the limitations of her training!

Molly fills the house. The cat has taken shelter upstairs on our bed all week, surprising since she's normally an evil beast (in the best possible feline way, of course). Fortunately we live in a small Oxfordshire village so open countryside is only a couple of minutes away, otherwise Mollie would just be too big for our tiny house and frenetic lives.

It's taking time to get used to working with her curled on the mat beside my desk, but the benefits will outweigh the difficulties: the life of a consultant/developer/writer is essentially a sedentary one, especially now my daughters have reached the staying-away and the-only-exercise-I-get-is-on-WII (how that sounds like ennui) ages respectively. The early morning walk over the lanes (Mollie seems to have a particular propensity for tracking through mud) are almost a tonic to match the double expressos that usually start my day.

So the mornings are now a chance to reflect before I start on the day's tasks. Thinking time has been in short supply recently, with the pressures to complete contract work, finish the new book and get the next set of products under way. With the result that the latter two have not been progressing as well as they should. Thinking time is important in any project, and stepping back has been difficult.

Thought for the Day: have a Thought for the Day.

Fundamentals of MultiValue (4)

2009-02-22T08:41:00.000-08:00

In the last instalment I wrote about the Dictionary, the meta-data that resides at the heart of a MultiValue database. The dictionary defines the physical content of the database, and extends this further to encompass virtual and derived information such as common calculations, synonyms and lookups.

The purpose of the Dictionary is to present as much potential information as possible, in as simple a manner as possible. This user-friendly approach to presenting the system is mirrored in the next piece of the puzzle: the MultiValue Enquiry language.

Annoyingly, at this point one thing definitely does not provide for user-friendly content: the protective attitude of the former generation of MultiValue vendors. There is no single, standard way to refer to the enquiry language: each MultiValue platform uses its own, proprietary and usually trademarked name for what is essentially exactly the same facility. The Enquiry language is known as RetrieVe or UniQuery under IBM UniVerse and UniData respectively; for D3 and jBASE we have AQL and jQL; while learning the technologies I used Access, English and Inform. No wonder nobody gets a handle on this technology!

Whatever you call it, the enquiry language itself is straightforward and flexible. The main commands are the LIST and SORT commands, used as below:

LIST BOOK_TITLES WITH AUTHOR_NAME LIKE “Eoin...”
SORT BOOK_SALES BY SURNAME BY FORENAME TOTAL BALANCE TOTAL SALES_TAX

Unlike SQL, which is predicated on a mathematical model and aimed principally at engineers, the MultiValue enquiry languages have always been pitched at power users. Several things distinguish these languages:

They harness the power of the Dictionary.

All of the elements referred to in the queries, other than the source file name, are defined through the Dictionary. Meaning that the enquiry languages can leverage real and virtual data using the dictionary expressions. In the first example above, AUTHOR_NAME could be (and for the standard demonstration database, really is) a lookup to another file (the BOOK_AUTHORS file). In the second the SALES_TAX might likewise be calculated from the sales order total.

The point of the dictionary is, that to the end user the derivation of the information is entirely irrelevant so long as the definitions are present. Whilst it is possible to use in-line expressions just as in SQL, there should rarely if ever be a need to do so.

They are flexible.

SQL in particular forces a predictable order for a query. In the MultiValue world, these three statements are synonymous and perfectly valid:

LIST BOOK_TITLES BY DEPT BY GENRE WITH UNITS > 10
LIST BOOK_TITLES WITH UNITS > 10 BY DEPT BY GENRE
LIST BOOK_TITLES BY DEPT WITH UNITS > 10 BY GENRE

This makes it easier to build up more complex enquiries in pieces as you go.

They are helpful.

The extent to which this statement in true depends very much on the specific implementation, but generally the MultiValue Enquiry languages will attempt to make sense of what you intend.

The following, for example, would normally be synonymous:

LIST BOOK_TITLES WITH GENRE = “HISTORY” OR WITH GENRE = “FANTASY”
LIST BOOK_TITLES WITH GENRE “HISTORY” “FANTASY”

They are based on reporting.

Back in the world of fixed pitched printing, the enquiry languages were principally used for reporting. This means that they support a range of layout keywords and handle such useful features as subtotals and detail rows:

SORT BOOK_TITLES BY DEPT BY GENRE BREAK.ON DEPT_GENRE TOTAL UNITS HEADING “Books in Stock 'D'”


Books in Stock 22 Feb 2009

Title......................... Dept & Genre........ Units
My Year                        ADULT-BIOGRAPHY         10

                               ***                     10

The 7 Habits of Highly         ADULT-BUSINESS           9
Effective People
Grass Roots Management         ADULT-BUSINESS          12
5 Steps to Successful Selling  ADULT-BUSINESS          15
Principle Centred Leadership   ADULT-BUSINESS          23
The One Minute Manager         ADULT-BUSINESS           1
The Leader in You              ADULT-BUSINESS           4
Elizabeth                      ADULT-BUSINESS           1
7 Steps to Fearless Speaking   ADULT-BUSINESS          18

                               ***                     83



Great Expectations             ADULT-CLASSIC            4
The Importance of Being        ADULT-CLASSIC           22
Earnest
David Copperfield              ADULT-CLASSIC            9

These are simple examples of a truth: the MultiValue enquiry languages are designed to appeal to business users who want answers from a system.

Behind these stand one other features that is far more important to the designers of a system. So important, in fact, that it provides the basis for the name 'MultiValue'.

MultiValued Data

The relational database model is founded on certain axioms. Like all such models, these axioms are matters of faith and convenience: they provide a fundamental statement of what the model is about, but are not statements of truth beyond that model.

The particular axiom that most separates the MultiValue and Relational models is this:

Each row (aka record or tuple) represents a single fact and each row contains information about just one fact.

This may be true in the rarefied atmosphere of mathematical system design but to quote a recent Gartner report; not all data is flat. In fact, the MultiValue model makes extensive use of repeating fields where it makes sense to do so: that is, everywhere you don't want to incur the unnecessary expense of performing parent/child table joins.

Consider a sales order. In the relational world this is divided into (at least) two tables: one for the order header and one for the order lines. When you need to access it, these need to be joined together once again, an expensive and wasteful operation.

In the MultiValue world, a sales order is represented by a single file. The order lines are held in repeating fields known as an association. Here is how the MultiValue world might represent a simple sales order:


   Date
   Time
   Customer_Code
   Delivery_address
   Product_Code
   Quantity
   Price
   Line_Total

In this scheme, the order header is represented by the Date, Time, Customer_Code and Delivery_Address. Note that this does not preclude the use of relations: the Customer_Code would normally point to a customers table just as in a relational model. The Delivery_Address, however, need not be split into separate components (unless it makes sense to do so in the context).

The order lines are represented by the Product_Code (another foreign key), Quantity, Price and (calculated) Line_Total. These will be repeating fields. Each Product_Code fields might contain zero or many product codes, each with an associated entry in the Quantity and Price fields. Where a field contains multiple entries, these are referred to as 'values' – hence the name, MultiValue.

This makes it exceptionally easy to access and to manipulate three dimensional data like a sales order without needing to invest in joins. Note that for the following enquiry to work, the calculated LINE_TOTAL automatically vectors the calculation across each MultiValued PRICE and QTY :

LIST BOOK_SALES SALE_DATE SALE_TIME SURNAME FORENAME TITLE_ID QTY PRICE TITLE_NAME LINE_TOTAL


Sale Date.. 26 MAY 2005
Sale Time.. 15:30
SURNAME.... MAGGS
FORENAME... CARL

TITLE_ID QTY PRICE Title............... Line Total
      49   1  9.99 Jingo                      9.99
      47   1 49.99 Harry Potter and the      49.99
                   Goblet of Fire (Book
                   4 - Unabridged 14
                   Audio Cassette Set)
      11   1 12.99 I'm Sorry I Haven't       12.99
                   a Clue: Vol 8 (BBC
                   Radio Collection)

Do repeating fields make sense? Yes: just look at XML.

In fact, the UniVerse and UniData enquiry languages are specialized to produce XML:


<root>
<book_sales>
<sale_date>26 MAY 2005</sale_date>
<sale_time>15:30</sale_time>
<surname>MAGGS</surname>
<forename>CARL</forename>
<sale_items-mv>
<title_id>49</title_id>
<qty>1</qty>
<price>9.99</price>
<title_name>Jingo</title_name>
<line_total>9.99</line_total>
</SALE_ITEMS-MV>
<sale_items-mv>
<title_id>47</title_id>
<qty>1</qty>
<price>49.99</price>
<title_name>Harry Potter and the Goblet of Fire (Book 4 - Unabridged 14 Audio Cassette Set)</title_name>
<line_total>49.99</line_total>
</SALE_ITEMS-MV>
<sale_items-mv>
<title_id>11</title_id>
<qty>1</qty>
<price>12.99</price>
<title_name>I'm Sorry I Haven't a Clue: Vol 8 (BBC Radio Collection)</title_name>
<line_total>12.99</line_total>
</SALE_ITEMS-MV>
</book_sales>
</root>

XML, with its support for nested entities, is a perfect fit for the MultiValue model. Coupled with client side technologies like LINQ for XML, this is fast becoming the route of choice for MultiValue developers in extracting and sharing data.

So have we finished our tour of the fundamentals of MultiValue here? Not yet! The enquiry languages are used for just what their names suggest – for enquiring on the information in the system. To manipulate that information, these platforms do not use queries. In the next instalment I will write about the server side development language that turns these into a scalable and efficient business platform.

Fundamentals of MultiValue (3)

2009-02-07T03:07:00.000-08:00

In my last post I blogged about the MultiValue file store. In this post I will introduce one of the four defining aspects of a MultiValue system - the file dictionary.

MultiValue files, unlike traditional database entities, may or may not define their content. For the most part, having a full content definition for files that contain regular data, such as customers or sales orders, is a good thing. But that raises an interesting question - just what is a full content definition?

Defining Information

For a database engineer this would simply indicate the physical file content and its data types, with possibly a few constraint thrown in for good measure. But that isn't really the full story.

For one, the same content may mean different things to different people, or to different groups within an organization. A payments department might refer to a customer identifier as an account code, where the same identifier might be known as a client code to the sales team. Subtle distinctions, perhaps, but these things can be politically important and can cause confusion.

Sometime data has different meanings in different contexts, or may require different forms of presentation. A simple thing like a date might require different formatting for reports, exports to third party systems, or analysis based on periods. A code might be built up from several pieces of useful information, as in the case where an accounting period and department mnemonic form part of an invoice reference: in some instances this may need to be seen as a whole or in others broken into its constituent parts.

Even then, you are only looking at a very small subset of the information that is potentially available. Where a file contains base data, what about all the information that can be derived from it? If an order detail line holds a quantity, price and tax code it makes sense to calculate the line totals before and after tax, and to make that part of the 'information domain' as the academics like to call it.

A file may include data that forms relations to other data. Not just between files in the case of a foreign or shared key that can be used as a lookup - like a relational join or a vlookup() in a spreadsheet - but also between fields in the same file that naturally form groupings. A file holding a full specification of customer details might include the title, first name and last name for a customer: together these form a grouping that makes up the full customer name.

As you can see, it is possible to enhance the information available about a file content beyond just a listing of the physical data: but with that comes resposibilities. As the information domain extends, so does the need for control. Not all departments may be privileged to see all of the information; some may be meaningless or only required in special circumstances. Creating customized subsets of that information can help to maintain security and makes the content presented easier to navigate.

The Dictionary

All this is handled by the Dictionary in a MultiValue system.

Every file in the system has - potentially at any rate - at least one and possibly several associated dictionaries, through which the developer or DBA, and in some cases power users, can make sense of the content of the system.

A dictionary contains four types of definitions:

Data Definitions deal with the real stuff held in the file. These act like the column definitions in a SQL schema, describing the data content, format, type and size. Recall, however, that not all columns need to be defined, and not all may be defined across all dictionaries: only the ones that are actually useful.

Synonym Definitions present the same content under different names, or providing different formatting. Here you might specify alternate date and period formats, extract meaningful pieces from composite codes, handle issues of casing or number formatting.

Association Definitions identify groupings of fields within the file. These act as shortcuts and as controllers for complex data.

Interpreted Definitions are the most potent symbol of the MultiValue model. These define virtual content: information that is derived from the real data either by means of calculations or lookups. An Interpreted Definition might perform a calculation, retrieve a piece of related information from a secondary file by means of a join or call a stored procedure.

In a perfect world, a Dictionary should lay out everything you might need to know or understand based on a file - not just the content, but the whole information domain that can be derived from that file. No need for the enquirer to understand and to navigate the underlying data model. No need to resort to views or joins. All there, conveniently held in one place.

Enough theory - let us consider an example of a simple sales order. In a relational database, this might be presented as a set of related tables:

In a MultiValue database this is presented in a single dictionary, providing access to the related information and adding calculated content:

Dictionaries make the system easy to understand, and easy to extend. If new information is required, it may be just a simple definition away: consider the effect of this on the cost of running a MultiValue system as opposed to the cost of making programming changes or employing costly database architects to craft new views and enquiries.

Dictionaries are more than just metadata. They are about making the system approachable and easy to use. In this spirit they are the driving force behind the second defining aspect of a MultiValue system and the subject of my next post: the user-friendly enquiry language.

Fundamentals of MultiValue (part 2)

2009-01-21T12:31:00.000-08:00

In my last post, I promised to explain more about 'MultiValue' as a data model. So in this post I will start our tour of the fundamental aspects of this technology.

MultiValue systems are generally referred to as 'databases', and are sometimes given the designation 'MVDBMS' (MultiValue DataBase Management Systems) to differentiate them from Relational Databases (usually with the RDBMS acronym).

I always feel uncomfortable with this designation since the term 'database' conjures up a set of expectations about what the technologys is expected to deliver - and, more to the point, what it does not deliver. The fact is, MultiValue platforms generally offer a much wider range of functionality, and play a much fuller participatory role in their solutions, than you might expect from a simple 'database'. For this reason, I prefer to refer to these as 'business platforms', being a more accurate reflection of the jobs that they perform.

However, MultiValue platforms do share some similarities with mainstream databases, not least of which are the central roles played by the data storage and enquiry facilities. So it seems reasonable to begin our tour with the most basic aspect of the model: the data store.

The Data Store

Data storage and retrieval sits at the heart of the MultiValue model. MultiValue platforms have their own fully featured data store containing at first glance all the elements you might associate with any data model: files (tables), indices, records (rows) and fields (columns). Some MultiValue databases - notably IBM UniVerse - even offer out-of-the-box support for the most popular enquiry language - SQL - making them appear instantly familiar to database users.

However, this is really giving a false impression: these are not relational databases and designing a MultiValue database using standard data modelling and applying the normal techniques of relational theory will end up in tears. Under the hood the storage and retrieval models are very different. UniVerse SQL may follow standard syntax, but this is still using the features of the MultiValue architecture.

To understand how this makes a difference consider this: a truck and a Formula 1 racing car both have wheels, gears, axles and engines; both can be driven on the road or track; but the designs for each are aimed at different solutions. A Formula 1 car may be much quicker, but might not be the best choice for towing a heavy load! MultiValued data is often heavily loaded with information, and so is designed to be accessed in a particular way: and that often counts against the database when used with industry standard protocols such as ODBC. To understand further, we first need to understand the difference in layout between a mainstream database and the MultiValue file and record structures.

Files and Records

MultiValue platforms hold data in files. These are similar in nature to tables in other databases, but with one very important difference: in a mainstream database, tables are created based on a definition of the structure of their content. In a MultiValue database, they are not.

If you wish to create a new table in a relational database, the tool of choice is the SQL CREATE TABLE command:

CREATE TABLE MYTABLE
(ID INT NOT NULL PRIMARY KEY,
SOME_NAME VARCHAR,
SOME_DATE DATE);

The CREATE TABLE command is a slight misnomer since it is really not defining the table at all, but rather defines the column structure of the rows that will make up the data in that table. It may be extended to hold details about the storage patterns, volume placement and other paraphernalia associated with a particular database, but its primary purpose is to set out the data layout. Once defined, only data that supports that structure can be stored in that table: the table is said to be 'homegeneous'.

MultiValue databases define files based on storage structure alone. The equivalent CREATE.FILE statement labels the file and determines the internal structure (more on which later) and optionally the initial storage size to reserve for best performance.

CREATE.FILE MYFILE DYNAMIC

The CREATE.FILE statement does not define the data layout. Why should this be?

MultiValue files hold data in the form of records, each of which is identified by a primary key and each of which can be subdivided into its constituent fields.

But here is the first shock - a MultiValue datastore does not require you to define the data you will be holding. In fact, the MultiValue model explicitly allows you to create files that have no known layout, and to use these to store records that can be accessed generically, without regard to the column layouts or to data types.

That may seem a very uncomfortable concept for DBAs - after all, if you cannot define the format how can you have faith in the structure? Avoiding the obvious corrollary - that plenty of systems founded on relational databases seem to end up with problems and cost overruns despite the strong definitions - the fact is that the MultiValue model does allow you to define the data format, but does not make that definition mandatory or prescriptive. And just as the ability to use generic or variant types has its uses in programming languages (where appropriate), so there are some real advantages to that in the data store.

For example, it enables MultiValue developers to design files that are not necessarily homogeneous. Not all data held in a system is structured: look into any sizeable application and you will find it littered with parameter records, control values, settings - either sitting in tables with single rows, tables offering key/value pairings or shipped out into INI or XML files. In a MultiValue system, these control records can reside side by side in a general parameter file.

Second, it allows developers to use data stores whose content is not known until run time. These may be work files, intermediate files used during processing or more often files used for the storage and manipulation of data that flows through the system from elsewhere. If the data store does not require the layout to be known, it opens the door for the application logic to work easily with 'soft-coded' data definitions - such as the definitions of streams of data that may be captured from external systems, and even 'dirty' data that needs to be cleansed and formalized before sending it on. Because of this, the MultiValue model can be used as a foundation for complex data analysis, transformation and extraction tools, rules based systems and anywhere that the data formats are defined based on context.

One particular advantage of this can be found in the use of systems development tools. Using generic data stores makes it possible to build applications that are themselves data interpreters - leading to a market in which fourth generation languages and advanced system generation tools play a prominent role, being able to harness the ability to work with such generics at the storage and enquiry levels whilst applying their own rules and interpretations of the data at a higher level. 4GLs and interpreted systems have had occasional renaissances amongst RDBMS vendors, but often have to fight against the restrictions imposed by the data model, rather than working comfortably with them.

There are other uses for opaque data structures, but the most powerful one is this: the MultiValue model forces a separation of the definition of the data (the 'metadata') from the data itself. An equivalent in the mainstream world might be the separation of a DTD or schema from an XML document. By separating the two, and by removing the dependencies between the storage and definition of the data, the model is better adapted to suit the ever changing nature of data-centered operations in the real world.

This separation opens up a whole realm of opportunities for presenting and shaping information based on context, including the ability to derive 'virtual' data from the information held in the data store. This in turn makes it far easier (and therefore cost-effective) for building agile systems that can quickly respond to changing requirements and to supporting new business processes.

And that will be the subject of the next post when we will take a look at the metadata and enquiry languages.

About MultiValue (Part 1)

2009-01-01T06:00:00.000-08:00

First and foremost, Happy New Year to everyone!

You may - I hope - have noticed the title of this blog and be curious as to the meaning of the term 'MultiValue'. So before I pursue my musings further, let me set out my stall and explain the derivation and compass of this technology.

Now this is too large a task for a single posting, so I will take pity on my readers and break this down into bite sized pieces. In this series I will do my best to define terms, encourage you to take an interest in this model, and point out some of the advantages (and occasional disadvantages) it offers over other, more widely understood, data practices. All opinions will, of course, be my own.

The term MultiValue refers to a database model, and more.

The term was first coined by Gus Giobbi, then editor of International Spectrum Magazine, as a shorthand through which to refer to a wide range of products from a number of disparate suppliers, all of which shared the same data model and, to a large extent, the same technologies and languages.

Unfortunately, each of these manufacturers had for a long time pushed their own proprietary derivations of the same data model: a situation not unlike the relational world at that time, but with the significant exception that these manufacturers were doing their utmost not to even acknowledge the existence of their competitors. So unlike the relational world in which elements like SQL form a well -known standard (albeit with many different variations), in this technology group there were no standard terms that were not trademarked to one or other manufacturer, and no standard way in which to refer to the technology in a generic way.

This led to two practical problems:

1. Speaking about MultiValue becomes a pain in the backside, when referring to a key part of the technology such as the enquiry language means reeling off a list of different names to ensure the whole audience understands what the speaker is talking about.

2. For outsiders, it appears even today that this is a small and localized set of technologies. Without a standard name, commentators cannot get a sense of the actual size and impact of this market.

The story of the MultiValue industry is an interesting (and often debated) one: Dawn Wolthuis of Tincat Group has done a great job in researching this topic. Suffice it to say that MultiValue technology is found in every industry sector, in every size of company from one or two user systems to systems with tens of thousands of concurrent users, from finance to manufacture, ecommerce, government and military, education, publishing, healthcare - whatever sector you choose to name, there will be MultiValue systems in evidence, usually in mission critical capacities.

So why is the technology so little known?

The blame for this can be laid at two doors: the MultiValue vendors and the technology.

For most of its history, vendors have fought over the MultiValue market rather than taking an active role in attempting to increase it. The lack of any central marketing for the technology - in the way of a marketing board or a single nomenclature - has left it lagging far behind the public consciousness.

Technically, MultiValue is also a peculiar victim of its own success. MultiValue systems suffer from being exceptionally stable and feature rich. A typical MVDBMS (to give the regular abbreviation) requires little to no maintenance, and may run for years with nothing more than an automated backup routine to concern those using the system. I often liken MVDBMS to a highly capable secretary, doing his or her work quietly in the background, and consequently frequently overlooked. Systems that need constant hand-holding are meat and drink to consultants and to the assembled ranks of the IT industry, and so tend to get high profiles. Systems that just work make it difficult to pay the bills!

MVDBMS is primarily an embedded technology - hence the requirement for exceptional stability - and so end users are generally aware of the applications being supported by the technology as opposed to the technology itself.

Today, the MultiValue market has consolidated into a handful of significant vendors. Top of that list is IBM, with the lions share of the market and two of the leading MVDBMS platforms - UniVerse and UniData. To assist in the general confusion, some bright spark in IBM Marketing decided that combining the two under the marketing name of U2 would be a good idea, and we have suffered from that insanity ever since. Most of my posts will concentrate on these two platforms, since they form most of my work.

There are alternatives however that deserve a mention: OpenInsight from Revelation Technologies is a PC based MVDBMS with a significant and loyal following; OpenQM is an open-source offering from Ladybridge Systems; jBASE has a high profile in the banking sector in particular; D3 from Raining Data/Tiger Logic is still found in many industries; and there are a number of smaller systems enjoying a long shelf life.

What these have to offer, and why the name 'MultiValue' came about, will be the subject of the next post.

UniRPC and Security

2008-12-21T05:02:00.000-08:00

No blog last weekend since I was reminded by Clif Oliver that it was a submission deadline for International Spectrum Magazine. Spectrum has long waved the MultiValue banner, and I will be posting a short history of the magazine and the organization in a later blog.

Security has been on my mind this week, prompted by two rather different incidents.

The first was taking the train into London from my home in Oxfordshire, something I am forced to do all too frequently. I habitually sit in the quiet carriage, which for those unfamiliar with our creaking rail system, is a carriage in which mobile phones, personal stereos and loud conversation are banned whilst commuters sleep, read or consult their notebooks. This particular morning a chap in the seat in front of mine was reading some papers, clearly unaware that there was anyone behind him. The papers were labelled 'RESTRICTED' and from the glance I took related to the foreign office. Had I wished (and I will quickly point out that I had better things to do) I could have easily read every word.

The second was a posting on the U2-Users list from a colleague on the U2UG board, asking whether the UniRPC protocol had been published. UniRPC is the underlying protocol used to connect to UniVerse and UniData databases, and chance would have it that I had asked exactly the same question of IBM on various occasions over the last few years, and for exactly the same reason: to create a PHP driver direct to the UniRPC deamon without the need to install UniObjects or InterCall.

Each time I have asked for this the result has been negative.

There are other ways to connect PHP with IBM U2: using the java or COM bridges to run UniObjects or WebDE or by calling U2 web services, but these are not always practical particularly for systems that use external hosting. Plus IBM's insistance on penalizing developers wanting to build web based solutions by requiring connection pooled licences makes web services an expensive option. Personally I think IBM are shooting themselves in the foot with this policy - no other database vendor requires this, and hosted solutions are the best option for the future of U2, since they play to the traditional strengths of fast, low level data access and minimal resource consumption coupled with a application-centric and feature rich runtime.

Returning to the theme, however: I suggested that the protocol could be easily worked out from tracing the network. I have traced UniObjects in the past for support reasons and the traffic is easily readable: it would take time but not too much work to deconstruct the protocol.

Which led me to think -

a) how many U2 sites are running the UniRPC daemon/service without SSL?
b) how many administrators are running UniRPC without realizing the hole it makes in their security?

Most U2 administrators are traditionalists: that combined with the age of most U2 applications mean that security is too often considered from the point of the application (menu options, for example). UniObjects blows a hole in that - from a UO connection (easy from any Windows PC) anyone with authority to connect to the server can browse, view and update any part of the application. UniData offers some security with the (helpfully undocumented) UOLOGIN subroutine that can deny UniRPC access to an account based on logic. UniVerse does not seem to have such a beast. At the very least, adminstrators should be implementing the Data Encyption At Rest functionality now available for sensitive U2 data.

Like the chap reading restricted material on the train, many U2 administrators may be unwittingly leaving the door open for anyone to view their systems. That is a frightening thought.

New Toy

2008-11-28T00:21:00.000-08:00

I have a new toy.

I've finally settled - at least for now - on a Samsung NC10 to take on the train. A fraction of the price of the Vaio, and so following a quick (and easy) memory upgrade, a not-so-quick Vista Ultimate installation, a slow Visual Studio 2008 setup and the inevitable wait-for-geological-time VS SP1 upgrade (why does it take sooooo long), and I'm ready for action.

With, of course, a copy of IBM UniVerse and Silverlight.

I've been waiting for Silverlight for a long time. Not specifically that product, but for an RIA that is really pitched at developers. There have been some good toolkits - OpenLaszlo and the excellent GOA WinForms for example - but even these haven't quite hit the spot. OpenLaszlo was too limited, and GOA - at the time - too tied into its own technology.

Flash, of course, has ruled the roost for a long time, but has never hit the developer spot: ActionScript just doesn't read like a real language and the IDEs were too - foreign - to developers weaned on C++, Delphi and Visual Studio. And when it comes to IBM U2, products like mvScript and WebDE can make use of AJAX but it's a lot of work for little real reward.

When we designed mvQuery for MultiValue databases such as IBM UniVerse and IBM UniData, we were hamstrung by our decision to make the product into a real reporting tool - one that generated complex reports, not just simple tables of results. That of course meant building a report designer, and one that was powerful enough to handle the vagaries (for which read massive flexibility) of the MultiValue data model. And that meant using an environment that offfered the level of user interaction required by a graphical report painter. And that, in turn, meant that it could not be web based. We did, eventually, settle on a compromise: developers and power users got the Windows based client to design the reports, and other users got a web based tool through which they could schedule and run those reports, either viewing or emailing the results as PDF.

Now, at long last, Microsoft has stepped up to the mark with a full technology stack for RIAs that suits the needs of designers and developers alike, and I have started work full throttle on a new reporting tool that will both deliver and design UniVerse and UniData reports through Silverlight.

It's been a long time since programming has been fun, and I'm looking forward to this one.

Has Sony Gone Mad?

2008-11-23T10:00:00.000-08:00

I've been looking for a new laptop for the past weeks, and it is not proving easy.

On my desk at the moment I have two laptops: an IBM ThinkPad and a Sony Vaio. The two premier brands of the laptop world, according to most reviewers. (I wouldn't consider anything without reading trusted reviews).

The ThinkPad is a workhorse: solid construction, professional and practical, a laptop that oozes confidence. The Vaio is - well, you must know what Vaios are like. Pure beauty.

That is where the real difference comes. The ThinkPad I can use - but it simply does not make me feel the same way as the Vaio. When I get out the Vaio it just feels special. And it makes me feel special. It is expensive, yes, but that feeling is a price worth paying given the hours I spend in front of the screen.

But - the TT series? I could fall in love with that - except for the insane pricing. A couple of hundred pounds extra for an SDD would be reasonable, but almost six hundred for that upgrade? Two hundred pounds for an extra two GB RAM? And all for a machine that is slower and heavier than the Samsung NC10 - at almost two thousand pounds less.

Sorry Sony, but not this time.

Starting Out

2008-11-22T05:53:00.000-08:00

Well I guess it had to happen sooner or later.

Three things convinced me to finally start a blog:

A request from IBM, who had noticed that my profile as an IBM Data Champion did not include a blog entry. As if the website(s), books and wiki weren't enough.
Reading the thoughts of an old friend, Robert Altinger, on his blog.
The feeling that the U2UG may be back on track again.

The latter probably needs some explanation.

For the last months we have been floundering a little - we have all been busy with other matters, I've been occupied writing the first two volumes of my series on IBM UniVerse, and more importantly, we have lost Janet Oswald, our IBM liaison, who has gone on to better things with a fully deserved promotion. We wish her every success.

This week we met our new liaison - Martha, whose background includes previously working for the Informix user group. Having IBM attend the board meetings is crucial in meeting our obligations to act as a conduit between the U2 technology users and IBM.

There is plenty of work to be done at U2UG: the website, whilst functional, needs a redesign; the wiki needs serious content and there are moves afoot to extend the Learner Pack with samples using as many U2 technologies as possible.

So there should be exciting times ahead.