The Bigger Picture

Session 4

The time had finally come to start piecing the tiles together.  Using the new user control created in the last session I removed the existing user control from the designer and created a method on the form load to create a 10 x 10 grid of ASCIITilePanel controls.  Each of the new controls is then setup to point at the required file and given a unique ID. 

The goal was to load an entire two letter tile set into the application which will give coverage of 100km squared.  With that in mind  I moved all the location details into a settings file so that if the project or data move then it is all relative and added some code to enable the two letter grid reference to be entered.  For the present time I am staying with SU to keep things simple.

After that was completed I started loading the data to ensure it was all appearing in the correct position.  I soon noticed that things didn't look quite right.  After a few minutes of investigation I remembered that the tile data starts in the bottom left but realised when I create the image of the terrain for display I was using the top left.  To remedy this I simply rotated the image 90 degrees and this solved the issue.  However, the tiles now looked correct north to south but east to west the tiles were not lining up.  My decision to use an area of terrain I recognised paid off when I saw that the tiles were in the wrong order.  Instead of displaying left to right they should be right to left.  After fixing the bug I reloaded the data and was presented with a pretty cool looking visualisation of Southampton water and the east of the New Forest.




Multiple tile display

 

A couple of things I noticed from the image was that at the higher points of the terrain I was getting black spots so will need to think about the range of the colour picker algorithm.  The other alteration I made was to store the bitmap image offline.  The code then checks when the data is loaded to see if a bitmap exists for the tile and loads the bitmap rather than generate the image from scratch.  This prevents the application from regenerating data each time it is run.

Session 4 finally gave me a bit a of a geek buzz and to see the image presented really got me excited for the following 3 sessions. The next session will be a clean up of the code and I will start to think about how to deal with the data as a whole before passing it to the A* algorithm.  The main task will be to look at how I can improve the terrain class that holds the tiles so that I can pick points out and do some whole data operations.   

2D or not to 2D

Session 3

By session 3 I had a single OS terrain 50 data tile loaded using the ASCII file format.  However, it didn't look that great.  The reason for this is that using the Perlin noise algorithm for generating the terrain in the original application, meant that I didn't need a technique to display terrain at a particularly high fidelity.  At this point I came to a minor cross roads.  I had previously used an algorithm written in C++ that created an OpenGL triangle mesh from DTED.  I was going port it to C# and use it to display the terrain in 3D but decided that it would probably take too much time and wasn't really a core objective of the sessions.  This was recorded as another ideas for phase 2.

After ditching the idea for the triangle mesh and keeping to my simple UI approach I started to look around for some easy to implement alternatives.  I considered a few options, including some cool HTML 5 pages, but again chose to keep it simple and use a heat map and the cyclical nature of hue.  I have seen this technique used a couple of times on other projects and remembered that it was fairly simple to implement.  

The algorithm (which is described in detail here) can be used to set a range of acceptable values and then interpolates the hue value between 0 and 360 degrees.  This is then converted to RGB using the algorithm and it gives a  range of values in the same way a colour picker is displayed in a paint application.  So using this technique and applying it to the terrain model you can now see the detail of the OS terrain 50 data.  The range goes from red at its lowest point and purple to its highest. 

The same OS terrain 50 data as displayed previously but now using the colour picker technique for display.

After implementing the new display algorithm, I decided to delay the multiple tile implementation until tomorrow but prepare a new user control to enable multiple tiles instead.  To display multiple tiles I know that I will need to create the user controls in code (current plan is to display 100 tiles or all of the SU tile set) and not in the designer.  The new user control is an ASCII tile panel and inherits from the double buffered panel.  It contains a tile ID, X and Y coordinate of the tile and the file that contains the tile content.  I am not 100% happy with the design of this but for now it will be a good way to test the loading of multiple tiles.  I may revisit this in time so that the user control is valid across any tile content type.

In summary, session 3 has thrown up issues that I hadn't thought were that important initially (display of the tile data) to the main goal but I have managed to implement a timely solution and kept on track with attempting to get a decent test bed for the A* algorithm up and running in a weeks time.  Tomorrow will be using multiple tiles and ensuring that the application doesn't suffer too much from storing all the data in memory.

Winforms UI and the Double Buffered Panel

Session 2

The main task of this session was to link the new tile loader package to the WinForms user interface and display the new OS Terrain 50 data and existing Perlin noise terrain on the UI.  The first task was to remove the duplicated classes that were now in the tile loader package and to upgrade the UI so it can be used to display more than tile.  It also needs a bit of tidying up as it was developed fairly rapidly as a test bed for the A* algorithm.


The existing UI

Once the tile loading classes had been removed I reorganised the code behind to operate using the asynchronous tile loading package.  The main thing about user interfaces is that the main UI thread cannot be locked so the asynchronous calls to load tiles are crucial.

The other hurdle I faced was that the existing UI uses a basic panel to display the tile on.  When the UI scrolls the panel flickers so I had to create a new user control that is double buffered but inherits from the main .NET panel object.  This user control is then created dynamically on the UI loading so that it is tiled on the main pane. 

The second session overran slightly as I had underestimated how long it would take to change the UI over to using asynchronous calls. However, by the end of it I had the first sight of the terrain 50 data in the UI.  I had chosen the SU area tiles to begin the testing for two reasons, 1 it has a clear coast line which will help test the orientation is correct and 2 I am familiar with the area.

After session 2, the application only loads a single tile using the menu buttons so tomorrow I will be looking to load multiple tiles and change the UI slightly to load the data by double clicking the desired tile on the UI.  The background threads worked well and the performance is pretty good.  I will also be looking to improve the colour algorithm to give it more detail as the new data has a better fidelity which is not shown using the existing algorithm.

OS Terrain 50 in application

Esri ASCII Grid Format

Session 1

The first session goal was to load an OS Terrain 50 data file into a terrain object.  The existing structure of the terrain object is a simple C# dictionary using a 2D point as a key and the height data as the value.  This was redesigned to allow for multiple tiles and to allow different types of loaders to populate the data, i.e. Perlin noise, GML, ASCII grid etc.

The new structure contains an overarching terrain object.  The terrain object can contain 0..* tiles.  The tile contains 1..1 tile content object. The tile content object is an abstract class that contains 0..* tileData objects and a virtual method, LoadContent. 

First design of the tile loader package

There are two items of note.  The first is that the tile is given the tile content object so that I can use lazy loading, i.e. the tile can exist without any data.  This way I can create all the tiles I require and position them before loading the data of the tile. The second point is the the tile object can be oblivious to where its data came from, i.e. file loaded, auto generated etc.  This is a first draft of the design and I expect to change as things develop.

ESRI ASCII Grid Format

I have decided to start with the simplest format for the first data file which is the ASCII grid format.  The details of the format can be found here but here are the main points:

The format has a header block containing the following items:

ncols - The number of columns in the data
nrows - The number of rows in the data
xllcorner - The X offset from the lower left corner
yllcorner - The Y offset from the lower left corner
cellsize - The cell size in metres
NODATA_value - Value that indicates no data recorded.

After the header block is the main data starting in the upper left corner.

To read the data in I am using the C# stream reader to read the file in line by line.  I have created a new object that inherits from TileContent called ASCIITileContent and implements the LoadContent method.  The load content method creates a background worker thread to read the file and load into the terrainData object.  For this I had to create a new event that tells the tile when its content is loaded which is raised by the background worker thread upon completion.  I added the new event to the TileContent class as I am planning to use the same technique for the rest of the classes.

I have also modified the Perlin algorithm to fit into the new design.  I have created a new class, PerlinTileContent, which inherits from TileContent and generates a random terrain.  This is then saved in the terrainData and an event is raised by the background worker.

For the tile loading I have created a new package in the project called tile loader that contains all the tile classes so that I can keep the data and UI loosely coupled.  

Overall, the first session went very well.  The data can now be read in successfully using the background worker threads and the class design seems to work well.  I think I may tweak it tomorrow to allow more information to be passed back and forth (mainly to report on progress whilst loading the file) but for now I am happy that I have met the objectives of first session.  Reading the ASCII files was very simple job but I may have to think about using a 3rd party zip library so that I don't have to keep unzipping the data files from the master zip but that may be a task for phase 2.  Tomorrow, I am hoping to link it up to the WinForms UI and look at how the UI can be improved.

Podcasts

I have had a daily commute of around an hour to work each day for a number of years now.  When I started this commute, a lot of this time felt like wasted time as I listened to the same songs on the radio or just daydreamed.  However, a few years ago I was given an IPod for my birthday and discovered pod casts.

Initially I listened to everything and downloaded any pod cast of interest to me and my interests of software, comedy, books and running.  The list below represents the pod casts I currently subscribe to or have found to be the most entertaining and thought provoking.

.NET Rocks - .NET Development and all subjects around the Microsoft tech stack.

The Infinite Monkey Cage - Science and comedy chat

Outriders - Web news show

Desert Island Discs - Always entertaining

Click - Technology news show by the BBC

TED Radio - Lectures on Technology, Engineering and Design

Anything by Richard Herring the Podfather of comedy

OS OpenData

I recently came across the Ordnance Surveys OpenData products and decided to look into how I could use some of the data in some of my existing projects.  In particular I have a personal project that I have been developing that uses an A* path finding algorithm with a heuristic that avoids steep inclines and declines, picking the most efficient route through a 3D terrain.

At the current time I have been using a Perlin noise algorithm to randomly generate terrain.  The weakness of this algorithm for generating the terrain is that, without some particular exceptions, it cannot produce certain terrain effects such as cliffs, inlets and sharp changes in terrain.  There are some techniques that can be employed to produce these effects after the Perlin noise has run but again they take some time to get a realistic looking terrain.  Using real terrain would be a great test for the algorithm.



Perlin noise generated terrain

A*algorithm avoiding the water and raised terrain

The OpenData products come in a number of formats and are continually updated to give information from postcodes and boundaries to digital maps.  The product that caught my eye was the OS Terrain 50 which gives height data at 1:50 scale across the whole of the UK. 

OS Terrain 50 is available as:

• 50 metre grid in ASCII grid and GML 3.2.1
• 10 metre contours in Esri® shapefile or GML 3.2.1

ASCII grid is a simple file format that specifies height data by storing tile information in a header block then a table of data in the body containing the payload. Geography Markup Language (GML) is an XML grammar to express geographical features. Esri shapefile is a geospatial vector data format for specifying shapes as points, lines and polygons that can be used to represent rivers, water wells and lakes.

My goal is to use one (or possibly all) of these file formats to produce some real terrain data to test my A* algorithm.  To do this I will use my existing test bed application written using WinForms and C# and modify it to read in the OS Terrain 50 data.  The existing test bed is a very simple application that produces a terrain object which is rendered in 2D and is colour coded to show different heights in different colours.  The terrain object is then used by the A* algorithm to calculate the most efficient route through the terrain.

The main tasks are:

• A GML/ASCII/Shapefile reader will be developed.  There are a number of readers available but I am planning to write my own so that I can learn more about the file format.
• The application will be redesigned so that a new layer is added that can feed the main application with either a terrain object that was produced by the Perlin noise algorithm or a terrain object built from the OS terrain 50.  In this way the A* algorithm will not be affected by the type of terrain data loaded.
• The main user interface will be changed to display multiple tiles;
• The application will be changed to use threads to load the tiles in the background.
• The colour code will be updated to produce a better quality display.

I am planning to tackle these tasks in hour long sessions over a period of a week and see where I am after the week has finished.  The main risk to completion are finding the time in a week for an hour of coding and half an hour of blogging. 

Creating the ASP.net Membership Tables In SQL Server

Because I always forget and I rarely have to create the asp.net membership tables (usually backup from exisitng database) I decided to put the command in a blog post. So the command:

aspnet_regsql -S -d -E -A mr 

-S and -d are obvious command line options. -E authenticates using the Windows credentials of the currently logged-on user. - A adds support for one or more ASP.NET application services. Service identifiers can be specified together or separately. The following identifiers are used for ASP.NET application services: m - Membership r - Role Manager

Microsoft Virtual Academy

I've been looking at the Microsoft Azure offerings lately and was really looking for a route in to start looking at what it could offer when I had a discussion with a colleague who mentioned the Microsoft Virtual Academy. The resources on their training academy are superb and I soon found a few courses and added them to my training plan.

I followed the Azure lesson and was blown away by the features that Azure had.  From my perspective, using the platform as a test bed would be fantastic but combined with MS Sync Framework and the ability to sync with on site instances of SQL server would be incredible.  I have started working on s tech demo to explore Azure further and to demonstrate to colleagues how we could use it in our day to day working.