Wednesday, October 22, 2014

Breaktroughts lectures at university of Saskatchewan

A remarkable initiative at University of Saskatchewan, has been initiated, under the impulse of Jeff McDonnell. He invited many top hydrological scientists to express their opinion and ideas about various topics in modern hydrologic research. Fortunately, the lectures are subsequently posted on a proper channel in Youtube.
I have to say that I do not share all the ideas presented. However, they constitute a corpus that is useful to know.


The video lectures, can be found here.



Thursday, October 9, 2014

A couple of new things from Hydrologis

My former students of Hydrologis, which whom I collaborated in doing the Horton Machine contained in the uDig Spatial Toolbox, came out recently with a few good news.

The first is Stage an application that makes the Spatial Toolbox available alone, in meanwhile uDig migration to Location Tech is ongoing. Besides it offers a way to save and store Geopaparazzi projects in your personal Computer. A more extensive description of what Stage does, can be found here.
The second is Lesto, part of the work of Silvia Franceschi for her Ph.D. at University of Bolzano. A set of tools for extracting features from LIDAR data. Information about Lesto can be found here
For getting more information, please contact info <at> hydrologis.com

Wednesday, October 8, 2014

CISLAM

CISLAM is the simplified hydrological model produced by Cristiano Lanni during his P.h. D which was devoted to the study of landslide triggering. The theory behind the code is commented in a Hydrological Processes Paper, and the original code was written in R, but Marco Foi ported it to the Jgrasstools during a fortunate Google Summer of Code.

  • The CISLAM Manual can be found here
  • Source code can be found here.


You can find a jar file ready to be used within a hacked version of JGrassTools 0.7.7: in fact, Marco had to modify a little JGrasstools to get it working. These changes were, so far, never introduced in version 0.8, and therefore for using CISLAM, it is necessary to use this version of JGrasstools: 


The tool has been tested to on the data set that can be found here, the same described in the manual. Other tests, would be necessary indeed. 

Friday, October 3, 2014

Naming things in hydrological models

Yesterday I could meet with Olaf David, Scott Peckham. Scott is a well known scientists either among hydrological modellers than geomorphologists. In the first field because of his recent work on CSDMS project (and his own model Topoflow), in which he was one of the leader scientists, in the latter thanks to his work on river network topology and the construction of Rivertools, one of the best suite of tools for watershed delineation and analysis.

The reason to meet was friendship and just talking and exchanging  what we are doing, and the meeting, closed in one (actually two) of the small breweries of Fort Collins, was really successful. 

One of the recent things Scott is pursuing is to understand what models have inside, and the approach he took, was to categorise all the variables they contain, and define in a manner as clear as possible.  His efforts can be found and well described in here.

“CSDMS asks that contributed models should be provided with a Basic Model Interface (BMI) which includes mapping input and output variable names to CSDMS Standard Names and providing model metadata. …  A good introduction to the CSDMS Standard Names is provided by Peckham (2014). A somewhat outdated, high-level overview of the CSDMS Standard Names is also available as aPowerpoint presentation.”

Scott and coworkers did not forgot netCDF parallel effort with its CF convention, but he realised that the coverage of hydrology was poor, and he want to built the vocabulary from scratch. The effort, is by far not useful to his project, but also for other models and infrastructures. With our model GEOtop we started a parallel, and much more limited work, in identifying keywords related to hydrological quantities and to control the model’s workflow (see GEOtop’s manual), and I plan to provide soon a matching between CSDMS names and GEOtop names (and, I will repeat the operation inside my lectures, modifying my slides).

Having a common vocabulary for identify things in models would certainly make easier to choose names for quantities, even if, clearly the internal variable names should be shorter for practical purposes,   identify code chunks that treat the same phenomena. Also search model through the web would facilitate with standard names for search.


Here below a brief description of the whole Scott’s effort.
While it is always a good idea to use existing standards whenever possible, CSDMS discovered that other naming conventions, such as the CF Convention Standard Names were not well-suited to the needs of component-based modeling. This section explains our motivation for developing a new standard.
This section provides some background and basic information about the CSDMS Standard Names.
This section provides numerous examples of CSDMS Standard Names, organized by the main object under consideration and its parts or "subobjects".
The CSDMS Standard Names follow an object + quantity pattern with an optional operation prefix applied to the quantity part. This section provides the basic rules for constructing CSDMS Standard Names.
This section provides a set of templates and rules for constructing the object name part of a CSDMS Standard Name.
This section provides a set of templates and rules for constructing the quantity name part of a CSDMS Standard Name. Many quantity names include the name of aphysical process and information about constructing process names along with numerous examples are given on the CSDMS Process Names page.
This section provides a set of templates and rules for constructing the optional operation part of a CSDMS Standard Name.
This section provides information on CSDMS Model Coupling Metadata (MCM) files and provides standardized model/variable metadata names for units, ellipsoids, datums, projections, "how modeled" and assumptions. It links to an extensive set of CSDMS Assumption Names and includes An Example Model Coupling Metadata file.


Thursday, October 2, 2014

Machine Learning

I found this informative blog post on Element of Statistic Learning, a fundamental book by Trevor Hastie and Rob Tibshirani.

I reproduce verbatim the blogpost:

"In January 2014, Stanford University professors Trevor Hastie and Rob Tibshirani (authors of the legendaryElements of Statistical Learning textbook) taught an online course based on their newest textbook, An Introduction to Statistical Learning with Applications in R (ISLR). I found it to be an excellent course in statistical learning (also known as "machine learning"), largely due to the high quality of both the textbook and the video lectures. And as an R user, it was extremely helpful that they included R code to demonstrate most of the techniques described in the book.
If you are new to machine learning (and even if you are not an R user), I highly recommend reading ISLR from cover-to-cover to gain both a theoretical and practical understanding of many important methods for regression and classification. It is available as a free PDF download from the authors' website.
If you decide to attempt the exercises at the end of each chapter, there is a GitHub repository of solutions provided by students you can use to check your work.
As a supplement to the textbook, you may also want to watch the excellent course lecture videos(linked below), in which Dr. Hastie and Dr. Tibshirani discuss much of the material. In case you want to browse the lecture content, I've also linked to the PDF slides used in the videos.

Chapter 1: Introduction (slidesplaylist)

Chapter 2: Statistical Learning (slidesplaylist)

Chapter 3: Linear Regression (slidesplaylist)

Chapter 4: Classification (slidesplaylist)

Chapter 5: Resampling Methods (slidesplaylist)

Chapter 6: Linear Model Selection and Regularization (slidesplaylist)

Chapter 7: Moving Beyond Linearity (slidesplaylist)

Chapter 8: Tree-Based Methods (slidesplaylist)

Chapter 9: Support Vector Machines (slidesplaylist)

Chapter 10: Unsupervised Learning (slidesplaylist)

Interviews (playlist)

Wednesday, September 17, 2014

GEOtop binaries for Mac OS

After a few hacking I was able to transport my compiled version of GEOtop and libraries to a different machine. It is still a rough approximation of what it can be. However, it works. No compiling is required as before, just installations:

#You have to know, first, that in Mac OS there exist an application called "Terminal" in /Applications/Utilities/ folder. If you do not know browse a little the web to understand what is is about.  From Terminal, then,

# Install the following packages (or verify you have them already):

Install,  Homebrew and other required libraries.

#Install needed libraries

brew install boost
brew install gdal
brew install proj

#Install GEOtop components and libraries: this require, at the moment, a little of hand working

Download the executable zipped file from here.

Once unzipped the file look at the directories created and place the files you find in the "/Libraries" and "/usr" directories into the directories with the same name in your Mac hard disk. Both the directories can be found from the root with the Finder and while Library is visible, "/usr" is usually not. To make it visible, follow the instructions here.

As you can see, once unzipped the files, I chose to create a "/usr/local/GEOtop" directory where all the real executable are (has to be placed). This directory is actually not a standard one, However, I placed a symbolic link into /usr/local/bin, which is a directory usually already present in the Terminal path (to make a symbolic link by yourself, see here). Differently of the /Library and the directory "/usr/local/lib" and "/usr/local/include" that already exist, "/usr/local/GEOtop" has to be created, and therefore you can simply drag and drop the unzipped directory to copy it.

At this point everything should be set, if everything works fine, you should just execute GEOtop from any place simply by issuing:

$ geotop

If it runs, at this point it should give a runtime error:

Error::Run time error
Error::You need to export the WorkingPath environment variable before to run the program
........exiting...

But this is what it should be^1.  Now, that you are doing it right:

#Download the examples

You can find them here. Unzip them  in your directory, for instance ~/GEOtop, under your Home directory ("~" stands for your Home directory or,  given as absolute path for  "/Users/YourHomeNameHere/GEOtop").

#Try geotop examples

cd to the directory (folder) where the tests are), for instance:

~/GEOtop/tests/test_sample_run/small_example

Issue:

$ geotop ./

If you did it well and the goddess of programmers assisted you, it should work, as it did for me and in Tim's machine.

_________________________________________________________________________________

^1 - If it does not run try:


/usr/local/GEOtop/geotop/geotop

and you should have the same runtime error. This just means that all is correct except for, maybe, the symbolic link or your PATH in Terminal shell, that, for some reason, is not containing  "/usr/local/bin".

Monday, September 15, 2014

Opinions on the GEOtop roadmap

Just opinions indeed. For who landed here occasionally, information on GEOtop can be found here.

1 - Infrastructure

Mountain-eering and Exact-lab companies are doing a visible and positive effort in having a clean C++ code, after the huge work made by Stefano Endrizzi and Stephan Gruber to arrive to GEOtop 2.0. However, I believe the work on C++  should make some further steps, which I delineate in one of my previous posts here.
I deem necessary to do at least the first step described there, but the others would be also necessary in order that many people can work collaboratively on the project, and to maintain what we have while producing alternatives, and progress. My solution is, obviously just a proposal, and others (that I do not know) could exist. I plan to work myself part time on this from now to February, but if I can have help from someone this will be great. While changing the basic structure of the model's main( ), if  I will be really able to do it in this favourable astral conjunction,  I will take notes, and provide information to everybody. 

2 - Algorithms

On the side of the algorithms I believe we can work in several directions.  Using an unstructured grid will be a necessary intermediate step to pass through. 
Obviously this would imply a subsequent complete rewrite of the code, and in this rewriting we could take the occasion to: 1 - solve Richards equation with Casulli’s method, and 2 - better integrate surface waters with it. If resources would be available, even freezing soil could be rewritten, and the successful 2011 paper already contains how to do it.

3 - Processes

Among the processes, it would be important if the work by Stefano Dellachiesa on vegetation, and if the work of Florian Marshall on contaminant transport could be really embedded in the GEOtop main stream. My claim for more OO structure above, will go also in the direction to include these achievements, without cluttering the code and/or making more complicate the IO.
Personally, and in perspective, I am interested to include in GEOtop a modeling of the low atmosphere, integrating Navier-Stokes equation. I did some preliminary approach to the problem working conjointly with Dino Zardi and Michael Dumbser. A lot of resources are needed but I can work to grab them.

4 - OMS - Web services

My personal academic interest is also to push further  the evolution of GEOtop’s (beyond step 1) informatics for embedding GEOtop in OMS. This would allow splitting GEOtop  in components that can be joined at run-time through a scripting language, i.e. Groovy. This would enormously  increase the ability to run different modelling solutions, facilitate simulations reports, incapsulate even more the code. I am not claiming here, or pursuing, that this must be the main road followed by the GEOtop community. A certain “genetic variability” in the GEOtop versions and ecology would be indeed of benefit, and in favour of the model eventual survival. A passage necessary for OMS integration will be that task 1above will be pursued. That I think, should be really accomplished right away.

5 - Commitment and Executables

I already manifested my opinion to the developer group. Which is the following: in this moment using GEOtop require the total commitment of the guy/gal the want to embrace it. This is not acceptable if we want that our product would have some spreading. In this moment compiling GEOtop, still remains a challenge for a new users, and even for expert users, because they usually have a lot of other things to do, and they are pissed out when something that they consider trivial does not work as they expect. 
Downloading the source code directly from trunk should work for developers, but not for users. A tagged version, known to compile under the three major operating systems, should be used instead. More than that, a tarred zipped file, with the source code should be made available without passing through git. The best experience for a user would be, however, to download directly the executable for her/his machine and run it. I think that this move, together with improving the manual and documentation, will really boost GEOtop use. And, IMHO is the presence of a lot users that create a fertile soil for businesses.  Certainly a different pathway would be to implement GEOtop as a remote web-service. Recently at CSU,  I was impressed on what te eRams platform can offer, and how it helps in getting data and simulations together, and make easier collaborative work of groups.

6 -Testing

GEOtop comes with a series of tests, and, I verified, all of them compile, and all except one converge to the right solutions. This is useful for making users confident of what they are doing, and developers certain that they did not screwed up the system with their last minute modifications.  However, a special attention should be given to compare GEOtop with the tests provided by the increasing community of process-based model developers. They call their virtual experiment “the superslab” and you can find its definition here [link to be set up].  Being engaged with those guys would be really important, and would bring furhter recognition the our model. We could possibly be able also to challenge that community and propose tests that cope with the cryosphere, since none of its models does it. 

7 - Manual

The manual is not out-of -date but not up-to-date too. Efforts to get it improved would be necessary. I will try to put some students on it, if the occasion comes.

8 -License and Community

GEOtop is GPL and I believe it has to remain GPL. However, not in name of a single Author, but most probably in name of many. The creation of a GEOtop by components, could help in this direction, in the sense that single components, could be copyrighted then by single persons or group different from the original ones. The establishment of some GEOtop foundation with some organisation, and which determines some rules of behavior and/or an etiquette to follow for GEOtop use, will be a desired goal for future. I consider myself not eligible for any position in it though. 

9 - Courses

If a few of the conditions above would be met, it would not be difficult to start some schools either in Europe and in the Americas (or elsewhere). This also should be a goal to be pursued. Schools for Ph.D. students and Researchers, could not be very expensive, however some income can be generated.  Courses for professionals should be more expensive ... but maybe these could be directed to use web-services, and therefore the real income could arrive from subsequent use. 


10 - And then ?


Then it is matter to find resources and man/months to have all of it working.