This post is for those who start an internship or a PhD. It summarizes a couple of advices I give to my students.

2 principles:

To start with, here are two principles that should guide you:

1. "In 4 years, someone I don't know now should be able to easily find information in my stuff". This means that you should consider that you work as part of a team. In practice, it means for example: extensively comment you script, add readme.txt files to your folders to explain what's inside or where it comes from, don't use words like "toto","tutu","tata","tmp","old","very_old","new" for files or folders for more than a week, name files and folders for what they are or contain, etc ...
2. "Think to future tasks in order to organize present ones". I think the core principle here is modularity. It applies to folder's organization as well as code's structure. For instance, if you analyze a given dataset for a given project, the scripts for the analysis should be in a project folder and the data files in another. May be later, you'll conduct the same analysis on a different dataset or you use the same dataset to conduct a different analysis. In a code structure, this means that you should use functions and/or classes as soon as you copy/pasted the same code snippet more than 3 times !

1 example:

To set-up a clean working environment, you first need a root folder. It could be your home directory (it's often the case), but it could also be any folder from your disk. Just select it and place EVERYTHING underneath. Side effects will be a much easier handling of backups and a clean split between your work and private life if you use a single computer. Now that you have your root folder, let's take an example to illustrate the two principles stated above. Let's say you start an internship in a lab. You're going to use Matlab to conduct your analysis and visualization and you're going to use a dataset called Argo downloaded from a ftp server. Under the root folder, I suggest you have the following folders:

+ data
+ matlab
+ projects

Now, let's expand these folders:

- data
  - ARGO
    readme.txt
    + ftp
    + docs
    - matlab
      plot_one_float_trajectory.m
- matlab
    m_map -> m_map_1.4
  + m_map_1.2
  + m_map_1.4
  + copoda
  startup.m
- projects
  - my_first_analysis_of_Argo_data
    - analysis
      + figures
      load_coordinates.m
      map_profiles.m
    - reports
      + figures
      version0.doc
  + my_second_project_as_a_phd

Under the data folder, we have a folder dedicated to the Argo dataset. It contains:

• a readme.txt where you would say on which ftp server you downloaded the dataset and when
• a ftp folder where the data files are
• a docs folder where you will put documents related to this dataset such as the file format description for instance
• a matlab folder where you will place scripts that are not related to a scientific analysis or a specific project, but are related to generic tasks on the dataset

Whatever your use or analysis of this dataset, this folder is a standalone source for it. You can then share this folder to your team or get back to it in 4 years easily. Matlab has its own folder at the root level because in this way you will be able to handle your own toolboxes and customized environment easily. In this example I added several sub-folders.The "m_map" folder is in fact a link to the "m_map_1.4" one. This simple trick could be use to simply handle several versions and updates to a similar toolbox. Everywhere you need to refer to this toolbox, you point to the "m_map" folder (see startup.m file below). Then, with the link you point to the desired version. The "copoda" folder doesn't make use of this method because it's a toolbox handled using a svn repository. Versioning is thus delocalized. Last, you have the "startup.m" file. It will automatically be run by Matlab when starting a new session. In this file you will tell Matlab to load any toolbox. It could look like:

% Startup file for Matlab
% Last update: 2013/07/01
disp('Hello you !')

% Load toolboxes:
addpath('~/matlab/m_map')

% End of startup

Last, the Projects folder will contain all the real science work ! If there should be only thing to back up here, it would be this folder; all the others could be regenerated. Note that I created two sub-folders for the first project: "analysis" and "reports". The folder "analysis" will contain all the scripts and figures produced during the research phase, the folder "reports" will contain all the reports and hopefully articles about your work. Of course, all of this is an example to help beginners organize their work a minima. It all depends on your tools and common practices in your lab, although the two principles stated above should be kept in mind whatever your situation.

A while ago I wrote a post on how to create a clean netcdf file from Matlab using the Unidata netcdf toolbox. Starting from release 2010a, Matlab is now delivered with a built-in netcdf toolbox. And the API is not quite similar. So, I thought it would be nice top update the how-to as well. Here it is…

This is a Matlab script showing how to create a clean netcdf (4) file, including groups.

Create dummy data to put into the netcdf file

% Define axis, for example:
X = 0:2:360; % Longitude
Y = -90:2:90; % Latitude
Z = -5000:100:0; % Depth
T = datenum(1958:2006,1,1,0,0,0); % Time

% Create a random field to record in the cdf file dimensions are (Z,Y,X):
C = rand(length(Z),length(Y),length(X));

File standard set-up

Open file and insert dimensions

Note I exclude the temporal axis for now. So let’s create a new netCDF4 file:

scope = netcdf.create('myfirstcdffile.nc','netcdf4');

I call the netcdf pointer a scope and not ncid or nc because in netCDF4, we can create groups with their own dimensions and variables. So we need to know where we are when defining dimensions and so on. It will become clear in the next section.

% Define useful constants:
NC_GLOBAL = netcdf.getConstant('NC_GLOBAL');
fillValue = -9999;

% Define dimensions:
dimidZ = netcdf.defDim(scope,'depth',length(Z));
dimidY = netcdf.defDim(scope,'latitude',length(Y));
dimidX = netcdf.defDim(scope,'longitude',length(X));

% Define axis:
varid = netcdf.defVar(scope,'depth','double',[dimidZ]);         
netcdf.putAtt(scope,varid,'standard_name','depth');
netcdf.putAtt(scope,varid,'long_name','Vertical distance below the surface');
netcdf.putAtt(scope,varid,'units','m');
netcdf.defVarFill(scope,varid,false,fillValue);
netcdf.putVar(scope,varid,Z);

varid = netcdf.defVar(scope,'latitude','double',[dimidY]);
netcdf.putAtt(scope,varid,'standard_name','latitude');
netcdf.putAtt(scope,varid,'long_name','Station latitude');
netcdf.putAtt(scope,varid,'units','degrees_north');
netcdf.defVarFill(scope,varid,false,fillValue);
netcdf.putVar(scope,varid,Y);

varid = netcdf.defVar(scope,'longitude','double',[dimidX]);
netcdf.putAtt(scope,varid,'standard_name','longitude');
netcdf.putAtt(scope,varid,'long_name','Station longitude');
netcdf.putAtt(scope,varid,'units','degrees_east');
netcdf.defVarFill(scope,varid,false,fillValue);
netcdf.putVar(scope,varid,X);

Insert global attributes

These are just my stuff, you need to customize. But note attributes related to the convention. This is very useful for you netcdf file to be read by anybody else than you.

netcdf.putAtt(scope,NC_GLOBAL,'title','My first netcdf file')
netcdf.putAtt(scope,NC_GLOBAL,'long_title','Self explanatory, isnt''it ?')
netcdf.putAtt(scope,NC_GLOBAL,'comments','All variables are just noise !')
netcdf.putAtt(scope,NC_GLOBAL,'institution','LPO/Ifremer')
netcdf.putAtt(scope,NC_GLOBAL,'source','my_model')

netcdf.putAtt(scope,NC_GLOBAL,'Conventions','CF-1.6')   
netcdf.putAtt(scope,NC_GLOBAL,'Conventions_help','http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html')

netcdf.putAtt(scope,NC_GLOBAL,'CreationDate',datestr(now,'yyyy/mm/dd HH:MM:SS'))
netcdf.putAtt(scope,NC_GLOBAL,'CreatedBy',getenv('LOGNAME'))
%netcdf.putAtt(scope,NC_GLOBAL,'MatlabSource',which('your_matlab_script_generating_this_file'))

Check what we’ve just done

We can now close the netcdf file (optional):

netcdf.close(scope)

and go in a terminal to look at it with:

ncdump -h myfirstcdffile.nc

or simply from Matlab display a minimum of informations:

>> ncdisp('myfirstcdffile.nc')
Source:
           /Users/gmaze/work/Production/How to : Tutorials/myfirstcdffile.nc
Format:
           netcdf4
Variables:
    depth    
           Size:       51x1
           Dimensions: depth
           Datatype:   double
    latitude 
           Size:       91x1
           Dimensions: latitude
           Datatype:   double
    longitude
           Size:       181x1
           Dimensions: longitude
           Datatype:   double

or a maximum:

>> ncdisp('myfirstcdffile.nc','/','full')
Source:
           /Users/gmaze/work/Production/How to : Tutorials/myfirstcdffile.nc
Format:
           netcdf4
Global Attributes:
           title            = 'My first netcdf file'
           long_title       = 'Self explanatory, isnt'it ?'
           comments         = 'All variables are just noise !'
           institution      = 'LPO/Ifremer'
           source           = 'my_model'
           Conventions      = 'CF-1.6'
           Conventions_help = 'http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html'
           CreationDate     = '2013/02/14 11:03:09'
           CreatedBy        = 'gmaze'
Dimensions:
           depth     = 51
           latitude  = 91
           longitude = 181
Variables:
    depth    
           Size:       51x1
           Dimensions: depth
           Datatype:   double
           Attributes:
                       standard_name = 'depth'
                       long_name     = 'Vertical distance below the surface'
                       units         = 'm'
                       _FillValue    = -1e+04
    latitude 
           Size:       91x1
           Dimensions: latitude
           Datatype:   double
           Attributes:
                       standard_name = 'latitude'
                       long_name     = 'Station latitude'
                       units         = 'degrees_north'
                       _FillValue    = -1e+04
    longitude
           Size:       181x1
           Dimensions: longitude
           Datatype:   double
           Attributes:
                       standard_name = 'longitude'
                       long_name     = 'Station longitude'
                       units         = 'degrees_east'
                       _FillValue    = -1e+04

Note that the ncdisp function can be used only with the file name. Here I used the complete syntax to highlight the specificity of netCDF4, with the slash /, we ask for information at the top location of the file. We don’t have groups yet, but we could have also asked for informations about a specific group.

Insert a 3D field

If the file was closed, we need to reopen it:

scope = netcdf.open('myfirstcdffile.nc','WRITE')  % Rq: here we use 'WRITE' because the file already exists

The variable we want to record is in the Matlab field C but we have to give it a name and define metadata for the netcdf file:

varname = 'my_first_field';
long_name = 'A long description of what this is, well simply noise !';
unit = 'm/s'; % for example of course

And now we can write it:

varid = netcdf.defVar(scope,varname,'double',[dimidZ, dimidY, dimidX]);
netcdf.putAtt(scope,varid,'name',varname);
netcdf.putAtt(scope,varid,'long_name',long_name);
netcdf.putAtt(scope,varid,'units',unit);
netcdf.defVarFill(scope,varid,false,fillValue);
%netcdf.defVarDeflate(scope,varid,true,true,9);

v = C;
    v(isnan(v)) = fillValue;
    netcdf.putVar(scope,varid,v);

We can now close the netcdf file (optional):

netcdf.close(scope)

and from Matlab:

ncdisp('myfirstcdffile.nc')

and see the new variable we just added !

Insert a 2D field

The procedure is similar to the 3D case, except that when defining the new variable the dimensions are different:

iz = 1; % Define the level
varname = sprintf('%s_level%i',varname,iz);
long_name = 'Still a long description of what this is, well simply noise !';
unit = 'm/s';
varid = netcdf.defVar(scope,varname,'double',[dimidY, dimidX]);
netcdf.putAtt(scope,varid,'name',varname);
netcdf.putAtt(scope,varid,'long_name',long_name);
netcdf.putAtt(scope,varid,'units',unit);
netcdf.defVarFill(scope,varid,false,fillValue);
v = squeeze(C(iz,:,:));
    v(isnan(v)) = fillValue;
    netcdf.putVar(scope,varid,v);

Handling the time axis

You probably use the native Matlab format for your date/time information: datenum. But this is not how conventional netcdf handles time.

Insert the new dimension

Usually, I define my netcdf time axis as the days (can be fractional) since 1900–01–01. So let’s reopen the file:

scope = netcdf.open('myfirstcdffile.nc','WRITE');  % Rq: here we use 'WRITE' because the file already exists

and add the new time axis. First, define the dimension:

dimidT = netcdf.defDim(scope,'time',length(T));

Then add the variable:

varid = netcdf.defVar(scope,'time','double',[dimidT]);          
netcdf.putAtt(scope,varid,'standard_name','time');
netcdf.putAtt(scope,varid,'long_name','Time of measurements');
netcdf.putAtt(scope,varid,'units','days since 1900-01-01 00:00:00');
netcdf.defVarFill(scope,varid,false,fillValue);
netcdf.putVar(scope,varid,T-datenum(1900,1,1,0,0,0));

NOTE THAT THIS IS ONLY MY CHOICE, YOU NEED TO CUSTOMIZE WITH YOUR REQUIREMENTS

Insert a time varying field

Now we create a dummy variable, supposedly a timeserie of an horizontal 2D fields:

D = rand(length(T),length(Y),length(X));

And record it:

varname = 'my_time_serie';
long_name = 'This is a time serie of noise !';
unit = 'degC';

varid = netcdf.defVar(scope,varname,'double',[dimidT,dimdY,dimdX]);         
netcdf.putAtt(scope,varid,'long_name',long_name);
netcdf.putAtt(scope,varid,'units',unit);
netcdf.defVarFill(scope,varid,false,fillValue);
v = D;
    v(isnan(v)) = fillValue;
    netcdf.putVar(scope,varid,v);   

We can now close the netcdf file (optional):

netcdf.close(scope)

and from Matlab:

ncdisp('myfirstcdffile.nc')

Variant with groups

This is one of the main new possibility of netCDF4: you can put some structure into the netcdf file itself. I won’t go into the details, but this simply like sub-folders into the file. Let’s see with an example. Imagine that we want we the same netcdf file to contain model output (or observations) as well as some results of their analysis, like say a principal component timeserie from an EOF analysis. We could define the netCDF4 file structure like:

global_scope  = netcdf.create('myfirstcdf4file.nc','netcdf4');
netcdf.putAtt(global_scope,NC_GLOBAL,'Comment','This file is not a mess ! It is organized by group')

data_scope    = netcdf.defGrp(global_scope,'Raw_data');
netcdf.putAtt(data_scope,NC_GLOBAL,'Comment','This group or part of the file contains raw data')

another_scope = netcdf.defGrp(global_scope,'Analysis');
netcdf.putAtt(another_scope,NC_GLOBAL,'Comment','This second group or part of the file contains analysis data')

and try:

ncdisp('myfirstcdf4file.nc')

wait ! this throw an error:

Error using netcdflib
The NetCDF library encountered an error during execution of 'open' function - 'Unknown file format (NC_ENOTNC)'.

Error in netcdf.open (line 60)
        [varargout{:}] = netcdflib ( 'open', filename, varargin{1} );

Error in internal.matlab.imagesci.nc/openToRead (line 1258)
            this.ncRootid = netcdf.open(this.Filename,'NOWRITE');

Error in internal.matlab.imagesci.nc (line 122)
                    this.openToRead();

Error in ncdisp (line 51)
ncObj   = internal.matlab.imagesci.nc(ncFile);

Ok, this was simply to show you that there is a difference between what Matlab knows or wants to do with the actual file state on disk. We need to synchronize both with a sync:

netcdf.sync(global_scope)
ncdisp('myfirstcdf4file.nc')

which returns:

Source:
           /Users/gmaze/work/Production/How to : Tutorials/myfirstcdf4file.nc
Format:
           netcdf4
Groups:
    /Raw_data/

    /Analysis/

this is where the ncdisp parameter location is useful, we can display information for a single group, try, for instance:

ncdisp('myfirstcdf4file.nc','Raw_data','full')

Now, each group can contain its own dimensions and variables, and filling in the data, is as simple as with the previous case, we just need to specify the correct scope of where we want to put information. Consider the following script implementing our use case scenario:

%% Setup data for entire file:

netcdf.putAtt(global_scope,NC_GLOBAL,'title','My first netCDF4 file')
netcdf.putAtt(global_scope,NC_GLOBAL,'long_title','Self explanatory, isnt''it ?')
netcdf.putAtt(global_scope,NC_GLOBAL,'comments','Groups are very useful !')
netcdf.putAtt(global_scope,NC_GLOBAL,'institution','LPO/Ifremer')
netcdf.putAtt(global_scope,NC_GLOBAL,'source','my_model')
netcdf.putAtt(global_scope,NC_GLOBAL,'Conventions','CF-1.6')    
netcdf.putAtt(global_scope,NC_GLOBAL,'Conventions_help','http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.4/cf-conventions.html')
netcdf.putAtt(global_scope,NC_GLOBAL,'CreationDate',datestr(now,'yyyy/mm/dd HH:MM:SS'))
netcdf.putAtt(global_scope,NC_GLOBAL,'CreatedBy',getenv('LOGNAME'))
%netcdf.putAtt(global_scope,NC_GLOBAL,'MatlabSource',which('your_matlab_script_generating_this_file'))

%% Put data into the first group:

% Define dimensions:
dimidT = netcdf.defDim(data_scope,'time',length(T));    
dimidY = netcdf.defDim(data_scope,'latitude',length(Y));
dimidX = netcdf.defDim(data_scope,'longitude',length(X));

% Define axis:
varid = netcdf.defVar(data_scope,'time','double',[dimidT]);         
netcdf.putAtt(data_scope,varid,'standard_name','time');
netcdf.putAtt(data_scope,varid,'long_name','Time of measurements');
netcdf.putAtt(data_scope,varid,'units','days since 1900-01-01 00:00:00');
netcdf.defVarFill(data_scope,varid,false,fillValue);
netcdf.putVar(data_scope,varid,T-datenum(1900,1,1,0,0,0));

varid = netcdf.defVar(data_scope,'latitude','double',[dimidY]);
netcdf.putAtt(data_scope,varid,'standard_name','latitude');
netcdf.putAtt(data_scope,varid,'long_name','Station latitude');
netcdf.putAtt(data_scope,varid,'units','degrees_north');
netcdf.defVarFill(data_scope,varid,false,fillValue);
netcdf.putVar(data_scope,varid,Y);

varid = netcdf.defVar(data_scope,'longitude','double',[dimidX]);
netcdf.putAtt(data_scope,varid,'standard_name','longitude');
netcdf.putAtt(data_scope,varid,'long_name','Station longitude');
netcdf.putAtt(data_scope,varid,'units','degrees_east');
netcdf.defVarFill(data_scope,varid,false,fillValue);
netcdf.putVar(data_scope,varid,X);

% Insert data:
varid = netcdf.defVar(data_scope,'SST','double',[dimidT, dimidY, dimidX]);
netcdf.putAtt(data_scope,varid,'name','Sea_Surface_Temperature');
netcdf.putAtt(data_scope,varid,'long_name','Temperature at the surface of the ocean');
netcdf.putAtt(data_scope,varid,'units','degC');
netcdf.defVarFill(data_scope,varid,false,fillValue);
v = D;
    v(isnan(v)) = fillValue;
    netcdf.putVar(data_scope,varid,v);


% Synchronize:
netcdf.sync(data_scope)

% Look at what we've just done:
disp('------ ncdisp of file (after data insertion in the global scope and one group):')
ncdisp('myfirstcdf4file.nc','/','full')
disp('------')

Now, let’s put data into the second group. Image we performed an EOF analysis only over the first 40 snapshots of the variable named SST and recorded in the Raw_data group. We want to save the first principal component of the analysis:

%% Put data into the second group:

% Define dimensions:
T2 = T(1:40);
dimidT = netcdf.defDim(another_scope,'time',length(T2));    

% Define axis:
varid = netcdf.defVar(another_scope,'time','double',[dimidT]);          
netcdf.putAtt(another_scope,varid,'standard_name','time');
netcdf.putAtt(another_scope,varid,'units','days since 1900-01-01 00:00:00');
netcdf.defVarFill(another_scope,varid,false,fillValue);
netcdf.putVar(another_scope,varid,T2-datenum(1900,1,1,0,0,0));

% Insert data:
varid = netcdf.defVar(another_scope,'PC','double',[dimidT]);
netcdf.putAtt(another_scope,varid,'name','Principal_Component');
netcdf.putAtt(another_scope,varid,'units','none');
netcdf.defVarFill(another_scope,varid,false,fillValue);
v = rand(length(T2),1);
    v(isnan(v)) = fillValue;
    netcdf.putVar(another_scope,varid,v);

% Synchronize:
netcdf.sync(another_scope)

Note how a dimension and a variable can be defined in 2 groups and have different properties. Before netCDF4 this would have forced us to create a dimension and a variable with a different name. So the final structure of our netCDF4 file is:

>> ncdisp('myfirstcdf4file.nc','/','full')

Source:
           /Users/gmaze/work/Production/How to : Tutorials/myfirstcdf4file.nc
Format:
           netcdf4
Global Attributes:
           Comment          = 'This file is not a mess ! It is organized by group'
           title            = 'My first netCDF4 file'
           long_title       = 'Self explanatory, isnt'it ?'
           comments         = 'Groups are very useful !'
           institution      = 'LPO/Ifremer'
           source           = 'my_model'
           Conventions      = 'CF-1.6'
           Conventions_help = 'http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.4/cf-conventions.html'
           CreationDate     = '2013/02/14 12:21:35'
           CreatedBy        = 'gmaze'
Groups:
    /Raw_data/
        Attributes:
                   Comment = 'This group or part of the file contains raw data'
        Dimensions:
                   time      = 49
                   latitude  = 91
                   longitude = 181
        Variables:
            time     
                   Size:       49x1
                   Dimensions: time
                   Datatype:   double
                   Attributes:
                               standard_name = 'time'
                               long_name     = 'Time of measurements'
                               units         = 'days since 1900-01-01 00:00:00'
                               _FillValue    = -1e+04
            latitude 
                   Size:       91x1
                   Dimensions: latitude
                   Datatype:   double
                   Attributes:
                               standard_name = 'latitude'
                               long_name     = 'Station latitude'
                               units         = 'degrees_north'
                               _FillValue    = -1e+04
            longitude
                   Size:       181x1
                   Dimensions: longitude
                   Datatype:   double
                   Attributes:
                               standard_name = 'longitude'
                               long_name     = 'Station longitude'
                               units         = 'degrees_east'
                               _FillValue    = -1e+04
            SST      
                   Size:       49x91x181
                   Dimensions: time,latitude,longitude
                   Datatype:   double
                   Attributes:
                               name       = 'Sea_Surface_Temperature'
                               long_name  = 'Temperature at the surface of the ocean'
                               units      = 'degC'
                               _FillValue = -1e+04

    /Analysis/
        Attributes:
                   Comment = 'This second group or part of the file contains analysis data'
        Dimensions:
                   time = 40
        Variables:
            time
                   Size:       40x1
                   Dimensions: time
                   Datatype:   double
                   Attributes:
                               standard_name = 'time'
                               long_name     = 'Time of measurements'
                               units         = 'days since 1900-01-01 00:00:00'
                               _FillValue    = -1e+04
            PC  
                   Size:       40x1
                   Dimensions: time
                   Datatype:   double
                   Attributes:
                               name       = 'Principal_Component'
                               units      = ''
                               _FillValue = -1e+04

I don't use the vi/vim editor much and I found myself constantly looking for useful commands. Here they are, thanks to lagmonster.org.

I've been a Firefox user for several years, and to be honest, since I had my first Sun station to work with ! I moved to Mac OS and kept using Firefox because I never fell in love with Safari. But as new updates keep pilling up new functions and a bunch of other stuff, I started to find Firefox buggy, slow and unstable. So, I've been patient and as soon as Google released a Mac OS version of its navigator Chrome I gave it a try. What a charm ! It is fast, stable, works intuitively and on my small 13' MacBook screen optimizes the space taken by the webpage.

I started aggregating a bunch of useful sites to look for GFD informations on the web, some kind of specialized Google search engine.
You can find its homepage here:
http://www.google.com/cse/home?cx=006346700993241352089:qyczjxt9rgc
or here:
http://www.ifremer.fr/lpo/files/gmaze/gfd_search.html

If you want to insert it on your webpage, visit this page, or if you want to add it to your Google home page click on this button:
But if you want to add it to your Firefox Custom Search Engines, visit this page (which will reveal the engine parameters) and simply try to change the search engine on your Firefox Search toolbar, you should see an Add "GFD Search" line, click on it and you're all set !
The xml file is here.

MIT provides a very nice method to access articles from publishers. It's through their library proxy. I used to use a simple trick, simply adding .libproxy.mit.edu at the end of the article url hostname. For example I changed: