Planet Octave

strel class

octave> im1 = im2 = randp (5, 2000) > 15;
octave> t = cputime ();
octave> im1 = conv2 (im1, true (20), "same") > 0; # dilation by 1 square
octave> cputime () - t
ans =  2.6402
octave> t = cputime ();
octave> im2 = conv2 (im2, true (20, 1), "same") > 0; # dilation by 1 column
octave> im2 = conv2 (im2, true (1, 20), "same") > 0; # dilation by 1 row
octave> cputime () - t
ans =  0.52803
octave> isequal (im1, im2)
ans =  1

Non-flat morphology

Spatial transformations

Have you ever looked up at the sky on a clear, moonless night? When all you can see is an endless field of stars, glittering like an array of celestial diamonds speckled across a cosmic fabric? Did it make you think, "I wonder if I could simulate this in the browser using transparent images and CSS"?

That has never happened to me either, actually. But it was the best segue into this subject that I could think of.

Recently, I built the frontend for a visualisation of Christmas wishes on Twitter. My biggest consideration when thinking of a design was that I wanted to make it somewhat Christmas-themed while avoiding kitschiness as much as possible. I settled on a night sky backdrop, with falling snowflakes simulated using a modified version of this jQuery plugin (okay, a little kitschy, but I'd like to think it embodies a certain classiness as well). The backdrop ended up looking like this:

It looks simple enough, but the technique is actually rather involved. Why? Because the sky is meant to expand to fill the user's browser. Creating a fixed-size image containing randomly-positioned stars would not suffice, because then that image would either have be tiled (which would be very noticeable, ruining the illusion of the endless starfield) or stretched (which could result in very large and pixellated stars for users with larger screens). Neither solution is desirable.

So what's the alternative, if you want to preserve the illusion that these stars go on forever without having to actually generate an endless starfield? My solution was to separate the stars into three different layers: small stars, medium stars, and large stars. Each layer would be a mostly-transparent PNG image, with white dots for stars. And most importantly, the widths and heights of each layer would have to be relatively prime.

In my original design (the one pictured above uses images that are half the size of the original ones, for illustration purposes), the dimensions were:

• Small stars layer: 1103 wide × 541 high
• Medium stars layer: 1009 wide × 479 high
• Larger stars layer: 1051 wide × 524 high

How many pixels of "randomness" do we get before it repeats? Well, all the numbers except 524 are prime, and we can see immediately that 541, 479 and 524 are relatively prime. So the non-repeating area has a width of 1103 × 1009 × 1051 = 1,169,686,277 pixels (the lcm of the individual widths), and a height of 541 × 479 × 524 = 135,788,836. The result is more than hundred million pixels of vertical scroll space and more than a billion pixels of horizontal scroll space before we encounter a pattern of stars that we've seen before. Not bad for less than 14KB of images. (In fact, these numbers are a lot higher than they need to be - anything above 300 or 400 pixels would probably work just as well, with the period being well over a million pixels.)

So that's the basic idea. DesignFestival calls this the cicada principle, which is a catchy if somewhat silly name for something that is a basic property of numbers (there are some really cool examples here). If you want to use this concept in your own project, the HTML structure looks like this:

<div id="stars">
    <div class="large"></div>
    <div class="medium"></div>
    <div class="small"></div>
</div>

Feel free to grab the CSS and the images (stars_large.png, stars_medium.png, stars_small.png), though making your own images is pretty fun as well (I spent a few minutes pasting white dots at random positions in an image document in Inkscape; it was a blast). If you do use this in your own project, I would love to hear about it.

• dependencies on other packages should be automatically solved
• pkg() will not load packages automatically
• an option to install packages given a URL will be added
• the source of installed packages will be kept in disk for times installations
• it will be possible to have multiple package lists that can be merged or replaced
• support for different packages version and different octave versions will be added
• pkg() will stay written in the Octave language
• the -local and -global options will be removed
• a header will be added to the octave list files warning that they should not be manually edited

Breakpoints

Interrupts

i = 0;
while 1
  i += 1;
endwhile

This past week has been full of midterms and assignments so this update will be shorter than usual. I made a few small changes (feature additions and bug fixes) to the bundle feature, mostly based on feedback from the mailing list.

This week

The full commit log is available here: http://inversethought.com/hg/hgwebdir.cgi/agora-dellsystem/

• If an uploaded bundle contains a file named DESCRIPTION, and the bundle's description is empty, and the bundle is indicated to follow Octave packaging formatting standards, then the contents of that file should be used as the description (commit 184, commit 187)
• If the user has already created a bundle with the specified name, then the form should display a validation error rather than 500'ing (commit 183)
• Fixed the submit button text in the bundle form (commit 182)

Next week

I'll continue working on bundles and housekeeping changes (see last week's update). I'll also look into documentation integration (suggested by Carlo de Falco via the mailing list).

In other news, I got an email from ESA a few days ago informing me that I have successfully completed SOCIS 2012. Much thanks to my mentor Jordi Gutiérrez Hermoso for all of his help as well as the positive feedback. I will continue to work on Agora Octave after the end of the official coding period (which I just realised is tomorrow, making this my last official blog post :() for as long as it takes to get the project in a usable state, although the frequency of my blog posts may decrease as I get closer to exam period. I'm also interested in eventually getting more involved with Octave as a whole, and hopefully even contributing to Octave itself one day — at the very least, it would be a great opportunity to learn C++. This will probably have to wait until next semester, or at least until I finish my exams (middle/end of December).

Package instrument-control

• Fixed a nasty bug (r11317) which was hard to find as I failed to initialize structures to default values which meant it appeared/disappeared pretty much randomly
• Fixed serial compatibility with Mac OSX (r11344) where it sometimes hanged while opening the interface
• Miscellaneous bug fixes and improvements (r11315, r11316)

I worked primarily on the bundle feature this week. I added a rudimentary versioning system and made some other smaller changes, including fixing the bundle explore page, making bundle descriptions optional and fixing the 404 and 500 error page templates.

This week

The full commit log is available here: http://inversethought.com/hg/hgwebdir.cgi/agora-dellsystem/

Bundle versioning and editing

You can now upload a new version of a bundle (either as an archive or as a plain text file - commit 177). Files in previous versions of the bundle can be accessed by viewing the bundle at a particular version.

You can also now edit the details (currently only the description and license - the name is not editable as it constitutes part of the URL) of a bundle that you are the creator of.

Additionally, bundle descriptions are now optional (commit 176).

Smaller changes

• Added help text to name field for bundles, explaining that the name is meant to be a slug (part of the URL) (commit 171)
• Made form field labels bold (commit 172
• Added the ability to whitelist certain mimetypes that do not begin with text/ (currently only XML - commit 179)
• Added number of views to snippet details page (commit 175)
• Added publish date and number of views to snippet explore page (commit 173)
• Fixed 404 and 500 error pages (hadn't yet converted the template files over to the new template system - commit 174)

Next week

More on bundles

Here's last week's version, with the completed ones removed:

• Ensuring that file uploading is secure (for instance, it needs to be able to handle zip bombs without exploding)
• Displaying images and allowing downloads for individual files
• Adding license information to uploaded files (see this Perl script)
• Downloads (so that users can easily download the contents of a bundle as an archive file)
• Rating/feedback functionality
• Mercurial integration (eventually)

Here are other things that need to be done:

• Showing the timestamp and an optional comment for new versions of bundles
• Fixing the left bar for listing files in a bundle (if the structure is deeply nested or if the filenames are particularly long, the user will have to scroll horizontally, which can be unwieldy)

Housekeeping changes

(Continued from last week, since I didn't have time to do this for the entire codebase.) Deleting unused code, making use of Django shortcuts and other conveniences when possible, checking concordance with PEP 8, etc.

Position in timeline

• August 1-15: Check out codebase, get it running; read all the code and documentation; start learning Mercurial
• August 16-31: Start working on design; start mocking up models and routes for the components
• September 1 - October 15: Work on the core components, with as much test coverage and documentation as is feasible; start implementing the views, models and template
• October 16-31: Ensure that documentation is thorough and up to date; write any remaining tests that need to be written; test out the user interface and fix any bugs

(This timeline isn't really relevant anymore, but I'll keep including it for reference.)

Package instrument-control

• Added i2c interface support for FreeBSD (r11276). Which means that the FreeBSD platform is now fully supported
• Confirmed Serial support for Mac OSX (it is surprisingly time consuming to get a OSX on <ehem> macbook running...)
• Investigated i2c/parallel support for OSX. I do not want to promise anything, but things are looking positive so far, will update on this matter next week.

Lots of progress on the bundle feature this week. I also significantly revised my plans for the feature, based on some very helpful feedback from the mailing list. Some other smaller changes - bug fixes, minor feature additions, cosmetic changes - were committed as well.

This week

Bundles, continued

I've committed all the progress I've made over the past couple of weeks on bundles. See commit 151. It's still incomplete, but most of the basic functionality is there. Feel free to test it out for yourself.

Optimising dynamic line-number insertion

The dynamic line-number insertion was extraordinarily slow for long snippets. Optimising it was pretty simple, though - I just had to minimise the number of DOM manipulations and to change the $.each() call to a standard for loop (see commit 141). I should really have done this from the beginning, but at the time I thought that the initial method was neater and that performance wouldn't be an issue, which turned out to be very naive.

LESS compilation

I added server-side LESS compilation as an option (commit 143). To trigger it, set compile_less = yes in agora.conf. Now, stylesheets load faster, still work with Javascript turned off, and aren't cached in localStorage when you don't want them to be.

Contributions sidebar for user profiles

There is now a sidebar on user profile pages showing what snippets and bundles that user has created (commit 152).

Fixing registration

There was a bug in registration before relating to an incorrect usage of the django-registration app. It should be fixed now (commit 160).

Other cosmetic changes

Minor cosmetic changes include:

• Adding word-wrap: break-word to diffs and snippets (so that extremely long words don't break out of the box - commits 158 and 161)
• Adding overflow-x to the snippet history list in the sidebar, to ensure that it never gets too cramped (commit 145)
• Adding a top margin to the diff box to ensure that it is aligned with the sidebar (commit 146)
• Preventing the browser from trying to jump to # when you click the "Highlight code" link (commit 142)

Next week

More on bundles

Here's a revised version of the to-do list for bundles from last week:

• Ensuring that file uploading is secure (for instance, it needs to be able to handle zip bombs without exploding)
• Displaying images and allowing downloads for individual files
• Whitelist for certain mimetypes (e.g., XML) that don't begin with text/ but should still be considered plain text
• Versioning (the ability to upload subsequent versions of a bundle, each associated with a single incrementing version number)
• Adding license information to uploaded files (see this Perl script)
• Downloads (so that users can easily download the contents of a bundle as an archive file)
• Rating/feedback functionality
• Mercurial integration (eventually)

I'll be working on the downloads feature first, then versioning and licensing. I'll probably also come up with other small features while working on these.

Housekeeping changes

Deleting unused code, making use of Django shortcuts and other conveniences when possible, checking concordance with PEP 8, etc.

Position in timeline

• August 1-15: Check out codebase, get it running; read all the code and documentation; start learning Mercurial
• August 16-31: Start working on design; start mocking up models and routes for the components
• September 1 - October 15: Work on the core components, with as much test coverage and documentation as is feasible; start implementing the views, models and templates
• October 16-31: Ensure that documentation is thorough and up to date; write any remaining tests that need to be written; test out the user interface and fix any bugs

(This timeline isn't really relevant anymore, but I'll keep including it for reference.)

Package instrument-control

• Added Parallel interface support for FreeBSD (r11219)
• Updated Octave Wiki with a compatibility table
• Changed Makefile so now it tries to compile all possible interfaces even if some of them fail (see below)

I made quite a bit of progress on bundle/module uploading this week. A lot of it is behind-the-scenes work, and there are still a few things that need to be done before it's production-ready, so unfortunately I don't have a working demo yet. For a sneak preview of what's coming, check out the screenshots below.

This week

Bundles

My main focus was working on the bundle uploading feature. The idea is that you should be able to upload either a plain text file or an archive (zip, tar, etc) containing the files you want to share. If you upload an archive, then it should be unpacked on the server, and the contents of the archive should be restored with their original filenames and directory structure, and shown on the page with syntax highlighting (if possible).

The naive approach to extracting the archive file is to just unpack it immediately after it is uploaded and display the contents of the archive afterwards. However, extracting archive files is something that can take quite a bit of time, particularly for large files. Handling this in the server thread would result in long wait-times and, in the case of a single-threaded server, would prevent others from loading the site at all while a zip file is being extracted. This is, of course, unacceptable for any site that needs to be used in production.

The solution? Using an asynchronous task queue system. I chose celery because I've used it successfully in previous projects and because it integrates well with Django. Although this definitely increases the complexity of Agora, in terms of deployment, dependencies, and the codebase, it's not really avoidable due to the nature of the functionality that Agora requires. Other ways of executing tasks (like archive extraction) asynchronously, such as using the subprocess module, are simply not feasible because of the post-extraction processing that needs to be done.

The result is that creating a bundle only takes as long as uploading the file. Once the bundle has been created, the user is redirected to the bundle view page. If the file is still being processed, the user will be shown something like this:

Once the file processing is complete, the message will disappear, and users can view the files in the bundle. If a single plain-text file was uploaded, its contents (with syntax highlighting) will be displayed on the page, along with its filename:

If, instead, an archive file containing multiple files were uploaded, then the contents of all the plain-text files (of reasonable length) will be shown instead:

The interface will be very similar to that of snippets, with the main differences being that: 1) filenames (and directory structures) are incorporated; and 2) each bundle can contain multiple "snippets".

Line numbers revisited

I took another look at using CSS counters for displaying line numbers, which I mentioned briefly in my last post. It turns out that I was incrementing the counter incorrectly last time - instead of doing:

.line:before {
    counter-increment: line;
    content: counter(line);
}

I should have been doing this:

.line {
    counter-increment: line;
}

.line:before {
    content: counter(line);
}

(and then resetting the counter for each snippet encountered).

The problem with this technique is that there doesn't appear to be a way to make the line numbers hyperlinks. One way around this is to use Javascript to make the line itself a hyperlink, so that when you click anywhere on a line of code you're taken to the relevant anchor. Of course, this still requires Javascript, so I don't know if it's really a better solution. It's also a bit harder to style. The current implementation (using Javascript to display the line numbers) should suffice for now, so I'm going to table this issue and only come back to it if it turns out there's a major problem with the current method.

Next week

Finishing up bundles

Sadly, bundle uploading is far from complete. I haven't committed the changes I've made yet, partly because I had to do a fair bit of restructuring of the way imports are done in order to get celery to work, which seems to have broken the snippet feature. It's also very much a work-in-progress - certainly not stable enough to be used in production - and I want to get it in a more acceptable state, with adequate documentation, before committing it.

Here are some of the things that I am prioritising for next week:

• Updating the README and the requirements file to account for the new dependencies and setup instructions
• Fixing the snippet functionality (it has to do with imports being changed from from agora.apps.etc import Etc to from apps.etc import Etc, which causes the mptt app to register the Snippet model twice)
• Testing edge cases (empty archive files, different types of compression, maliciously named files, etc)
• Displaying images, and allowing users to download other types of files
• Committing the changes and updating my demo

Some more long-term goals for the bundle feature include:

• Ensuring that file uploading is secure (for instance, it needs to be able to handle zip bombs without exploding)
• A bundle management system (so that users can rename, delete, and add files after creating a bundle)
• Downloads (so that users can easily download the contents of a bundle as an archive file)
• Figuring out terminology (bundles, modules or something else entirely?)
• Mercurial integration (eventually)

I'm going to be buried in schoolwork this week, so I won't be able to focus on Agora again until Friday. The plan is to finish up the most important aspects of this feature on Friday, commit, and then start working on the others. There are also some housekeeping changes I'd like to make, so hopefully I'll have time to get to those as well (in terms of improving code organisation and the like).

Position in timeline

• August 1-15: Check out codebase, get it running; read all the code and documentation; start learning Mercurial
• August 16-31: Start working on design; start mocking up models and routes for the components
• September 1 - October 15: Work on the core components, with as much test coverage and documentation as is feasible; start implementing the views, models and templates
• October 16-31: Ensure that documentation is thorough and up to date; write any remaining tests that need to be written; test out the user interface and fix any bugs

Package instrument-control

• Retested package which resulted in a few small fixes (r11181, r11180, r11161)
• Submitted package for approval/inclusion to Octave Forge for Public Release
• Updated Octave Wiki with a few examples (diff)

Let me share with you a little tale involving line numbers.

One of the required features for Agora Octave (which I've been working on for the past couple of months, as part of the European Space Agency's Summer of Code 2012) is the ability to display syntax-highlighted code. Ideally, the code would come adorned with correctly-aligned line numbers as well (which can be linked to directly), and the code would be wrapped around so that you don't have to scroll horizontally to view the end of long lines.

Here's an example of what it should look like:

My first attempt towards this noble goal was a fairly lackluster one that resulted in a horizontal scrollbar whenever there was one line that was too wide for the box:

It was pointed out in the mailing list that enabling line-wrapping by default might be better, which I was able to accomplish by simply setting the white-space CSS attribute to pre-wrap. Unfortunately, when you add line numbers to the mix, this is what happens:

One way to fix that is to output the line number within the same element as the contents of the line. Using a table, this involves showing each line within a <tr>, with one <td> for the line number and one <td> for the contents of the line. This is the result:

Which, as I posted about several weeks ago, seemed to resolve the problem.

A wild complication appears

The day after I pushed the line-number alignment fix, I was informed via IRC that what I thought to be a bulletproof solution actually broke a very important feature: the ability to highlight a code snippet and then copy and paste it elsewhere. See, the fact that the line numbers were now integrated with the code itself meant that when you try to highlight a piece of code, you end up getting the line numbers as well:

At first, I panicked, thinking that all my efforts were for nought and that I would never get to live in a world in which Agora had a decent way of displaying code.

Then, I recalled the training I had received at boot camp, from my short stint in the Web Developer Paramilitary Forces: "Don't panic. There is probably some CSS attribute for that. Also, it probably doesn't work in Internet Explorer, but who cares. Does anyone even use that anymore?"

So I stopped panicking and did a search for css disable selection. The top result was a StackOverflow post that happened to be exactly what I wanted. Essentially, the trick is to make use of the user-select CSS attribute (and all of the vendor-prefixed variations, for the different browsers out there) on the table cells containing the line numbers:

-webkit-touch-callout: none;
-webkit-user-select: none;
-khtml-user-select: none;
-moz-user-select: none;
-ms-user-select: none;
user-select: none;

Two minutes later, I had pushed a fix, and the world was beautiful once more. (In my haste to get out a fix, I made a typo, which was corrected twenty minutes later. Not sure why I didn't just copy and paste; StackOverflow doesn't even have line numbers.)

Here's what it looked like after the fix had been applied:

If that were the end of it, I would probably have forgotten about the topic entirely and would certainly never have written a blog post about it. By which I mean, that was not the end of it.

The user-select property is a lie

About three days ago, my SOC mentor Jordi informed me that although line numbers no longer appeared to be selected, they would still show up when you tried to copy and paste a snippet. Once again, my world was shattered. Looking into this more, I discovered that this issue has been reported as a bug for Chromium and Firefox (and probably other browsers). In some reports, the issue is said to have been fixed. However, the Mozilla Developer Network specifications for the user-select property seem to indicate that this is actually intended behaviour:

Controls the appearance (only) of selection. This does not have any affect on actual selection operation.

After shedding some tears, I decided I had several options. One was to go back to the original system, with the horizontal scrollbar, which seems to be the most popular way of aligning line numbers (see: Github, Bitbucket). Another was to simply ignore the fact that users couldn't select text without catching the line number as well, which is what Trac and the Mercurial web server seem to be doing. And then there's SourceForge, which, I believe, is a strong proponent of the innovative "If the line is too long, then we'll just cut it off, lol" method. Pastebin.com comes the closest I've ever seen to solving this problem, with the line numbers properly aligned and not selected when you highlight the code (even if they appear to be), but they introduce weird whitespace to the beginning of lines and to be honest I have no idea what they're doing.

I ended up going with the other option: making it work. It took a while, and it's not the most elegant one, but it does work. You can view a demo here: http://agora.dellsystem.me/snippet/8DVW/ (although if it doesn't work for you, I'm starting to think that I'd rather not know).

The final solution

Although I hate the idea of using Javascript to accomplish what really should be possible without it, it seemed like I had no choice. Line numbers are now inserted via Javascript. The container for the snippet is relatively positioned, and the line numbers are absolutely positioned, with their top offset set to the top offset of the line they are meant to indicate. To see how this is done, check out the commit.

Other solutions

• Stash, a Git management tool by Atlassian, apparently has a solution for this, which they blogged about here. This would have been perfect, except I was not able to reproduce their solution, and since Stash is one of those products that require you to buy a license first I wasn't able to test it out, either. (They do ostensibly have a "try it for free" option, but that page seemed broken when I looked at it.) If I do manage to get that working, it will almost certainly render my Javascript technique obsolete. But hey, progress is good.
• Sourceforge, despite its flaws, actually has a pretty clever of handling linking to a specific line. Click on any line and you'll jump to that anchor. This opens the door to a different solution: if the line numbers don't need to be clickable, they could be hidden (in terms of z-index) behind the lines themselves, which might prevent them from being selected. I tested this out a bit, and though I wasn't able to get it working, it remains a possibility. Sourceforge's implementation of this requires Javascript, which makes it hardly any better than mine, but it could probably be reworked to function without it.
• Update: It turns out Pastebin uses an ordered list to display the numbers. That is actually extremely clever and even sort of semantic. I don't know why I didn't realise this earlier. The only problem with this solution is that when you highlight the text, the line numbers appear to be in the selection, even though they don't show up when you copy and paste it anywhere. Even user-select can't handle this. There might be another way, but for now, I'm going to give this method a pass.

Closing remarks

Was this worth it? Well, maybe. I learned some things about CSS (and, in the process, myself!!11). In any case, it's not like I had a better way to spend my Friday night. Unfortunately, this does mean that I wasn't able to finish the features for Agora that I promised last week. Expect an update on that in the next few days.

My hope is that someone, someday, who happens to be looking at an Octave snippet on Agora, will notice my painstakingly crafted solution to the hardest problem in computer science. And, instead of thinking, "Wow, I can't believe someone went to the trouble of making this combination of features work together, that's such a pointless thing to do", this person will instead think, "Wow, what a brilliant solution to an otherwise intractable problem. This web developer is clearly a genius, and we should give her a job."

At the very least, I should be able to get a paper out of this. I'm thinking Nature.

Package instrument-control

To-Do:

Offtopic:

This week, I added the ability to edit your account settings, and the ability to upload a file to create a snippet. I've been overloaded with assignments lately (hence the late update) and wasn't able to spend much time on the module upload feature, so I'll have to push that back again. It will be my top priority for next week, though, so stay tuned.

Also, I regenerated my private key and the new public key hasn't yet been updated on the server hosting the mercurial repository, so I wasn't able to push my changes there. As a temporary workaround, I've exported the commit history to git and pushed it to github: https://github.com/dellsystem/agora-octave

This week

Ability to upload files as snippets

This feature works pretty well so far. There is a small issue in terms of handling binary files uploads, at least if you're running SQLite (this shouldn't happen with production databases). I implemented a temporary fix for this by checking the content-type header of the uploaded file. However, this can be changed by the user, so it's not completely trustworthy, as explained in the Django documentation. Unfortunately it looks as if there is no foolproof way to detect the encoding a file in Python, according to this StackOverflow post. I will continue to look into this, though.

View a demo »

Profiles

You can now edit your profile details. I cleaned up the view code a bit and replaced the manual form processing with the use of a ModelForm.

Next week

Module upload

My priority for next week.

More profile stuff

I believe there are some other user-specific settings (font size, line height?) that users should be able to edit from their profile page. Also, the URL for the editprofile view should probably be changed to something other than /users/editprofile, or else anyone who registers an account with the username "editprofile" will never be able to view his own profile.

Spam prevention

I will work on implementing the techniques mentioned last week.

Position in timeline

• August 1-15: Check out codebase, get it running; read all the code and documentation; start learning Mercurial
• August 16-31: Start working on design; start mocking up models and routes for the components
• September 1 - October 15: Work on the core components, with as much test coverage and documentation as is feasible; start implementing the views, models and templates
• October 16-31: Ensure that documentation is thorough and up to date; write any remaining tests that need to be written; test out the user interface and fix any bugs

Package instrument-control

This will be a short update, partly because of the follow-up update I posted three days ago (which covers most of what I accomplished this week) and partly because school is now in full swing and so I've been spending a lot of time trying to type up my notes. Only minor changes this week. I'll definitely do something cool by next week, though, so stay tuned.

This week

Summary of changes:

• added an emacs pygments style
• fixed some W3C validation errors
• fixed my first merge conflict with hg, using vimdiff (it was a bit scary)

Next week

Module upload

I wasn't able to make too much progress on this yet. Next week!

Profiles

Same as above.

Ability to upload files as snippets

In the IRC channel, jwe suggested adding the ability to upload a file to be posted as a snippet, in addition to the current method of pasting it into a text box. This sounds like a useful feature, and I will work on that for next week.

Spam prevention

I'll look into ways of dealing with spam (the old installation at agora.octave.org was flooded with spammy snippets). Jordi suggested not publicly listing anonymous snippets, and requiring a CAPTCHA (possibly using this app) for registration.

Position in timeline

• August 1-15: Check out codebase, get it running; read all the code and documentation; start learning Mercurial
• August 16-31: Start working on design; start mocking up models and routes for the components
• September 1 - October 15: Work on the core components, with as much test coverage and documentation as is feasible; start implementing the views, models and templates
• October 16-31: Ensure that documentation is thorough and up to date; write any remaining tests that need to be written; test out the user interface and fix any bugs

This is the follow-up to update #5 (a little bit later than planned). Summary of changes this week: updating the code page to make it slightly more useful (at least in terms of listing the snippets), fixing vertical alignment of lines and their corresponding numbers on the snippet view page (more on that below), and adding a pop-up window for a more streamlined login and registration process.

This week

Updates to the code page

This is nothing major, I just changed the layout slightly (the buttons have been moved), changed some of the terminology (bundles/singles are now known collectively as "modules", for now), and added the list of the 5 most recent snippets, as well as some placeholder text for forge and module sections. View demo »

Line number alignment

This one might require a bit of background. Previously, the line numbers were being output in a separate loop from the actual lines themselves, within one giant <pre> tag nested within a <th>. The problem with this method is that if a line is ever too wide for the box, then it gets hidden and you have to scroll horizontally to see it, as in the following screenshot:

This method worked for me, because I have a touch pad with horizontal scroll enabled, so I didn't think much of it. However, it was brought up on the mailing list that this can be inconvenient for users who don't have a way to easily horizontally scroll, so I decided to look into it again. Although there existed a "toggle wordwrap" link, which users could click to toggle the line-wrapping of the snippet (and so eliminate the need to scroll horizontally), that feature also caused the line numbers to disappear:

I had to do it this way because any line that is wider than the box itself would, when wrapped, push down the content after it, without affecting the placement of the line numbers as well. In other words, you would end up with something like this:

The line numbering here is just plain wrong. In the end, I simply changed the way line numbers and lines are outputted by using a single loop and a <tr> for each line, and now it looks much better (with word-wrapping used by default):

The only thing that was difficult was getting the styling to match how it looked before, as the change in the HTML structure made it a bit tricky. It should be pretty much the same now, although the corners look a bit off with a dark background.

Streamlined login and registration

We now have a slightly more streamlined login/registration system: simply click any login or registration link and a window should pop up asking you to fill out the combined login/registration form. If you are successfully logged in, you will be returned to the page you were previously looking at. For a demo, click the "Login or register" link at the top right of the code page. (Please don't actually register because the registration app sends out an activation email by default, and email isn't enabled on my server so it just won't work. To test logging in, use the username test and the password test.)

Next week

(Technically, this week. Next blog post will be on Friday, September 14.)

Module upload

I've started working on an app for handling module uploads. It will be similar to snippets in the way that it is structured, but it will be more powerful in that users will be able to upload files rather than being limited to copying and pasting (either a single file in plain text, or an archive containing multiple files). I will try to get the bulk of this done by my next update.

Profiles

I will continue working on user profiles: layout, features (including the ability to change your default syntax highlighting colour scheme, as mentioned in a previous update), and editing your own profile.

Position in timeline

• August 1-15: Check out codebase, get it running; read all the code and documentation; start learning Mercurial
• August 16-31: Start working on design; start mocking up models and routes for the components
• September 1 - October 15: Work on the core components, with as much test coverage and documentation as is feasible; start implementing the views, models and templates
• October 16-31: Ensure that documentation is thorough and up to date; write any remaining tests that need to be written; test out the user interface and fix any bugs

Package instrument-control can now speak Parallel

General progress 

I've been pretty busy this week, with classes resuming for fall semester and a major project that I'm wrapping up. This week, I made some improvements to the code explore page and started on the file upload functionality for singles and bundles (tentatively known collectively as "modules", although this is subject to change), but haven't had a chance to commit the changes yet. I will work on finishing what I planned to do in last week's update in the next couple of days, and will commit the changes and post a more comprehensive update (update 5.1?) by the end of the weekend. I'll also update the demo and will help to update agora.octave.org to the new codebase.

Position in timeline

• August 1-15: Check out codebase, get it running; read all the code and documentation; start learning Mercurial
• August 16-31: Start working on design; start mocking up models and routes for the components
• September 1 - October 15: Work on the core components, with as much test coverage and documentation as is feasible; start implementing the views, models and templates
• October 16-31: Ensure that documentation is thorough and up to date; write any remaining tests that need to be written; test out the user interface and fix any bugs

Merge of i2c and serial packages to instrument-control

General progress 

Offtopic

Update #4 for Agora Octave. I worked on completing the integration of the snippet functionality with the new design, added the ability to change the syntax highlighting colour scheme, and made some minor housekeeping changes. See below for the exciting details

This week

Integrating snippets and the new design

I added back the "view snippet" view, with all of its functionality: "view raw", delete, toggle word-wrapping, line numbers, and syntax highlighting (more on that below). I also made a small change to the lexer field to ensure that when you revise a snippet, the chosen lexer is the same as the lexer for the original snippet, which is probably what you would expect.

Syntax highlighting colour schemes

There are now 8 different syntax highlighting colour schemes, and you can switch between them when you view a snippet. Most of them are the default stylesheets that come with Pygments, with one being mostly-custom ("vibrant"). View a demo » (use the syntax highlighting style dropdown near the top)

Housekeeping changes

I've been slowly cleaning up the code wherever I can, both to meet PEP 8 (the de facto coding style guide for Python) and to just simplify or improve the code wherever possible. This week, I made some changes in the order of imports, made use of a custom shortcut method on a model, fixed some whitespace issues, and moved the setting of "choices" for a field from the form to the model, for logical reasons and to make it easier to get the human-readable name in any situation. I also upgraded the Django dependency to 1.3, because 1.3 is really quite a bit better (for one, it has the render shortcut function, which eliminates the need to manually create a RequestContext).

Next week

Profiles

Next week, I will work on completing the profile view as well as the edit profile view, and integrating them with the new style. I'll also add the ability to set a default syntax highlighting colour scheme in your profile that will be applied to all the snippets you view (although it will still be possible to choose a different colour scheme on a per-snippet basis). Additionally, I will work on improving the usability of the login and registration pages by combining them into one, with a window popping up if you have Javascript enabled (sort of like how Reddit does it).

Code upload

I'll start working on the code upload functionality, for uploading single files as well as bundles. I will also work on the code "explore" view (at /code) - the intention is to create a page that gives a good overview of what is happening around the site, so users can view popular or recently-uploaded code snippets and modules.

Position in timeline

• August 1-15: Check out codebase, get it running; read all the code and documentation; start learning Mercurial
• August 16-31: Start working on design; start mocking up models and routes for the components
• September 1 - October 15: Work on the core components, with as much test coverage and documentation as is feasible; start implementing the views, models and templates
• October 16-31: Ensure that documentation is thorough and up to date; write any remaining tests that need to be written; test out the user interface and fix any bugs

Well, the email from Melange could be less ambiguous, but I'm proud to say I completed Summer of Code. As soon as I get information on where to upload the code sample, I'll be happily sending a .tar.gz of my summer's work! This is just a quick note to say thank you, to everyone who was following my project this summer—and to GNU Octave, for making this excellent experience possible.

I will still be contributing to my code and to Octave in general; I'm going to take a little time off to enjoy the rest of the summer, but I'm going to get back to working on the wavelet functions with some more reading when I have time during the semester. I'll also happily respond to questions and updates about the code that's already present, just send them along to me.

Happy computations!

I've been pretty sick this week and, as a result, haven't managed to get as much done as I would have liked. I did manage to get the new design mostly integrated with Django, and made some changes to it based on feedback from the mailing list. I've pushed the commits, but it's still very much a work in progress. There is a demo running at agora.dellsystem.me.

This week

Changes to the design

I removed the sombrero graphic from the banner, as it was mentioned that there were too many logos on the front page. I also changed the order of the menu items in the header, from about - code - discuss - help to code - discuss - help - about, again as the result of discussion on the mailing list (although this can be easily changed later on if necessary). The layout, wording, and associated icons of the headings on the front page were changed as well, with the Octave logo replacing the "Share your code" icon.

Integrating the design into Django's template system

I reorganised the static files directory, creating css/, img/, and js/ subdirectories and temporarily removing all the existing files. I then converted the modified design mockup into HTML, LESS, and image files. Some of the existing pages still need to be styled or have the javascript added back, which I will work on next week.

Next week

Finishing up the design integration

Next week, I will convert all the existing pages over to the new design and static file organisation. This will require adding new CSS, Javascript and possibly images, as well as integrating some of the existing code into the new system. I'll also finish the layout of the code-browsing portal and any other pages that need to be finished.

File upload functionality

After the design integration is complete, I'll work on the code-sharing features (bundle and single-file upload). At the moment I am not completely sure how all the various code-sharing components are related, and what the correct nomenclature should be. I will clarify that with my mentor and will aim to have these components more or less functional by next week.

Position in timeline

• August 1-15: Check out codebase, get it running; read all the code and documentation; start learning Mercurial
• August 16-31: Start working on design; start mocking up models and routes for the components
• September 1 - October 15: Work on the core components, with as much test coverage and documentation as is feasible; start implementing the views, models and templates
• October 16-31: Ensure that documentation is thorough and up to date; write any remaining tests that need to be written; test out the user interface and fix any bugs

Public initial commit of i2c package

It's been a little long since my last post here; I've mostly been staring at code and thinking about phrasing recently. The best and worst part of documentation strings is making something sensible out of 80 characters—doubly hard when the name of a function is the Lomb Normalized Periodogram. That's 27 characters right there!

Admittedly, not everything has been documentation. With Carlo's help, I've replaced the guts of the lscomplex and lsreal functions with code that can take advantage of vector processing methods; the differences between the results were minimal, and the code became far more manageable to read, so it's now the main form of the functions. (Thanks, Carlo!)

Also from Carlo are two texts on wavelet transforms that I'm going to look into; I've set aside hopes of completing the wavelet functions currently—other than lswaveletcoeff and lscorrcoeff; those two work just fine, however they're limited in their capabilities, as they each only operate over a single window defined by the user. Carlo provided these references:

Abramovich, F., Bailey, T.C. & Sapatinas, T. (2000). Wavelet analysis and its statistical applications. The Statistician, Vol. 49, 1-29. http://www2.ucy.ac.cy/~fanis/Papers/statistician2000.pdf

Guy Nason, Wavelet Methods in Statistics with R, Springer, 2008.

As a closing note, the biggest change that I've made in the latest release (lssa-0.1.2) is adding all sorts of input checking code to all of the functions; if you give a function the wrong number of variables, it will complain—but I haven't come up with a way to check the window function for those functions that take a window function as input. So be careful with the window function.

This week, I mainly worked on a mockup for a redesign (available below). I also made some small changes to the readme and pip-requirements, which have been committed.

This week

Design mockup

Here's a sneak preview of what I have in mind for the front page (click for higher-resolution):

Things I'm still working on:

• The location of login/logout buttons and other navigation items (for easily accessing the pastebin or code bundles)
• Font for the header (currently Museo Sans - suggestions would be nice)
• Position/size of the Octave sombrero
• The colouring of the logo. I changed the colour to make it fit in more with the icon scheme, but I'm not completely happy with it.
• The layout for logged-in users

I made some small changes to the colour scheme I mentioned last week, partly to better fit the colours used in the Octave logo:

• Dark grey: #333333
• Medium grey: #808080
• Light grey: #E6E6E6
• Dark orange (from the Octave logo; currently not used elsewhere): #D45500
• Orange: #FF7F2A
• Light blue: #60CAE1
• Medium blue: #22A2CA
• Dark blue: #1B749D

CSS

I'll be using the LESS CSS preprocessor, with client-side compilation for now. The files will be contained in a css directory within the static files root, and will be organised like this:

• variables.less - the colour scheme, dimensions of major elements, and other variables will be defined here
• mixins.less - will contain mixins for things like vendor-prefixed CSS properties, gradients, and other frequently-used features
• code.less - for the syntax highlighting colour scheme
• agora.less - the main stylesheet file
• imports.less - imports all the other files; is the only file that needs to be included in the HTML

New dependency: OpenID

I installed django-openid-auth through pip and added it to the list of dependencies in the readme and in the pip-requirements file. I also updated the readme with instructions on using pip to install all the dependencies.

Continuous integration

As the result of this message on the mailing post, I'm shelving the issue of continuous integration for now. The task of learning how to use Hydra will be left up to Jordi.

Basic code maintenance

I'll hold off on this until I actually need to work with the relevant code (i.e. the week after next).

Next week

Finishing the design

I'll be spending some more time on the design next week to finish it up and make any necessary improvements. The reason I want to focus on the design so early on is to ensure that the foundation is good, making things much more painless in the long run. If there's a solid, clean, and easily-extensible design framework in place, then adding new pages and features becomes much easier, with immediate visual feedback. Plus, the larger and more complex the project, the harder it becomes to effect a redesign, and so I'm doing it now, while the codebase is still manageable.

Integrate the design with Django

This entails implementing the design with HTML, CSS and images, and then integrating the HTML files into Django's templating system. The existing pages will be converted over at this stage as well. I'll finish as much as I can by next week's blog post; after next week, I'll start working on adding new features.

Position in timeline

• August 1-15: Check out codebase, get it running; read all the code and documentation; start learning Mercurial
• August 16-31: Start working on design; start mocking up models and routes for the components
• September 1 - October 15: Work on the core components, with as much test coverage and documentation as is feasible; start implementing the views, models and templates
• October 16-31: Ensure that documentation is thorough and up to date; write any remaining tests that need to be written; test out the user interface and fix any bugs

Public initial commit of Serial package

This week, I read more about the specifications and code for Agora Octave and started investigating some of the solutions. I also made my first few commits to the Mercurial repository (most were minor changes involving settings.py).

This week

Reading specifications and code

To get a better understanding of the project, I spent some time reading and re-reading the current README file, the somewhat spartan wiki page, and the code (all the .py files). I also read through this recap of Agora-related discussion during OctConf 2012, which was brought to my attention via the mailing list.

Changes to settings.py

Most of the edits were whitespace-related (changing 2-space indentation to 4-space indentation, adding/removing it in accordance with PEP 8, etc).

There was also a small compatibility issue with Django 1.4 that I experienced when I first tried to run it, which I fixed in this commit. More information on that can be found here.

Managing dependencies with pip

pip is a widely used package installer for Python. It has a great system for managing project-wide dependencies, and works very well in conjunction with virtualenv.

To get the list of required dependencies, I first set up a virtual environment with virtualenv and then installed all the required modules through pip. I then exported the requirements to a file named requirements.txt with pip freeze > requirements.txt, as is often the convention (the file was later renamed to pip-requirements, which does look nicer).

Installing the required dependencies is then a simple matter of running pip install -r pip-requirements. This system is especially useful when operating within a virtualenv in which only the default Python modules (and pip) are installed.

Mercurial

Before making my first commit, I read through the Mercurial commit message guidelines on the Octave wiki. They seemed pretty similar to the conventions I use for git, so I didn't have any problems there. One thing that I found somewhat unsettling at first was the lack of output colours, but enabling the colour extension was a simple matter of adding

[extensions]
color =

to my ~/.hgrc, so that was fine.

Next week

OpenID for authentication

It was brought up on the mailing list that we should use openID for authentication. Luckily, there are many libraries for accomplishing this in Django - this StackOverflow answer suggests that django-openid-auth is a good choice. Available uner the BSD license, like some of the current dependencies. Next week I try it out, and if it all goes well I will add it to the dependencies.

Design mockup

For next week's update, I'll include some mockups (images, maybe a static webpage) of redesign proposals. The theme will be consistent with Octave's - #D45500 (orange), #0791C1 (blue), #808080 (grey) - as well as the light grey used in the logo (#EEEEEC). I'll also send it out to the mailing list for feedback.

Continuous integration

My initial plan for ensuring that unit tests are run regularly, against multiple version of Django and Python, was to use the continuous integration tool Travis. However, since it integrates exclusively with Github, I sent an email to the octave-maintainers mailing list about other possibilities. Hydra (a paper about it is available here) was brought up. Next week, I'll look into it more to see if it's the right tool for the job.

Updating the readme

I'll edit the readme to reflect the improved dependency management system with pip and the new pip-requirements file, and will ensure that the to-do list is up-to-date.

Basic code maintenance

When reading through the existing code, I noticed some areas that could be improved, as well as some minor organisation and whitespace-related changes that could be made. I'll make any such changes next week.

Position in timeline

• August 1-15: Check out codebase, get it running; read all the code and documentation; start learning Mercurial
• August 16-31: Start working on design; start mocking up models and routes for the components
• September 1 - October 15: Work on the core components, with as much test coverage and documentation as is feasible; start implementing the views, models and templates
• October 16-31: Ensure that documentation is thorough and up to date; write any remaining tests that need to be written; test out the user interface and fix any bugs

y = A(1, sin (end));

y = A(1, 2, B(sin (end), 5));

type     identifier index count
function sin        0     1
matrix   B          0     2
matrix   A          2     4

This summer, I'll be working with Octave as part of the European Space Agency's Summer of Code in Space 2012. My focus will be on completing Agora Octave, which is a code submission and collaboration website for Octave-related projects built using the Django web framework.

I'm very excited about the opportunity to work on Agora Octave. I love building websites, especially with Django, and this will be a fun project for me that hopefully will also be of benefit to the Octave community. I've identified the major components that I will be working on this summer, and have sketched out a rough timeline of when I expect to complete the various steps, which you can find below.

Project components

Mercurial integration

When uploading a bundle of code, a user should have the option to either create an associated hg repository or import an existing one. I've never worked with Mercurial before, so I will have to spend some time familiarising myself with it and determining the best way to integrate it with Django.

User authentication system

A user authentication system is required to ensure that only registered users contribute code, comments, and rankings. This can be done by making use of the built-in auth module in Django as well as the django-registration module. Much of this component has already been done.

Bundle upload and code highlighting

This would be the main feature in Agora. Users should be able to share their code (either as a snippet, or as a bundle) under a particular license, with syntax highlighting done using Pygments. Parts of this component have already been completed.

Comment system

A comment system is needed to allow authenticated users to post comments on bundles and code snippets. This will be accomplished using the django-threadedcomments module.

User interface

I plan to develop a clean, consistent, and usable user interface for the website. I will be using LESS for the stylesheets (with server-side compilation) and will create any necessary graphics using Inkscape.

Unit tests

Since there are some features with potentially complicated interactions, I will make sure that any code whose validity isn't readily apparent is thoroughly tested. If the need arises, I will also use the continuous integration system Travis, in order to ensure that tests are always run.

Documentation

I will endeavour to keep my code as well-documented as possible without going overboard, in the form of inline comments, docstrings, notes in the README, and external documentation if need be.

Timeline

• August 1-15: Check out codebase, get it running; read all the code and documentation; start learning Mercurial
• August 16-31: Start working on design; start mocking up models and routes for the components
• September 1 - October 15: Work on the core components, with as much test coverage and documentation as is feasible; start implementing the views, models and templates
• October 16-31: Ensure that documentation is thorough and up to date; write any remaining tests that need to be written; test out the user interface and fix any bugs

Progress updates, code snapshots, and a mirror

I plan to write a new blog post every Friday with a brief summary of my progress so far. Although I'll be committing my code primarily to a Mercurial repository, I will also mirror the repository on Github because I like Github's user interface and the tools it provides.

I plan to develop locally, but will make my changes publicly available at agora.dellsystem.me.

About me

I'm an undergraduate student going into my third year towards a Bachelor of Science in Math and Computer Science at McGill University, in Canada. I first started using (and fell in love with) Django last summer, and have been creating and designing websites for about 8 years (of progressively better quality). I've been involved in open source since I joined the phpBB team in 2007, initially as a moderator, and now as part of the website team as well. I love web design and development, and am currently working part-time for a music technology lab at McGill in which I write software and design interfaces.

You can find out more about my projects at www.dellsystem.me.

Agora

Single and Bundle

Forge

• have at least one package maintainer;
• install and work with the latest Octave release;
• released under a GPL compatible license;
• not dependent on a non-GPL compatible libraries or applications;
• all functions (except private) must be documented in TexInfo;
• if a doc section exists it must be written in TexInfo;
• a NEWS file must exist listing changes for each release.

• no shadowing of Octave core functions;
• no direct inclusion of external dependencies.

Snippets

pkg()

• removal of the autoload option. No package will be able to automatically load itself and its value on the DESCRIPTION file will be ignored. This prevents users from inadvertently shadowing functions (even from other packages) and will increase aware on the role of packages.
• implementation of a new flag, -url, to specify URLs for a package tarball.
• automatic download and install of dependencies if those are part of Forge.
• keep the source of installed packages. This will allow to reinstall a package when Octave is updated as well as run the tests on C++ code.
• implementing an option to run the integrated function tests when installing a new package.
• a new organization for the installed packages on the system. This will include the removal of --global and --local flags (which will be handled automatically) and is meant to to allow:
• different versions of the same package
• different versions of Octave using the same packages
• global package installs in relation to the Octave installation, not to the system.
• automatic build of a package documentation in HTML, PDF and info format from TexInfo formats, similarly to what happens when building Octave.

Octave has acquired two students for the European Space Agency’s Summer of Code In Space.

Wendy Liu is a Django enthusiast who will be working on finishing Agora Octave. She is a capable webdev, having already built several successful websites, and she is also no stranger to free software and free development, as she is involved with the phpBB community.

Andrius Sutas is familiar with hardware interfaces and will be working on our low-level I/O project for Octave. He has been active in his university robotics student team which has recently announced the development of BLUEbot. In addition to this, he is a capable Unix sysadmin and is familiar with C++ and low-level C and assembly.

Let’s welcome them both and help them achieve success with their projects.

P. S. я играю на гармошке…

First off, OctConf was awesome! Meeting the other GSoC students and all the core developers was an awesome experience, and I hope I can go to more events in the future.

Now then. It is safe to say that the implementation of nurealwavelet() is in fact completely broken and may never have been meant to work. There's an infinite loop, there's an unused pointer, there's so many things wrong that I wonder how this article made it into print. On that note, I'm going to go over the entire function as it is represented in the paper here, and try to understand it. (In the following, the series is represented as a (t,ξ) series of complex values associated with a time series.)

The problem that I've been avoiding so far is that the window changes as the frequency drops; a lower frequency requires a much larger window to define it, but this means the number of windows tested decreases, thus a matrix for storing results doesn't make sense, as it will become progressively more full of junk data/excess unused elements. I'm going to look for other wavelet transforms as they're performed with Octave to see what I can learn from others on how to implement this.

OctConf 2012 was a success. Over the course of five days, we had 20 participants involved, and all but two were there for at least four days. Big thanks to the Centre de Recherches Mathématiques who helped us with all of the organising and logistics.

The first two days were informational and dedicated to getting everyone up to speed on development methodologies. The next three days were a more involved open discussion and development. The schedule was followed with almost no modification. We were sadly unable to record talks due to not being able to secure proper equipment, but we do have the slides.

The following talks took place:

What is Octave? by Jordi Gutiérrez Hermoso

Introductory talk. Raised a few points for future discussion.

Octave’s architecture by Jordi Gutiérrez Hermoso

A discussion about the source code diretory layout and how to find code. Prompted a wiki page about debugging.

Status of Octave by John W. Eaton

Guiding talk about principles and policies plus general state of Octave development given by lead dev.

Octave Speed and Cell Arrays by Daniel Sebald

Raised problems with execution speed related to cell arrays along with possible solutions. Prompted some ideas for how to proceed.

JIT Compiler by Max Brister

First GSoC talk. An explanation of JIT compiling and the important single static assignment (SSA) technique.

The Octave GUI by Jacob Dawid

Second GSoC talk. The upcoming GUI design and layout. Implementation details in Qt. Lots of discussion about how it should look and what Qt can do.

Agora Octave and Packaging by Carlo de Falco, David Pinto and Juan Pablo Carbajal.

An open discussion about how we need to reorganise Octave and code distribution. Resulted in moving Agora to the agora.octave.org domain, and a roadmap for how to proceed.

Bringing Least-Squares Spectral Analysis to Octave by Ben Lewis

Third GSoC talk. Discussion of the method and implementation details. Lots of healthy discussion.

BIM package by Carlo de Falco

An interesting discussion about how to solve an diffusion-advection reaction boundary value problem with Octave tools and other free software.

In addition to the talks, many other important ideas were discussed and decisions about the development of Octave took place. A few highlights:

• OctConf will now take place annually, with the next one taking place in Europe. The details should be ready within six months from today.
• Various suggestions on where to improve documentation.
• Rearrangements of source tree layout
• Distribution of Windows and Mac OS X binaries will occur in the future, but we will charge for them on a pay-what-you-want scale.
• Paid support for Octave and specialised binaries (rpms, debs) for enterprise setting will be produced and promoted.
• Max’s JIT compiling branch got merged back into Savannah. It will be released with Octave 3.8 as an experimental compile-time feature, disabled by default.
• Have a public list of release goals, editable only by project admins.
• Better advertisement for Octave. Who uses it, how, write a semiannual newsletter about new developments in Octave.

Lastly, there was also much informal socialising. Lunch every day at various locations, and on Tuesday we went for beer at L’Amère à Boire and that same night saw Greece’s fireworks in Montreal’s international fireworks competition. The final farewell on Saturday was a Mexican brunch at El Sombrero with a few people who stayed for the weekend.

Much fun was had by all.

So, fastlscomplex and fastlsreal have had some problems. Lots of problems, in fact, and I wasn't really happy with my code. However, after rewriting most of fastlscomplex, I've got it correct on the first result, while further approximations appear to run afoul of compounding error. At the very least, however, this is far improved from the previous behaviour, and I'll work on getting fastlsreal rewritten.

I really haven't updated recently. Then again, most of my recent work has been in fixing minor errors and adding documentation. So, as it stands, none of the tests I have written work if you call test() and the info strings (which I tried to make informative) also don't work. I have no clue why; Jordi thinks it's the negation field and I'm agreeing with him for now. I'm sure I'll find a way to fix it shortly. In the meantime, I'm reading the Introduction to Wavelets article and working on my presentation.

This will all be pretty cool, I think.

Oct file

#include <octave/oct.h>
#include <octave/parse.h>

DEFUN_DLD (oct_loopy, args, , "TODO")
{
  feval ("tic");

  octave_value ret;
  int nargin = args.length ();
  if (nargin != 2)
    print_usage ();
  else
    {
      NDArray data = args(0).array_value ();
      octave_idx_type nconsec;
      nconsec = static_cast<octave_idx_type> (args(1).double_value ());

      if (!error_state)
        {
          double *vec = data.fortran_vec ();
          octave_idx_type counter = 0;
          octave_idx_type n = data.nelem ();
          for (octave_idx_type i = 0; i < n; ++i)
            {
              if (vec[i])
                ++counter;
              else
                {
                  if (counter > 0 && counter < nconsec)
                    std::fill (vec + i - counter, vec + i, 0);

                  counter = 0;
                }
            }

          if (counter > 0 && counter < nconsec)
            std::fill (vec + n - counter, vec + n, 0);

          ret = octave_value (data);
        }
    }

  feval ("toc");
  return ret;
}

Results

So, a quick update: the lombcoeff function, which implements the Lomb unnormalized transform at one frequency, has been written and has a doc string; there was actually no problem with my code, I needed to reboot Octave; I may try compiling all of the Fortran libraries for x64 (or check if they already are, since I installed a 64-bit only system on the computer in question) and build Octave with 64-bit support; finally, I'll be writing a batch of tests this weekend based around a sum of a few sine curves, just to show how the function works (and I'll apply them across the whole suite, too. Once I've got sample data, it's pretty quick to extend.) The next step there is to write a test script using the Vostok data, and then I need to write a documentation file clearing up the source of that data (collected and documented by JR Petit et al, published in Nature, and available in tab-separated values from the NOAA's paleoclimatology site.) Currently in the /data folder is a CSV export of the .rda archives from the Mathias paper, but I think I may re-export them with a text file explaining each column, since the R export included various nonsense values around the data.

As for the nurealwavelet and fastnurealwavelet commands, I will need to study the code and the paper a few more times (I think I've read it at least five times so far) because neither section actually makes any sense to me. There is no nurealwavelet.R file, so nurealwavelet() in fastnu.c is not actually accessible from R, while the fast- version seems to have other problems. I get the feeling that I'm better off writing using only the paper as a guide when it comes to this transform, and I may not implement the fast- variation, since thus far the "fast" implementations have been orders of magnitude slower than their supposedly slower brethren (although I have a few ideas about accelerating them, mostly by changing how some math is handled and changing to switches from my current control structure. The second is easier to implement, and I will add that change before the more intricate changes I'm planning.)

In short, I'm back to work as usual, except for scratching my head over the wavelet transform. (I'll take some time now to read the paper Mathias cited, An introduction to wavelets, as it is available online and will most likely make my life easier.)

Input generation

function result = gen_test (n)
  result = double (rand (1, n) > .01);
endfunction

Vectorized

function z = vectorized (A, K)
  tic;
  temp = ones (1, K);
  z = conv (A, temp);
  z = z > K-1;
  z = conv (z, temp);
  z = z(K:end-K+1);
  z = z >= 1;
  toc;
endfunction

Loopy

function z = loopy (A, K)
  tic;

  z = A;
  n = numel (A);
  counter = 0;
  for ii=1:n
    if z(ii)
      counter = counter + 1;
    else
      if counter > 0 && counter < K
        z(ii-counter:ii-1) = 0;
      endif
      counter = 0;
    endif
  endfor

  if counter > 0 && counter < K
    z(end-counter+1:end) = 0;
  endif
  toc;
endfunction

Results

Well, the methods all run. That's a nice thing. The not-so-nice thing is that their results are not exactly in line with what the R code suggests they should be. Specifically, in the case of lscomplex, all of the R results are compressed into the first 400 values, taking omegamax = 1, 20 octaves and 100 coefficients per octave, while the next 1600 values are all infinitesimally small frequencies which are in essence garbage values ... oh, and my code generates values 283.42 times larger than the R results. (Other oddities are cropping up in the fast methods as well, but I'd rather break it down in the ordinary methods first.)

In short, I'm slightly confused and need to work out what exactly is going wrong here. I have a feeling this will involve the rest of the day.

So, I ran mkoctfile on the fast transforms this morning; fastlscomplex works fine and cleanly (and fast, on this processor!) Unfortunately, I've left some error in fastlsreal still, and it segfaults with a SIGSEGV 11 — Address Boundary Error; I'll figure that out this morning, and I'll check the output of fastlscomplex against the reference R implementation.

UPDATE: Found the problem, fixed it. There was an accidental call to a zero address (whoops.)

UPDATE 2: There's now the excellent question of what to do about the fact that the results produced vary in some areas, occasionally by spectacular amounts. I'm going to do some analysis on the results today, including generating the cosine functions indicated and comparing them to the data.

1. File editor does not complain anymore when saving file.
2. Terminal code cleaned up, now determines a proper fixed size itself.
3. Implemented basic debugging controls in the main window as well as in the file editor.
4. Added iconset from Crystal Project.
5. Fixed some weird behaviour with the current working directory.
6. Fixed collapsing/expanding the workspace tree.
7. Rik helped me to get more variable types get detected correctly.
8. Added a performance indicator to the status bar to see how much time the GUI spends in the octave thread. It's important to keep an eye on that, because we do not want to punish GUI users with horrible performance.
9. Fixed mess with rivalising shortcuts. Through ambivalences with shortcuts Qt was not able to determine the correct action for a certain shortcut.

Well, I'm about to commit some code that compiles without a hitch for fastlsreal, definitely a plus. Once I set up a clean environment to build the necessary R code, I'll generate the results to compare against my .oct files (as I've gotten behind on that front.) Finally, I'll write the documentation for the functions from the past two weeks, and that'll be it!

The Octave Language

A = [1 2 3];
B = A;
A(1) = 100;
disp (B(1));

Interpreter Implementation

A = [1 2 3];
B = A; # No copy
A(1) = 100; # Copy, as both A and B refer to [1 2 3]

JIT

A(idx) = x; # Where idx and x are scalar

if (isint (idx) && idx >= 1 && i < nelem (A) && count (A) == 1)
  A(idx) = x;
else
  A = handle_odd_assign (A, idx, x);

So, on Wednesday I had a bunch of stuff stolen out of my locker at the gym, including my laptop and various other important things. This halted my work for a little while as I reassembled my life, but I have a computer again and am currently running ./configure to build Octave; I should be completely set up to get to work again later today.

In code-related news, everything I've written up to this point (except for edits to the lombnormcoeff.m file) was stored in the octave-forge svn, so nothing was lost (other than time, money, a sense of privacy, and my general peace of mind) when the thief got at my stuff. That's the upside of offsite storage! To my mentors, I'll be back at work tomorrow, and once I get everything really rebuilt I'll have more code to submit (I'll test fastlscomplex tomorrow and then write fastlsreal as it's just a few additional steps on top of fastlscomplex.)

octave_value retval;
octave_value lhs = // compute lhs
if (! error_state && lhs.is_defined ())
{
  octave_value rhs = // compute rhs
  if (! error_state && rhs.is_defined ())
    retval = ::do_binary_op (op_type, lhs, rhs);
}
return retval;

a = (a + b) / c;

block0:
  temp0 = a0 + b0
  check_error block1, done
block1:
  temp2 = temp0 / c0
  check_error block2, done
block2:
  c1 = temp2
  goto done
done:
  c2 = phi (block0: c0, block1: c0, block2: c1)
  # yield a0, b0, and c2 as results

i1 = i0 + i0;

I just uploaded fastlscomplex.cc in its current version, and it works. It generates what looked to me to be correct results, I'll do a few spot-tests later. So far, it doesn't implement any sanity checks, so if run with bad input, it will segfault right now; again, I'll add sanity checks this afternoon. (The other thing is, it's much slower than lscomplex, most likely because I didn't hard-code many optimisations yet. Some functions could be accelerated by changing the number of operations used, for instance. I'll look into reducing the complexity of some sections during the buffer time before OctConf, while I'm also cleaning up documentation.

In the meantime, I'm just happy to have a working function!

So, I decided on Monday that the best option was to completely rewrite fastlscomplex from scratch; the code took a day to write, and on Wednesday morning, it compiled just fine. One minor problem is that it causes segfaults. (Okay, more than just minor. But it runs, sometimes.) I've figured out where the segfault is, after abusing std::cout as a way to check that it's running. The segfault-causing error is somewhere in the merging code, so I'm going to remove the output I added to check on what output was generated so far, and focus my attention on the merging code so that I can determine why it's failing.

The other problem with this code currently is that it doesn't actually output results. When it does run, it generates a long vector of the correct size, all of whose elements are zero. That's ... not exactly the result I was expecting, so while I know the actual answer-generating code is not killing the process, there's obviously a problem in it somewhere. That's what I'll be focusing on this afternoon, after I find a way to keep the code from randomly going ballistic on me.

class octave_event
{
  public:
    octave_event (octave_event_observer& o)
      : _octave_event_observer (o)
    { }

    virtual ~octave_event ()
    { }

    virtual bool perform () const = 0;

    void accept ()
    { _octave_event_observer.event_accepted (this); }

    void reject ()
    { _octave_event_observer.event_reject (this); }

  private:
    octave_event_observer& _octave_event_observer;
};

/** Implements a change directory event. */
class octave_change_directory_event : public octave_event
{
  public:
    /** Creates a new octave_change_directory_event. */
    octave_change_directory_event (octave_event_observer& o,
                                   std::string directory)
      : octave_event (o)
    { _directory = directory; }

    bool perform () const
    { return octave_env::chdir (_directory); }

  private:
    std::string _directory;
};

octave_link::instance ()->post_event
   (new octave_change_directory_event (*this, "/home/example"));

void
observer_class::event_accepted (octave_event *e)
{
  if (octave_change_directory_event *oce
        = dynamic_cast <octave_change_directory_event *> (e))

    {
      // Directory has been changed.
    }
  delete e;
}