nukep / opticnav Goto Github PK

This can be considered my post-post-mortem.

@IdoShalev and @Kay-Bernhardt (Jacky has no GitHub yet...), feel free to comment on this issue with your opinions/suggestions.
#1. Preamble

As the group members and instructors involved would know, this project was the end result of an 8-month Capstone course at SAIT Polytechnic. Although I consider the project a success, it's not maintainable in its current form.

Knowing what we know now, we can do better.
I believe we can improve on the original prototype to become something that's a) maintainable down the line, b) useful to more people, and c) more intuitive to use.

The general idea stays the same: develop the infrastructure to track physical user actions (e.g. location) so it can be extended by other applications. The web app and Android app are meant to showcase the functionality and to be as immediately useful to "as many people as possible".

The top mistakes (or why this needs to be rewritten)

1. Writing the documentation took 4 months, as part of a course requirement.
   • All documentation had to be written before any coding was allowed to happen.
   • It didn't allow us to organically and iteratively develop our use cases.
2. The webapp tastes like spaghetti.
   • Java EE's template engine is verbose and an eyesore. Alternatives such as Handlebars or ERB are more minimal and are less subject to easy-to-make mistakes (i.e. not properly escaping HTML).
   • The SQL code was more complicated than it needed to be; an ORM would have sufficed.
   • Java never lent itself well to web development; not even Spring can save Java. (disclaimer: In our humble opinion).
3. The network protocol is terrible (see section 2.4)
   #2. The proposed plan

2.1. Develop a preliminary list of use cases that will evolve and grow organically over time

I think we should start off with a very small set of use cases that relate to the user-tracking aspect of the project. Once it's clear what the user-tracking can do, we can develop meaningful applications around it.

Developing the infrastructure first will allow us to identify fundamental issues before they become too intertwined with the rest of the project.

2.2. Rewrite the web front-end using Ruby on Rails

One of the biggest problems plaguing the prior web development process was its slow iteration cycle. It was difficult to make changes to the HTML, JavaScript or CSS because Glassfish required waiting up to 40 seconds before we could see the results. Interruptions can diminish a designer's creative spark, and they violate the principles of rapid development (see Inventing on Principle - Bret Victor).

Because use of Glassfish was required by the course, we were stuck with its (frankly terrible) webapp deployment system. To deploy a webapp, Glassfish would teardown the existing webapp, repackage all Java classes and static assets (all HTML, JavaScript and CSS), and send the webapp to the Glassfish server. Again, this whole ordeal could take up to 40 seconds; this is a Bad Thing for a test environment. As we found out much later, Tomcat is much better for hot-swapping static assets (instantaneous changes!).

Another problem that I'm not convinced has been solved in any Java web framework is the problem of verbosity. Java is a verbose language. Java's unique flavour of verbosity can be good for many line-of-business applications, but verbosity in general is not what you want for front-end development.

Ruby + Ruby on Rails facilitates rapid development and can speed up the prototyping and development process. There are other options such as JavaScript + node.js that have similar offerings, but I'm proposing Ruby on Rails because of its ubiquity.

2.3. Rewrite the daemon using Rust

One-line Rust pitch: Rust is an upcoming systems programming language from Mozilla that statically guarantees memory safety without a garbage collector. The result is native, safe programs with performance comparable to C++.

As a functional programming language, Rust emphasizes the importance of invariants (what cannot change) and how it makes reasoning about code easier. For example:

• All variables in Rust are immutable by default.
  • Java has no concept of immutability outside of final and the getter/setter pattern.
• No data races across threads; safe concurrency.
  • Safe and concurrent programming is really hard in most programming languages, Java included.
  • Thread safety is statically checked.
  • This is exactly what you need in a program that talks over the network.
• Global variables are a pain to use.
  • Rarely would I praise a language for making something harder, but this is a special exception. Global variables don't quite jive with Rust's concept of unique ownership and borrowing rules.

Other goodies:

• Pattern matching
• Algebraic data types, tagged unions
• No null value - Option<T> instead
• No undefined behavior (i.e. no strict aliasing, buffer overflows, use-after-free)
• unsafe code blocks for when you really need to play with pointers
• Package manager and build tool: Cargo at https://crates.io/ and bundled with all Rust installations
• THE original use case for Rust is Mozilla's prototype web layout engine: Servo

As of writing, Rust is currently at a reasonably stable 1.0 alpha. One year ago, Rust underwent breaking changes almost daily, so it would have made very little sense to use it back then.

I may be a Rust hipster, but it's only because I know it does everything C++ can do, but simpler and safer. In my experience, I've been much more productive writing Rust programs in the first two months learning it, than I've been writing C++ programs in the five or so years I've known C++.

2.4 Reimplement Device <-> Daemon protocol in HTTP(S) + JSON

The prior protocol was implemented in a hand-written binary specification over TCP. [My] rationale for using it over an alternative such as JSON was a) smaller packet sizes, b) two-way communication without reopening connections, and c) reduced latency due to the lack of polling.

I didn't realize that all I needed to do was open an HTTP server. Instead, what I wrote was convoluted yet "clever".

"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian Kernighan

The network layer's "cleverness" was next to impossible to debug. Low latency is a noble goal, but like all micro-optimizations, there is a trade-off.

Furthermore, I don't feel that using a stateless protocol such as HTTP will negatively impact performance.

If packet size matters, we can always use gzip or a JSON serializer such as MessagePack: http://msgpack.org/

2.5. Android app stays the same?

Native Android development sucks, but there's nothing else like it on Android.

Activities in Android feel very volatile to code against because of their inescapable dependency on global state. As I stated in the Rust section, side effects such as global variables make code hard to reason about. Any side effects are confusing and slow down development.

One solution would be to use a framework like PhoneGap, which makes UI development simpler with the use of HTML, JavaScript and CSS. However, I'm afraid that PhoneGap might abstract too much control away from us. I'm not sure.

This isn't a topic I'm too versed in, so I'll leave this topic for future consideration.