The thinking is that a better way to get information to users would be to build one or more information services that will assist – either iPhone/Droid apps, or tailored web sites.  If you could have any piece of conceivable information, what would be most useful?

This would include both the Mall as well as the surrounding area and all of the new Metro stops. We’ll be building a proposal and prototype to help out the local government. What do you think we should have in/on it?

For the Mall:

1. Map of parking area with updates on available parking spots.
2. Means to mark your spot on the map and some way to trace your route back to your car.
3. Layout of mall. You are here identification.
4. Recommendations for non-busy times.
5. Senior specials. Maybe special area for Seniors: exercise suggestions; food suggestion; matinee movie suggestions;
6. Movie listings.
7. Special sales or activities in mall stores. Coupons would be good. A way to hold your iPhone to a store scanner for them to read your coupon. Maybe a way to have your name and phone number on your phone be scanned in by stores or sales people for drawings like for a new car or whatever.
8. Holiday or special activities or presentations.
9. Food Court identification and other eating facilities. Ratings on cleanliness. “Eat This Not That” recommendations (maybe tie-in to app for that.) Vegan or Vegetarian suggestions.
10. Restaurant reservation system.
11. Restroom locations. Elevator or handicap identification.
12. Mall-walker distance and route suggestions for exercise purposes.
13. Search capability for items for sale with stores that have availability. For instance “shoes” or “fondue pot”
14. Feedback capability to make suggestions to mall managers and/or stores.

For Tysons:

1. Traffic patterns and route planning
2. Augmented Reality of bus stops, traffic directions, points of interest
3. Metro delays
4. Metro and construction status, dollars spent, current plans, expected benefits
5. Events in the area
6. Highlight businesses to sponsor key companies in the area that are supporting
7. Coupons or store bonuses for reporting suggestions or road improvements (light out, pot hole, graffiti, car accident, etc)
8. Carpooling recommendations

I’d love more requirements or prioritization of what would be best to build!  What would you find useful in the most congested/largest Mall area of the country?

People who looked at this item also looked at…

• About Jay
• Crossler.com
• Orrery – 3D Planet simulator
• Fictional Writing
• App Store for the Army

Related items

• HTML5 Databases + Canvas
• 3d iPhone web apps
• iPhone Geotags
• App Store for the Army
• Updated FlickrProxy

(built on previous Chumby iRemote post here)

Overview
This past month, I extended the web server on my older-style Chumby to add a lot of functionality to my house. For a while, I’ve wanted to automate a number of my home systems – X10 to control our home lights (specifically the Living Room lights), the Chumby to play music with, and web cameras to watch our dogs during the day. As the new Chumby just came out, I thought it would be useful to build all of the control into the Chumby device, so that others could also try the same thing.

I also build a feedback system that would try to monitor the amount of times our dogs jumped and barked at cars, then change music and turn on/off lights to find the combination that calmed them the most. Unfortunately, I found that there is no magic configuration – rather the dogs seem to relax whenever the lights and music change (possibly because they feel that someone is home). Either way, it was a cool experiment that I learned a lot from. It is a bit “fragile”, meaning that there’s a good amount of custom configuration that you would need to use to reuse everything. Nonetheless, I’m posting all code and configuration samples here in case people want to set parts up themselves. The X10 webpage, for example, is pretty useful even if it is the only piece loaded.

Details
I call this marvelous contraption the “Puppy Ubiquitous Pacifying Preference Engine” (PUPPE). If you want to be all research, the Technology Goal was to build an automation engine that optimizes environmental support for dogs left home alone. The Research question I set to achieve was: Can an automatic feedback engine be used to optimize relaxation for dogs by controlling home automation electronics? I was partially able to achieve this goal, though in a way I completely didn’t plan for. Rather than find an optimal configuration, I found that the more I randomly changed the lights and music throughout the day, the more relaxed our dogs seemed to behave.

The tools I used for this were:

• X10 Wall Light Switch
• X10 Application Switch
• X10 Home Automation software on Windows XP (old laptop LAP1)
  • X10 Controller software listening on TCP port 6003
• Mobiscope Desktop Webcam controller on Windows XP (old laptop LAP2)
  • copSSH – ssh server for secure remote connections
  • ipsecCmd and IP Security Policies to filter IP traffic
• Two cheap USB webcams
• Mobiscope web site for hosting video
• Weather Web Service for determining sunset time
• Chumby running ChumbyOS Linux (CHUMBY3)

Client devices:

• User Laptops (LAP4 and LAP5)
• iPhones (PHONE6 and PHONE7)

There were a number of moving pieces to this configuration, and each piece took a while to determine the best approach to implement. I’ll describe the solutions and tricks I came up with below. Many people probably won’t have two extra laptops laying around, so the configuration will likely change depending on circumstance. I spent a good portion of overall effort intentionally trying out different approaches and security option, so there are many other ways of having the same effect. The X10 Commander interface is shown below:

Technical Architecture

The picture above shows the logical architecture I came up with. All the software actually ran on the Chumby (except the Mobiscope viewing page that is hosted on that company’s web server). I firmly believe that as systems become more complex (some people call this a Service Oriented Architecture – which I’ve studied in depth, but don’t want to go into here), it’s important to build very simple interfaces. I implemented three main types of API service interfaces, each very simple to work with. I call these “ramps” or “loose couplers.”

Loose Couplers:
HTTP GET-based web service. In the simplest configuration, a web service is a web page that a machine returns when it receives a certain input. The Chumby has a CGI-based web server that can be used to run simple sh shell scripts. We pass in variables through an HTTP querystring (the part following a ? in a URL). This way, when an event.cgi?setVolume0 is called, the cgi file shell command executes a command to mute the radio. We use x10.cgi to listen for light-switch control messages, event.cgi for Chumby-specific events, and cameras.cgi for commands to control the web cameras.

The Chumby also can act as a hosting web server, using an index.cgi page that points to all of the other commands. The interface looks like this:

And the hosting code looks like (the launch javascript is declared in the full file, see attached code):

SSH-based camera control service. Secure Shell or SSH is a network protocol that allows data to be exchanged using a secure channel between two networked devices (as described in RFC 4252). Most linux or unix-based machines (including the Chumby) have an ssh client or an ssh server built in. As we want to control services on an Windows XP machine, we needed to add an ssh server, so I chose copSSH after testing multiple other options. By using IP Security policies, we can add a firewall filtering policy to the machine from the command line with a batch file (Cam-Block.bat) that runs `ipseccmd –w REG –p “BlockMobiscope” –x`, where BlockMobiscope is a windows IP filtering policy as shown below:

To make remote triggering as simple as possible, we want to enable PKI key exchange on both machines. This allows a single shell command `ssh UserName@192.168.2.3 ‘./Cam-Block.bat’` run on the Chumby to securely turn on or off traffic from USB cameras being sent to the Mobiscope web server. The home router is configured to always map 192.168.2.3 to the camera server (LAP2), which saves a lot of effort when machines don’t properly identify their hostnames.

One of the most painful steps is setting up the SSH server to run without entering a password at every instance. Building this PKI key-exchange process uses ssh-keygen command to generate a client and server key. Because the Chumby does not have ssh-keygen installed, we need to build the ssh keys on another machine and remotely configure both the chumby and LAP2 server with these keys. Once we do this, we can be reasonable assured that only the Chumby can interact with the XP laptop (LAP2). Because this machine controls transferring video from our USB cameras out to the web, we want the most amount of security around this interface.

TCP-based network injection service. By using the X10 Active Home software with the X10 Commander program, iPhones can send TCP messages to a laptop to turn X10 devices on and off. Rather than using the iPhone client, we can spoof TCP messages to emulate the same commands that the client would send. By using WireShark, I found that the TCP commands are simple messages sent on port 6003:

Using NetCat ‘nc’ – (my new favorite linux command!), these messages can easily be replicated using the shell command: echo ‘DEVICE~sendplc~”C4 ON”‘ | nc 192.168.2.4 6003 (this is to the LAP1 Home Server). This is not very secure, but is based on inherent security limitations of the X10 Commander software. Given time, I might rewrite a more secure X10 controller. As only the lights are controlled from this server, I found that was a worthwhile privacy tradeoff.
Experimentation

Dog owners frequently ask “What are my dogs doing while I’m at work?” My wife specifically wanted to watch our dogs while she was at work (and her company blocked the firewall connection). To give her this capability of watching our dogs that are blocked off in the kitchen each day, I set up a pair of USB cameras on an old laptop and purchased the $20 Mobiscope software that is iPhone compatible. She is able to watch her dogs anytime she wishes on her iPhone.

After observing our dogs for a few days, we noticed that they would become more and more agitated throughout the day, and bark at cars and pedestrians walking outside our window. Some days they were more relaxed, and some days they were more agitated. Every day we played different radio stations, and surmised that different music at different times would help calm them. Also, we noticed that with shorter winter days, the sunlight dimming early in the evening seemed to agitate them more. We surmised that if we could modulate lighting conditions along with music, we could help minimize the amount of times they got up to bark out the window.

The experimentation had two phases. During the first phase, I used the two USB cameras to record the dogs in time-stopped video (3 frames/sec). By trying a number of algorithms, I determined that a good way to rate the number of “agitations” was to identify an active area of the camera view and to count the amount of pixel change over a period of a second. As shown in the picture above, if more than 84% of all pixels in the active area changed in a second (out of a max of 300% as there are 3 frames x 100%), then a recording is triggered and an “agitation” is recorded. Due to varying lighting conditions, this number was not optimal in every case – and recommendations are not to use the Mobiscope to do this process in the future.

Average # of agitations/hour at time-of-day

The # of agitations/hour was then used to build an algorithm to remotely turn on and off lights and to change radio stations in the kitchen. Unfortunately, there was no reliable way to detect agitations as they occurred. The Mobiscope web page hosted only cycling images and passed recording #s through javascript. A few attempts to “Screen scrape” the number of recordings only worked intermittently. I used a greasemonkey script that would read the data, but had many issues with reliability. (As a note, I just realized yesterday that each recording of an agitation produced a new data file in a directory – and that directory could be watched for updates to trigger the algorithm. This would be useful in follow on work).

I worked through 5 algorithms that were all sh shell scripts fed through the web page and greasemonkey script. This proved problematic, as the script never ran concurrently throughout the day. Greasemonkey just wasn’t robust enough to handle web errors or dropped connections from the Mobiscope server. As these were working locally inside my home intranet, I didn’t have a reliably secure method to tweak and monitor these while I was out of the house (and whenever I was in the house, the dogs’ behavior would change dramatically).

When the algorithms did work, it became quickly apparent that every change was a “good” one. Any time there was an agitation, if the radio was changed or a light level was adjusted, the dogs immediately calmed down for a period. It didn’t matter which radio station or which light level was set. Any change seemed to reduce agitations. The algorithm I now use just rotates radio stations and light levels randomly, without any inputs. On average, it seems to have reduced # of agitations by 45% (though I haven’t yet had enough testing to ensure it is statistically accurate or will remain lower).
Privacy Engine

In order to use the system well, we wanted to add two extra capabilities:
1) When either my wife or I come home, turn off the web cams and radio, and turn on the kitchen lights
2) As a backup, turn off any recording at sunset (in case the first doesn’t trigger)

To implement the first capability, we need a way to detect when either of us comes home. I initially tried using an X10 motion sensor in the door way, but it was quite cheap and would sometimes trigger when there was direct sunlight on it, and frequently not trigger when people walked by too quickly.

I’ve noticed through watching Wireshark that when an iPhone connects to a wifi network, it sends out a few IGMP/Multicast DNS packets to 224.0.0.251 (local multicast to all machines, as specified in RFC 3171). By listening for this message, I have the Chumby look for the correct identifiers and send out the “turn cameras off”, “mute radios”, and “turn lights on” commands.

I would have liked to detect a method for turning the cameras on when no one was home (as we frequently forgot to do so in the morning), but couldn’t find a reliable method.

To implement the second capability of always turning cameras off at sunset, I had a script that pinged a web service daily and asked for the sunset time of the current locale. The below script illustrates this method (but condenses some of the logic for readability).

(Note: That should read “sunset” not “sunrise” in the web service request above… otherwise things get a little wonky.)

Conclusion

This was a fun project that had a lot of moving pieces. I built some useful functions that we still use daily. The light and radio controls are very reusable, and I will be releasing all of the source code for others to do so. I was able to partially prove out that modifying the environment modifies our dog behavior. More importantly, I gave my wife tools to feel more attached to her dogs throughout the day (and justify my spending money on technical toys).

The code is here, please download – released fully creative commons/open source (BY 3.0 license).

People who looked at this item also looked at…

• App Store for the Army
• Fictional Writing
• Building applications for Tysons Corner
• About Jay
• Moon Base One Novel

Related items

• 3d iPhone web apps
• Chumby iRemote
• Orrery Game: ver 2
• HTML5 Databases + Canvas
• Updated FlickrProxy

Also, many soldiers are age 18-25… and wind up having a large amount of time on their hands when deployed. I’ve been thinking that many of the useful apps would help with training, study, first aid, language lessons, and other skill development. Also, the device should contain public domain books and note-taking software.

Assuming that we could spend $100 per device to download apps, which ones should we use on the devices? Should apps be preinstalled, or should we load a $100 iTunes app account for each device and let soldiers download what they want (probably dangerous… but could help discover some really good use cases).

The newly released Army iPhone App

Here’s the list that I’ve come up with so far for iPhones. Please post any comments on what I’m missing, or what I should use for Droid or Windows Mobile phones:

Combat Assistance
Sniper ($39.99)
Ballistic ($9.99)
KAC Bullet Flight

Navigation
GPS Sport ($2.99)
Google Earth
Weatherbug
wxHere
Map and Land Navigation Reference

Fitness and Health
Army APFT Calculator
Army Body Fat Percentage Calculator
US Army First Aid Manual
MedVideo First Aid
Hand2Hand (Combat training)

Enterprise Support
Good email client
PerDiem calculator
FedTravel ($0.99)
2Do Lite
Note Taker Lite

Training
The Army App
Army Board Study Guide
Language Translators ($14.99)
Arabic Language trainer ($150)
ASVAB Exam (Study guide)
Armystudyguide.com/mobile
Languages – Arabic, Chinese, Japanese, Korean, Spanish, German,

School and Science
Science Glossary
101 Science
SAT Vocab Challenge
SAT Math prep
Conversation English
iCertify (Cisco routers)
Stanza (book reader)
Shakespeare (books)
23,000 great quotes

Government Knowledge
Military Aircraft Recognition Quiz
Army News (general news)
US Armed Forces (General Knowledge)
Squared Away (general Army Knowledge)
US Military Handbook (general DoD knowledge)
DoDComm (News)
White House
Constitution for iPhone
US Military Handbook ($9.99)
Military Slang – ($0.99) – review
Congress+ ($9.99)
CIA World Countries Factbook ($0.99)
US Armed Forces ($0.99)
The Law Pod ($0.99)

Good Citizenship
Amber Alert
FBI Most Wanted ($1.99)

Air Force
Military Pilot Logbook ($19.99)

Hardware for a kit
Case – Otterbox
Stylus
Extra Charger

People who looked at this item also looked at…

• About Jay
• Interview on GarageGames
• Fictional Writing
• Building applications for Tysons Corner
• Orrery Game: ver 2

Related items

• Building applications for Tysons Corner
• HTML5 Databases + Canvas
• 3d iPhone web apps
• iPhone Geotags
• Updated FlickrProxy

I’ve posted an updated version of the Orrery Torquescript library. Thanks for all of the great feedback. This new version has a few features:

• Fully Object Oriented, so you can add multiple orreries or systems
• Renders correct positions of the moon
• Has a datetime object to show the current universal epoch (date/time)
• Allows for different animation speeds and interactions

The code is at: http://wecreategames.com/apps/planets/Orrery.v2.zip. Feel free to download and play with it (it currently works only with Torque3D).

I’ve also built a version for Torque Gaming Engine. It’s a fully compiled project, so should work on any Mac. Leave a comment if you’d like me to build a version to work with PC. It’s a 100Mb download, but has all the files within. There are a few extra math functions in this one that weren’t included in TGE (but were in T3D), so I added them to the Functions_Math.cs.

Getting Torquescript (which is a scripting layer on top of Torque’s C++ engine to support OO behavior was a bit painful, so I used a few of the concepts from JavaScript to make pseudo-classes.

I’ve released all the code under Creative Commons BY 3.0 license – you can use and make money from this, just give me credit, please!
Here are the steps to implement:

//=====================================
1) Create a new mission, named Orrery

//=====================================
2) Add Orrery.mis to Orrery/game/levels/
3) Delete the other missions (simple.mis NewMission.mis) in this directory
4) In Orrery/game/scripts/client/init.cs:
Within this function, change:

function loadDefaultMission()
{
Canvas.setCursor(”DefaultCursor”);
createServer( “SinglePlayer”, “levels/simple.mis” );

to:

createServer( “SinglePlayer”, “levels/Orrery.mis” );

//=====================================
5) Add PlanetsInit.cs and the Functions*.cs to Orrery/game/scripts/server/
6) Update Orrery/game/scripts/server/scriptExec.cs
add to end of file:

//Supporting functions
exec(”./Functions_Math.cs”);
exec(”./Functions_Time.cs”);
exec(”./Functions_Objects.cs”);
//Initialize planets
exec(”./PlanetsInit.cs”);

//=====================================
7) Copy planets.cs to Orrery/game/art/datablocks/
exec(”./planets.cs”);
8 ) Update Orrery/game/art/datablocks/datablockExec.cs
add to end of file:
exec(”./planets.cs”);

9) Add materials.cs to Orrery/game/art/materials.cs
10) Copy the planets directory to Orrery/game/art/shapes/ (so it’s Orrery/games/shapes/planets)

//=====================================
11) Copy the sky_night directory to Orrery/game/art/skies
12) Copy the three rocks jpgs (rocks_65.jpg, rocks_dark.jpg, rocks_meddark.jpg) to the Orrery/game/art/terrains/WarriorCamp directory
NOTE: If you open the world and there are big orange patches, go into World Editor, Terrain Painter, then select the “Warning Material” and choose Edit. Navigate to the rocks_65 and rocks_dark textures)

Walking forward, you can step on each of the three rocks to change the animation speed of the Orrery.

If you want to make your own orreries through script:
You need to (1) Create an Orrery, (2) Initialize planets in the correct location, (3) Animate them:

//Orrery_Create(%orreryHandler, %name, %center, %radius, %numplanets,
%spacingmode, %planetarchivefile, %timestep, %planetsizing,
%show_children_of, %levels_of_children, %animationstep)
Examples:
Orrery_Create("orrery1");
orrery1.build_planets();
orrery1.begin_animation();
Orrery_Create("orrery2", "Mini System", "-233.581 -60.61 196.786", "5 5 5", 5, 4, "", "hour:2", 0.2, "Sol", 1,500);
orrery2.build_planets();
orrery2.begin_animation();

If you are editing code, a shortcut is to type “r” in the console, which reloads the PlanetsInit.cs. This way, you don’t need to quit Torque to make sure code compiles/updates correctly

The Orrery/game/art/shapes/solar_system.cs has all of the planet/moon information in it
- You can copy this and make your own solar system (or update to new information).
- Important fields:
//For a planet
new ScriptObject() {
planetName = "Pluto"; //Name of the planet that's shown, also used for lookups
planetOrder = 9; // The order of the planet out from the star (used as an estimate, as sometimes planet orbits are elongated and some move within others)
satelliteOf = "Sol"; // What does this orbit around?
shapeName = "art/shapes/planets/pluto.dts"; // Which DTS file is used? Materials.cs has to describe the texture as well
scale_mult = "1 1 1"; // Some planets are squished (especially gas giants, which have a smaller Z axis. Jupiter's is "1 1 .92" for example)
orbit_calc_method = "major_planet"; // How should the orbits be calculated?
dist_from_parent = 5900000000; // Average distance from parent. Not yet used
days_sidereal_rot = 90767.11; // How many earth days in a rotation? Not yet used
diameter_km = 2302; // Diameter, Not yet used
obliquity = 119.61; //How much tilt? Not yet used
//As of 1 Jan 2000, Julian Date = 2451545
a_semimajor_axis = 39.48168677; //6 variables are used to calculate planetary axis
e_eccentricity = 0.24880766;
i_inclination = 17.14175;
O_ecliptic_long = 110.30347;
w_perihelion = 224.06676;
L_mean_longitude = 238.92881;
a_per_cy = -0.00076912; //This is how much does they vary per century
e_per_cy = 0.00006465;
i_per_cy = 11.07;
O_per_cy = -37.33;
w_per_cy = -132.25;
L_per_cy = 522747.90;
//Only the Earth's moon is currently implemented - others will be in release #3

People who looked at this item also looked at…

• Orrery – 3D Planet simulator
• Interview on GarageGames
• Cloud Computing paper
• Source code to make your own “Lunar Academy”
• New MoonBaseOne game

Related items

• Orrery – 3D Planet simulator
• Interview on GarageGames
• Cloud Computing paper
• Chumby Advanced Tricks
• 3d iPhone web apps

The interview was very well done, and they were kind enough to post about our successes in publishing a space game for kids. In fact, they posted it on the 40th anniversary of the Moon Landing, which was pretty cool.

I hope you enjoy the write-up. I captured a number of the lessons learned and experiences working on the project.

People who looked at this item also looked at…

• Orrery Game: ver 2
• Orrery – 3D Planet simulator
• New MoonBaseOne game
• Source code to make your own “Lunar Academy”
• App Store for the Army

Related items

• Orrery – 3D Planet simulator
• Orrery Game: ver 2
• New MoonBaseOne game
• Cloud Computing Game Vid
• Moon Base One v3 Video

In honor of this auspicious occasion, I’m releasing a new “Orrery” resource for the Torque programming language. This isn’t yet a full-fledged game, rather just a library of celestial mechanics.

I wrote the instructionsfor integrating within Torque 3D (which is currently in a closed Beta) — this is a new version of the video Game engine that I’ve used on many projects. If you want to integrate this into an older version of Torque (such as TGE/TGEA), it should be pretty easy to do so.

I’ve intentionally kept all of the calculations in TorqueScript, rather than using the engine C++ code. This will allow you to make quick changes without having to recompile the game. Torque uses two layers, a “crunchy shell” of TorqueScript (a C++ derivative scripting language) and a “molten core” of C++ (which is much faster yet more complex). If you want to get more accurate calculations of the planets, you should move the code into the C++ engine. For example, the calculations now are only 32-bit Floating Point, so please don’t use these to launch interplanetary voyages… the planetary positions will only be accurate within 1 or 2% for the next 1000 years, and break down to chaos after that.

The way it works is by specifying a Julian Date – a time specified in centuries down to the second. You can pass in convertDateTimeToEpoch(2000,1,1,12,0,0) and it returns a JD that is the number of centuries since January 1st, 2000 at noon.

The Instructions code describes how to use this:
Orrery_Create("orrery1", "Mini System", "-233.581 -60.61 196.786", "5 5 5", 5, 4, "", 0.00004, 0.1, "Sol", 1);
Orrery_InitPlanets("orrery1");
Orrery_BeginAnimation("orrery1");

• This will create an orrery at the world points at “-233.581 -60.61 196.786″.
• It will stretch the orrery to fit within 5 cubic game units (mostly likely 5 cubic meters if you use the standard game settings).
• It will only show the first 5 planets
• It will render the planets in a realistic configuration
• It will use the default planets file (so you can pass in your own settings)
• It will animate at 0.00004 (about 2 days every 10th of a second)
• It will scale all planets to 10% (0.10) of their scaled size
• It will only show children of the object named “Sol” (so you can have hundreds of planets and moons in the same file and only use the ones you want
• It will only show 1 level of child objects (2 would show planets and moons)

You can have multiple Orreries in the same game. The more you have, the more choppy it gets. You might want to change the animation to move faster that every 10th of a second.

Let me know if it works for you, or you use it in a game! I’m working on another version, where moons and satellites will render. Let me know if there are any features I can add.

People who looked at this item also looked at…

• Orrery Game: ver 2
• Interview on GarageGames
• New MoonBaseOne game
• Cloud Computing Game Vid
• Mining Robots control code

Related items

• Orrery Game: ver 2
• Interview on GarageGames
• Mining Robots control code
• Cloud Computing Game Vid
• Cloud Computing paper

There are some new security challenges, though. The drafters of the specification were pretty smart in thinking through some of these, which makes it more tricky to code things just right.

Below, I’ll show a demo that uses only a web page to:

1. Go out to a web site, pass in the lat/long
2. Get a list of images and download them
3. Write the list of images to a local SQL database
4. Serialize, and add the images themselves to the database
5. Allow the user to unconnect their internet connection and still pull the images

You might note that this is the same technology that Google Gears offers… but built standard into the browser. Apple seems to really be embracing these technologies as an alternative to loading Gears or Flash onto the iPhone.

You can see the demo in action at: http://wecreategames.com/apps/imagesave/. Also check it out on your iPhone. Here’s a link to a zip file of all the source code.

Also, here’s a video summarizing how it all works:

Offline apps – The Manifest File

I’ve described the manifest file in the past as a way to tell the browser which files are important for caching and which shouldn’t be cached. For this app, we only have two files (index.html and imagesave.js) that need to be cached, and then two proxy files that are called to pull data about images and the images themselves. The manifest can do alot of other things, but we’re using it in it’s simplest form here:

CACHE MANIFEST
#ver 0.1
CACHE:
index.html
imagesave.js
NETWORK:
ws_image.php
ws.php

Working with the local browser database

The local database is a SQLlite database that’s built into Firefox, Safari, Opera, and Mobile Safari. It’s very powerful and easily makes sense if you’ve built SQL applications in the past. Very similar to Google Gears, the local database gives you a secure local storage of complex content. We’re going to use it to store metadata about the images we pull down (lat, longs, authors, etc) as well as serialized versions of the images themselves.

function picsInitDatabase() {
try {
if (!window.openDatabase) {
console.log('Databases are not supported in this browser');
} else {
var shortName = 'picsGeoDB';
var version = '1.0';
var displayName = 'Pictures Geotagged database';
var maxSize = 5000000; // in bytes
picsDB = openDatabase(shortName, version, displayName, maxSize);
console.log("Database is setup: "+picsDB);
}

In this instance, I’m telling the db to be 5Mb large. Any more than that and Mobile Safari asks for permission from the user.

Running SQL is the done through asynchronous transactions:
function picsCreateTables(){
picsDB.transaction(
function (transaction) {
/* The first query causes the transaction to (intentionally) fail if the table exists. */
transaction.executeSql('CREATE TABLE geopictures(id INTEGER NOT NULL PRIMARY KEY, secret TEXT NOT NULL DEFAULT "747060c06c", server INTEGER NOT NULL DEFAULT "3368",farm INTEGER NOT NULL DEFAULT "4", title TEXT NOT NULL DEFAULT "1 Infinite Loop", latitude TEXT NOT NULL DEFAULT "37.331666", longitude TEXT NOT NULL DEFAULT "-122.030834", accuracy INTEGER NOT NULL DEFAULT "16", datetaken TEXT NOT NULL DEFAULT "2009-06-14", ownername TEXT NOT NULL DEFAULT "avon11");', [], nullDataHandler, errorHandler);
transaction.executeSql('CREATE TABLE photodata(picid INTEGER NOT NULL PRIMARY KEY, encodedtext TEXT NOT NULL);', [], nullDataHandler, errorHandler);
}
);
}
function picsUpdateTables(dataID){
picsDB.transaction(
function (transaction) {
var p = data[dataID];
transaction.executeSql("INSERT INTO geopictures (id, secret, server, farm, title, latitude, longitude, accuracy, datetaken, ownername) VALUES (?,?,?,?,?,?,?,?,?,?);",
[p.id, p.secret, p.server, p.farm, p.title, p.latitude, p.longitude, p.accuracy, p.datetaken, p.ownername] );
transaction.executeSql("INSERT INTO photodata (picid, encodedtext) VALUES (?, ?)", [p.id, serializeCanvasByID(p.id)] );
}
);
}

It’s a bit tricky building your SQL, as you don’t get any detailed error messages on why things failed. I recommend you first build it on a local SQL Lite database and then copy in the SQL exactly, otherwise you’ll spend hours trying to track down small mistakes.

It’s very easy to call straight SQL commands, though make sure you don’t build the SQL on the fly and pass it in to be run. These local databases are susceptible to SQL injection attacks, just like any database. As you can see above, I’m passing in the values as an array.

SELECTing items is a little bit trickier, as you need to build some asynchronous calls. Also realize that you’re never sure of the order these calls will finish in.

function picsPhotodataSelectHandler(transaction, results)
{
console.log("Photodata had Results: "+results.rows.length);
// Handle the results
for (var i=0; i<results.rows.length; i++) {
var row = results.rows.item(i);
var newImage = new Object();
newImage.picid = row['picid'];
newImage.encodedtext = row['encodedtext'];
canvasData.push(newImage);
}
showImagesFromDB();
}
function picsSelectAll(){
picsDB.transaction(
function (transaction) {
transaction.executeSql("SELECT * from photodata;", [], picsPhotodataSelectHandler, errorHandler);
}
);
}

The above code runs a select statement, and passes in a function for what to do when things are finished successfully and another for when there is an error. The picsPhotodataSelectHandler is called once results are returned.

Drawing pixel by pixel in Javascript: the Canvas

The last piece of magic is to take images that we download, and to map them onto a canvas tag. The Canvas object is pretty amazing – it lets you either draw on it directly, or to read off individual pixels. You can also serialize the entire canvas out as an ascii string, which can then be saved to a database and later read back in. This is how we’re maintaining full control of our downloaded images. There are a few other methods of doing this that are a bit easier, but this gives you the most control.

function serializeCanvasByID(picID) {
var canvas = document.getElementById('cx'+picID);
var serializedVal = 'data:,';
if (canvas.toDataURL) serializedVal = canvas.toDataURL();
return serializedVal;
}
function drawImageOnCanvas(data, canvas) {
var img = new Image();
img.onload = function() {
canvas.width = img.width;
canvas.height = img.height;
canvas.getContext("2d").drawImage(img, 0, 0);
};
img.src = data;
}
function getCanvasDataByID(ID){
for (var i=0; i<canvasData.length;i++){
if (canvasData.picid == ID) {
return canvasData.encodedtext;
}
}
return '';
}
function showImagesFromDB() {
var container = document.getElementById('img-container')
container.innerHTML='';
for (var i=0; i<data.length; i++){
var canvasImg = document.createElement("canvas");
var imgID = data.id;
canvasImg.setAttribute("id", 'cx'+imgID);
canvasImg.setAttribute("title", data.desc);
drawImageOnCanvas(getCanvasDataByID(imgID), canvasImg);
container.appendChild(canvasImg);
}
}

The canvas.toDataURL(); call serializes out the image data. You can look at the string and see that it starts with “data:image/png” and then a bunch of characters.

The second piece of magic is in the drawImageOnCanvas function. It’s basically creating an img with the src either equal to a URL or to the “data:image/png…” text. Once the image is loaded into memory, it’s fed into the canvas’s 2d context to be shown on screen.

Summary

These three technologies can enable some pretty amazing things. You’ll be able to build very sophisticated web applications that are fully standards-compliant (so should hopefully work in iPhone, Android, Palm, etc). This could save thousands of $ as you no longer need to have experts in building apps for each platform. I could see this as being extremely valuable for making offline mapping applications.

(Back to last post in the series: Nearly2, iPhone app with 3D transforms)

People who looked at this item also looked at…

• App Store for the Army
• Orrery – 3D Planet simulator
• Building applications for Tysons Corner
• Orrery Game: ver 2
• Interview on GarageGames

Related items

• 3d iPhone web apps
• iPhone Geotags
• Building applications for Tysons Corner
• Updated FlickrProxy
• Chumby Advanced Tricks

I got the idea to put the Flickr image results into a 3D “ring” from Antoine Quint, a developer for Apple Europe that I met at WWDC this year. These transform-3d talks were all the rage at the conference, and I plan to heavily use them in future projects.

You can see the application if you point your iPhone browser to http://wecreategames.com/apps/nearly2. Notice that if you look at this page in Firefox, it looks like gibberish (though it does ask for your permission to send geoLocation data). Even in Safari 4 on Snow Leopard, it doesn’t look exactly right, but it does give you GeoLocation and the ability to test through logs with the new excellent Web Inspector (you have to go into Preferences, and turn on Developer tools first). The apple guys assured me that a patch coming out for desktop Safari next week will fix these 3D rendering issues.

The code is available here (please note that some of it is reusing Apple’s example code. While my stuff is Creative Commons, please don’t tread on their copyright).

Speeding up page loads:

So, here are the important new parts of code:
<html manifest="default.manifest">

This tells the page to use a manifest, which is basically a document listing which files should/souldn’t be cached by the application. The application is just a text file that looks like:
CACHE MANIFEST
#ver0.03
CACHE:
index.html
square.html
style.css
script.js
data.js
ui/back_button_touched.png
ui/back_button.png
ui/back_button_touched.png
ui/back_button.png
gradient_bottom.png
ui/gradient_top.png
ui/play_button.png
ui/play_button_ring.png
apple-touch-startup-image.png
apple-touch-icon.png
NETWORK:
ws.php

This is telling the browser (mobile safari in this case) to download and cache all the files at the top of the list, and that the ws.php file shouldn’t be cached (as it will constantly be changing and returning different image results). This is really nice for iPhones, as large websites can be all cached on the first version, drastically speeding up the load time. I think that after 5mb, the user is prompted to see if the site is allowed to use more space. Also, there are a few weird things when using the manifests:

1. When you update a file inside (such as data.js), the app won’t always notice. Instead, you have to physically change the .manifest file – I do this by changing the version # inside the file comment
2. It takes 2refreshes of the app for the new manifest to take hold. This makes sense, as when a page is opened in the browser it will keep on loading. After it’s finished loading, it then looks for the new manifest information (so as not to slow things down). The next refresh then has the new manifest data
3. You need to change your webserver’s mime type to return .manifest files as mime type of ‘text/cache-manifest’. Usually this is done through cPanel or whatever tool you’re using to manage your site.

Customizing the icon

On the iPhone, when users click the “+” button at the bottom of the screen, they can add the app to the home page. When they do, a screen shot of the app is created as the icon. With HTML5, you can specify which 57px x 57px icon you want to use with the code:
<link rel="apple-touch-icon" href="apple-touch-icon.png">

When the user adds the icon to the iPhone home page, the icon is given a glassy overlay. You can use ‘apple-touch-icon-precomposed’ if you don’t want the glass added.

Tailoring the page to look best on iPhone

The following css is in style.css:
body {
-webkit-touch-callout: none;
-webkit-text-size-adjust: none;
-webkit-tap-highlight-color: rgba(0,0,0,0);

On mobile safari, when you hold down on a link or piece of text for a few seconds, a callout pops up to ask if you want to copy/paste, or email. The first line disables this for everything in the document, which also helps with catching our custom touch events. Apple recommends a better practice is to only put these on certain objects on your page, not everything.

Mobile safari also tries to resize text to be helpful to users if it thinks the aspect ratio is too small. The second line turns off this adjustment.

The third line tells safari to highlight things in black if you click on them to show an action (rather than the default gray).

Making the 3D ring

The following code was all written by Antoine Quint, so I’ll only breiefly describe them.

In script.js, the code to calculate the rotation of the wheel and the position back in “Z” space is:
var angle = -i * (FULL_ROTATION / NUM_POSTERS);
item.style.webkitTransform = 'rotateX(' + angle + 'rad) translateZ(' + RADIUS + 'px)';

and in style.css, there are a ton of 3d-specific calls. Here are some important ones:
#ring-container {
/* make the perspective origin be the center of the screen */
-webkit-perspective-origin: 160px 208px;
/* set up the amount of perspective on the children of this container */
-webkit-perspective: 1000;
/* base transform for this container, need a value to transition from */
-webkit-transform: translateX(0);
}
#ring-aligner {
/* make sure the content lives in 3D space */
-webkit-transform-style: preserve-3d;
/* push back the content in the Z axis to cancel individual transforms set on ring items by script */
-webkit-transform: translateZ(-235px);
}
/* this matches all items within the ring */
#ring > li {
/* hide the back of the elements so that we don't see anything but the items currently in the front of the ring */
-webkit-backface-visibility: hidden;

The first call here is to add a “perspective” so that in 3d space, it looks like items are falling-back into the screen and getting smaller the farther away they are.

The last line applies to all list items with the #ring item. Because items are being rotated around in a circle, we turn off the back faces so that we don’t see them off in the distance.

-webkit-box-shadow: 0px 0px 3px rgba(255, 255, 255, 0.7);
-webkit-border-radius: 5px;

One of my favorite additions to HTML5 is how easy it is to add a shadow or rounded corners. Also, notice the 4th number above is the transparency – making it very easy to create blends.

Merging image lookups with the Ring

It can be a little painful debugging on your iPhone. The best tool I’ve found is to use Safari’s Web inspector, Firefox’s Firebug, or to just write messages out to the console. I’m also going to start playing with DashCode (on the mac), as it seems to have some nice debugging features. Instead of calling alert()s every time I want to output a variable, I use console.log(’Error text’). Because Firefox doesn’t have this function, we add it:
if (!console){
var console = new Object();
console.log = function(intext){alert(intext);};
}

Most of the rest of the code was pretty easy to move over from the first example into the FingerTips example. One thing I quickly noticed was that as you walk around, new images are being pulled in to the array and it’s hard to know which ones to show. I’ll attack this more on the next post, but for now, a work-around is to just show the last images in the array:
var data_offset = 0;
if (data.length>NUM_POSTERS) {
data_offset = (data.length-NUM_POSTERS);
console.log('Data Offset is: '+data_offset);
}

How’s it look? I would love any feedback or suggestions!

(Back to last post in the series: FlickrProxy)
(Next post in the series: Full Offline iPhone apps with HTML Canvas and DBs)

People who looked at this item also looked at…

• Building applications for Tysons Corner
• App Store for the Army
• Interview on GarageGames
• Orrery – 3D Planet simulator

Related items

• iPhone Geotags
• HTML5 Databases + Canvas
• Building applications for Tysons Corner
• Chumby Advanced Tricks
• Updated FlickrProxy

You can replace the ws.php into the previous Nearly1 example and you’ll get instant improvements and functionality. This will be the cornerstone of a few of the other iPhone Web Apps I’ll be releasing over the next few days.

Also, I found that to best test this code and make sure it was returning valid results, I added an option to render the results as KML.  This is nice because you can then test it with Google Maps or Google Earth.  Check out an example of this in action on Google Maps.

So, without further ado, I’ll jump into describing how the FlickrProxy works.

require_once("phpFlickr/phpFlickr.php");
// Create new phpFlickr object
$f = new phpFlickr("3055e96cbe8a5b33c248afb201bca822");
$f->enableCache("fs", "/home/bigyak/public_html/apps/nearly2/cache");

First, I’m using the phpFlickr API. There are good ways of installing it using PEAR (if you’re a bit more advanced of a PHP user), but it works just as well to copy the directory structure that I’ve provided. Enter in your Flickr API key, then put in a path to your cache directory (basically, just make a directory with write access turned on). You can comment that line out if you’re not using the cache.

if ($_GET["tags"]=="") {
$tags = "";
} else {
$tags = $_GET["tags"];
}
if ($_GET["daysago"]=="") {
if ($tags != "") {
//If they've entered tags, then don't be so restrictive on the date range
$daysago = 60;
} else {
$daysago = 7;
}
} else {
$daysago = $_GET["daysago"];
}
...
if ($_GET["format"]=="") {
$format = "json";
} else {
$format = $_GET["format"];
}
Next, we check all the variables that the user sent in through the querystring. If they put in tags to search on, we’ll look for those. If they entered a time range, we’ll search on that, then we’ll look via Latitude and Longitude as well as some other variables. For format, you can enter html, json, or kml. I’ve found that HTML is good for looking at quick results, KML is great for testing with Google Maps, and JSON is the type that our web apps can most easily use.

$RADIUS_EXPANSION = 10;
$DATE_EXPANSION = 4;
$PHOTOTRIES = 12;
$radius = 0.01;
for ($i = 1; $i <= $PHOTOTRIES; $i++) {
if($photos_found >= $MINPHOTOS) break;
$dateminformat = date("Y-m-d", mktime(0, 0, 0, date("m"), date("d")-$daysago, date("Y")));
$args = array("tags"=>$tags, "lat"=>$lat, "lon"=>$lon,"sort"=>"interestingness-desc", "per_page"=>$PHOTOTRIES, "radius"=>$radius, "min_taken_date"=>$dateminformat, "extras"=>"geo,date_taken,owner_name");
$photos_now = $f->photos_search($args);

This is the main loop of our code. We’re going to keep asking Flickr until we get the number of pictures back that the user has asked for, or until we’ve reached the 20mil limit of search. Notice that each successive query to Flickr is getting a little bigger and asking for older pictures. We’re ordering these by interestingness, but that doesn’t always return the most interesting pictures.

if(count($photos_now['photo']) > 0) {
foreach ($photos_now['photo'] as $photo) {
//Look through existing photos
$foundmatch=false;
foreach ($photos['photo'] as $lookingat) {
if ($photo[id]==$lookingat[id]) {
$foundmatch=true;
break;
}
}
if ($foundmatch==false) $photos['photo'][$photos_found++] = $photo;
}

$photos_found=count($photos['photo']);

The final part of the loop has us compare all returned pictures with our existing array of pictures. We throw away any duplicates and increment the count for checking later.

if ($format=="json") {
if(count($photos['photo']) == 0) {
if($backup=="yes"){
$html = file_get_contents('./test.json', true);
} else {
$html = 'jsonFlickrApi({"photos":"none", "stat":"ok", "requestedlat":"'.$lat.'", "requestedlon":"'.$lon.'"})'; }
} else {
$html = 'jsonFlickrApi({"photos":'.json_encode($photos).', "stat":"ok", "requestedlat":"'.$lat.'", "requestedlon":"'.$lon.'"})';;
}
...

I’ve found that sometimes you just don’t get any photos returned. As a last ditch effort, I have a backup file of json that I’ll just echo out if there are no results returned. For future versions, if none are returned in the a geo-area, I’ll experiment with increasign the geo box (rather than the radius).

If you look through the rest of the code, you’ll see the loops that manually build the KML or HTML for return to the users.

In summary, I’ve now improved the FlickrProxy to make it easier to ask for nearby images.  For future work, I’ll be looking at how to further improve the iPhone interface to take advantage of HTML5.

(Back to last post in the series: Nearly1, the iPhone app)

(Next post in the series: Nearly2, iPhone app with 3D transforms)

People who looked at this item also looked at…

• Orrery – 3D Planet simulator
• Building applications for Tysons Corner
• Orrery Game: ver 2
• Open Source – Functions.asp
• IESP research

Related items

• iPhone Geotags
• HTML5 Databases + Canvas
• 3d iPhone web apps
• Building applications for Tysons Corner
• Chumby Advanced Tricks

This is the first in a series of posts about how to write HTML5 applications and the power that they bring to the iPhone and iPod Touch. I’ll mostly be concentrating on iPhone applications (ones that use a GPS), and will try to build something useful that you can all have to make even better applications with. Like everything on this site, we’ll be releasing this as Open Source – so feel free to edit, sell it, make it better – but please just let me know what you create with it.

Nearly, version 1 (4 of the closest images pulled from Flickr)

For the first writeup in this series, we’ll create a simple application that uses HTML 5 to ask your browser for the location, and then passes that to Flickr to ask for 4 images. There is an example at: http://wecreategames.com/apps/nearly1/ – you can use this in your iPhone, or in Safari or Firefox Beta (it will ask permission to get your location, which you have to allow)

To get the code working on your own PHP server:

1. Download the Nearly1.zip file, unzip it.
2. Upload this to a directory in a php server that you control. (notice, this could easily be written in RoR, .Net, or anything else).
3. Register for a Flickr API, and then copy the API Key (which is a text string)
4. Edit the ws.php file and replace the API key in quotes with yours.
5. Optionally, add a cache directory, which will make requests run faster
6. Navigate to http://your.web.server/nearly (or whatever you named it) on a newer browser or your iPhone (notice, doesn’t work with IE)

Here are relevant and important pieces of code that get this to work:

The first line tells Mobile Safari not to allow the user to scroll or zoom the window using touch events when shown on an iPhone. The second says that it’s OK that when added to a home page it should function like an app, and not show the navigation bar on an iPhone:
<meta name="viewport" content="width=device-width, minimum-scale=1.0, maximum-scale=1.0">
<meta name="apple-mobile-web-app-capable" content="yes">

To enable geocoding, we first check if the browser has a navigator object. If so, we ask it for the current position and pass in two functions: what to do if the location is found, and what to do if there’s an error. In all error cases, I have it fake a location (which makes testing much easier). Notice that just for simplicity sake, I’m writing out error messages within the innerHTML of the rectangles on screen.
function getLocOnce() {
if(navigator.geolocation){
navigator.geolocation.getCurrentPosition(function(position) {
current_position=position;
showLoc(position);
}, function(error) {
document.getElementById('image1').innerHTML = "You are off the radar! Faking location. Error:" + error.code + error.message;
var position = new Object();
position.coords = new Object();
position.coords.latitude = DEFAULT_LAT;
position.coords.longitude = DEFAULT_LON;
showLoc(position);
}, {enableHighAccuracy:true}
);
} else {
document.getElementById('image1').innerHTML = "No location on your browser";
var position = new Object();
position.coords = new Object();
position.coords.latitude = DEFAULT_LAT;
position.coords.longitude = DEFAULT_LON;
showLoc(position);
}
}
Notice that because we’re passing in functions, we’re setting up the JavaScript to make Asynchronous calls when results are known. The process is:

1. Ask the browser for the location
2. When the browser responds (seconds later), give it a block of code to run
3. Show the results of the location

HTML5 has many of these similar asynchronous call structures in it – which is more complex but enables a lot of advanced functionality.

The showLoc function pulls in values and coordinates from the browser:
function showLoc(position){
if (position.coords) {
var pos1 = "Lat: " + position.coords.latitude + "<br/>Long:" + position.coords.longitude + "<br/>Accuracy:" + position.coords.accuracy;
var pos2 = "Altitude: " + position.altitude + "<br/>Accuracy:" + position.altitudeAccuracy;
var pos3 = "Speed: " + position.coords.speed + "<br/> Heading: " + position.coords.heading;
var pos4 = "Timestamp: " + position.timestamp;
document.getElementById('image1').innerHTML = pos1;
document.getElementById('image2').innerHTML = pos2;
document.getElementById('image3').innerHTML = pos3;
document.getElementById('image4').innerHTML = pos4;
}
}

The last piece of geoPosition code that we’ve added is the watchLoc and clearWatch functions. When calling watchLoc() function, you’re setting up another asynchronous callback function. Basically, you’re asking for a continuous GPS update whenever the phone can give you one. It’s very important that you turn off this watching process as it really drains the batteries quickly. I make sure to clear it when the page ends and as often as possible.
var watchId = null;
function watchLoc() {
btn = document.getElementById('geo_btn');
if (btn.value == 'Track') {
watchId = navigator.geolocation.watchPosition(showLoc);
btn.value='Stop';
} else {
clearWatch();
}
}
function clearWatch(){
if(watchId) {
navigator.geolocation.clearWatch(watchId);
watchId = null;
}
document.getElementById('geo_btn').value='Track';
}

Telling the page to always unload the request for GPS polling:
<body onunload="clearWatch()">

The final piece that might require some explanation is not new in HTML5 – it’s a standard JSON query back to a server to ask for data. In this case, we’re making an XMLHttpRequest call to ask for a list of nearby Flickr Images. The results are returned to the browser and eval()ed – which isn’t the safest way to write JavaScript but works for this demo.

Because you can’t make requests to multiple servers in the same JavaScript page, I’ve put in ws.php, which uses the phpFlickr library to ask Flickr for images. I’ve found there are a number of places (the horror!) that don’t have Flickr images geotagged nearby, so the php code tries a number of times to ask for images, each time widening the radius of allowed results.

It’s a really good idea to turn caching on (uncomment the line) in the ws.php file, as that will speed up response times and lighten the load on Flickr. I like to join multiple strings together to build a querystring, as it’s just easier to maintain in the long run.
var api_req = 'ws.php?';
var opt_lat = 'lat='+ lat;
var opt_lon = 'lon='+ lon;
var opt_num = 'num=4';
var opt_tag = 'tags=fun,travel,iphone';
var requestString = api_req + [opt_lat, opt_lon, opt_tag].join('&');
loadJavascript(requestString);

Flickr has a pretty useful tool for testing these REST-based API calls. I spent a lot of time testing with it, which helped me realize there must be tags sent in with the calls, or else no photos are returned.

(Next post in the series: An Updated Flickr Proxy)

People who looked at this item also looked at…

• Building applications for Tysons Corner
• Orrery – 3D Planet simulator
• Open Source – Functions.asp
• App Store for the Army
• Star Wars Chess

Related items

• 3d iPhone web apps
• HTML5 Databases + Canvas
• Updated FlickrProxy
• Building applications for Tysons Corner
• Chumby Advanced Tricks